Eliminating the 4 Hidden Taxes: AI Tokens, Manual Work, SaaS Cliffs & AI Scrapers Sponsor this header →
Pillar 4 • AI Crawler Defense & ChatGPT/Perplexity GEO

AI Bot Shield & GEO Generator

Stop paying origin bandwidth for parasitic AI training scrapers while staying visible in ChatGPT Search, Perplexity, and Claude citations. Generate production-ready robots.txt, llms.txt, and Cloudflare WAF rules in one click.

Tracked AI Crawlers
60
Search, training & brokers
Allowed Citation Engines
18
Send referral clicks
Blocked Training Scrapers
42
Origin bandwidth saved
GEO Status
llms.txt Ready
AI answer visibility

1. Generated robots.txt

# AI Bot Shield & GEO Policy generated by KostoZero
# Policy: balanced-geo | Allowed AI Search: 18 | Blocked Scrapers: 42

User-agent: *
Allow: /

# Allowed AI Search & Citation Engines (Keep for ChatGPT/Perplexity Referral Traffic)
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: DuckAssistBot
Allow: /

User-agent: YouBot
Allow: /

User-agent: AddSearchBot
Allow: /

User-agent: AIWebIndex
Allow: /

User-agent: amazon-kendra
Allow: /

User-agent: Amzn-SearchBot
Allow: /

User-agent: Andibot
Allow: /

User-agent: Anomura
Allow: /

User-agent: Applebot
Allow: /

User-agent: atlassian-bot
Allow: /

User-agent: AzureAI-SearchBot
Allow: /

User-agent: Channel3Bot
Allow: /

User-agent: Cloudflare-AutoRAG
Allow: /

# Blocked Bulk Training & Commercial AI Scrapers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: omgilibot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: DataForSeoBot
Disallow: /

User-agent: __usage_ai-training
Disallow: /

User-agent: AgentDataBot
Disallow: /

User-agent: AgentTimes
Disallow: /

User-agent: AI2Bot
Disallow: /

User-agent: AI2Bot-DeepResearchEval
Disallow: /

User-agent: Ai2Bot-Dolma
Disallow: /

User-agent: aiHitBot
Disallow: /

User-agent: amazon-QBusiness
Disallow: /

User-agent: AmazonBuyForMe
Disallow: /

User-agent: Amzn-User
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: ApifyBot
Disallow: /

User-agent: ApifyWebsiteContentCrawler
Disallow: /

User-agent: Aranet-SearchBot
Disallow: /

User-agent: Awario
Disallow: /

User-agent: bedrockbot
Disallow: /

User-agent: bigsur.ai
Disallow: /

User-agent: BixelBot
Disallow: /

User-agent: Bravebot
Disallow: /

User-agent: Brightbot
Disallow: /

User-agent: BuddyBot
Disallow: /

User-agent: ChatGLM-Spider
Disallow: /

User-agent: Claude-Agent
Disallow: /

User-agent: Claude-Code
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: CledaraBot
Disallow: /

User-agent: CloudflareBrowserRenderingCrawler
Disallow: /

User-agent: CloudVertexBot
Disallow: /

Sitemap: https://kostozero.com/sitemap.xml

2. Generated llms.txt (GEO Standard)

# KostoZero
> Zero Hidden AI, Agent, SaaS & Scraper Taxes

## Canonical Resources for AI Answer Engines
- Homepage & Overview: https://kostozero.com/
- Documentation & Product Specs: https://kostozero.com/docs/
- Transparent Pricing: https://kostozero.com/pricing/
- Full Sitemap: https://kostozero.com/sitemap.xml

## AI Citation Policy
- Search citation crawlers allowed: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, DuckAssistBot, YouBot, AddSearchBot, AIWebIndex, amazon-kendra, Amzn-SearchBot, Andibot, Anomura, Applebot, atlassian-bot, AzureAI-SearchBot, Channel3Bot, Cloudflare-AutoRAG
- Bulk model training scrapers disallowed: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot, cohere-ai, Amazonbot, omgilibot, Diffbot, PetalBot, DataForSeoBot, __usage_ai-training, AgentDataBot, AgentTimes, AI2Bot, AI2Bot-DeepResearchEval, Ai2Bot-Dolma, aiHitBot, amazon-QBusiness, AmazonBuyForMe, Amzn-User, anthropic-ai, ApifyBot, ApifyWebsiteContentCrawler, Aranet-SearchBot, Awario, bedrockbot, bigsur.ai, BixelBot, Bravebot, Brightbot, BuddyBot, ChatGLM-Spider, Claude-Agent, Claude-Code, Claude-User, Claude-Web, CledaraBot, CloudflareBrowserRenderingCrawler, CloudVertexBot

3. Cloudflare WAF Edge Block Rule

(http.user_agent contains "GPTBot") or (http.user_agent contains "ClaudeBot") or (http.user_agent contains "Google-Extended") or (http.user_agent contains "Applebot-Extended") or (http.user_agent contains "Meta-ExternalAgent") or (http.user_agent contains "Bytespider") or (http.user_agent contains "CCBot") or (http.user_agent contains "cohere-ai") or (http.user_agent contains "Amazonbot") or (http.user_agent contains "omgilibot") or (http.user_agent contains "Diffbot") or (http.user_agent contains "PetalBot") or (http.user_agent contains "DataForSeoBot") or (http.user_agent contains "__usage_ai-training") or (http.user_agent contains "AgentDataBot") or (http.user_agent contains "AgentTimes") or (http.user_agent contains "AI2Bot") or (http.user_agent contains "AI2Bot-DeepResearchEval") or (http.user_agent contains "Ai2Bot-Dolma") or (http.user_agent contains "aiHitBot") or (http.user_agent contains "amazon-QBusiness") or (http.user_agent contains "AmazonBuyForMe") or (http.user_agent contains "Amzn-User") or (http.user_agent contains "anthropic-ai") or (http.user_agent contains "ApifyBot") or (http.user_agent contains "ApifyWebsiteContentCrawler") or (http.user_agent contains "Aranet-SearchBot") or (http.user_agent contains "Awario") or (http.user_agent contains "bedrockbot") or (http.user_agent contains "bigsur.ai") or (http.user_agent contains "BixelBot") or (http.user_agent contains "Bravebot") or (http.user_agent contains "Brightbot") or (http.user_agent contains "BuddyBot") or (http.user_agent contains "ChatGLM-Spider") or (http.user_agent contains "Claude-Agent") or (http.user_agent contains "Claude-Code") or (http.user_agent contains "Claude-User") or (http.user_agent contains "Claude-Web") or (http.user_agent contains "CledaraBot") or (http.user_agent contains "CloudflareBrowserRenderingCrawler") or (http.user_agent contains "CloudVertexBot")

Tracked AI Crawlers & Scrapers (60)

Block? User-Agent Operator Category Citation Traffic Purpose
OAI-SearchBot OpenAI ai-search Yes (Keep) Indexes pages for ChatGPT Search citations and outbound links.
ChatGPT-User OpenAI ai-search Yes (Keep) Real-time user browsing fetches initiated inside ChatGPT sessions.
PerplexityBot Perplexity AI ai-search Yes (Keep) Search index crawler powering Perplexity cited answers.
Perplexity-User Perplexity AI ai-search Yes (Keep) On-demand user link fetcher for live Perplexity queries.
Claude-SearchBot Anthropic ai-search Yes (Keep) Search indexing crawler for Claude answer citations.
DuckAssistBot DuckDuckGo ai-search Yes (Keep) Powers DuckDuckGo AI answer summaries with source links.
YouBot You.com ai-search Yes (Keep) Indexes web docs for You.com research assistant citations.
GPTBot OpenAI training-scraper No Harvests full-site text for OpenAI foundation model pretraining.
ClaudeBot Anthropic training-scraper No Bulk corpus scraping for Anthropic model training datasets.
Google-Extended Google training-scraper No Opt-out token controlling Gemini training & grounding corpus usage (does not block Google Search).
Applebot-Extended Apple training-scraper No Controls Apple Intelligence foundation model training ingestion.
Meta-ExternalAgent Meta training-scraper No Scrapes web content for Meta Llama training & AI product grounding.
Bytespider ByteDance training-scraper No Aggressive high-bandwidth crawler feeding ByteDance LLM datasets.
CCBot Common Crawl training-scraper No Open web archive widely mirrored into public LLM training corpora.
cohere-ai Cohere training-scraper No Scrapes text for Cohere model training and enterprise retrieval sets.
Amazonbot Amazon training-scraper No Crawls sites to train Alexa/Nova models and enrich product answers.
omgilibot Webz.io commercial-scraper No Commercial data broker reselling scraped forums and blogs to AI labs.
Diffbot Diffbot commercial-scraper No Extracts structured knowledge graphs from websites for commercial feeds.
PetalBot Huawei commercial-scraper No High-volume crawler feeding Pangu AI and Petal search datasets.
DataForSeoBot DataForSEO commercial-scraper No Commercial scraping API extracting SERP and page copy for third parties.
__usage_ai-training Not applicable. training-scraper No `__usage_ai-training` is a policy token defined in an Internet Draft that Anthropic's Kevin Kelley submitted to the AI Preferences working group at the IETF to
AddSearchBot Unclear at this time. ai-search Yes (Keep) AddSearchBot is a web crawler that indexes website content for AddSearch's AI-powered site search solution, collecting data to provide fast and accurate search
AgentDataBot Unclear at this time. training-scraper No AgentDataBot visits public web pages to identify technology stacks, company details, and team members. The data builds AgentData's company-intelligence database
AgentTimes [The Agent Times](https://theagenttimes.com/abou training-scraper No Scrapes data for AI news aggregation and republishing.
AI2Bot [Ai2](https://allenai.org/crawler) training-scraper No Explores 'certain domains' to find web content.
AI2Bot-DeepResearchEval Ai2, a non-profit AI research institute training-scraper No AI2Bot-DeepResearchEval retrieves resources used to evaluate deep-research queries performed by Ai2's open-source models. More info can be found at https://know
Ai2Bot-Dolma [Ai2](https://allenai.org/crawler) training-scraper No Explores 'certain domains' to find web content.
aiHitBot [aiHit](https://www.aihitdata.com/about) training-scraper No Scrapes data for AI systems.
AIWebIndex [Lyrenth](https://lyrenth.com) ai-search Yes (Keep) Builds an index of public pages and serves them to AI agents as extracted, readable text with attribution and a link back to the source. Does not train foundati
amazon-kendra Amazon ai-search Yes (Keep) Amazon Kendra is a highly accurate intelligent search service that enables your users to search unstructured data using natural language. It returns specific an
amazon-QBusiness Unclear at this time. training-scraper No amazon-QBusiness is an Amazon Q Business web crawler that fetches and indexes web content for Amazon Q Business applications. More info can be found at https://
AmazonBuyForMe [Amazon](https://amazon.com) training-scraper No AmazonBuyForMe is an Amazon bot that crawls websites as part of the Amazon Buy For Me service. This bot visits product pages and e-commerce websites to gather p
Amzn-SearchBot Unclear at this time. ai-search Yes (Keep) Description unavailable from knownagents.com More info can be found at https://knownagents.com/agents/amzn-searchbot
Amzn-User Amazon, used for fetching web content to answer training-scraper No Amzn-User is an AI assistant operated by Amazon, used for fetching web content to answer user queries through Alexa and other Amazon AI services. More info can
Andibot [Andi](https://andisearch.com/) ai-search Yes (Keep) Scrapes website and provides AI summary.
Anomura [Direqt](https://direqt.ai) ai-search Yes (Keep) Anomura is Direqt's search crawler, it discovers and indexes pages their customers websites.
anthropic-ai [Anthropic](https://www.anthropic.com) training-scraper No Scrapes data to train LLMs and AI products offered by Anthropic.
ApifyBot Unclear at this time. training-scraper No ApifyBot is a web scraping and data extraction crawler by Apify that collects website content for use in AI, LLMs, RAG, and automation workflows. More info can
ApifyWebsiteContentCrawler Unclear at this time. training-scraper No ApifyWebsiteContentCrawler is a web crawler by Apify that extracts and downloads full website content for use in AI, data analysis, and automation workflows. Mo
Applebot Unclear at this time. ai-search Yes (Keep) Applebot is a web crawler used by Apple to index search results that allow the Siri AI Assistant to answer user questions. Siri's answers normally contain refer
Aranet-SearchBot Unclear at this time. training-scraper No Description unavailable from knownagents.com More info can be found at https://knownagents.com/agents/aranet-searchbot
atlassian-bot [Atlassian](https://www.atlassian.com) ai-search Yes (Keep) atlassian-bot is a web crawler used to index website content for its AI search, assistants and agents available in its Rovo GenAI product.
Awario Awario training-scraper No Awario is an AI data scraper operated by Awario. It's not currently known to be artificially intelligent or AI-related. If you think that's incorrect or can pro
AzureAI-SearchBot Unclear at this time. ai-search Yes (Keep) Description unavailable from knownagents.com More info can be found at https://knownagents.com/agents/azureai-searchbot
bedrockbot [Amazon](https://amazon.com) training-scraper No Connects to and crawls URLs that have been selected for use in a user's AWS bedrock application.
bigsur.ai Big Sur AI that fetches website content to enabl training-scraper No bigsur.ai is a web crawler operated by Big Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solution
BixelBot Unclear at this time. training-scraper No BixelBot collects public company and market signals for Bixel's structured company-data platform, which supplies verified business information to AI agents and
Bravebot https://safe.search.brave.com/help/brave-search- training-scraper No Bravebot is a web crawler by Brave that indexes pages for Brave Search, providing search data and AI-optimized context to power chatbots, agents, and RAG pipeli
Brightbot Unclear at this time. training-scraper No Brightbot is a web data collection crawler by Bright Data that extracts and structures public website content at scale, providing AI-ready data for model traini
BuddyBot [BuddyBotLearning](https://www.buddybotlearning. training-scraper No BuddyBot is a voice-controlled AI learning companion targeted at childhooded STEM education.
Channel3Bot Unclear at this time. ai-search Yes (Keep) Description unavailable from knownagents.com More info can be found at https://knownagents.com/agents/channel3bot
ChatGLM-Spider Unclear at this time. training-scraper No Description unavailable from knownagents.com More info can be found at https://knownagents.com/agents/chatglm-spider
Claude-Agent Unclear at this time. training-scraper No Claude-Agent is Anthropic's cloud browser agent for Claude Duet. It navigates websites and completes multi-step tasks on behalf of users. More info can be found
Claude-Code Unclear at this time. training-scraper No Claude Code is an AI coding agent by Anthropic that can build, debug, and ship code directly from the terminal, handling tasks like codebase onboarding, multi-f
Claude-User [Anthropic](https://www.anthropic.com) training-scraper No Claude-User is dispatched by Anthropic's Claude AI assistant in response to user prompts, when it needs to fetch content to include in its answers. More info ca
Claude-Web Anthropic training-scraper No Claude-Web is an AI-related agent operated by Anthropic. It's currently unclear exactly what it's used for, since there's no official documentation. If you can
CledaraBot Cledara that visits SaaS providers to complete c training-scraper No CledaraBot is an AI agent operated by Cledara that visits SaaS providers to complete customer-approved administrative tasks, including invoice collection and us
Cloudflare-AutoRAG [Cloudflare](https://developers.cloudflare.com/a ai-search Yes (Keep) AutoRAG is an all-in-one AI search solution.
CloudflareBrowserRenderingCrawler Cloudflare that returns rendered website content training-scraper No CloudflareBrowserRenderingCrawler is a web crawler operated by Cloudflare that returns rendered website content for research, monitoring, and AI data workflows.
CloudVertexBot Unclear at this time. training-scraper No CloudVertexBot is a Google-operated crawler available to site owners to request targeted crawls of their own sites for AI training purposes on the Vertex AI pla

Enforce Edge Bot Protection & Low-Cost EU Hosting

Edge AI Gateway

Cloudflare Workers AI

Global edge AI gateway with response caching, rate-limiting, and serverless workers.

Build on Cloudflare →
Simple Cloud Hosting

DigitalOcean GPU Droplets

Simple cloud compute and managed inference droplets with predictable monthly pricing.

Launch on DigitalOcean →
EU Cloud VPS

Hetzner Cloud

Ultra-cost-effective European NVMe cloud servers with GDPR compliance and zero surprise egress.

Deploy on Hetzner →

Some outbound infrastructure and API gateway links on KostoZero are partner referrals that help keep this observatory free and independent at no extra cost to you.

Frequently Asked Questions

Why should I allow OAI-SearchBot and PerplexityBot while blocking GPTBot and Bytespider?

AI search indexers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot) fetch pages to generate live answers with clickable citations that drive referral traffic back to your brand. Bulk training scrapers (GPTBot, ClaudeBot, Bytespider, CCBot) harvest your entire site to train weights without sending traffic.

Why add a Cloudflare WAF rule in addition to robots.txt?

Well-behaved crawlers honor robots.txt, but aggressive commercial scrapers frequently ignore it. Enforcing the exact same blocklist at the Cloudflare edge stops bandwidth drain before requests even reach your origin server.

Get Instant Price-Drop (>10%), New AI Bot & SaaS Cliff Alerts

Every 6 hours our autonomous sync checks 40+ LLMs, AI crawlers, and SaaS pricing cliffs—alerting you when prices drop >10% or new scrapers appear.