AI Bot Shield & GEO Generator
Stop paying origin bandwidth for parasitic AI training scrapers while staying visible in ChatGPT Search, Perplexity, and Claude citations. Generate production-ready robots.txt, llms.txt, and Cloudflare WAF rules in one click.
# AI Bot Shield & GEO Policy generated by KostoZero # Policy: balanced-geo | Allowed AI Search: 18 | Blocked Scrapers: 42 User-agent: * Allow: / # Allowed AI Search & Citation Engines (Keep for ChatGPT/Perplexity Referral Traffic) User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: DuckAssistBot Allow: / User-agent: YouBot Allow: / User-agent: AddSearchBot Allow: / User-agent: AIWebIndex Allow: / User-agent: amazon-kendra Allow: / User-agent: Amzn-SearchBot Allow: / User-agent: Andibot Allow: / User-agent: Anomura Allow: / User-agent: Applebot Allow: / User-agent: atlassian-bot Allow: / User-agent: AzureAI-SearchBot Allow: / User-agent: Channel3Bot Allow: / User-agent: Cloudflare-AutoRAG Allow: / # Blocked Bulk Training & Commercial AI Scrapers User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: Amazonbot Disallow: / User-agent: omgilibot Disallow: / User-agent: Diffbot Disallow: / User-agent: PetalBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: __usage_ai-training Disallow: / User-agent: AgentDataBot Disallow: / User-agent: AgentTimes Disallow: / User-agent: AI2Bot Disallow: / User-agent: AI2Bot-DeepResearchEval Disallow: / User-agent: Ai2Bot-Dolma Disallow: / User-agent: aiHitBot Disallow: / User-agent: amazon-QBusiness Disallow: / User-agent: AmazonBuyForMe Disallow: / User-agent: Amzn-User Disallow: / User-agent: anthropic-ai Disallow: / User-agent: ApifyBot Disallow: / User-agent: ApifyWebsiteContentCrawler Disallow: / User-agent: Aranet-SearchBot Disallow: / User-agent: Awario Disallow: / User-agent: bedrockbot Disallow: / User-agent: bigsur.ai Disallow: / User-agent: BixelBot Disallow: / User-agent: Bravebot Disallow: / User-agent: Brightbot Disallow: / User-agent: BuddyBot Disallow: / User-agent: ChatGLM-Spider Disallow: / User-agent: Claude-Agent Disallow: / User-agent: Claude-Code Disallow: / User-agent: Claude-User Disallow: / User-agent: Claude-Web Disallow: / User-agent: CledaraBot Disallow: / User-agent: CloudflareBrowserRenderingCrawler Disallow: / User-agent: CloudVertexBot Disallow: / Sitemap: https://kostozero.com/sitemap.xml
# KostoZero > Zero Hidden AI, Agent, SaaS & Scraper Taxes ## Canonical Resources for AI Answer Engines - Homepage & Overview: https://kostozero.com/ - Documentation & Product Specs: https://kostozero.com/docs/ - Transparent Pricing: https://kostozero.com/pricing/ - Full Sitemap: https://kostozero.com/sitemap.xml ## AI Citation Policy - Search citation crawlers allowed: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, DuckAssistBot, YouBot, AddSearchBot, AIWebIndex, amazon-kendra, Amzn-SearchBot, Andibot, Anomura, Applebot, atlassian-bot, AzureAI-SearchBot, Channel3Bot, Cloudflare-AutoRAG - Bulk model training scrapers disallowed: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot, cohere-ai, Amazonbot, omgilibot, Diffbot, PetalBot, DataForSeoBot, __usage_ai-training, AgentDataBot, AgentTimes, AI2Bot, AI2Bot-DeepResearchEval, Ai2Bot-Dolma, aiHitBot, amazon-QBusiness, AmazonBuyForMe, Amzn-User, anthropic-ai, ApifyBot, ApifyWebsiteContentCrawler, Aranet-SearchBot, Awario, bedrockbot, bigsur.ai, BixelBot, Bravebot, Brightbot, BuddyBot, ChatGLM-Spider, Claude-Agent, Claude-Code, Claude-User, Claude-Web, CledaraBot, CloudflareBrowserRenderingCrawler, CloudVertexBot
(http.user_agent contains "GPTBot") or (http.user_agent contains "ClaudeBot") or (http.user_agent contains "Google-Extended") or (http.user_agent contains "Applebot-Extended") or (http.user_agent contains "Meta-ExternalAgent") or (http.user_agent contains "Bytespider") or (http.user_agent contains "CCBot") or (http.user_agent contains "cohere-ai") or (http.user_agent contains "Amazonbot") or (http.user_agent contains "omgilibot") or (http.user_agent contains "Diffbot") or (http.user_agent contains "PetalBot") or (http.user_agent contains "DataForSeoBot") or (http.user_agent contains "__usage_ai-training") or (http.user_agent contains "AgentDataBot") or (http.user_agent contains "AgentTimes") or (http.user_agent contains "AI2Bot") or (http.user_agent contains "AI2Bot-DeepResearchEval") or (http.user_agent contains "Ai2Bot-Dolma") or (http.user_agent contains "aiHitBot") or (http.user_agent contains "amazon-QBusiness") or (http.user_agent contains "AmazonBuyForMe") or (http.user_agent contains "Amzn-User") or (http.user_agent contains "anthropic-ai") or (http.user_agent contains "ApifyBot") or (http.user_agent contains "ApifyWebsiteContentCrawler") or (http.user_agent contains "Aranet-SearchBot") or (http.user_agent contains "Awario") or (http.user_agent contains "bedrockbot") or (http.user_agent contains "bigsur.ai") or (http.user_agent contains "BixelBot") or (http.user_agent contains "Bravebot") or (http.user_agent contains "Brightbot") or (http.user_agent contains "BuddyBot") or (http.user_agent contains "ChatGLM-Spider") or (http.user_agent contains "Claude-Agent") or (http.user_agent contains "Claude-Code") or (http.user_agent contains "Claude-User") or (http.user_agent contains "Claude-Web") or (http.user_agent contains "CledaraBot") or (http.user_agent contains "CloudflareBrowserRenderingCrawler") or (http.user_agent contains "CloudVertexBot")
Tracked AI Crawlers & Scrapers (60)
| Block? | User-Agent | Operator | Category | Citation Traffic | Purpose |
|---|---|---|---|---|---|
OAI-SearchBot |
OpenAI | ai-search | Yes (Keep) | Indexes pages for ChatGPT Search citations and outbound links. | |
ChatGPT-User |
OpenAI | ai-search | Yes (Keep) | Real-time user browsing fetches initiated inside ChatGPT sessions. | |
PerplexityBot |
Perplexity AI | ai-search | Yes (Keep) | Search index crawler powering Perplexity cited answers. | |
Perplexity-User |
Perplexity AI | ai-search | Yes (Keep) | On-demand user link fetcher for live Perplexity queries. | |
Claude-SearchBot |
Anthropic | ai-search | Yes (Keep) | Search indexing crawler for Claude answer citations. | |
DuckAssistBot |
DuckDuckGo | ai-search | Yes (Keep) | Powers DuckDuckGo AI answer summaries with source links. | |
YouBot |
You.com | ai-search | Yes (Keep) | Indexes web docs for You.com research assistant citations. | |
GPTBot |
OpenAI | training-scraper | No | Harvests full-site text for OpenAI foundation model pretraining. | |
ClaudeBot |
Anthropic | training-scraper | No | Bulk corpus scraping for Anthropic model training datasets. | |
Google-Extended |
training-scraper | No | Opt-out token controlling Gemini training & grounding corpus usage (does not block Google Search). | ||
Applebot-Extended |
Apple | training-scraper | No | Controls Apple Intelligence foundation model training ingestion. | |
Meta-ExternalAgent |
Meta | training-scraper | No | Scrapes web content for Meta Llama training & AI product grounding. | |
Bytespider |
ByteDance | training-scraper | No | Aggressive high-bandwidth crawler feeding ByteDance LLM datasets. | |
CCBot |
Common Crawl | training-scraper | No | Open web archive widely mirrored into public LLM training corpora. | |
cohere-ai |
Cohere | training-scraper | No | Scrapes text for Cohere model training and enterprise retrieval sets. | |
Amazonbot |
Amazon | training-scraper | No | Crawls sites to train Alexa/Nova models and enrich product answers. | |
omgilibot |
Webz.io | commercial-scraper | No | Commercial data broker reselling scraped forums and blogs to AI labs. | |
Diffbot |
Diffbot | commercial-scraper | No | Extracts structured knowledge graphs from websites for commercial feeds. | |
PetalBot |
Huawei | commercial-scraper | No | High-volume crawler feeding Pangu AI and Petal search datasets. | |
DataForSeoBot |
DataForSEO | commercial-scraper | No | Commercial scraping API extracting SERP and page copy for third parties. | |
__usage_ai-training |
Not applicable. | training-scraper | No | `__usage_ai-training` is a policy token defined in an Internet Draft that Anthropic's Kevin Kelley submitted to the AI Preferences working group at the IETF to | |
AddSearchBot |
Unclear at this time. | ai-search | Yes (Keep) | AddSearchBot is a web crawler that indexes website content for AddSearch's AI-powered site search solution, collecting data to provide fast and accurate search | |
AgentDataBot |
Unclear at this time. | training-scraper | No | AgentDataBot visits public web pages to identify technology stacks, company details, and team members. The data builds AgentData's company-intelligence database | |
AgentTimes |
[The Agent Times](https://theagenttimes.com/abou | training-scraper | No | Scrapes data for AI news aggregation and republishing. | |
AI2Bot |
[Ai2](https://allenai.org/crawler) | training-scraper | No | Explores 'certain domains' to find web content. | |
AI2Bot-DeepResearchEval |
Ai2, a non-profit AI research institute | training-scraper | No | AI2Bot-DeepResearchEval retrieves resources used to evaluate deep-research queries performed by Ai2's open-source models. More info can be found at https://know | |
Ai2Bot-Dolma |
[Ai2](https://allenai.org/crawler) | training-scraper | No | Explores 'certain domains' to find web content. | |
aiHitBot |
[aiHit](https://www.aihitdata.com/about) | training-scraper | No | Scrapes data for AI systems. | |
AIWebIndex |
[Lyrenth](https://lyrenth.com) | ai-search | Yes (Keep) | Builds an index of public pages and serves them to AI agents as extracted, readable text with attribution and a link back to the source. Does not train foundati | |
amazon-kendra |
Amazon | ai-search | Yes (Keep) | Amazon Kendra is a highly accurate intelligent search service that enables your users to search unstructured data using natural language. It returns specific an | |
amazon-QBusiness |
Unclear at this time. | training-scraper | No | amazon-QBusiness is an Amazon Q Business web crawler that fetches and indexes web content for Amazon Q Business applications. More info can be found at https:// | |
AmazonBuyForMe |
[Amazon](https://amazon.com) | training-scraper | No | AmazonBuyForMe is an Amazon bot that crawls websites as part of the Amazon Buy For Me service. This bot visits product pages and e-commerce websites to gather p | |
Amzn-SearchBot |
Unclear at this time. | ai-search | Yes (Keep) | Description unavailable from knownagents.com More info can be found at https://knownagents.com/agents/amzn-searchbot | |
Amzn-User |
Amazon, used for fetching web content to answer | training-scraper | No | Amzn-User is an AI assistant operated by Amazon, used for fetching web content to answer user queries through Alexa and other Amazon AI services. More info can | |
Andibot |
[Andi](https://andisearch.com/) | ai-search | Yes (Keep) | Scrapes website and provides AI summary. | |
Anomura |
[Direqt](https://direqt.ai) | ai-search | Yes (Keep) | Anomura is Direqt's search crawler, it discovers and indexes pages their customers websites. | |
anthropic-ai |
[Anthropic](https://www.anthropic.com) | training-scraper | No | Scrapes data to train LLMs and AI products offered by Anthropic. | |
ApifyBot |
Unclear at this time. | training-scraper | No | ApifyBot is a web scraping and data extraction crawler by Apify that collects website content for use in AI, LLMs, RAG, and automation workflows. More info can | |
ApifyWebsiteContentCrawler |
Unclear at this time. | training-scraper | No | ApifyWebsiteContentCrawler is a web crawler by Apify that extracts and downloads full website content for use in AI, data analysis, and automation workflows. Mo | |
Applebot |
Unclear at this time. | ai-search | Yes (Keep) | Applebot is a web crawler used by Apple to index search results that allow the Siri AI Assistant to answer user questions. Siri's answers normally contain refer | |
Aranet-SearchBot |
Unclear at this time. | training-scraper | No | Description unavailable from knownagents.com More info can be found at https://knownagents.com/agents/aranet-searchbot | |
atlassian-bot |
[Atlassian](https://www.atlassian.com) | ai-search | Yes (Keep) | atlassian-bot is a web crawler used to index website content for its AI search, assistants and agents available in its Rovo GenAI product. | |
Awario |
Awario | training-scraper | No | Awario is an AI data scraper operated by Awario. It's not currently known to be artificially intelligent or AI-related. If you think that's incorrect or can pro | |
AzureAI-SearchBot |
Unclear at this time. | ai-search | Yes (Keep) | Description unavailable from knownagents.com More info can be found at https://knownagents.com/agents/azureai-searchbot | |
bedrockbot |
[Amazon](https://amazon.com) | training-scraper | No | Connects to and crawls URLs that have been selected for use in a user's AWS bedrock application. | |
bigsur.ai |
Big Sur AI that fetches website content to enabl | training-scraper | No | bigsur.ai is a web crawler operated by Big Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solution | |
BixelBot |
Unclear at this time. | training-scraper | No | BixelBot collects public company and market signals for Bixel's structured company-data platform, which supplies verified business information to AI agents and | |
Bravebot |
https://safe.search.brave.com/help/brave-search- | training-scraper | No | Bravebot is a web crawler by Brave that indexes pages for Brave Search, providing search data and AI-optimized context to power chatbots, agents, and RAG pipeli | |
Brightbot |
Unclear at this time. | training-scraper | No | Brightbot is a web data collection crawler by Bright Data that extracts and structures public website content at scale, providing AI-ready data for model traini | |
BuddyBot |
[BuddyBotLearning](https://www.buddybotlearning. | training-scraper | No | BuddyBot is a voice-controlled AI learning companion targeted at childhooded STEM education. | |
Channel3Bot |
Unclear at this time. | ai-search | Yes (Keep) | Description unavailable from knownagents.com More info can be found at https://knownagents.com/agents/channel3bot | |
ChatGLM-Spider |
Unclear at this time. | training-scraper | No | Description unavailable from knownagents.com More info can be found at https://knownagents.com/agents/chatglm-spider | |
Claude-Agent |
Unclear at this time. | training-scraper | No | Claude-Agent is Anthropic's cloud browser agent for Claude Duet. It navigates websites and completes multi-step tasks on behalf of users. More info can be found | |
Claude-Code |
Unclear at this time. | training-scraper | No | Claude Code is an AI coding agent by Anthropic that can build, debug, and ship code directly from the terminal, handling tasks like codebase onboarding, multi-f | |
Claude-User |
[Anthropic](https://www.anthropic.com) | training-scraper | No | Claude-User is dispatched by Anthropic's Claude AI assistant in response to user prompts, when it needs to fetch content to include in its answers. More info ca | |
Claude-Web |
Anthropic | training-scraper | No | Claude-Web is an AI-related agent operated by Anthropic. It's currently unclear exactly what it's used for, since there's no official documentation. If you can | |
CledaraBot |
Cledara that visits SaaS providers to complete c | training-scraper | No | CledaraBot is an AI agent operated by Cledara that visits SaaS providers to complete customer-approved administrative tasks, including invoice collection and us | |
Cloudflare-AutoRAG |
[Cloudflare](https://developers.cloudflare.com/a | ai-search | Yes (Keep) | AutoRAG is an all-in-one AI search solution. | |
CloudflareBrowserRenderingCrawler |
Cloudflare that returns rendered website content | training-scraper | No | CloudflareBrowserRenderingCrawler is a web crawler operated by Cloudflare that returns rendered website content for research, monitoring, and AI data workflows. | |
CloudVertexBot |
Unclear at this time. | training-scraper | No | CloudVertexBot is a Google-operated crawler available to site owners to request targeted crawls of their own sites for AI training purposes on the Vertex AI pla |
Enforce Edge Bot Protection & Low-Cost EU Hosting
Cloudflare Workers AI
Global edge AI gateway with response caching, rate-limiting, and serverless workers.
DigitalOcean GPU Droplets
Simple cloud compute and managed inference droplets with predictable monthly pricing.
Hetzner Cloud
Ultra-cost-effective European NVMe cloud servers with GDPR compliance and zero surprise egress.
Some outbound infrastructure and API gateway links on KostoZero are partner referrals that help keep this observatory free and independent at no extra cost to you.
Frequently Asked Questions
Why should I allow OAI-SearchBot and PerplexityBot while blocking GPTBot and Bytespider?
AI search indexers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot) fetch pages to generate live answers with clickable citations that drive referral traffic back to your brand. Bulk training scrapers (GPTBot, ClaudeBot, Bytespider, CCBot) harvest your entire site to train weights without sending traffic.
Why add a Cloudflare WAF rule in addition to robots.txt?
Well-behaved crawlers honor robots.txt, but aggressive commercial scrapers frequently ignore it. Enforcing the exact same blocklist at the Cloudflare edge stops bandwidth drain before requests even reach your origin server.
Get Instant Price-Drop (>10%), New AI Bot & SaaS Cliff Alerts
Every 6 hours our autonomous sync checks 40+ LLMs, AI crawlers, and SaaS pricing cliffs—alerting you when prices drop >10% or new scrapers appear.