AI bot allowlist — robots.txt + Cloudflare configuration.

Why explicit allowlist (vs implicit)

robots.txt defaults to "allow all" if you don't mention a User-Agent. So technically, an emptyrobots.txt would allow GPTBot, ClaudeBot, PerplexityBot, etc.

In practice we recommend an explicit allowlist for two reasons:

• Signal intent. An explicit Allow: / per crawler makes your decision readable to anyone auditing the file, including you six months from now. It also documents which bots you considered.
• Defensibility against accidental blocks. If you add a global User-agent: * rule later (for example to block scraping bots), an explicit AI-bot allowlist above it survives the change.

Retrieval bots vs training bots

AI companies now run separate crawlers for separate jobs, and each one reads its own User-agent group in your robots.txt. That lets you make two different decisions:

• Retrieval (search and user) bots fetch pages to answer a question and link to the source. These are the ones that produce citations. Blocking them removes you from AI answers.
• Training bots collect content that may be used to train future models. Blocking them does not remove you from AI search answers.

The tokens below come from each vendor's own crawler documentation (checked September 2026):

• OpenAI. OAI-SearchBot surfaces websites in ChatGPT search. ChatGPT-User fetches pages for user actions in ChatGPT and Custom GPTs. GPTBot crawls content that may be used for training. OpenAI notes that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.
• Anthropic. Claude-SearchBot improves search result quality. Claude-User visits pages when people ask Claude questions. ClaudeBot collects content that could contribute to training.
• Perplexity. PerplexityBot surfaces and links websites in Perplexity search results and, per Perplexity, is not used to crawl content for AI foundation models. Perplexity-User visits pages for user questions.
• Google. Google-Extended is a control token, not a separate crawler. It decides whether content Googlebot crawls may be used for Gemini training and grounding. It does not affect Google Search.
• Apple. Applebot-Extended opts content out of training Apple's foundation models. Apple states it does not crawl webpages itself; the regular Applebot does the crawling.
• Common Crawl. CCBot collects the open Common Crawl web archive.
• Meta. meta-webindexer improves Meta AI search results. meta-externalagent indexes content for purposes such as training AI models.
• Amazon. Amzn-SearchBot and Amzn-User support search and Alexa answers and are not used for generative AI training, per Amazon. Amazonbot may be used to train Amazon AI models.

The canonical robots.txt block

Here's the allow-everything block for a reference site that wants maximum AI visibility. Paste it at the top of your robots.txt, before any other User-agent rules:

# AI search + user-action bots (these produce citations) User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: meta-webindexer User-agent: Amzn-SearchBot Allow: / # AI training bots User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Amazonbot User-agent: Bytespider User-agent: Meta-ExternalAgent Allow: / # Standard search User-agent: * Allow: / Sitemap: https://yoursite.com/sitemap.xml

Several User-agent lines above one rule form a single group, which the robots.txt standard (RFC 9309) allows. The standard wildcard block lets all other crawlers (Bing, DuckDuckGo, etc.) in. Put your real sitemap URL on the last line.

Allow AI search, block AI training

If you want to be cited in AI answers but don't want your content used to train models, allow the retrieval bots and disallow the training bots. This is a common choice for publishers:

# Allow: AI search + user-action bots User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User Allow: / # Block: training-only bots (optional) User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: CCBot Disallow: / # Everyone else, including Googlebot and Bingbot User-agent: * Allow: / Sitemap: https://yoursite.com/sitemap.xml

Two things to know. Blocking Google-Extended leaves Google Search untouched, because Googlebot follows its own group. And bots acting for a user may not follow these rules at all: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Meta says the same of its user-triggered fetcher. So a Disallow is a request to well-behaved crawlers, not access control.

The two Cloudflare settings that silently override

If your site is behind Cloudflare (and most are), there are two dashboard settings that can silently block AI crawlers even with the perfect robots.txt above. They live at Overview → AI crawlers in the CF dashboard:

• Block AI training bots — has three modes:
  • Block on all pages (the default for new zones as of 2025) — HTTP 403 to GPTBot/ClaudeBot/etc. at the edge before your robots.txt is even consulted
  • Block only on hostnames with ads — conditional block
  • Do not block (allow crawlers) — this is what you want
• Manage your robots.txt — has three modes:
  • Content Signals Policy (default) — Cloudflare INJECTS a # BEGIN Cloudflare Managed content block at the top of your served robots.txt with Disallow: / for all major AI crawlers
  • Instruct AI bot traffic with robots.txt — same injection pattern
  • Disable robots.txt configuration — this is what you want

Both settings must be at the "allow crawlers" / "disable configuration" values for your applicationrobots.txt to actually serve. If either is set to a blocking mode, your robots.txt is overridden at the edge and citation becomes structurally impossible regardless of how well-optimized your content is.

The same applies outside Cloudflare. A WAF rule, a bot-management product, or a hosting plan's "block bad bots" toggle can return 403 to these user agents. Check every layer that sits in front of your site.

How do you verify it's working?

Four terminal commands:

# 1. Read the robots.txt that is actually served; look for a "Cloudflare Managed" block at the top curl -sL https://yoursite.com/robots.txt | head -20 # 2. ChatGPT search bot should get 200, not 403 curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" https://yoursite.com/ | head -1 # 3. Perplexity search bot curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://yoursite.com/ | head -1 # 4. A training bot, to confirm your block or allow decision took effect curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://yoursite.com/ | head -1

Expected results:

• Step 1: your robots.txt starts with your User-Agent blocks (not the Cloudflare-managed wrapper)
• Steps 2–4: HTTP/2 200 (a 301/302 redirect is fine — what you don't want is 403)

A curl with a bot user agent only tests rules that match on the user-agent string. Rules that verify a bot by its IP address can treat your test differently from the real crawler. So also open your CDN or WAF dashboard and confirm no "block AI bots" setting or custom rule is on, and check your server logs for real hits from these user agents.

The Citation Readiness Score runs an automated version of this check as part of the Bot-Crawl Health dimension.

When (rarely) you should block a specific bot

For most reference + comparison + calculator sites, we recommend allowing the AI crawlers above. Niche exceptions:

• Sensitive content / private data. If your site exposes content that must not appear in LLM training (e.g. user-uploaded PII, paid-only content cached publicly), block specific crawlers — but better, gate the content behind auth.
• Bandwidth-cost concerns. Some crawlers (Bytespider has been historically aggressive) can hammer your server with hundreds of thousands of requests. If you observe rate-related cost pressure, throttle via CF or block. But that's a bandwidth-mitigation decision, not a citation decision.
• Legal / licensing concerns. If your jurisdiction requires opt-out from AI training (some European publisher regulations), block the relevant training crawlers explicitly and document the legal rationale. The search-only block above does exactly that while keeping you citable.

For everything else: allow them. Treat AI crawler inventory as a marketing channel, not a cost.

Questions people ask