Does allowing AI crawlers mean my content trains their models?

Depends on the crawler. GPTBot, Google-Extended and CCBot are training crawlers — allowing them means your content may enter training corpora. OAI-SearchBot, PerplexityBot and Claude-SearchBot are retrieval crawlers serving live search citation, not training. You can control them separately: allow retrieval, block training.

Does blocking AI crawlers protect my content?

Only partially. robots.txt is a gentleman's agreement — major vendors honor it, gray-zone scrapers do not. For a lead-generation website the practical outcome of blocking is: legitimate AI engines cannot cite you (lost traffic), while determined copycats scrape you anyway. Unless your content is a paid asset, keep retrieval crawlers open.

Will AI crawlers overload my server?

Not a static site. AI crawlers fetch far less frequently than search spiders, and static HTML is cheap to serve. For dynamic sites, use crawl-delay (partially supported) or rate-limit at the CDN/gateway — do not block outright.

How do I verify AI crawlers actually visit?

Filter your access log by user agent: grep -i "GPTBot\|PerplexityBot\|ClaudeBot" access.log. Validate IPs against the vendors' published ranges (OpenAI and Perplexity both publish official IP lists) to filter out spoofed user agents.

The AI Crawler Whitelist: 13 User Agents Your robots.txt Should Allow (2026)

Blocking AI crawlers means forbidding AI engines from citing your content. This is the 2026 list of 13 mainstream AI crawler user agents (OpenAI, Perplexity, Anthropic, Google), what each one actually does, a copy-paste robots.txt config and how to decide about training bots.

Bottom line first

Blocking AI crawlers in robots.txt shuts down the “cited by AI” traffic channel with your own hands — the most common self-inflicted wound in GEO, because generic “hardening” guides block every non-search bot by default.

For a lead-generation website the right posture is: allow all retrieval crawlers; decide about training crawlers based on your content strategy.

1. The 13 user agents

User-Agent	Vendor	Purpose	Recommendation
`OAI-SearchBot`	OpenAI	ChatGPT search index	✅ Allow
`ChatGPT-User`	OpenAI	Live fetch during user sessions	✅ Allow
`GPTBot`	OpenAI	Training corpus	⚖️ Your call
`PerplexityBot`	Perplexity	Search index	✅ Allow
`Perplexity-User`	Perplexity	Live fetch during user sessions	✅ Allow
`Claude-SearchBot`	Anthropic	Claude search index	✅ Allow
`Claude-User`	Anthropic	Live fetch during user sessions	✅ Allow
`ClaudeBot`	Anthropic	Training corpus	⚖️ Your call
`Google-Extended`	Google	Gemini training & grounding	⚖️ Your call
`Googlebot`	Google	Classic search (feeds AI Overview)	✅ Must allow
`Bingbot`	Microsoft	Bing search + Copilot	✅ Must allow
`CCBot`	Common Crawl	Public corpus (training source for most open models)	⚖️ Your call
CN engines	Baidu/ByteDance/Alibaba	Baiduspider also feeds ERNIE; Doubao and Qwen have their own UAs	✅ Allow

Three distinct behaviors:

Retrieval bots (the SearchBot family) index for AI search; citations link back to you — this is GEO traffic. A lead-gen site has no reason to block them.
Session bots (the -User family) fire when a user pastes your link into a conversation or an agent visits on a user’s behalf. Blocking them turns away visitors who are already at your door.
Training bots (GPTBot / ClaudeBot / Google-Extended / CCBot) collect corpora. No direct traffic; the upside is indirect (models “know” your brand), the cost is corpus inclusion.

2. Copy-paste robots.txt (this site’s production config)

# AI retrieval crawler whitelist (blocking these = forbidding AI citation)
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: CCBot
Allow: /

# Everything else
User-agent: *
Allow: /
Sitemap: https://aigcharness.com/sitemap-index.xml

We allow the training bots too — this is a lead-generation content site, and everything on it is public collateral; being “known” by models is pure upside for us. If your site carries paid content or proprietary data, flip the four training UAs to Disallow: / — retrieval stays open and unaffected.

3. Two traps

Trap 1: assuming group order matters. robots.txt matching is by most-specific user-agent group, so an explicit whitelist above User-agent: * works. The common failure is the inverse: shipping only User-agent: * Disallow: / and forgetting the whitelist entirely. Always verify with the vendors’ robots testing tools after changes.

Trap 2: blocking Googlebot to opt out of AI. Google’s AI Overview draws from the main index; blocking Googlebot kills your classic search presence too. To say no to Gemini training only, block Google-Extended — it does not affect ranking.

4. The companion pieces

robots.txt only decides who gets in the door. What AI crawlers see once inside is decided by:

llms.txt — the human-readable site manifest for AI
The full GEO stack — JSON-LD, FAQ schema, BLUF structure and the rest of the 7 layers

All three together are the foundation of “citable by AI.”