Most robots.txt files treat AI crawlers as one category. They are not one category. A single vendor can run three separate bots, and each one decides something different about how your content shows up. Blocking all of them because you read a headline about training data is the most common way sites make themselves invisible by accident.
The three jobs a bot can have
Every crawler in this space does one of three things. Once you can tell them apart, robots.txt stops being guesswork.
- →Search indexing. Classic web crawlers that build an index. Googlebot and Bingbot. Blocking these removes you from ordinary search and from the AI features built on top of it.
- →Answer-time retrieval. Bots that fetch a page because a user asked a question right now. ChatGPT-User, Claude-User, OAI-SearchBot and PerplexityBot. Blocking these means you cannot be cited in an answer, even if the model already knows who you are.
- →Model training. Bots that collect content for future training runs. GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended and meta-externalagent. Blocking these has no effect on whether you get cited today.
The distinction that matters: training access is a licensing decision, retrieval access is a distribution decision. You can say no to the first and yes to the second.
The bots, by vendor
| Token | Vendor | What it controls |
|---|---|---|
| GPTBot | OpenAI | Training data collection. Not used to answer live questions. |
| OAI-SearchBot | OpenAI | Builds the index behind ChatGPT search. Block this and you drop out of ChatGPT's search results. |
| ChatGPT-User | OpenAI | Fetches a page when a user follows a link or asks ChatGPT to read it. |
| ClaudeBot | Anthropic | Training data collection. |
| Claude-User | Anthropic | Fetches a page during a live conversation. |
| PerplexityBot | Perplexity | Indexes for Perplexity answers and citations. |
| Google-Extended | Consent token for Gemini training and AI Overviews grounding. Does not affect Search ranking. | |
| Googlebot | Ordinary Search indexing, which also feeds AI Overviews. | |
| Bingbot | Microsoft | Bing index, which feeds Copilot. |
| CCBot | Common Crawl | Public crawl archive used to train many models, not just one vendor's. |
| Applebot-Extended | Apple | Consent token for Apple Intelligence training. |
| meta-externalagent | Meta | Meta AI training. |
The mistake that costs the most
Google-Extended is the one people get wrong most often. It is a consent token for training and grounding, and it has no effect on your Search ranking. Sites block it thinking they are protecting content from Gemini, which is true, and then separately wonder why they are not appearing in AI Overviews. Those are related outcomes, and the trade is real, but it should be a decision rather than an accident.
The second most common mistake is blocking OAI-SearchBot along with GPTBot because both start with the same three letters. GPTBot is training. OAI-SearchBot is the ChatGPT search index. One of those decisions is reversible in a week, the other costs you a quarter of visibility while the index catches up.
A robots.txt that says no to training and yes to answers
If your position is that your content should be findable and quotable but not absorbed into a training set, that is a coherent position and it is expressible:
# Training: no
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Retrieval and search: yes
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Disallow:That last line is worth a note. A bare `Disallow:` with no path means nothing is disallowed. It is the conventional way to say allow everything. `Disallow: /` is the opposite and blocks the entire site. One character between those two.
robots.txt is only half the answer
A crawler can be allowed on paper and blocked in practice. Bot protection at the CDN layer, aggressive rate limiting and WAF rules routinely return 403 to any user agent that looks automated, including the ones your robots.txt explicitly welcomes. Nothing in your robots.txt tells you this is happening.
This is why our check sends a live request using each crawler's real user-agent string and only counts a bot as allowed when both signals agree. If you take one thing from this post: read your robots.txt, then verify it from the outside.