Publishers keep getting told they have two options: let AI companies train on everything, or disappear from AI answers. That framing is wrong, and it is wrong in a way that pushes people into whichever decision they were most afraid of.
Two decisions wearing one coat
Training access decides whether your content is absorbed into a model's weights. It is a licensing question. Whether you say yes depends on what your content is worth, whether you have a syndication deal, and how you feel about the terms on offer.
Retrieval access decides whether an assistant can fetch your page and cite it while answering a question, today. It is a distribution question, and it behaves like search traffic. If you would want to rank on Google for a query, you want to be retrievable for the same query in ChatGPT.
The bots are different, the robots.txt tokens are different, and the consequences are different. Nothing forces you to answer both questions the same way.
What the big publishers actually do
Look at how large news organisations configure this and you see the split in the wild. Many block every training crawler while leaving search crawlers open, because search still sends traffic and training does not. Some negotiate a paid licence with one vendor and keep the rest blocked. Very few block everything, because blocking everything means nobody can cite you and the citation is the only part that sends a reader.
The pattern is not ideological. It follows the money: allow what sends traffic, charge for what does not.
Why most visibility scores punish you for this
Most AI visibility tools count blocked crawlers and subtract points. Block GPTBot, ClaudeBot, CCBot and Google-Extended on purpose and your score drops, then the tool tells you to unblock them. That is advice to reverse a deliberate licensing decision, generated by a tool that never asked what the decision was.
We split the two. Search and answer engines drive the AI Visibility score, because those genuinely decide whether you can be found and cited. Training crawlers are reported as a consent posture, open, selective or opted out, and they are worth zero points either way. A site that blocks every training bot and welcomes every answer engine scores exactly as well as a site that allows everything.
The configuration, in order
- →Decide your training position first. It is a business question, not a technical one, and the answer changes what everything else looks like.
- →Allow every retrieval and search bot regardless of that answer: Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot.
- →Apply your training decision to the training tokens only: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent.
- →Verify from outside. Check that your CDN is not returning 403 to the bots you just allowed, because robots.txt cannot tell you that.
- →Publish an llms.txt so the assistants that do reach you get a clean summary of what your site covers.
One caveat worth stating plainly
Blocking Google-Extended does reduce your presence in AI Overviews, because that token governs grounding as well as training. There is a real trade there and we are not going to pretend otherwise. It is still your call to make with the trade in front of you, rather than a number on a dashboard telling you that you failed.