Training vs. retrieval crawlers: the block decision

Blocking GPTBot stops OpenAI from using your content to train its models, and it does nothing to whether you appear in ChatGPT’s search answers. That is the whole answer to “should I block GPTBot” at the level of mechanics: it is a rights decision with no documented visibility cost, as long as you leave the search crawler alone. The hard part is not that one line. It is knowing which of the other dozen agents in your logs belong to the same decision.

Every AI vendor now separates its crawling into jobs, and each job is a separate decision with a separate price. This post sorts the agents by job, states what blocking each kind actually costs, and gives the robots.txt for the four policies most sites end up choosing. Who each individual bot is, in detail, lives in GPTBot vs OAI-SearchBot: a field guide to the AI crawlers; this post is the decision that sits on top of that reference.

What are the three kinds of AI crawler?

AI agents do one of three jobs: collect content that may train a model, crawl to build the index an assistant retrieves from, or fetch a single URL because a person asked for it. The vendors document these as independent controls. OpenAI’s bot documentation says so in five words — “Each setting is independent of the others” — and Anthropic, Perplexity, Google, and Apple draw the same lines in their own terms.

Sources: OpenAI, Anthropic, Perplexity, Google common crawlers, Google user-triggered fetchers, Apple, Common Crawl, Meta.

Read the table as three blocks, not fourteen rows. The top block is the training decision. The middle block is the distribution decision. The bottom block is mostly not a robots.txt decision at all.

Should I block GPTBot?

Block GPTBot if you do not want your content used to train OpenAI’s foundation models, and allow it if you do not mind. Either choice leaves your ChatGPT search visibility exactly where it was, because that is governed by OAI-SearchBot, and OpenAI documents the two as independent.

Two details from OpenAI’s page change how you should read your logs afterwards. If you allow both GPTBot and OAI-SearchBot, OpenAI says it may use the results of one crawl for both purposes — so a drop in GPTBot requests after a block tells you little about how often the search side reads you. And OpenAI notes that it can take roughly 24 hours from a robots.txt update for its search systems to adjust, so a same-day check proves nothing.

The same reasoning applies to ClaudeBot. Anthropic’s documentation says disabling it signals that your future materials should be excluded from training; indexing for Claude’s search is a separate agent, Claude-SearchBot.

What does blocking a training crawler cost?

Nothing that any vendor documents on any answer surface today. Training tokens govern future models, not current retrieval, and none of the vendor pages linked above describe a visibility penalty for opting out of training while leaving search crawlers allowed.

There is an open question the vendors do not answer: whether excluding your content from training changes how future models describe your brand or your topic from memory, without retrieval. No vendor publishes anything on that, and no number exists that would let you price it. Treat it as unknown in both directions. If someone quotes you a figure, ask for the source.

What blocking a training crawler buys is equally bounded. It is a request, honored by crawlers that honor requests, and it applies from the moment it is read — it does not remove anything already collected. Anthropic’s wording is precise about this: the block concerns your site’s future materials.

What does blocking a search crawler cost?

It removes you from that vendor’s answers. OpenAI’s documentation states that sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers.” Anthropic says blocking Claude-SearchBot prevents indexing for search and may reduce your visibility in user search results. Perplexity describes PerplexityBot as the agent that surfaces and links sites in its results, and recommends allowing it.

Google is the expensive case, because there is no separate AI search crawler to block. Google’s guidance on AI features says robots.txt directives for Googlebot are the control for crawling for Search, and that a page must be “indexed and eligible to be shown in Google Search with a snippet” to appear as a supporting link in AI Overviews or AI Mode. Blocking Googlebot to keep AI out removes you from Google Search. The levers for limiting what AI features show are snippet controls — nosnippet, data-nosnippet, max-snippet, noindex — and they apply to classic results too.

That asymmetry is the core of the decision. A training block costs you a say in future models, possibly. A search block costs you a distribution channel, definitely.

Does blocking a user-triggered fetcher do anything?

Partially, and less each year. These agents fetch one URL because a person asked an assistant about it, and most vendors document that robots.txt does not reliably govern them. OpenAI says rules “may not apply” to ChatGPT-User; Perplexity says Perplexity-User “generally ignores” them; Google says its user-triggered fetchers, including Google-Agent, “generally ignore robots.txt rules”; Meta says Meta-ExternalFetcher “may bypass” them. Anthropic is the exception — it documents Claude-User as respecting a block, which then prevents retrieval of your content in response to a user query.

Two consequences. If you need a user-requested fetch to fail, that is an authentication or network-layer job, not a robots.txt one. And if you want to be read, these fetchers are not how anyone finds you: they only visit a URL the assistant already has. Discovery runs through the search crawlers.

Which agents can’t be split cleanly?

Three vendors bundle jobs in ways a single line cannot separate, and each one needs a deliberate call rather than a reflex.

• Google. Search, AI Overviews, and AI Mode ride one crawl. Google-Extended separates Gemini training and grounding, and Google states it does not affect inclusion in Search. There is no token that keeps you in Search and out of AI Overviews.
• Apple. Apple says Applebot powers search in Spotlight, Siri, and Safari, and that its data may also be used to train Apple’s models. Applebot-Extended does not crawl; disallowing it opts your content out of training, and Apple says such pages “can still be included in search results.”
• Meta. Meta documents Meta-ExternalAgent as crawling for “training foundation AI models or improving products by indexing content directly.” One token, two jobs. Blocking it is broader than a training opt-out.

CCBot sits in its own category. It collects for Common Crawl’s open web corpus, so blocking it is a decision about everyone who builds on that corpus rather than about one vendor.

How do I decide which policy fits?

Answer two questions in order. First: does your business depend on being found? If yes, every search crawler stays allowed, and the remaining question is only about training. Second: is the value of some of your content its exclusivity — paid research, proprietary data, material you license? If yes, the training block, or even the search block, belongs on those sections rather than on the whole site.

The working files follow. Before you paste any of them, remember the matching rule that breaks more AI files than any policy choice: a bot named in its own group stops reading your User-agent: * group entirely, so any hygiene rules there — /admin/, /search/, /cart/ — must be restated inside every named group. robots.txt patterns for AI covers that rule and the other two in detail.

Policy 1: open, on the record

# AI search crawlers — allowed deliberately (reviewed 2026-10-10)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /admin/

# Everyone else, including training crawlers
User-agent: *
Disallow: /admin/

An empty file allows everything too. Naming the search crawlers buys you a dated record of the decision and protection from a wildcard someone tightens later.

Policy 2: search yes, training no

# Model-training controls — blocked (reviewed 2026-10-10)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /admin/

OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, and Applebot go unmentioned and fall to the wildcard, which allows everything except /admin/. Meta-ExternalAgent is left out on purpose, because blocking it also blocks indexing; add it if you have decided that trade is acceptable.

Policy 3: training blocked on the sections that matter

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /research/
Disallow: /reports/
Disallow: /admin/

User-agent: *
Disallow: /admin/

Training crawlers can read the blog and the docs and cannot read the two exclusive sections. Search crawlers can read everything, which means the exclusive sections can still be retrieved and quoted in answers. If that is also unacceptable, those paths need policy 4’s treatment or a login.

Policy 4: closed to AI, kept in Google Search

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Meta-ExternalAgent
Disallow: /

User-agent: *
Disallow: /admin/

Be precise about what this buys. It is a request to crawlers that honor requests. It does not cover the user-triggered fetchers that document ignoring the file, and it cannot keep Googlebot-crawled pages out of AI Overviews — only snippet controls on the pages themselves do that, at the cost of the same snippets in ordinary results. A block that has to hold belongs at the network layer, which AI crawler access: the technical layer of AI search covers.

How do I check the policy actually holds?

Read the file the way each bot reads it, on every host you serve. Open https://yourdomain.com/robots.txt in a browser, then the same path on every subdomain that serves content — robots.txt is per host, and Anthropic’s documentation says an opt-out must be repeated on every subdomain it should cover. For each named group, read it top to bottom as if it were the only text in the file, because for that bot it is.

Then test specific URLs against specific agents. The free robots.txt tester evaluates a file against any user-agent and path, which is the fastest way to confirm that OAI-SearchBot can reach your key pages and GPTBot cannot reach /research/. In Search Console, the robots.txt report shows the file Google last fetched and its status for every verified property. Across many hosts, a spreadsheet of hosts against agents, filled in from those two checks, is the audit; a crawler that evaluates rules per URL replaces the spreadsheet when the site outgrows it. Finally, confirm the edge agrees with the file — a CDN rule can block a bot the file allows — using the checks in how to check whether AI crawlers can actually reach you.

Put a date on the review. Vendors add, rename, and retire agents; a Disallow aimed at a token nobody sends any more is a comment, not a block.

GEOBee checks AI bot access per URL as a weighted signal on the GEO lens, so a rule that quietly closed an important section to a search crawler shows up as a finding on the affected pages — access, not citation, which no crawler can promise. The signal and its weight are listed in the three lenses; download GEOBee to run it against your own site.