GPTBot vs OAI-SearchBot: a field guide to the AI crawlers

GPTBot collects content that may train OpenAI’s models. OAI-SearchBot decides whether your site can be surfaced in ChatGPT’s search answers. They are separate settings, and blocking the first does nothing to the second. Almost every confused robots.txt in this category comes from treating those two as one decision.

The same split exists at every vendor now. OpenAI documents four agents, Anthropic three, Perplexity two, and Google runs the whole thing through Googlebot with one control token bolted on the side. This page is the reference for which is which, written as the questions people actually ask.

The whole list, at a glance

Sources: OpenAI, Anthropic, Perplexity, Google.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot is for training and OAI-SearchBot is for search, and OpenAI states that each setting is independent of the others. Disallowing GPTBot indicates your content should not be used in training OpenAI’s generative AI foundation models. Disallowing OAI-SearchBot means your site will not be shown in ChatGPT’s search answers, though it can still appear as a navigational link.

One wrinkle worth knowing if you read logs: if you allow both, OpenAI may use the results of a single crawl for both purposes rather than crawling twice. So the volume of GPTBot hits in your logs is not evidence about how often the search product is reading you.

Does blocking GPTBot remove me from ChatGPT?

No. Blocking GPTBot affects training only. The agent that governs whether you appear in ChatGPT search answers is OAI-SearchBot, and it is unaffected by a GPTBot rule.

The failure that actually causes disappearance is subtler. A blanket rule written before OAI-SearchBot existed — a User-agent: * disallow, or a bot-management policy that categorized all AI crawlers together — can catch the search crawler without anyone editing a line. That is the common shape of “we blocked training and vanished from ChatGPT”: nobody blocked search on purpose.

Also note the propagation lag. OpenAI says it can take roughly 24 hours from a robots.txt update for its systems to adjust for search results, so checking the same afternoon proves nothing either way.

Which Anthropic bot decides whether Claude can find me?

Claude-SearchBot does. Anthropic’s documentation splits its crawling into the same three jobs: ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for pages fetched because a user asked a question.

Disabling Claude-SearchBot prevents Anthropic from indexing your content for search, which Anthropic says may reduce your visibility and accuracy in user search results. Disabling Claude-User prevents retrieval of your content in response to a user query. Disabling ClaudeBot signals that your future materials should be excluded from training datasets, and nothing else.

Anthropic also supports the non-standard Crawl-delay directive, which most search engines ignore outright. If you have a genuine load concern rather than a policy one, that is one of the few places the line does something:

User-agent: ClaudeBot
Crawl-delay: 1

Does PerplexityBot train a model?

No. Perplexity states that PerplexityBot is designed to surface and link websites in search results and is not used to crawl content for AI foundation models. Perplexity documents no training crawler at all.

That makes the decision unusually simple. If you want to be findable in Perplexity, allow PerplexityBot; there is no training trade-off attached to it. The second agent, Perplexity-User, fires when a user’s question requires visiting a page, and Perplexity says it generally ignores robots.txt because a user requested the fetch.

What does Google-Extended actually control?

Google-Extended controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps. It does not control Google Search.

It is also not a crawler. Google states plainly that Google-Extended has no separate HTTP request user agent string — crawling is done with existing Google user agents, and the token exists only as a control in robots.txt. You will never see it in your access logs. Google says directly that it does not impact a site’s inclusion in Google Search and is not used as a ranking signal.

User-agent: Google-Extended
Disallow: /

That rule is a statement about Gemini training and grounding. It is not a statement about Search, and it is not a statement about AI Overviews.

Does Google-Extended keep me out of AI Overviews?

No, and this is the single most common misconception in the category. Google’s guidance on AI features in Search is explicit: AI is built into Search, and robots.txt directives for Googlebot are the control for how sites are crawled for Search. To limit what is shown from your pages, the controls are the ordinary snippet directives — nosnippet, data-nosnippet, max-snippet, or noindex.

Eligibility follows the same logic. To be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet. Google states there are no additional technical requirements.

The practical consequence catches a lot of sites: a max-snippet:0 directive set years ago, for a results page of ten blue links, still applies to a surface that did not exist when it was written. The page ranks and is structurally ineligible for the feature, and nothing tells you.

Why do some of these bots ignore robots.txt?

Because a person asked for the specific page, and the convention has always treated user-requested fetches differently from automated crawling. OpenAI says robots.txt rules may not apply to ChatGPT-User for that reason. Perplexity says Perplexity-User generally ignores them for the same reason. Google formalizes the distinction in its own taxonomy, sorting its agents into common crawlers that always respect robots.txt for automatic crawls, special-case crawlers that may or may not, and user-triggered fetchers that ignore it because the fetch was requested by a user.

Two consequences worth internalizing. A Disallow will not reliably stop someone from pasting your URL into an assistant and reading the contents back — if that matters, it is an authentication or network-layer decision, not a robots.txt one. And a user-initiated fetcher cannot find you; it only visits a URL the assistant already has. Discovery runs entirely through the search crawlers.

Which of these actually decide whether I get found?

Four agents, and only four: OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot. Those are the crawlers that build the indexes assistants retrieve from when they need current information. Everything else on the list either governs training, validates ads, or fetches a URL somebody already handed over.

This is the asymmetry to design around. The training crawlers are the ones with a loud public debate attached, so they get blocked deliberately and often. The search crawlers are quiet, and get blocked by accident — by a wildcard rule, an inherited bot-management policy, or a subdomain nobody checked. The bot people argue about is not the bot that decides whether they exist.

What about anthropic-ai, Claude-Web, CCBot, and Bytespider?

anthropic-ai and Claude-Web are not in Anthropic’s current documentation, which lists three bots: ClaudeBot, Claude-SearchBot, and Claude-User. Those older tokens spread widely through copy-pasted blocklists during 2023 and 2024. A Disallow naming an agent that no longer identifies itself that way is not a block; it is a comment.

CCBot, Bytespider, and Amazonbot are a different category. None of them is an answer-engine search crawler, so none of them affects whether an assistant can cite you. CCBot collects for Common Crawl, the public web corpus that has been used as training data across the industry, which is why it shows up in training blocklists. The others belong to companies with their own separate reasons for crawling. Decide on them as a training-and-scraping question, and keep that decision away from the four agents above.

The general rule: if a user-agent token is not currently documented by the vendor whose behavior you are trying to change, the line does nothing. Audit against the vendor page, not against a blocklist you inherited.

How do I confirm a bot is really who it says it is?

Match the request against the vendor’s published IP ranges, because the user-agent header is a string anyone can send. Every vendor here publishes current ranges as JSON for exactly this purpose:

Google additionally supports reverse DNS verification: its common crawlers resolve to crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. Anthropic publishes an IP list too, but warns against using it as an opt-out mechanism — blocking their IPs also blocks them from reading the robots.txt file that carries your preference, so the rule you wrote never gets seen.

If you are writing WAF rules, combine both signals: match the user-agent string and the published IP range, and set the allow rule at a higher priority than your general bot-mitigation rules. Perplexity publishes step-by-step guidance for Cloudflare and AWS WAF that generalizes cleanly to the other vendors.

Which ones should I allow?

If being found matters to your business, allow all four search crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot — and make the training question separate. That is the position the vendors built the split tokens to support:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Nothing in that block touches search. Note in particular what is absent: Googlebot. Blocking Googlebot to keep AI out removes you from Google Search entirely, because on Google’s side it is the same crawl.

If you have a real reason to withhold content — proprietary research, paid material, anything whose value is exclusivity — do it at the section level rather than site-wide, and be honest with yourself about which of the two decisions you are making. Access is not a moral position; it is a trade between exposure and control, and the tokens exist so you can price the two separately.

Whichever position you take, the thing to verify is that the file says what you think it says on every host you own. AI crawler access: the technical layer of AI search walks through the checks — fetching your own pages with real bot user-agent strings, catching CDN rules that override robots.txt, and finding the snippet directives that quietly gate AI Overviews. The three lenses lists every agent GEOBee checks per URL and what AI bot access is worth on the GEO score.