Every other question in AEO and GEO is downstream of one: can the thing reading for the assistant fetch your page at all? Answer blocks, schema, entity clarity, citability — all of it assumes a successful fetch. If the fetch fails, or never happens, the rest of the work is unread.
This is the least glamorous layer of AI search and the only one that is fully deterministic. You cannot control whether an assistant cites you. You can control, and verify, whether it is allowed to read you. What follows is the whole layer: which bots exist, what each one actually governs, where robots.txt stops being the truth, and how to check any of it on your own site.
The bots do three different jobs
Before any list of user agents is useful, the jobs have to be separated, because a single vendor runs several bots that do different things and are controlled independently.
Training. Collecting content that may go into a model’s training data. Blocking this has no effect on whether you appear in that vendor’s search or answer product today. It is a statement about future models.
Search indexing. Crawling to build the index the assistant retrieves from when it needs current information. This is the one that governs whether you can be surfaced and linked in an answer. Blocking it removes you from that product’s answers.
User-initiated fetching. A person asked the assistant something, and the assistant goes and fetches a specific URL to answer it. There is no crawl schedule; there is one request caused by one human. Several vendors state plainly that these fetchers may not follow robots.txt, precisely because a user asked for the page.
The common mistake is treating these as one decision. A site that blocks everything because it does not want to train a model also removes itself from the search index that assistants read at answer time, and finds out months later. A site that allows everything because it wants citations has also agreed to training. Both are defensible positions. Neither should be an accident.
Who’s who, as of July 2026
OpenAI
OpenAI documents four agents, and states that each setting is independent of the others.
| User agent | Job | What blocking it does |
|---|---|---|
OAI-SearchBot | Search | Your site is not shown in ChatGPT search answers, though it can still appear as a navigational link |
GPTBot | Training | Content is not used to train OpenAI’s foundation models |
ChatGPT-User | User-initiated fetch | Governs pages ChatGPT visits because a user’s question required it |
OAI-AdsBot | Ad landing-page validation | Only visits pages submitted as ads; not used for training |
Three details from that page are worth carrying around. First, if you allow both OAI-SearchBot and GPTBot, OpenAI may use the results of a single crawl for both purposes rather than crawling twice — so log volume is not a reliable signal of which product read you. Second, robots.txt changes take roughly 24 hours to be reflected in search results, so a same-day check after an edit proves nothing. Third, ChatGPT-User is explicitly not the control for search: OpenAI directs you to OAI-SearchBot for that.
Anthropic
Anthropic documents three bots, split along the same three jobs.
| User agent | Job | What blocking it does |
|---|---|---|
ClaudeBot | Training | Signals that future materials should be excluded from model training datasets |
Claude-SearchBot | Search | Prevents indexing for search, which may reduce visibility and accuracy in user search results |
Claude-User | User-initiated fetch | Prevents retrieval of your content in response to a user query |
Two operational notes on that page. Anthropic supports the non-standard Crawl-delay directive, which most search crawlers ignore, so it is one of the few places that line does something. And Anthropic states directly that blocking by IP address may not work reliably as an opt-out, because it also blocks them from reading the robots.txt file that carries your preference. A rule you have made unreadable is not a rule.
Perplexity
Perplexity documents two agents and, notably, no training crawler.
| User agent | Job | What blocking it does |
|---|---|---|
PerplexityBot | Search | Your site is not surfaced and linked in Perplexity search results |
Perplexity-User | User-initiated fetch | Controls which sites user requests can reach; generally ignores robots.txt because a user requested the fetch |
Perplexity states that PerplexityBot is not used to crawl content for AI foundation models. It also publishes something the others do not: explicit WAF configuration guidance for Cloudflare and AWS, and per-bot JSON endpoints listing current IP ranges. That guidance exists because the failure it addresses is common, which we will come back to.
Google is a different shape, and this is where most of the confusion in the category lives.
Google-Extended has no separate HTTP user agent string. Crawling is done with existing Google user agents; the token exists only as a control in robots.txt. It governs whether content Google crawls may be used to train future Gemini models, and for grounding in Gemini Apps. Google states explicitly that Google-Extended does not impact a site’s inclusion in Google Search and is not a ranking signal.
What it also does not do is control AI Overviews or AI Mode. Google’s documentation on AI features in Search is unambiguous: AI is built into Search, robots.txt directives for Googlebot are the control for how sites are crawled for Search, and the way to limit what is shown from your pages is the ordinary snippet controls — nosnippet, data-nosnippet, max-snippet, or noindex.
The eligibility rule follows from that: to be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet. There are, in Google’s words, no additional technical requirements.
That sentence is worth sitting with, because it means a page carrying max-snippet:0 is technically ranking and structurally ineligible to be surfaced in the feature everyone is now worried about. Snippet controls set years ago, for reasons that made sense against a results page of ten blue links, are still in force against a surface that did not exist when they were written. Nobody gets an alert about it.
The user-initiated fetchers are a separate category
Fetchers that act on a user’s explicit request generally do not follow robots.txt, and both OpenAI and Perplexity say so in their own documentation. OpenAI’s wording is that robots.txt rules “may not apply” to ChatGPT-User because the action is initiated by a user; Perplexity’s is that Perplexity-User “generally ignores robots.txt rules” for the same reason.
The practical consequences run in both directions.
If you are blocking, a Disallow line will not reliably stop a person from pasting your URL into an assistant and getting its contents back. Blocking there is a network-layer or authentication decision, not a robots.txt one.
If you are trying to be read, the fetcher is not your route to being found. It only fires on a URL the assistant already has. Discovery still runs through the search crawler — OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot — which is the one people most often leave blocked while allowing the fetcher they see in their logs.
That is the asymmetry to design around: the bots you can most easily block are the ones that matter least, and the bot that decides whether you exist to the product at all is the quiet one.
robots.txt patterns that actually say what you mean
Four positions cover almost every real site. Pick deliberately.
Open to everything. No AI-specific rules at all. Search crawlers and training crawlers both read you. This is the default for most of the web and the right default for anything whose business depends on being found.
Search yes, training no. The most common considered position, and the reason vendors split the tokens in the first place:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /Leaving OAI-SearchBot, Claude-SearchBot, and PerplexityBot unmentioned, and therefore allowed. Note what is not on that list: Googlebot. Blocking Googlebot to keep out AI removes you from Search entirely, because on Google’s side they are the same crawl.
Section-level. Allow the documentation and the blog, hold back the parts that are genuinely yours:
User-agent: GPTBot
Disallow: /pricing/
Disallow: /customers/
Allow: /docs/
Allow: /blog/Closed. Disallow: / for every named agent, understood as a statement about crawlers that honor the file. It does not cover user-initiated fetchers, and it does not cover anything that ignores the file entirely.
Three mechanics that cause more broken rules than any subtlety of policy. robots.txt is per host and per scheme, so blog.example.com needs its own file and does not inherit from example.com — Anthropic’s documentation says this explicitly, and it is the single most common miss on sites with a subdomain blog or docs. Most crawlers match the most specific user-agent group and only that group, so a bot named in its own block does not also inherit your User-agent: * rules. And a Disallow on a URL does not remove an already-indexed page; it only stops the fetch.
robots.txt is a claim, not a guarantee
The file states your intent. Whether the request actually succeeds is decided by everything in front of your origin — CDN rules, WAF policies, bot-management products, rate limiters, and origin-level blocks — none of which read robots.txt.
The disagreement runs both ways and both directions are worth checking.
A bot you have allowed can be blocked anyway. Managed bot-mitigation rulesets categorize AI crawlers as bot traffic and can challenge or drop them by default. From the vendor’s side this is not a 403 you will see in your own analytics; it is a fetch that quietly fails. This failure mode is common enough that Perplexity publishes step-by-step Cloudflare and AWS WAF instructions for allowing its agents, and Google’s own SEO guidance for AI features lists “ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure” as a single item, in that order.
A bot you have blocked can still be fetching. robots.txt is voluntary. Well-behaved crawlers from the four vendors above honor it; not everything crawling the web is one of those, and user-initiated fetchers are documented as not honoring it. If a block needs to be enforced rather than requested, it has to happen at the network layer.
There is also a middle case that is entirely self-inflicted: rules that were correct when written and no longer are. Vendors rename agents, split one bot into three, and retire tokens. A Disallow naming an agent that no longer identifies itself that way is not a block. It is a comment.
Checking it by hand
Do this before running anything at scale, on the three or four URLs you would most want an assistant to be able to read.
Read the file with fresh eyes. Fetch https://yourdomain.com/robots.txt and read it top to bottom, including every subdomain that serves content: blog, docs, help, support, shop. Confirm each AI agent you care about is either named deliberately or deliberately left to the wildcard group. Look specifically for rules that predate this year.
Confirm the request actually succeeds. Fetch your own page with the bot’s user-agent string and see what comes back. From the terminal:
curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
-s -o /dev/null -w "%{http_code}\n" https://yourdomain.com/your-page/A 200 means the edge let it through. A 403, a 503, or a challenge page means something in front of your origin is making a different decision than your robots.txt does. The vendors publish their full user-agent strings and their current IP ranges as JSON, so a proper allow rule can match both rather than trusting the header alone.
Check what came back, not just the status. A 200 that returns a JavaScript shell is not a readable page. Compare the byte count and the visible text of the bot fetch against a normal browser fetch. If your content only exists after a script runs, some pipelines will have it and some will not — what an answer engine reads covers what survives that trip and what does not.
Check your snippet controls. Grep the page source for nosnippet, max-snippet, and data-nosnippet, and check your X-Robots-Tag response headers as well as the meta tags. On Google’s surfaces these decide eligibility for AI Overviews and AI Mode directly, and they are frequently inherited from a template nobody has read in three years.
Read your logs for what is actually arriving. Filter access logs for the user-agent strings above over the last 30 days. The absence of a search crawler you believe you have allowed is the finding. So is a fetcher arriving at pages you thought were closed.
Checking it across a site
The by-hand method does not survive contact with a real site, for the ordinary reason: robots.txt is one file, but the effective answer is per URL. Path-level Disallow rules, section-level CDN policies, per-template snippet controls, and origin rules that apply to /docs/ but not /blog/ all mean the honest question is not “does my site allow GPTBot” but “which of my 40,000 URLs are actually reachable, and are the important ones in that set?”
That is a crawl. Whatever you use, the output you want is a per-URL answer to four questions: is this path allowed for each agent you care about, does a real request with that agent’s user-agent string succeed, does the response contain the content, and do the snippet controls on this page leave it eligible.
In GEOBee, AI bot access is a scored signal on the GEO lens, checked per URL across the crawlers that feed today’s assistants, so blocked-but-important pages surface as findings ranked against everything else rather than as a separate report nobody opens. The three lenses lists every agent it checks and what the signal is worth; findings and fixes shows how a blocked page turns into a ranked item with a fix attached.
Most robots.txt files are naming bots that no longer exist
If your AI rules were written in 2023 or 2024, they are probably wrong now, and wrong in a way that reads as correct.
Anthropic’s current documentation lists three bots: ClaudeBot, Claude-SearchBot, and Claude-User. Older tokens circulated widely in copy-pasted blocklists and are not in that list today. OpenAI’s list has grown from two agents to four, one of which — the search crawler that decides whether you appear in ChatGPT’s answers — did not exist when the first wave of “block GPTBot” advice went around. Perplexity split into two agents with different rules about whether robots.txt applies.
The consequence is a specific, quiet failure: a site that carefully blocked training in 2023, was silently opted out of search products that launched afterward because a wildcard rule caught them, and has been invisible in assistant answers for a year without a single line in its robots.txt looking wrong.
The fix is a calendar entry, not a technique. Re-read the four vendor pages linked above every quarter, diff them against your rules, and record the date you checked. Editorial dates are cheap; being wrong for a year is not.
What about llms.txt
llms.txt is a proposed convention for a Markdown file at your site root that points assistants at your most useful content. It is worth knowing about and it is not an access control — it grants nothing, blocks nothing, and no major vendor documents consuming it.
The strongest available evidence on this is Google’s, and it is blunt. Its guidance on AI features states that you do not need to create new machine-readable files, AI text files, or markup to appear in those features, and that there is no special structured data required. That is a first-party statement from the vendor with the largest answer surface, and it should carry more weight than any amount of adoption enthusiasm.
Publishing one costs an afternoon and breaks nothing. Just do not file it under access, and do not let it displace the work in this post. A perfect llms.txt on a site whose CDN drops OAI-SearchBot is a sign pointing at a locked door.
What access buys you, and what it doesn’t
Access is necessary and nowhere near sufficient. Allowing every AI crawler does not make you citable, quotable, or worth retrieving. It makes you eligible. What happens after the fetch — whether the page can be parsed cleanly, whether a passage answers the question it sits under, whether anything on it is worth lifting — is a different body of work, and it is the one where most of the remaining upside lives.
What makes access worth doing first is that it is the only part of this that is fully deterministic. Citation is probabilistic, multi-factor, and outside your control. Whether a bot can reach a URL, get a 200, and receive the content is a yes or no that you can check today, change this afternoon, and verify tomorrow. Fix that first, then argue about the rest.
Three things to do this week, in order. Read your robots.txt — every subdomain — against the four vendor pages linked here. Fetch three important URLs with real bot user-agent strings and confirm a 200 with content in the body. Grep your templates for snippet controls that predate AI Overviews. That is most of the layer, and almost nobody has checked it.
For what comes after the fetch, five things that keep a good page out of AI answers covers the faults that survive a perfectly successful crawl, and SEO, AEO, and GEO: what actually changed sets out which of the three lenses each of these signals belongs to.