robots.txt patterns for AI: allow, block, and the middle

A robots.txt that handles AI bots well makes two separate decisions: who may use your content to train models, and who may surface it in answers. The four patterns below are the four defensible combinations of those two decisions, each as a file you can copy. Most broken files are not broken on policy — the owner knew what they wanted — but on mechanics, because robots.txt matching has three rules that work nothing like people assume.

So this post is those three rules first, then the four patterns, then the maintenance that keeps the file true after the vendors rename things. Who the individual bots are and what each one governs is its own reference — GPTBot vs OAI-SearchBot: a field guide to the AI crawlers — and this post leans on it rather than repeating it.

Start with the decision, not the file

The tokens exist so that training and search can be priced separately. GPTBot, ClaudeBot, Google-Extended, and their peers govern model training; OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot govern whether an assistant can find you at all. Blocking the first group is a rights decision. Blocking the second is a distribution decision. A file that hasn’t separated them is expressing an accident, not a position.

Most of the web has not decided. When Cloudflare analyzed the top 10,000 domains in June 2025, 3,816 had a robots.txt at all, and only about 14% of those files mentioned any AI bot. In the files that did, GPTBot was simultaneously the most-blocked token (312 domains) and the most explicitly allowed (61 domains) — the sites that have thought about it have reached opposite conclusions, which is fine. The failure mode is the other 86%: a position by default, held without knowing it.

The three matching rules that break AI files

A named group switches the wildcard off. Under RFC 9309, a crawler obeys the group that names it, and falls back to the User-agent: * group only when no group does. The two are never combined. The moment you add a User-agent: GPTBot block, GPTBot stops reading every rule under your wildcard — including the Disallow lines you wrote years ago and still assume apply. Any path you want to keep closed to a named bot must be restated inside that bot’s own group.

The longest path wins. When Allow and Disallow rules both match a URL, the rule with the most octets — the longer path — applies, and a tie goes to Allow. This is what makes section-level files workable: Allow: /blog/ beats Disallow: / for everything under /blog/, regardless of the order the lines appear in. Order never matters; length does.

One file per host, per scheme. blog.example.com does not inherit from example.com, and Anthropic’s documentation says outright that an opt-out must be repeated on every subdomain it should cover. A policy that exists on the apex and nowhere else is a policy for the apex and nowhere else.

Pattern 1: open, and said out loud

If your business depends on being found, the right file may contain no AI rules at all — absence means allowed. Being explicit still buys you two things: the file documents a decision rather than an omission, and a named group protects that bot from a wildcard someone tightens later.

# AI search crawlers — allowed deliberately (reviewed 2026-08-17)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow:

One group, several User-agent lines, an empty Disallow — all legal under the standard. The caution from the matching rules cuts both ways here: these three bots now ignore your wildcard group, so if User-agent: * disallows /admin/ or /search/, restate those lines in this group. An explicit allow that silently un-applies your hygiene rules is the first gotcha of this pattern, and it is invisible until you look for it.

Pattern 2: search yes, training no

The most common considered position, and the one the split tokens were designed for. Block every training token, mention no search crawler, and the file reads:

# Model-training crawlers — blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: MistralAI-Training
Disallow: /

Every token in that block is a vendor-documented training control. The less obvious ones: CCBot is Common Crawl’s crawler, the public corpus that has fed training across the industry, and MistralAI-Training is documented on Mistral’s robots page alongside its search and fetcher siblings.

What is absent does the other half of the work. These tokens are unmentioned and therefore governed by your wildcard group — which, if you follow this pattern, should not disallow anything you want surfaced:

The last row is the expensive one: blocking Googlebot to keep AI out removes you from Search entirely, because on Google’s side search and AI features are the same crawl.

Two tokens in that block deserve a sentence each. Google-Extended is not a crawler — Google documents it as a control token with no user agent of its own, governing whether Googlebot-crawled content trains future Gemini models or feeds their grounding. Google states it does not affect Search inclusion or ranking, and it does not control AI Overviews either; the only lever there is snippet controls on Googlebot itself. Applebot-Extended works the same way: Apple’s documentation says it does not crawl, it only decides whether Applebot-crawled data trains Apple’s foundation models, and disallowing it leaves you discoverable in Spotlight, Siri, and Safari.

Amazonbot is the awkward one. Amazon says its data “may be used to train Amazon AI models,” but it is also a general crawler feeding Alexa’s answers — blocking it is broader than a pure training opt-out, so decide it separately rather than reflexively appending it to the block above.

Pattern 3: the middle — rules with a path on them

Section-level rules are where the longest-match rule earns its keep. The shape: hold back what is genuinely yours, keep open what earns you discovery.

User-agent: GPTBot
User-agent: ClaudeBot
Allow: /blog/
Allow: /docs/
Disallow: /

Every URL under /blog/ and /docs/ is allowed because those rules are longer than Disallow: /; everything else is blocked. The inverse shape — open by default, closed by section — is a plain Disallow list for /research/, /pricing/, whatever the exclusive material is, with no Disallow: / at all.

Two cautions. First, the standard requires support for only two special characters — * matching any octet sequence and $ anchoring the end of a path — so keep expressions simple and remember that a rule like Disallow: /*.pdf$ is honored by the major AI crawlers but is not guaranteed beyond them. Second, the restatement rule applies with full force: a training bot named here no longer reads your wildcard group, so the middle pattern is where a site most often re-opens /internal/ to a bot it was carefully restricting. After any edit, read each named group top to bottom as if it were the only text in the file — for that bot, it is.

Pattern 4: closed, and what closed cannot buy

Full block: every named agent plus the wildcard, all Disallow: /. It is a legitimate position for a site whose value is exclusivity. Be precise about what it purchases — a request, honored by crawlers that honor requests.

It does not cover user-triggered fetchers. OpenAI states that robots rules “may not apply” to ChatGPT-User because a person initiated the fetch; Perplexity says Perplexity-User “generally ignores” the file; Meta documents that Meta-ExternalFetcher “may bypass” it. This is the fastest-growing traffic in the category — Cloudflare’s 2025 Year in Review measured user-action fetching growing more than fifteen-fold during 2025 — which means the requests robots.txt governs least are the requests growing fastest. And it does not cover crawlers that publish no policy at all: ByteDance’s Bytespider has no crawler documentation to link — there is no vendor page stating whether it honors the file — and when Cloudflare measured AI crawler activity across its network in mid-2024, Bytespider was both the most active AI crawler and the most blocked.

A block that must hold, rather than merely be requested, is a network-layer job — WAF rules matching user-agent and published IP range together. AI crawler access: the technical layer of AI search covers that layer, including the inverse failure where your CDN blocks bots your file allows.

Keeping the file true

AI rules decay faster than any other part of robots.txt. OpenAI’s agent list has grown from two to four; Anthropic currently documents exactly three tokens, and older names still circulating in copy-pasted blocklists match nothing. A Disallow aimed at an agent that no longer identifies that way is dead text that reads as protection.

The maintenance is small and specific. Re-read the vendor pages linked here quarterly and diff their agent lists against your file — the field guide to the AI crawlers tracks the current list and what each token governs. Keep a dated comment in the file itself recording when it was last reconciled. Deploy the same decisions to every host you own, because each one is a separate scope. And ignore folklore directives: Crawl-delay is honored by Anthropic, explicitly not by Amazonbot, and ignored by most search crawlers — it is not where control lives.

Then verify, because the file states intent and the edge decides outcomes. How to check whether AI crawlers can actually reach you is the six-check pass: fetch your own pages with real bot user-agent strings and confirm the answer the file promises is the answer the server gives.

The file across every host you own

At one host this is an afternoon. The honest version of the question is per-URL: which paths are allowed for which agents, on which hosts, and did the last edit re-open something a named group was supposed to keep closed. Without a terminal, that audit is a spreadsheet of hosts against agents, filled in from each host’s /robots.txt in a browser and from Search Console’s robots.txt report, which shows the fetched file and its status for every verified property. At larger scale, a crawler that evaluates your rules per URL replaces the spreadsheet.

In GEOBee, AI bot access is a weighted signal on the GEO lens, evaluated per URL against the crawlers that feed today’s assistants — so a rule that quietly closed an important section surfaces as a scored finding on the affected pages, not as a line in a file nobody re-reads. The three lenses lists the agents it checks and what the signal is worth.