Reading your robots.txt tells you what you intended. It does not tell you whether an AI crawler can reach your pages, because everything between the internet and your origin — CDN rules, WAF policies, bot management, rate limits — gets a vote, and none of it reads robots.txt.
The only way to know is to make the request and look at what comes back. This is six checks, in the order that finds problems fastest. The whole pass takes about twenty minutes for one site.
Run it before anything else, because it is the only check here that can invalidate all the others. A page the crawlers cannot fetch fails every content test for a reason that has nothing to do with its content, and a quarter spent rewriting answer blocks on pages behind a WAF rule is a quarter spent on nothing. Twenty minutes now decides whether the rest of the work is worth scheduling.
Each step gives two routes: one using a browser and a crawler, one using a terminal. They test the same thing. Five of the six need nothing but a browser and the crawler you already own — only step 6 requires access somebody else controls, and that step says exactly what to ask for.
Step 1: Read robots.txt on every host you own
By the end of this step you will have a list of every AI agent your site names, per host, and every rule that predates this year.
robots.txt is per host and per scheme. blog.example.com does not inherit from example.com, and Anthropic’s documentation says so explicitly — opt-outs have to be repeated on every subdomain you want covered. This is the single most common miss on any site with a separate blog, docs, help centre, or shop.
List your hosts, then open /robots.txt on each one in a browser tab — example.com/robots.txt, blog.example.com/robots.txt, and so on. A missing file is a finding in itself: it means that host allows everything, whatever the parent domain says. Search Console’s robots.txt report shows the fetched file and its status for each property you own, which is the faster route if the hosts are already verified there.
From a terminal, the same sweep is one loop:
for host in example.com www.example.com blog.example.com docs.example.com; do
echo "--- $host"
curl -s "https://$host/robots.txt"
doneRead each file for three things: which AI agents are named, which are left to the User-agent: * group, and whether any rule names a token the vendor no longer documents. Rules written in 2023 and 2024 commonly name agents that have since been renamed or split, and a Disallow on an agent that no longer identifies itself that way does nothing at all.
Verification: you can say, for each host, whether OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot are allowed — and whether that was a decision or an accident.
Step 2: Confirm the rule you wrote is the rule that applies
By the end of this step you will know which group each crawler actually reads, which is frequently not the one you assume.
Crawlers match the most specific user-agent group and follow only that group. A bot named in its own block does not also inherit your wildcard rules. That cuts both ways, and both directions cause real bugs:
User-agent: *
Disallow: /internal/
User-agent: GPTBot
Disallow: /pricing/Here GPTBot is allowed on /internal/, because it matched its own group and stopped reading. The wildcard block never applies to it. Sites that carefully add an AI-specific block often widen access without noticing.
The inverse is more damaging. If your only AI-related rule is a wildcard Disallow: /, then every AI crawler — including the search crawlers that decide whether you appear in assistant answers — is blocked, and no line in the file mentions AI at all.
Verification: for each agent you care about, name the single group it will match and read that group’s rules on their own, ignoring everything else in the file.
Step 3: Make the request as the bot
By the end of this step you will know whether a real request with a real bot user-agent gets a 200 or gets stopped.
This is the check that catches the failures robots.txt cannot show you.
In a crawler. Set the crawler’s user-agent to one of the strings below, point it at a list of your important URLs, run it, and read the status column. Repeat once per agent — four runs, four columns, and the comparison between them is the finding. This is the better route even if you are comfortable in a terminal, because it tests many URLs rather than one.
In a browser, for a single URL. Open developer tools, find the network conditions panel, switch the user-agent from automatic to custom, paste one of the strings below, and reload. Watch the document request’s status in the network tab.
From a terminal. Request one of your important URLs with each agent’s published user-agent string and record the status code:
URL="https://example.com/your-important-page/"
declare -a AGENTS=(
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"
"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
"Mozilla/5.0 (compatible; Claude-SearchBot/1.0)"
)
for ua in "${AGENTS[@]}"; do
code=$(curl -s -A "$ua" -o /dev/null -w "%{http_code}" "$URL")
echo "$code ${ua:0:60}..."
doneA 200 means the edge let it through. A 403, a 429, a 503, or a redirect to a challenge page means something in front of your origin is making a different decision than your robots.txt does — and from the vendor’s side, that is not an error you will ever see in your own analytics. It is a fetch that quietly fails.
Anthropic does not publish full example user-agent strings, so the fourth entry above is a token match rather than a copy of a real string. That is enough to exercise user-agent-based WAF rules, which is what this step is testing.
Verification: four status codes. Any of them not 200 is a finding, and the layer that produced it is your next question, not your robots.txt.
Step 4: Check that the response actually contains the page
By the end of this step you will know whether the bot received your content or an empty shell.
A 200 is necessary and not sufficient. Two things commonly go wrong after a successful status code. The response can be a JavaScript application shell with no text in it, in which case a pipeline that does not execute JavaScript receives nothing usable. Or the response served to the bot user-agent can differ from the one served to a browser, which is what a bot-mitigation rule that soft-blocks rather than hard-blocks looks like.
Without a terminal. Two checks, both in the browser. First, open view-source: on the URL and use the find bar to search for a distinctive sentence from the body — one you know is on the page and appears nowhere else. View-source shows the raw response before any JavaScript runs, which is precisely what a non-rendering crawler receives. If your sentence is not there, the fetch succeeded and the read failed. Second, for the bot-versus-browser comparison, keep the user-agent override from step 3 in place, reload, and read the document request’s transferred size in the network tab. Note it, switch the user-agent back to the default, reload, and compare.
The crawler version of the same comparison: run the crawl twice, once with a bot user-agent and once with the default, and diff the word-count column. Any URL where the bot run reports substantially fewer words is serving something different to bots.
From a terminal, compare the two byte counts directly:
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" "$URL" | wc -c
curl -s -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/131.0.0.0 Safari/537.36" "$URL" | wc -cByte counts within a few percent of each other are fine. A bot response that is a fraction of the browser response means you are serving something different to bots. A bot response that is roughly the same size but contains no visible sentences means your content is rendered client-side, and what an answer engine reads covers what survives that and what does not.
Verification: a distinctive sentence from your page appears in the raw bytes returned to the bot user-agent.
Step 5: Check the snippet directives
By the end of this step you will know whether your pages are eligible to be shown in Google’s AI features, which is a separate question from crawling.
Google’s guidance on AI features is explicit that robots.txt directives for Googlebot control crawling, and the way to limit what is shown from your pages is nosnippet, data-nosnippet, max-snippet, or noindex. To be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown with a snippet — and Google states there are no additional technical requirements beyond that.
So a max-snippet:0 directive makes a page structurally ineligible for a surface everyone is now worried about, while the page continues to rank normally. These directives are often set in a template years ago, for reasons that made sense against ten blue links, and nobody gets an alert.
Check both places they can live — the meta tag and the HTTP header.
Without a terminal. For the meta tag, open view-source: and find name="robots". For the header, which never appears in the HTML, open developer tools, go to the network tab, reload, click the document request, and read the response headers for x-robots-tag. While you are in view-source, search for data-nosnippet as well — those attributes suppress specific blocks rather than the whole page and are easy to inherit from a component nobody has read recently. At scale, most crawlers extract the meta robots directive and the X-Robots-Tag header as standard columns, so a single crawl answers this for every URL at once.
From a terminal:
curl -s "$URL" | grep -i -o '<meta name="robots"[^>]*>'
curl -sI "$URL" | grep -i "x-robots-tag"Verification: you know, for your important templates, whether a snippet is permitted and how long it may be.
Step 6: Read the logs for what is actually arriving
By the end of this step you will know which agents have visited in the last 30 days, which is the only evidence that any of the above is working in production.
This is the one step that needs access you may not have, and it is the one most worth not skipping — every other check tells you what should happen, and this one tells you what did.
If the logs aren’t yours to read, this is a small and specific thing to ask for, and phrasing it precisely is most of the battle. Send this:
For the last 30 days, from the access logs: a count of requests grouped by user-agent and response status, filtered to user-agent strings containing any of
OAI-SearchBot,GPTBot,ChatGPT-User,Claude-SearchBot,ClaudeBot,Claude-User,PerplexityBot,Perplexity-User,Googlebot. A CSV is fine.
That is a five-minute request for whoever owns the logs, and it is phrased in their terms rather than in SEO terms, which is why it tends to come back. Two shortcuts worth trying first: many CDN dashboards report verified bot traffic by agent without anyone touching a log file, and some crawlers ship a log-file analysis module that takes an exported log and does the grouping for you.
From a terminal, filter for the tokens rather than full strings, since version numbers change:
grep -Eo "OAI-SearchBot|GPTBot|ChatGPT-User|Claude-SearchBot|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User|Googlebot" access.log \
| sort | uniq -c | sort -rnRead the result for absences, not presences. A search crawler you believe you have allowed that has not visited in 30 days is the finding. So is a user-initiated fetcher arriving at pages you thought were closed, which is expected behaviour rather than a bug — those fetchers are documented as not following robots.txt, because a person asked for the specific page.
One caveat on interpreting volume: OpenAI states that if you allow both OAI-SearchBot and GPTBot, it may use the results of a single crawl for both purposes rather than crawling twice. Log counts are evidence of access, not a measure of how often a given product read you.
Verification: a 30-day count per agent, and an explanation for every zero.
What a failure at each layer looks like
| Symptom | Layer | Where to fix it |
|---|---|---|
| Rule names an agent the vendor no longer documents | robots.txt | Rewrite against the vendor page |
| Agent allowed in its own block but blocked by wildcard, or the reverse | robots.txt group matching | Restate rules inside the specific group |
403 / 429 / challenge page on a bot user-agent | CDN, WAF, bot management | Allow rule matching user-agent and published IP range |
200 with a fraction of the browser byte count | Bot mitigation serving a different response | Same as above, at higher rule priority |
200 with no visible sentences in the body | Client-side rendering | Server-render the content that matters |
| Page ranks but never appears in AI Overviews | max-snippet / nosnippet | Snippet directives in the template |
| Subdomain has no AI rules at all | Per-host robots.txt | Publish a file on every host |
Match the symptom to the layer before changing anything. The most common wasted afternoon in this work is editing robots.txt to fix a problem that lives in a WAF rule, then waiting a week to see whether it helped.
Doing this across a whole site
Six checks on one URL is a diagnosis. It is not an audit, because the honest question is not “does my site allow PerplexityBot” but “which of my 40,000 URLs are actually reachable, and are the important ones in that set?” Path-level Disallow rules, section-scoped CDN policies, and per-template snippet directives all mean the answer varies by URL.
Whatever you run it with, the output you want is a per-URL answer to the same four questions: is this path allowed for each agent, does a real request with that agent’s user-agent succeed, does the response contain the content, and do the snippet directives leave the page eligible.
In GEOBee, AI bot access is a weighted signal on the GEO lens, evaluated per URL across the crawlers that feed today’s assistants, so a blocked page that matters surfaces in the same ranked list as everything else wrong with it rather than in a separate report. The three lenses lists every agent it checks and what the signal is worth; findings and fixes shows how one blocked URL turns into a finding with a fix attached.
The reason to run this before any other AEO or GEO work is that it is the only part that is fully deterministic. Whether an assistant cites you is probabilistic and outside your control. Whether it can fetch a URL and receive the content is a yes or no you can check this afternoon and verify tomorrow. AI crawler access: the technical layer of AI search covers the policy side — which agents to allow, what each one governs, and where robots.txt stops being the truth.