How to check whether AI crawlers can actually reach you

Reading your robots.txt tells you what you intended. It does not tell you whether an AI crawler can reach your pages, because everything between the internet and your origin — CDN rules, WAF policies, bot management, rate limits — gets a vote, and none of it reads robots.txt.

The only way to know is to make the request and look at what comes back. This is six checks, in the order that finds problems fastest. All of them run from a terminal on a laptop, no tooling required, and the whole pass takes about twenty minutes for one site.

Step 1: Read robots.txt on every host you own

By the end of this step you will have a list of every AI agent your site names, per host, and every rule that predates this year.

robots.txt is per host and per scheme. blog.example.com does not inherit from example.com, and Anthropic’s documentation says so explicitly — opt-outs have to be repeated on every subdomain you want covered. This is the single most common miss on any site with a separate blog, docs, help centre, or shop.

List your hosts, then fetch each one:

for host in example.com www.example.com blog.example.com docs.example.com; do
  echo "--- $host"
  curl -s "https://$host/robots.txt"
done

Read each file for three things: which AI agents are named, which are left to the User-agent: * group, and whether any rule names a token the vendor no longer documents. Rules written in 2023 and 2024 commonly name agents that have since been renamed or split, and a Disallow on an agent that no longer identifies itself that way does nothing at all.

Verification: you can say, for each host, whether OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot are allowed — and whether that was a decision or an accident.

Step 2: Confirm the rule you wrote is the rule that applies

By the end of this step you will know which group each crawler actually reads, which is frequently not the one you assume.

Crawlers match the most specific user-agent group and follow only that group. A bot named in its own block does not also inherit your wildcard rules. That cuts both ways, and both directions cause real bugs:

User-agent: *
Disallow: /internal/

User-agent: GPTBot
Disallow: /pricing/

Here GPTBot is allowed on /internal/, because it matched its own group and stopped reading. The wildcard block never applies to it. Sites that carefully add an AI-specific block often widen access without noticing.

The inverse is more damaging. If your only AI-related rule is a wildcard Disallow: /, then every AI crawler — including the search crawlers that decide whether you appear in assistant answers — is blocked, and no line in the file mentions AI at all.

Verification: for each agent you care about, name the single group it will match and read that group’s rules on their own, ignoring everything else in the file.

Step 3: Make the request as the bot

By the end of this step you will know whether a real request with a real bot user-agent gets a 200 or gets stopped.

This is the check that catches the failures robots.txt cannot show you. Request one of your important URLs with each agent’s published user-agent string and record the status code:

URL="https://example.com/your-important-page/"

declare -a AGENTS=(
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"
"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
"Mozilla/5.0 (compatible; Claude-SearchBot/1.0)"
)

for ua in "${AGENTS[@]}"; do
  code=$(curl -s -A "$ua" -o /dev/null -w "%{http_code}" "$URL")
  echo "$code  ${ua:0:60}..."
done

A 200 means the edge let it through. A 403, a 429, a 503, or a redirect to a challenge page means something in front of your origin is making a different decision than your robots.txt does — and from the vendor’s side, that is not an error you will ever see in your own analytics. It is a fetch that quietly fails.

Anthropic does not publish full example user-agent strings, so the fourth entry above is a token match rather than a copy of a real string. That is enough to exercise user-agent-based WAF rules, which is what this step is testing.

Verification: four status codes. Any of them not 200 is a finding, and the layer that produced it is your next question, not your robots.txt.

Step 4: Check that the response actually contains the page

By the end of this step you will know whether the bot received your content or an empty shell.

A 200 is necessary and not sufficient. Two things commonly go wrong after a successful status code. The response can be a JavaScript application shell with no text in it, in which case a pipeline that does not execute JavaScript receives nothing usable. Or the response served to the bot user-agent can differ from the one served to a browser, which is what a bot-mitigation rule that soft-blocks rather than hard-blocks looks like.

Compare the two directly:

curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" "$URL" | wc -c
curl -s -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/131.0.0.0 Safari/537.36" "$URL" | wc -c

Byte counts within a few percent of each other are fine. A bot response that is a fraction of the browser response means you are serving something different to bots. A bot response that is roughly the same size but contains no visible sentences means your content is rendered client-side, and what an answer engine reads covers what survives that and what does not.

Grep the bot response for a sentence you know is in the body. If it is not there, the fetch succeeded and the read failed.

Verification: a distinctive sentence from your page appears in the raw bytes returned to the bot user-agent.

Step 5: Check the snippet directives

By the end of this step you will know whether your pages are eligible to be shown in Google’s AI features, which is a separate question from crawling.

Google’s guidance on AI features is explicit that robots.txt directives for Googlebot control crawling, and the way to limit what is shown from your pages is nosnippet, data-nosnippet, max-snippet, or noindex. To be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown with a snippet — and Google states there are no additional technical requirements beyond that.

So a max-snippet:0 directive makes a page structurally ineligible for a surface everyone is now worried about, while the page continues to rank normally. These directives are often set in a template years ago, for reasons that made sense against ten blue links, and nobody gets an alert.

Check both places they can live — the meta tag and the HTTP header:

curl -s "$URL" | grep -i -o '<meta name="robots"[^>]*>'
curl -sI "$URL" | grep -i "x-robots-tag"

Also grep the rendered page for data-nosnippet attributes, which suppress specific blocks rather than the whole page and are easy to inherit from a component nobody has read recently.

Verification: you know, for your important templates, whether a snippet is permitted and how long it may be.

Step 6: Read the logs for what is actually arriving

By the end of this step you will know which agents have visited in the last 30 days, which is the only evidence that any of the above is working in production.

Filter your access logs for the tokens, not for full strings, since version numbers change:

grep -Eo "OAI-SearchBot|GPTBot|ChatGPT-User|Claude-SearchBot|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User|Googlebot" access.log \
  | sort | uniq -c | sort -rn

Read the result for absences, not presences. A search crawler you believe you have allowed that has not visited in 30 days is the finding. So is a user-initiated fetcher arriving at pages you thought were closed, which is expected behaviour rather than a bug — those fetchers are documented as not following robots.txt, because a person asked for the specific page.

One caveat on interpreting volume: OpenAI states that if you allow both OAI-SearchBot and GPTBot, it may use the results of a single crawl for both purposes rather than crawling twice. Log counts are evidence of access, not a measure of how often a given product read you.

Verification: a 30-day count per agent, and an explanation for every zero.

What a failure at each layer looks like

Match the symptom to the layer before changing anything. The most common wasted afternoon in this work is editing robots.txt to fix a problem that lives in a WAF rule, then waiting a week to see whether it helped.

Doing this across a whole site

Six checks on one URL is a diagnosis. It is not an audit, because the honest question is not “does my site allow PerplexityBot” but “which of my 40,000 URLs are actually reachable, and are the important ones in that set?” Path-level Disallow rules, section-scoped CDN policies, and per-template snippet directives all mean the answer varies by URL.

Whatever you run it with, the output you want is a per-URL answer to the same four questions: is this path allowed for each agent, does a real request with that agent’s user-agent succeed, does the response contain the content, and do the snippet directives leave the page eligible.

In GEOBee, AI bot access is a weighted signal on the GEO lens, evaluated per URL across the crawlers that feed today’s assistants, so a blocked page that matters surfaces in the same ranked list as everything else wrong with it rather than in a separate report. The three lenses lists every agent it checks and what the signal is worth; findings and fixes shows how one blocked URL turns into a finding with a fix attached.

The reason to run this before any other AEO or GEO work is that it is the only part that is fully deterministic. Whether an assistant cites you is probabilistic and outside your control. Whether it can fetch a URL and receive the content is a yes or no you can check this afternoon and verify tomorrow. AI crawler access: the technical layer of AI search covers the policy side — which agents to allow, what each one governs, and where robots.txt stops being the truth.