When your CDN or firewall blocks bots robots.txt allows

Your robots.txt can allow every AI search crawler and the crawlers can still be turned away, because the request never reaches the part of your stack that robots.txt describes. CDN bot-management settings, firewall rules, rate limiters, and challenge pages sit in front of the origin and make their own decision about each request. None of them reads robots.txt. When they say no, the crawler receives a 403, a 429, a 503, or a challenge page, and nothing appears in your analytics because the page never loaded.

This is the most common way a site that has done the policy work correctly still ends up unreachable, and it is invisible from a desk that only reads robots.txt. This post covers how the edge blocks bots, the specific case where it blocks robots.txt itself, how to detect it with and without a terminal, how vendors let you verify their crawlers, and the allow rule that holds.

It is the edge-layer detail behind AI crawler access: the technical layer of AI search, and step 3 of how to check whether AI crawlers can actually reach you is the quick version of the test below.

Why doesn’t robots.txt control this?

robots.txt is a request to well-behaved crawlers, not an access control, and the systems in front of your origin do not consult it. The standard says so directly: “These rules are not a form of access authorization” (RFC 9309, section 1). The crawler reads the file and decides what to request. The edge then decides, separately, whether to serve that request.

Google treats both as part of one requirement. Its guidance for AI features lists “ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure” as a single best practice (Google Search Central, last updated 2025-12-10). Perplexity’s crawler documentation goes further and publishes step-by-step instructions for writing allow rules in two firewall products, which says the problem is common enough to be worth documenting (Perplexity).

How does the edge block a crawler?

Five mechanisms account for most of it. Each produces a different response, which is how you tell them apart.

The challenge row is the one that hides best. A 200 that contains a challenge page instead of your content passes a status-code check and fails every content check. Look at the body, not just the code.

Any of these can be scoped by path or hostname, so a site can be open on /blog/ and blocked on /docs/, or open on the main host and blocked on a subdomain that sits behind a different configuration.

What happens when the edge blocks robots.txt itself?

When the edge challenges or errors on /robots.txt, a compliant crawler’s behavior is set by the status code it receives, and one of the outcomes is a full disallow. RFC 9309 specifies that if robots.txt returns a 4xx “unavailable” status, the crawler “MAY access any resources on the server,” and if it returns a 5xx “unreachable” status, the crawler “MUST assume complete disallow” (section 2.3.1).

That gives two failure shapes:

• 503 or 500 on robots.txt. A challenge or overloaded edge that returns a server error tells every standards-following crawler the whole host is off limits, whatever the file says. Your allow rules are never read.
• 403 on robots.txt. The crawler may treat the site as unrestricted, then hits the same edge rule on every page. You get the opposite of your policy in theory and a block in practice.

Google’s Crawl Stats report in Search Console shows this for Googlebot directly: its host-status panel tracks the failure rate for robots.txt fetches, alongside DNS and server-connectivity problems, and its response breakdown groups crawl requests by status (Search Console Help). It is under Settings, and only for root-level properties. There is no equivalent report from the other vendors, so for them the check is yours to run.

How to detect it without a terminal

There are three sources, and they answer different questions. Use all three.

1. Request as the bot. In browser developer tools, set a custom user-agent in the network conditions panel to a search crawler’s published string (the field guide lists them), then load /robots.txt and two or three important URLs. Read the status of the document request and look at what rendered. In a crawler, set the user-agent to the same string and crawl a list of important URLs; then crawl again with the default user-agent and put the two status columns side by side in a spreadsheet.

Be clear about what this test can and can’t show. It reproduces rules that match on user-agent. It does not reproduce rules that match on IP, because your request comes from your IP, not the vendor’s. A rule that allows only verified crawler IPs will block your spoofed request even though it would let the real crawler through. A 403 here is a lead, not a verdict; confirm it in the next two sources.

2. Read the CDN’s security or bot analytics. Most CDN dashboards have a security-events or bot-traffic view that can be filtered by user-agent or by verified-bot category. Filter for the search crawlers your policy allows and look at the action column: allowed, challenged, blocked, rate-limited. This is the most direct evidence available, because it shows the decision the edge made on real crawler requests. If you do not have access, ask for this exact view, filtered to the last 30 days.

3. Read the access logs, or ask for them. Origin logs only show requests the edge let through, so a crawler that is blocked at the edge may be absent rather than erroring. Absence is the finding. Request a 30-day count of requests grouped by user-agent and status, filtered to OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot. A search crawler your policy allows that has made zero successful requests in a month is almost always an edge problem.

How to detect it from a terminal

If you have a terminal, the user-agent test runs over a list of URLs in one pass. The same caveat about IP-based rules applies.

UA="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
for path in /robots.txt / /docs/ /pricing/; do
  printf "%s  %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -A "$UA" "https://example.com$path")" "$path"
done

Any line that is not 200, especially on /robots.txt, is worth taking to whoever owns the edge configuration.

How do vendors let you verify their crawlers?

Every major vendor publishes a way to confirm a request really came from it, which is what an allow rule should match on. User-agent strings can be copied by anyone; IP ranges and DNS cannot.

Perplexity’s guidance on the rule itself is the clearest any vendor publishes: match the user-agent and the source IP together, set the action to allow, give that rule higher priority than blocking rules, and automate updates of the IP lists, since “always use the most current IP ranges from the official JSON endpoints” (Perplexity). That pattern applies to every vendor in the table.

The fix: an allow rule that holds

The fix is a configuration change at the edge, not an edit to robots.txt, and it usually belongs to someone else. Phrase the request in their terms:

Create an allow rule, evaluated before any bot-management, challenge, or rate-limit rule, matching requests where the user-agent contains OAI-SearchBot, Claude-SearchBot, or PerplexityBot and the source IP is in that vendor’s published range. Refresh the IP lists automatically. Apply it to every hostname, and confirm /robots.txt returns 200 to these agents.

Three details make it hold. Match on both conditions, so the rule cannot be abused with a copied user-agent. Put it above the managed rules, because a managed category evaluated first will block before your allow is reached. And include robots.txt explicitly, since a challenge on that one file changes how compliant crawlers treat the whole host.

Which agents go in the rule is a policy decision. Search crawlers and training crawlers are separate tokens at every major vendor, and allowing one does not require allowing the other; training vs. retrieval crawlers sets out the options, and robots.txt patterns for AI has the matching files. The edge rule should say the same thing robots.txt says.

What about blocking at the edge on purpose?

Blocking at the edge is the only way to enforce a block, since robots.txt is voluntary. It has a cost worth knowing. Anthropic’s documentation says that alternate methods such as IP blocking “may not work correctly or persistently guarantee an opt-out,” because they can stop its crawler reading robots.txt, and recommends robots.txt for opting out (Anthropic). If the intent is to opt out of training, state it in robots.txt and keep robots.txt reachable, even if you also block at the edge.

Where this sits in an audit

Edge blocking is part of layer 1 of the audit loop, access, and it is the part a robots.txt review cannot see. Run the user-agent comparison on every hostname you own once a quarter and after any CDN or security configuration change, since a new managed rule or a plan change can switch on a bot category without anyone editing the site. How to audit a site for AEO and GEO puts it in sequence with the other layers.

A crawler that receives 200 and your content is reachable. Whether it does anything with what it fetched is a separate question, but no other work on the page matters until that first answer is yes.

GEOBee reads each page’s robots.txt verdict for eleven AI crawlers, and its user-agent is configurable in Preferences, so you can export one crawl per agent and compare the status columns in a spreadsheet. Download GEOBee to run it during the 7-day trial.