Status codes, redirects, and canonical drift on AI surfaces

Before an answer engine can extract anything from your page, it has to arrive at a URL that returns content, and before it can credit that content to you, something has to decide which of your URLs the content belongs to. Status codes answer the first question. Redirects and canonicals answer the second. When they disagree with each other, the content you wrote can end up attributed to a URL you did not intend, or to none.

This is old technical SEO, and most of it still works the way it always has. What has changed is the number of fetchers relying on it, and how little most of them document. This post covers what each class of response does to fetching and attribution, the specific ways canonicals drift, why a crawler should report a redirect or an error as “not scored” rather than as zero, and how to check all of it by hand and across a site. It belongs to the access layer described in AI crawler access: the technical layer of AI search.

Key Takeaways

• Google documents that content from a redirecting URL is ignored and the final target’s content is processed instead, and that content from 4xx and 5xx responses is ignored entirely.
• A canonical is a hint. Google states it “may choose a different page as canonical than you do,” and Search Console shows both the declared and the selected one.
• OpenAI, Anthropic, and Perplexity document nothing about how their crawlers handle status codes, redirects, or canonicals. Build so that it doesn’t matter.
• A redirect or an error is not a page that scored zero. It is a URL with no page to score, and an audit should say so.

Which URL does an AI surface actually fetch?

It fetches the URL it has, and ends up with content from whichever URL in the chain finally returns a 2xx. For Google, that rule is documented: “Any content Google receives from the redirecting URL is ignored, and the final target URL’s content is processed instead” (Google Search Central, retrieved 2026-10-10).

Google’s AI features inherit this, because they sit on the index. A page “must be indexed and eligible to be shown in Google Search with a snippet” to appear in AI Overviews or AI Mode (Google Search Central, retrieved 2026-10-10). Whatever decides which URL gets indexed decides which URL is eligible.

For the standalone AI crawlers, there is no documentation to check. OpenAI’s, Anthropic’s, and Perplexity’s crawler pages cover user-agent strings, robots.txt, and IP ranges, and nothing about redirects or canonicals (OpenAI, Anthropic, Perplexity, all retrieved 2026-10-10). The defensible position is to make every important URL resolve in one hop to a 200 whose canonical names itself — a configuration with only one reasonable interpretation, whatever the fetcher’s rules are.

What does each status code do?

Each class of response does something different to fetching and indexing, and Google documents all of them on one page:

Source: Google Search Central, HTTP status codes and network errors, retrieved 2026-10-10.

Two rows correct common assumptions. A 410 is not handled more aggressively than a 404 by Google: “All 4xx errors, except 429, are treated the same.” And a 5xx is not instantly fatal — already-indexed URLs “are preserved in the index, but eventually dropped” — which is why an intermittent server error looks harmless in a spot check and still costs you over a month.

What does a redirect chain cost?

Each hop is a fetch that returns no content, and the costs are documented at both ends. Google’s crawlers “follow up to 10 redirect hops,” and its crawl troubleshooting guide says to “watch out for long redirect chains, which have a negative effect on crawling” (Google Search Central, retrieved 2026-10-10). Search Console reports a “redirect chain that was too long” and “a redirect loop” as redirect errors.

For AI crawlers, the best public measurement is old but specific. Vercel and MERJ found ChatGPT’s crawler spending “an additional 14.36% of fetches following redirects,” against 1.49% for Googlebot, on top of 34.82% of fetches landing on 404s (Vercel, 17 December 2024). Their conclusion was that “the high 404 rates from AI crawlers highlight the importance of maintaining proper redirects.” Confirm the pattern in your own logs before relying on numbers that predate most of today’s crawlers.

The redirect type also matters, and two kinds deserve attention:

• Temporary redirects left in place. Google states that for a temporary redirect, “the indexing pipeline doesn’t use the redirect as a signal” that the target should be canonical (Google Search Central, retrieved 2026-10-10). A 302 installed during a migration and never changed leaves the old URL as a candidate.
• JavaScript redirects. Google warns that with a JavaScript redirect, it “might never see it if rendering of the content failed.” The standalone AI crawlers do not render JavaScript at all, per the Vercel and MERJ analysis — so to them a JavaScript redirect is a 200 whose body says almost nothing. What do AI crawlers extract from your page? covers the rendering gap.

Why are soft 404s worse than real ones?

A soft 404 is a page that says “not found” while returning 200. Google’s definition is “a URL that returns a page telling the user that the page does not exist and also a 200 (success) status code,” and such pages “are excluded from Search” (Google Search Central, retrieved 2026-10-10).

Google detects them and excludes them. Nothing in any AI vendor’s documentation says their crawlers do the same. A 200 is a success code, so a fetcher that does not run its own soft-404 detection has a successful response containing your error template — header, footer, a search box, and a sentence saying the page is missing. A real 404 or 410 is unambiguous to every client. Google’s own fix list is the right one: return 404 or 410 when a page is gone with no replacement, and a 301 when there is one.

How do canonicals drift?

A canonical drifts when the URL you declare stops being the URL every other signal points to. Google’s definition makes clear why that matters: the canonical is “the URL of a page that Google chose as the most representative from a set of duplicate pages,” and your declaration “is a hint, not a rule” (Google Search Central, retrieved 2026-10-10). Redirects and rel="canonical" are each a “strong signal,” sitemaps a “weak” one, and Google notes the methods “can stack” (Google Search Central, retrieved 2026-10-10). Stacking works in both directions — signals that disagree weaken each other.

The patterns that cause drift:

Search Console reports the result directly. In the Page indexing report, “Duplicate, Google chose different canonical than user” means “Google thinks another URL makes a better canonical” (Search Console Help, retrieved 2026-10-10). That is drift, observed.

Why should a crawler report these as “not scored”?

Because a redirect or an error has no page to score, and recording it as zero tells you something false. A 301 is often exactly what you wanted. A 404 might be a deleted page that should stay deleted. Neither is a page that failed an answer-readiness check, and folding them into an average as zeros drags the site-level number down for reasons unrelated to the content.

The opposite mistake is just as bad: scoring the content of a soft 404 or a redirecting URL’s interstitial as if it were a page. The audit then praises or condemns a template nobody intended anyone to read.

GEOBee takes the first position as a stated principle: “Redirects, errors, and non-HTML responses never fake a failing zero. Their score cells read as — (‘not scored’) so a 301 or a 404 can’t drag your averages down or hide behind a red that looks like a real failure” (Reading your scores). The status itself is still reported — 3xx as a warning, 4xx and 5xx as critical — so the URL stays on the list of things to look at without corrupting the scores of the pages that did return content.

How to check one URL by hand

You need a browser. No terminal required.

1. Watch the chain. Open developer tools, go to the Network tab, tick Preserve log, and load the URL. Every hop appears as its own row with its status code; select one and read the Location response header to see where it points. One 301 to a 200 is the target. Two or more hops, any 302 on a permanent move, or a 200 followed by a client-side navigation means there is work to do.

2. Read the declared canonical. Open view-source: on the final URL and search for rel="canonical". Confirm there is exactly one, it is in the <head>, it is absolute, and it uses the same protocol and host as the page.

3. Open the canonical. Paste the canonical URL into the address bar with the Network tab still open. It should return 200 directly — not redirect, not 404.

4. Compare with Google’s choice. In Search Console, run URL Inspection on the URL. It shows the “User-declared canonical” and the “Google-selected canonical” (Search Console Help, retrieved 2026-10-10). If they differ, Google has resolved your conflicting signals in a direction you did not choose.

How to check whether AI crawlers can actually reach you adds the bot-specific checks — requesting as each crawler’s user-agent, since a CDN can return a different status to a bot than to a browser.

How to check it across a site

In Search Console. The Page indexing report already groups the relevant failures: “Soft 404,” “Page with redirect,” “Redirect error,” “Not found (404),” “Server error (5xx),” and “Duplicate, Google chose different canonical than user.” Open each, export the example URLs, and you have a starting list. It reflects Google’s view only, and it shows examples rather than a full inventory.

In a crawler and a spreadsheet. Crawl the site and export the status code, the redirect target, the number of redirect hops if your crawler reports it, and the canonical link element. Then, in the spreadsheet:

• Filter status to 3xx, 4xx, and 5xx — these are the URLs to fix or to stop linking to.
• Look up each page’s canonical in the URL column and pull back its status, for example =XLOOKUP(D2, A:A, B:B, "not crawled"). Any canonical that resolves to something other than 200, or to “not crawled,” is drift.
• Flag rows where the canonical differs from the URL itself; most should be intentional, and the ones that are not are the finding.
• For soft 404 candidates, filter to 200 responses with a very low word count or a title containing “not found,” then open a handful.

From a terminal. To print every hop and status for one URL:

curl -sIL https://example.com/old-page/ | grep -iE '^(HTTP|location)'

Fix the links that point into chains as well as the chains themselves. A redirect kept for old inbound links is doing its job; an internal link that points at a redirect is an extra hop you chose, sent to every fetcher that follows it.

Getting this right does not get a page cited. It makes sure the page you wrote is the page that gets fetched, indexed, and credited — the precondition for anything else on this blog to matter.

To see status, redirect chains, and canonicals for every URL alongside answer-readiness scores, download GEOBee and run a crawl; redirects and errors come back marked “not scored,” not zero.