AEO for documentation sites

Documentation sites are closer to answer-ready than almost any other kind of page, because every docs page exists to answer a question somebody has while they are trying to do something. They still lose a lot at extraction, for reasons specific to how docs get built: pages titled after an API object instead of the task, answers that exist only as a code block, the same page published under five version paths, and platforms that render navigation and content tabs in the browser. Each of those is fixable in the template or the writing guide, which is where docs teams already have leverage.

This post covers the four places docs pages lose answer-readiness, how to check each one by hand and across a docs site, and where llms.txt fits. It applies the general method in Answer Engine Optimization: a page-level guide to docs specifically.

Why are documentation sites a special case?

Because their readers arrive with a task, and their pages are written for people who already know where they are. A developer who lands on /docs/v3/auth/keys from the sidebar knows they are in the authentication section of version 3. An extractor holding one passage from that page knows only what the passage says. Docs written for in-context navigation lean on that context constantly: “as above,” “this method,” “see the previous step,” a heading that just says “Usage.”

The other difference is that docs pages are often generated. Reference pages come from an OpenAPI file or code comments; guides come from a static site generator with a shared template. That cuts both ways. A fault in the template is on every page, and a fix in the template is too.

Should docs pages be titled after tasks?

Yes, for guides; reference pages can keep the object name if the title also says what the object is for. A task title states the question the page answers, which is the same property that makes a heading useful to an extractor.

The same applies one level down. A section heading called “Usage” gives a chunk nothing to stand on when it is separated from the page. “Retry a request after a 429” does. Question headings vs. keyword headings covers the heading side in more depth; on a docs site, a task-shaped heading usually works better than a literal question, because developers search the way they would describe the task.

Should the answer be prose or code?

Both, in that order. Put one or two sentences that answer the task in words directly under the heading, then the code, then what the code’s parameters do. The sentence is what an extractor can lift and quote; the code is what the reader copies.

A fictional example of the pattern, for a page on retries:

To retry a request that returned a 429, wait for the number of seconds in the Retry-After header, then resend the same request. The client does this automatically for up to three attempts when retries is set.

Then the code block, then a short table of the options. Compare that with a section that opens with the code block and explains nothing until after it. The facts are identical, but only one version states the answer in a sentence that survives without its surroundings. If the page defines a term, the opening sentence should be a definition; How to write a definition an assistant can lift covers that sentence shape.

Three code-specific rules:

• Code must be text. Put code in <pre><code>, never in a screenshot. Google’s guidance for AI features lists “making sure that important content is available in textual form” among its best practices, and an image of a terminal is not text.
• Name the subject inside the section. “Call this method with the ID” means nothing once the section is a standalone chunk. “Call client.orders.cancel() with the order ID” does. Chunking: why passages have to stand alone explains why.
• Keep parameter references in tables. A parameter table with name, type, default, and description extracts cleanly; the same information in a paragraph does not. Tables and lists that survive extraction has the markup rules.

What about language tabs and collapsed sections?

They survive if the text is in the HTML response and fail if it isn’t. Many docs platforms show one language at a time in tabs, or collapse long examples. If all the tab panels ship in the HTML and CSS hides the inactive ones, a non-rendering crawler still receives every language. If the inactive panels are fetched or built by JavaScript only when clicked, they were never in the document a crawler received. Check view-source, not the rendered page. JavaScript content and what never makes the trip covers how to tell the two apart.

How should versioned docs handle canonicals?

Decide, per page, whether the versions are the same document or different documents, and mark them accordingly. Versioned docs are the largest source of near-duplicate pages on most developer sites: /docs/v2/webhooks, /docs/v3/webhooks, and /docs/latest/webhooks can be byte-for-byte identical or completely different.

Google’s documentation on duplicate URLs ranks the signals it uses: redirects and rel="canonical" are strong signals that a URL should become canonical, and sitemaps are a weak one. It also says not to use robots.txt for canonicalization, and that using noindex to choose a canonical within a site is not recommended because it removes the page from Search entirely.

That gives a workable policy:

The second row is the one teams get wrong. If v2 and v3 describe different request formats and both canonicalize to latest, the v2 answer has been declared a duplicate of a page that contradicts it. An assistant that retrieves the v3 page for a v2 question has been handed the wrong answer by your own markup. Put “v2” in the title so the two pages are distinguishable as text, not just as paths. Status codes, redirects, and canonical drift on AI surfaces covers the general failure modes.

Where does llms.txt fit?

On a docs site, if anywhere. The proposal’s own site says llms.txt files “are used most heavily for software documentation,” with coding agents following the file to reach API references and tutorials. The proposal also suggests offering a Markdown copy of pages at the same URL with .md appended, discoverable through a rel="alternate" link.

Whether any given assistant reads the file is a separate question with a less tidy answer, and llms.txt: what it is, how it works, and whether to bother goes through it. For a docs site the cost is low, because the file is a curated index of pages you already maintain, and the llms.txt generator builds a first draft from a list of URLs. Treat it as an addition to crawlable HTML, never a replacement for it.

How do you audit a docs site by hand?

Pick one page from each template: a guide, a reference page, a tutorial, and the docs home. For each:

1. Read the title and H1 aloud. Does it name a task, or an object nobody outside the team would search for?
2. Read the first sentence under each H2. Does it answer the heading in words, or does it start with a code block or “This section describes”?
3. Open view-source. Search for a line from the second language tab and for a line inside a collapsed example. If either is missing, it is not in the response.
4. Check the canonical. Find rel="canonical" in the head and compare it with the version path you are on.
5. Open /llms.txt. If it exists, do the links resolve, and do they point to current versions?

How do you audit it across a docs site?

A crawler covers all of it without a terminal. Crawl the docs subdirectory, then add custom extraction rules:

• //h1 and //title to list every page title. Sort the export; object-named titles stand out as single words.
• count(//pre) against the main-content word count, to find pages that are almost all code.
• //link[@rel='canonical']/@href beside the crawled URL. A spreadsheet formula such as =IF(A2=B2,"self","points elsewhere") sorts version pages into the two policies in the table above; filter the “points elsewhere” rows and confirm each target is the same document.
• count(//h2) and the text of the first <p> after each H2, if your crawler extracts lists.

From a terminal, the canonical check across a URL list:

while read -r url; do
  printf '%s\t%s\n' "$url" "$(curl -s "$url" | grep -oE '<link[^>]+rel="canonical"[^>]*>' | head -1)"
done < docs-urls.txt

GEOBee scores each docs URL on the AEO lens, including the answer block (20), headings (15), extractability (15), and lists and tables (10), and the SEO lens checks the canonical; the three lenses lists every weight. If your docs platform renders navigation client-side, run a deep scan so JavaScript-discovered links are crawled; scan modes explains the difference.

To score every page in your docs and find the ones whose answers live only in code, download GEOBee and crawl the docs subdirectory during the 7-day trial.