Boilerplate ratio: how nav and footers shrink your page

Your page has a word count, and then it has the word count that survives extraction. The difference between the two is boilerplate: the header, the navigation, the footer, the cookie banner, the newsletter form, the related-posts rail. On a long article the gap is small. On a short product page wrapped in a mega-menu and a five-column footer, it can be most of the document.

That gap matters for two reasons, and only one of them is obvious. The obvious one is that boilerplate gets thrown away, so the page an extractor holds is smaller than the page you wrote. The less obvious one is that an extractor has to decide which block is the content, and a page dominated by template can push that decision the wrong way — keeping a sidebar, or keeping nothing.

This post covers what boilerplate is, how main-content extraction treats it, how to measure the ratio on one page and across a site, and the markup and template changes that make the content block easy to find. It sits under Answer Engine Optimization: a page-level guide, and it is the measurement half of a step what an answer engine reads describes in passing.

Key Takeaways

• Boilerplate is content repeated across a set of URLs — navigation, headers, footers, sidebars. Main-content extractors are built to discard it.
• Extraction is a heuristic, not a parser of your intent. Readability, the library behind Firefox Reader View, scores blocks partly on class and id names and penalizes link-heavy text.
• Boilerplate ratio is main-content text divided by whole-page text. There is no published threshold; it is useful for comparing templates against each other, not for passing or failing a page.
• The fixes are structural: one <main>, an <article> around the real content, <nav> and <footer> around the chrome, and fewer links in the template.

What counts as boilerplate?

Boilerplate is any text on a page that also appears, near-identically, on many other pages of the same site. MDN’s definition of the <main> element draws the line the same way: main content “should be unique to the document,” and “content that is repeated across a set of documents or document sections such as sidebars, navigation links, copyright information, site logos, and search forms shouldn’t be included” (MDN, retrieved 2026-10-10).

The practical list is longer than nav and footer:

The author bio is the row worth thinking about. If the only place your page states who wrote it and why they are qualified is a box that appears on two hundred other URLs, an extractor has good reason to treat it as furniture.

Why does main-content extraction discard it?

Extraction pipelines discard boilerplate because repeated template text is noise for anything that wants to know what a page is about. The people building training corpora say this directly. The FineWeb team, documenting how they built a large web text dataset from Common Crawl, report that Common Crawl’s own text-only files “retained too much boilerplate and menu text,” so they re-extracted from the raw HTML using the open-source trafilatura library instead (Penedo et al., The FineWeb Datasets, 2024, section 3.2). Trafilatura’s own paper frames the job the same way: “An essential operation in web corpus construction consists in retaining the desired content while discarding the rest” (Barbaresi, ACL 2021).

No answer engine documents which extractor it uses, so the useful question is how the published ones decide. Readability — “the readability library used for Firefox Reader View,” per its README (mozilla/readability, retrieved 2026-10-10) — is open source, so its rules can be read directly. Three of them explain most of what happens to a template-heavy page (Readability.js, commit ab4027a):

• Class and id names. Before scoring, it removes elements whose class or id matches a list of “unlikely candidate” words — including header, footer, menu, sidebar, breadcrumbs, related, social, and sponsor — unless they also match a rescue list such as article, content, or main. Elements with role="navigation", role="menu", or role="complementary" are removed the same way.
• Link density. Each candidate block’s score is multiplied by one minus its link density — the share of its text that sits inside links. The code comment states the assumption: “Good content should have a relatively small link density (5% or less).”
• A minimum size. By default it wants at least 500 characters of article text before it considers a result usable.

Read those together and the failure mode is clear. A navigation panel is almost all links, so it scores near zero. A footer is usually caught by name. But a content block that is itself short and link-heavy — a category page, a product page that is mostly “related” tiles — scores badly for the same reasons, and if it falls under the size floor, the extractor has very little to return.

That is one library, and it is not necessarily what any given assistant runs. What generalizes is the direction: every published approach treats repetition, link density, and naming as evidence, and none of them can read your intent. What do AI crawlers extract from your page? has the element-by-element inventory of what survives.

What is the boilerplate ratio?

Boilerplate ratio is the share of a page’s visible text that is not main content — or, stated the other way round, main-content text divided by whole-page text. A page with 900 words of text, 600 of them in the article, has two-thirds main content and one-third boilerplate.

There is no published threshold for a good ratio, and this post does not invent one. The ratio is useful comparatively. Measure your article template, your product template, your category template, and your landing-page template, and the one with the lowest share of main content is the one where extraction has the least to work with and the most chance of picking the wrong block.

Two cautions. Count words or characters of text, not bytes of HTML — inline SVG, scripts, and styles inflate the HTML without adding anything an extractor reads. And measure the response, not the rendered page, because content a script injects after load was never in the document a non-rendering crawler received.

How to measure it on one page

You need a browser and a spreadsheet. This takes about ten minutes for one URL.

1. Get the whole-page text. Open view-source:https://example.com/your-page/ to confirm the content is in the raw HTML at all — if your main copy is not in view-source, you have a rendering problem first and a boilerplate problem second. Then open the normal page, select all, copy, and paste into cell A2 of a spreadsheet as plain text.

2. Get the main-content text. Open the same page in Firefox and switch to Reader View. Because Reader View runs Readability, this is a reasonable stand-in for a heuristic extractor’s view of your page. Select all, copy, and paste into cell B2.

3. Count both. A word count that works in Sheets and Excel:

=IF(LEN(TRIM(A2))=0, 0, LEN(TRIM(A2)) - LEN(SUBSTITUTE(TRIM(A2), " ", "")) + 1)

Copy it across for column B, then divide B by A.

4. Read what Reader View kept. The ratio is the number; this is the finding. Check whether Reader View picked the right block, whether your first heading and its answer survived, and whether anything you consider content — a comparison table, a pricing note, an author credential — was dropped. If Firefox does not offer Reader View for the page at all, note it: its quick check for whether a page looks like an article said no. That check is, in its own documentation’s words, likely to produce “both false positives and false negatives,” so treat it as a prompt to look closer rather than a verdict.

Repeat for one URL per template. Four rows in a spreadsheet is enough to show which template is the problem.

How to measure it across a site

At scale, the method is the same — main text against whole-page text — with a crawler doing the fetching.

In a crawler, with custom extraction. Most desktop crawlers let you add extraction rules by XPath or CSS selector. Add two:

• Main content: //main — or //article, or the selector your templates actually wrap the body in.
• Whole page: //body.

Set both to extract text, crawl, and export to a spreadsheet. If your crawler accepts XPath expressions that return a value, string-length(normalize-space(//main)) and string-length(normalize-space(//body)) return character counts directly, which avoids pasting long text into cells. Divide one column by the other, then sort by template or URL pattern.

Two things this surfaces that a single-page check does not. Rows where the main-content column is empty are pages with no <main> element at all — a finding in itself, and a list you can hand to a developer. And rows where main content equals whole-page text usually mean <main> wraps everything, including the navigation, which tells an extractor nothing.

A useful companion column is count(//a) against count(//main//a). The difference is how many links the template adds to every page, which is the mega-menu problem stated as a number.

From a terminal. If you have Python available, trafilatura gives you a main-content extraction from the command line to set against a raw text count:

URL="https://example.com/your-page/"
trafilatura -u "$URL" | wc -w

Trafilatura is a different heuristic from Readability, and the two will not always agree. That disagreement is informative: a page both extractors handle the same way has an unambiguous content block.

GEOBee’s AEO lens measures a related property directly. Its Extractability signal, weighted 15 of 100, checks for “enough real text (aim for 300+ words) versus HTML noise,” and its grid reports the word count of the main content for every URL (the three lenses).

How to fix a high boilerplate ratio

The fixes fall into two groups: making the content block easy to identify, and making the template smaller.

Mark the content block. Use exactly one visible <main> — MDN states that “a document mustn’t have more than one <main> element that doesn’t have the hidden attribute specified.” Inside it, wrap the article itself in <article>, which MDN defines as “a self-contained composition… intended to be independently distributable or reusable” (MDN, retrieved 2026-10-10). Put navigation in <nav>, the footer in <footer>, and sidebars in <aside>. Readability’s cleanup step removes footer and aside elements from the content it keeps, so the chrome you label as chrome is chrome it can drop.

Name things plainly. Class names are evidence to heuristic extractors. A content wrapper called content or article matches the rescue list; one called wrapper-3 or a hashed utility class matches nothing. Do not put words like related, sidebar, or social on the element that holds your body copy.

Shrink the mega-menu. A navigation panel with hundreds of links adds the same text to every page you publish. Ask whether every link needs to ship in every response, or whether category pages can carry the deep links while the global menu carries the top level.

Keep content out of the template. Anything you need extracted belongs inside the main block: the author credential, the last-reviewed date, the disclaimer that qualifies the answer. If it only exists in a sidebar, it is formatted as boilerplate.

Do not move content into collapsed widgets to save space. Accordions and tabs are a common response to a long page, and whether they survive depends on where the text lives. Text that ships in the HTML and is hidden with CSS is still in the response; text fetched only when someone clicks was never there. The extraction inventory covers the distinction, and it is checked the same way as everything else here — in view-source.

Once the content block is identifiable, the next question is whether each section inside it survives being split; chunking: why passages have to stand alone picks up from there.

What a better ratio does and doesn’t buy you

A lower boilerplate ratio makes your page easier to extract correctly. It does not make the page more likely to be cited, and nobody publishing extraction research claims it does. What it changes is narrower and verifiable: whether a machine that reduces your page to one block of text ends up holding the block you meant. You can measure that before and after a template change, and the number will either move or it won’t.

To measure main-content share across every URL rather than one template at a time, download GEOBee — it scores Extractability alongside the other answer-readiness signals and exports the result to CSV.