llms.txt is a proposed Markdown file, served at /llms.txt, that gives AI agents a short, curated index of a site’s most useful content. It is worth an afternoon if your site is documentation that coding agents read, and it is a low-priority nice-to-have for almost everything else. No major answer-engine vendor documents reading it for search or citation, and Google’s own guidance says you do not need AI text files to appear in its AI features.
That is the answer. The rest of this post is the evidence behind it: what the proposal actually specifies, what changed in its second version, which systems are documented as reading the file, and how to tell whether anything reads yours.
What is llms.txt?
It is a convention proposed by Jeremy Howard on 3 September 2024 and revised as version 2 in August 2026. The problem it addresses is stated in the proposal plainly: web pages wrap their information in navigation, ads, and JavaScript, converting that back into clean text is “difficult and imprecise,” and context windows are still too small for most whole sites. The fix it proposes is a small Markdown file that summarizes the site and links to clean, LLM-readable versions of the pages that matter.
It is a proposal, not a standard. There is no RFC, no standards body, and no requirement on any crawler to fetch it. The specification lives in a public GitHub repository open for community input.
What goes in the file?
A file that follows the spec contains these sections, in this order:
- An optional byte-order mark.
- An H1 with the name of the project or site — the only required section.
- A blockquote with a short summary containing the key information needed to understand the rest of the file.
- Zero or more Markdown sections of any type except headings — paragraphs, lists — with more detail.
- Zero or more H2 sections, each a “file list”: a Markdown list where every item is a
[name](url)link, optionally followed by a colon and a note.
An H2 named Optional is used, by convention, for secondary links an agent can skip when it needs a shorter context. In v2 that is only a convention; the changes page says Optional sections “no longer carry mechanical semantics.”
Here is what that looks like for a hypothetical analytics product:
# Example Analytics
> Example Analytics is a self-hosted web analytics tool. It runs on a single
> server, stores data in PostgreSQL, and does not set cookies.
Pricing is per server, not per pageview. The API is REST, versioned at /v2/.
## Docs
- [Install guide](https://example.com/docs/install.md): Single-server setup on Linux
- [API reference](https://example.com/docs/api.md): Every endpoint, with request and response examples
- [Data retention](https://example.com/docs/retention.md): What is stored, for how long, and how to purge it
## Policies
- [Privacy policy](https://example.com/privacy.md): Data handling for the hosted version
- [Pricing](https://example.com/pricing.md): Per-server pricing and volume terms
## Optional
- [Changelog](https://example.com/changelog.md): Release notes since 2023Three things make a file like this useful rather than decorative. The summary states facts an agent would otherwise get wrong — no cookies, per-server pricing. Every link has a note saying what is behind it. And the links point at Markdown, not at the HTML pages the file was meant to route around.
What changed in version 2?
Version 2 made the file scoped, made Markdown pages discoverable, and dropped the tooling that expanded the file into one large context. Per the changes page, the main differences from v1 are:
| Area | Version 2 |
|---|---|
| Location | Root or any subpath; a file covers the pages under its path, and the most specific file applies — so /docs/llms.txt covers /docs/ |
| Markdown pages | Both page.html.md and page.md forms are accepted |
| Discovery | rel="alternate" type="text/markdown" points to a page’s Markdown version; rel="describedby" points to the covering llms.txt — as <link> elements or an HTTP Link: header |
| How agents use it | Agents are expected to “view or search” the file and follow relevant links, rather than expand it into one context |
| Optional section | A convention for skippable links, with no mechanical meaning |
The header form is the practical one for many sites, because it can be added in server or CDN configuration without touching templates:
Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"One thing not in the spec: llms-full.txt. Neither the current proposal nor its changes page defines it, so if you see one recommended, it is a convention around the convention, and nothing obliges an agent to look for it.
How is it different from robots.txt and sitemap.xml?
robots.txt states what automated clients may fetch; llms.txt suggests what an agent should read; sitemap.xml lists everything indexable. The proposal itself says robots.txt and llms.txt “have different purposes,” and that a sitemap is no substitute because it lists HTML pages, omits useful external links, and is generally too large to fit in a model’s context.
| File | Says | Enforces anything | Read by |
|---|---|---|---|
robots.txt | What may be crawled, per agent | A request honored by compliant crawlers | Every major crawler, documented |
sitemap.xml | Every URL you want indexed | Nothing | Search engines, documented |
llms.txt | What an agent should read first, and where the clean version is | Nothing | Undocumented for answer engines (see below) |
The consequence is the one to remember: llms.txt is not an access control. It grants nothing and blocks nothing. A URL listed in it and disallowed in robots.txt is still disallowed, and a page your CDN blocks for OAI-SearchBot is still blocked however prominently llms.txt links it. If your access layer is wrong, fix that first — AI crawler access: the technical layer of AI search covers it, and robots.txt patterns for AI has the files.
Which systems actually read llms.txt?
Very little is documented, and the honest summary is a short list of what is and a long list of what isn’t.
Documented. Chrome’s Lighthouse includes an llms.txt audit in its agentic browsing checks. It flags a server error when fetching the file, and treats a 404 as not applicable, because providing the file is, in Google’s words on that page, “optional at the moment.” That audit checks that the file is served without error; it says nothing about whether any Google product reads it.
Claimed by the proposal. The v2 text states that thousands of sites publish the file, that documentation platforms generate one automatically, and that the files are used most heavily for software documentation, where coding agents follow them to API references. Those are the author’s statements, linked here at their source; no independent count is cited.
Published, not consumed. OpenAI, Anthropic, and Google each publish an llms.txt for their own developer documentation, as the proposal notes. Publishing a file for your docs is not a statement that your crawler reads other people’s.
Not documented. As of this check, the crawler documentation from OpenAI, Anthropic, and Perplexity does not mention llms.txt at all. Whether OAI-SearchBot, Claude-SearchBot, or PerplexityBot fetch it, and whether anything they fetch from it affects retrieval or citation, is undocumented. Claims in either direction that do not link a vendor page are guesses.
Stated against. Google’s guidance on AI features in Search says: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” That is the clearest first-party statement in the category, from the vendor with the largest answer surface.
Is llms.txt worth it?
It depends on who reads your site, and the split is clean enough to decide quickly.
Worth doing: documentation, APIs, developer tools. This is the use the proposal was designed around and where its author reports it is used most. A coding agent asked to call your API benefits from a short index pointing at clean Markdown references. If your docs platform already generates the file, check its contents and keep it.
Cheap, low priority: marketing, publishing, e-commerce. Publishing one costs an afternoon and breaks nothing. What it does not do, on any documented evidence, is make an answer engine more likely to retrieve or cite you. Do it after the work that is documented to matter — crawler access, snippet controls, pages whose passages answer their headings — not instead of it.
Not worth it as a fix for anything. If you are reaching for llms.txt because you are not showing up in AI answers, the cause is almost certainly elsewhere, and the file will not reveal it.
The real cost is maintenance. A file that lists URLs you have since moved, or describes pricing you have since changed, is a curated source of wrong answers. Give it an owner and a review date, the same as robots.txt.
How do I write one, and tell whether anything reads it?
Write it by hand from the pages you would hand a new colleague: the five to fifteen that explain what you are, what you sell, and how it works. Use the format above, give every link a note, and point links at Markdown where you have it. The free llms.txt generator builds a spec-shaped file from a list of pages, which you should then edit by hand. Confirm it loads at https://yourdomain.com/llms.txt in a browser, and run Lighthouse in Chrome DevTools to confirm the audit does not flag a server error.
Telling whether anything reads it is a log question, and the only evidence that counts. Ask whoever owns your server or CDN logs for a 30-day count of requests to /llms.txt, grouped by user agent and response status. Some CDN dashboards let you filter requests by path, which answers the same question without a ticket. If no answer-engine crawler appears, you have learned something the vendors have not told you. Check again in a quarter.
GEOBee’s lenses contain no llms.txt signal. They score what is documented — per-URL AI bot access, extractability, and citability, with published weights in the three lenses — as readiness, not as a promise of citation. See pricing if you want to run it against your own site.