llms.txt: what it is, how it works, and whether to bother
Is llms.txt worth it? The proposal, the file format with a worked example, what is documented about who reads it, and when the afternoon pays off.
crawlers
8 posts
Status codes, redirects, and canonical drift on AI surfaces
How redirect chains, soft 404s, 4xx and 5xx responses, and conflicting canonicals decide which URL gets fetched, indexed, and attributed — and how to check.
Training vs. retrieval crawlers: the block decision
Should you block GPTBot? Which AI agents train, which search, which fetch for a user, what each block costs, and four robots.txt policies to copy.
robots.txt patterns for AI: allow, block, and the middle
Four positions cover almost every site. The copy-paste file for each, the matching rules that quietly break them, and the maintenance the file needs.
What do AI crawlers extract from your page?
An element-by-element inventory of what AI crawlers keep, discard, and only conditionally read — plus the parts you can exclude on purpose.
How to check whether AI crawlers can actually reach you
Reading robots.txt tells you what you intended. Six checks that tell you what actually happens when an AI crawler requests one of your pages.
GPTBot vs OAI-SearchBot: a field guide to the AI crawlers
Every AI bot worth naming in your robots.txt, what each one actually controls, and which ones decide whether an assistant can find you at all.
AI crawler access: the technical layer of AI search
Which AI bots exist, what each one actually controls, why robots.txt is only half the answer, and how to check whether assistants can reach your pages.