AI robots.txt tester.
Paste your robots.txt and a path. Every AI crawler is evaluated with the same rules the vendors document: the matching group, the longest rule, Allow on a tie. Nothing is uploaded.
Open yoursite.com/robots.txt in a browser and paste the whole file. Each host and subdomain has its own file.
| Crawler | Job | Result | Decided by |
|---|---|---|---|
| OAI-SearchBotOpenAI | Search | ||
| GPTBotOpenAI | Training | ||
| ChatGPT-UserOpenAI | User-triggered | ||
| OAI-AdsBotOpenAI | Ads | ||
| Claude-SearchBotAnthropic | Search | ||
| ClaudeBotAnthropic | Training | ||
| Claude-UserAnthropic | User-triggered | ||
| PerplexityBotPerplexity | Search | ||
| Perplexity-UserPerplexity | User-triggered | ||
| GooglebotGoogle | Search | ||
| Google-ExtendedGoogle | Control token |
Highlighted rows are the four search crawlers that decide whether an assistant can find you. User-triggered agents may not follow robots.txt at all. Crawler list verified against vendor docs on 30 July 2026; see the AI crawler directory.
What this checks
Whether each agent may fetch the path, under RFC 9309 matching: case-insensitive user-agent groups, merged duplicate groups, * and $ wildcards, longest rule wins.
What it can't see
CDN and firewall rules that block bots before robots.txt is read, and noindex or nosnippet on the page. How to check those.
Do it for every URL
GEOBee runs this check on every crawled page and scores AI bot access inside the GEO lens. Try it free for 7 days.