Reference · AI crawler directory

Who's knocking, and why.

Eleven agents, four vendors, five different jobs. Only four of them decide whether an assistant can find you. Verified against each vendor's documentation on 30 July 2026.

OpenAI Search

OAI-SearchBot

Decides whether your site can be surfaced in ChatGPT search answers.

robots.txt: Honors robots.txt
Finds you?: Yes — a discovery crawler

Keep it allowed

User-agent: OAI-SearchBot
Allow: /

OpenAI documentation ↗

OpenAI Training

GPTBot

Collects content that may be used to train OpenAI foundation models.

robots.txt: Honors robots.txt
Finds you?: No

To opt out

User-agent: GPTBot
Disallow: /

OpenAI documentation ↗

OpenAI User-triggered

ChatGPT-User

Fetches a specific URL because a ChatGPT user asked for it.

robots.txt: May not apply
Finds you?: No

To opt out

User-agent: ChatGPT-User
Disallow: /

OpenAI documentation ↗

OpenAI Ads

OAI-AdsBot

Validates pages submitted as ads.

robots.txt: Honors robots.txt
Finds you?: No

To opt out

User-agent: OAI-AdsBot
Disallow: /

OpenAI documentation ↗

Anthropic Search

Claude-SearchBot

Indexes content for search in Claude.

robots.txt: Honors robots.txt
Finds you?: Yes — a discovery crawler

Keep it allowed

User-agent: Claude-SearchBot
Allow: /

Anthropic documentation ↗

Anthropic Training

ClaudeBot

Collects content that may be used to train Anthropic models.

robots.txt: Honors robots.txt, plus Crawl-delay
Finds you?: No

To opt out

User-agent: ClaudeBot
Disallow: /

Anthropic documentation ↗

Anthropic User-triggered

Claude-User

Fetches a specific URL because a Claude user asked a question.

robots.txt: Honors robots.txt
Finds you?: No

To opt out

User-agent: Claude-User
Disallow: /

Anthropic documentation ↗

Perplexity Search

PerplexityBot

Surfaces and links websites in Perplexity search results. Not used for training.

robots.txt: Honors robots.txt
Finds you?: Yes — a discovery crawler

Keep it allowed

User-agent: PerplexityBot
Allow: /

Perplexity documentation ↗

Perplexity User-triggered

Perplexity-User

Visits a page when a user’s question requires it.

robots.txt: Generally ignores
Finds you?: No

To opt out

User-agent: Perplexity-User
Disallow: /

Perplexity documentation ↗

Google Search

Googlebot

Crawls for Google Search, including AI Overviews and AI Mode.

robots.txt: Honors robots.txt
Finds you?: Yes — a discovery crawler

Keep it allowed

User-agent: Googlebot
Allow: /

Blocking Googlebot removes the site from Google Search, AI Overviews included.

Google documentation ↗

Google Control token

Google-Extended

Controls use for Gemini training and grounding. Does not affect Google Search.

robots.txt: Control token only, never seen in logs
Finds you?: No

To opt out

User-agent: Google-Extended
Disallow: /

Google documentation ↗

The four that decide discovery

OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot build the indexes assistants retrieve from. Blocking a training crawler does not remove you from search, provided the same vendor's search crawler is still allowed.

The reasoning, question by question, is in GPTBot vs OAI-SearchBot: a field guide.

Test your own robots.txt against all eleven.

Open the tester →