Server logs are the only record of which AI crawlers actually requested your pages, what they asked for, and what your server sent back. Everything else in an access audit (reading robots.txt, testing a URL as a bot) tells you what should happen. The log tells you what did. Reading it comes down to four steps: get the right log, label each request by agent, verify that the agents are who they claim to be, and pivot the result by agent, status, and path.
This post covers each step with a spreadsheet as the main tool. The log-reading step in How to check whether AI crawlers can actually reach you is a one-line request for a count by agent; this is the full method behind it, for when a count isn’t enough.
What logs can’t tell you is just as important. A request from a search crawler is evidence of access, not of use. No log line records whether a fetched page was ever shown or cited anywhere.
Which log do you need: origin or CDN?
You usually need the CDN log, or both. A site behind a CDN has two places a request can end, and each log sees only its own.
| Event | Origin log | CDN (edge) log |
|---|---|---|
| Request served from the edge cache | Not recorded | Recorded |
| Request blocked or challenged at the edge | Not recorded | Recorded, with the block status |
| Request passed through to the origin | Recorded | Recorded |
Two consequences follow. Origin logs undercount crawler activity on any cached page. And origin logs can’t show a block that happened at the edge, because the blocked request never arrived. If the question is “is our firewall stopping OAI-SearchBot?”, the origin log will answer with silence, which looks the same as the crawler never trying. When your CDN or firewall blocks bots robots.txt allows covers what those edge blocks look like.
Most CDNs offer log exports or a log-search view in their dashboard, and some classify known bots in their analytics without any export at all. If you don’t have access, ask for the edge log, not the server log, and name the period.
What does a log line contain?
A standard access log line records the client IP, the time, the request, the response status, the response size, the referrer, and the user-agent. The Apache Combined Log Format is the common baseline, and many servers and CDNs export something close to it:
203.0.113.7 - - [10/Oct/2026:13:55:36 +0000] "GET /pricing/ HTTP/1.1" 200 18342 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"The IP and line above are illustrative; 203.0.113.0/24 is a range reserved for documentation. The user-agent is the form OpenAI documents for GPTBot. Five fields matter for this work: IP, time, path, status, and user-agent. The response size helps too, because a 200 response that is a fraction of a page’s normal size is often a challenge page served with a success code.
Which agents should you look for?
Look for the tokens the vendors currently document, matched as substrings, because full strings change with version numbers. OpenAI’s own examples carry a version (GPTBot/1.4) that it notes may change.
| Token | Vendor | Job | Published IP list |
|---|---|---|---|
OAI-SearchBot | OpenAI | Search | openai.com/searchbot.json |
GPTBot | OpenAI | Training | openai.com/gptbot.json |
ChatGPT-User | OpenAI | User-requested fetch | openai.com/chatgpt-user.json |
Claude-SearchBot | Anthropic | Search | claude.com/crawling/bots.json (one list for Anthropic’s bots) |
ClaudeBot | Anthropic | Training | same |
Claude-User | Anthropic | User-requested fetch | same |
PerplexityBot | Perplexity | Search | perplexity.com/perplexitybot.json |
Perplexity-User | Perplexity | User-requested fetch | perplexity.com/perplexity-user.json |
Googlebot | Search, including AI features | common-crawlers.json on developers.google.com |
Two tokens you won’t find. Google-Extended has no user-agent string of its own; Google says it is a robots.txt control only, so it never appears in a log. And tokens no vendor documents any more, such as anthropic-ai, won’t appear from the vendor’s current crawlers. GPTBot vs OAI-SearchBot explains each agent, and the AI crawlers reference keeps the list in one place.
How to label and pivot the log in a spreadsheet
A spreadsheet handles a filtered log comfortably. Spreadsheets have row limits, so if the raw log is large, ask for an extract filtered to lines containing any of the tokens above. That filter alone usually shrinks it to a size a spreadsheet can open.
1. Import with the double quote as the delimiter. In the Combined Log Format, the request, referrer, and user-agent are each wrapped in double quotes. Import the file as text and choose " as a custom separator. The line splits cleanly:
| Column | Contents |
|---|---|
| A | IP, identity, user, and [time] |
| B | Request: method, path, protocol |
| C | Status and size |
| D | Referrer |
| F | User-agent |
2. Add helper columns. With the split above:
- IP:
=LEFT(A2, FIND(" ", A2) - 1) - Date:
=MID(A2, FIND("[", A2) + 1, 11) - Path:
=INDEX(SPLIT(B2, " "), 1, 2)in Google Sheets, or=TEXTBEFORE(TEXTAFTER(B2, " "), " ")in current Excel - Status:
=VALUE(LEFT(TRIM(C2), 3))
3. Label each row by agent. Put the token list in a range, say Tokens!A2:A10, and add:
=IFERROR(INDEX(Tokens!$A$2:$A$10, MATCH(TRUE, ISNUMBER(SEARCH(Tokens!$A$2:$A$10, F2)), 0)), "other")In Google Sheets, wrap it in ARRAYFORMULA. Current Excel evaluates it as written.
4. Build four pivot tables. Each one answers a specific question:
| Pivot | Rows × columns | What it answers |
|---|---|---|
| Agent × status | Agent × status, count of requests | Which crawlers are getting 200s, and which are getting 403, 429, or 5xx |
| Agent × path | Agent × top paths | Whether the pages you care about are among those fetched |
/robots.txt by agent | Filter path = /robots.txt, rows = agent | Whether each crawler is reading your rules at all |
| Agent × date | Agent × day | Whether a crawler stopped arriving on a particular day, usually right after a deploy or a CDN change |
Read for absences first. A search crawler you believe is allowed and that hasn’t requested anything in the period is the finding. So is a crawler that fetches /robots.txt and nothing else, which is consistent with a disallow you didn’t intend. robots.txt patterns for AI covers how group matching produces those.
Then read the status column. A run of 403 or 429 responses to one agent and not to browsers is a bot-management rule. A run of 301s to one agent may be redirect handling that treats bots differently. Status codes, redirects, and canonical drift explains what each status does to a crawler.
How do you verify that a bot is real?
Match the request’s IP against the vendor’s published ranges. The user-agent header is a string any client can send, so a log line that says GPTBot proves nothing by itself. All four vendors above publish JSON lists of IP prefixes for this. Anthropic adds that if a crawler’s source IP is on its list, the crawler is coming from Anthropic. Google also documents reverse DNS verification: its common crawlers resolve to hostnames under googlebot.com.
Without a terminal. There are two clean paths, and one that needs help.
- For Googlebot, use Search Console. The Crawl Stats report, under Settings, is Google’s own record of its crawling: requests over time, broken down by response, file type, purpose, and Googlebot type. It’s available for Domain properties and root-level URL-prefix properties. It can’t be spoofed, because it isn’t built from your log.
- For the others, use your CDN’s bot classification if it has one. Some CDNs mark requests from known, verified bots in their analytics, which does the IP check for you.
- Otherwise, hand the list to a developer. In the spreadsheet, build a pivot of distinct IPs per agent. That list is usually short. The published prefixes vary in size, from single addresses to blocks covering about a thousand, so matching them by eye isn’t reliable. Send the distinct IP list and the vendor JSON URLs, and ask for each IP marked as inside or outside its vendor’s ranges.
From a terminal. A short Python script does the match against any of the vendor files, all of which use the same prefixes format:
import ipaddress, json, sys, urllib.request
url, ips_file = sys.argv[1], sys.argv[2] # vendor JSON URL, file with one IP per line
data = json.load(urllib.request.urlopen(url))
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in data["prefixes"]]
for line in open(ips_file):
ip = ipaddress.ip_address(line.strip())
print(ip, "verified" if any(ip in n for n in nets) else "NOT in published ranges")Run it once per vendor with the distinct IPs for that vendor’s agents. Requests that claim an agent’s name but come from outside its ranges are not that agent. Exclude them from every count, and treat them as a security question rather than an AI-access one.
What do user-requested fetches tell you?
ChatGPT-User, Claude-User, and Perplexity-User fetch a specific URL because a person asked an assistant something that needed it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because the fetch was requested by a user. So these agents can appear on paths you disallowed. That’s documented behaviour, not a bug.
Their presence on a URL is a narrow signal: at least one person, in one assistant, caused that page to be fetched. It isn’t a count of citations, and it can’t be read as one. The log records the fetch and nothing about what was done with it. It is still worth a column in the agent × path pivot, because the pages these agents request are pages someone thought to ask about.
How often should you do this?
Do it after any change that could affect access: a CDN or firewall rule, a robots.txt edit, a migration, a new bot-management setting. Keep a monthly snapshot of the four pivots for comparison. OpenAI notes that robots.txt changes can take about 24 hours to reflect in search results, and Perplexity says changes can take up to 24 hours to take effect, so compare windows that start after that delay.
Crawl volume also isn’t a quality measure. OpenAI notes that if you allow both GPTBot and OAI-SearchBot, it may use one crawl for both purposes. Fewer requests from one agent after a change can mean the other agent’s crawl is covering it. AI crawler access: the technical layer of AI search puts the logs alongside the rest of the access picture, and training vs. retrieval crawlers covers which agents you want in the first place.
Logs show what arrived. A crawl shows what each page offers once it does. To check robots.txt access for the AI crawlers on every URL of a site, download GEOBee and run it during the 7-day trial.