Log file analysis for AI crawlers: who visits, what they get, and which are fake

Log file analysis is the only way to see what AI crawlers really do on your site. robots.txt says what you allow, and analytics tools mostly miss bots, because bots rarely run the JavaScript those tools depend on. Your server logs record every request. This guide shows how to find the AI crawlers in them, how to tell real ones from fakes, and what to fix once you can see them.

What an access log line contains

Most servers write logs in the “combined” format, which both Apache and nginx define the same way: the client IP, the time, the request line, the status code, the bytes sent, the referrer and the user agent^[1][2]. The last field is the one that names the crawler.

log_format combined '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

The AI crawlers to look for

SEO log file analysis used to mean tracking Googlebot. Now it means three kinds of AI visitor as well, and they do different things. OpenAI documents four user agents with an IP list for each^[3]. Anthropic runs ClaudeBot for training, Claude-User for fetches a user asks for, and Claude-SearchBot for search, all honouring robots.txt, and publishes one IP list for all three^[4]. Perplexity runs PerplexityBot for its search index and Perplexity-User for user requests, which “generally ignores robots.txt rules”^[5].

Common Crawl belongs on the list because it publishes its crawl as an open archive that anyone can download. Its crawler identifies itself as CCBot/2.0, and Common Crawl warns that it is “aware of crawlers falsely identifying themselves as CCBot”^[8]. Google-Extended is missing from the table on purpose: it is a robots.txt token, not a crawler, so it never appears in a log.

Try it on your own log

Paste a few hundred lines from your access log. The analyser groups requests by crawler, shows the status codes each one received and the path it asked for most, and lists the check that verifies each IP. It runs entirely in your browser.

Real or fake: verifying the IPs

Anyone can put GPTBot in a user agent. Cloudflare puts it plainly: user agent headers are “easily spoofed and are therefore insufficient for reliable identification”^[9]. The fix is to check where the request came from. For the AI operators, that means matching the IP against their published lists. Google documents a DNS check: run a reverse lookup on the IP, confirm the name ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup and confirm it returns the same IP^[6]. Bing offers a Verify Bingbot tool and publishes its IPs as well^[7].

host 66.249.66.1
# → crawl-66-249-66-1.googlebot.com

host crawl-66-249-66-1.googlebot.com
# → must return 66.249.66.1

How much fake traffic should you expect? There is no reliable industry figure, and the percentages that circulate rarely trace back to a method. One field test is worth knowing about, with its limits stated. Duane Forrester logged a new, unpromoted site for two weeks and checked every request against the published IP lists: 27 of 33 “AI assistant” requests and 692 of 799 “Googlebot” requests came from elsewhere, and some of the fakes went looking for files like .env.production^[10]. He calls it one small site over 14 days, not a trend. The lesson is the method, not the number.

What to fix once you can see them

If you are deciding what to allow in the first place, should you block GPTBot works through the trade-offs, the AI crawler checker tells you what your robots.txt currently allows, and the robots.txt generator writes a new one. For the agents that act for users rather than crawl, see what replaced ChatGPT agent.

No access to raw logs?

On many managed hosts you cannot read the raw log. CDNs fill part of the gap: Cloudflare’s AI Crawl Control, generally available since August 2025, shows how AI crawlers use a site and lets paid customers answer them with a custom 402 Payment Required message instead of a plain block^[11]. It is not the same as your own log, but it answers the first question: who is coming, and how often.

Questions people ask

Sources