How to Check Server Logs for AI Crawlers

You have done the work. The schema is in place, the site is fast, robots.txt looks clean. Now there is one question that everything else depends on: is anything from OpenAI, Google, Perplexity, or Anthropic actually coming to read the pages?

Most of the places you would think to look cannot answer that. Google Analytics is built to ignore automated visitors, so a crawler leaves no trace in it. Search Console only reports Google's own crawler, not anyone else's. The AI visibility tools that check whether ChatGPT mentions your brand are running their own prompts on their own servers; they never touch your site, so they cannot tell you who is crawling it.

Your server log can. Every request to your site, human or bot, gets written to a file the moment it arrives, with the visitor's address, the exact page they asked for, the time, and what your server sent back. It is the one honest record of who is really showing up. This guide covers how to get at that record, which lines to look for, how to tell a real AI crawler from something wearing its name, and what the answers tell you to do next.

Why the log is the only place that knows

A quick tour of why the easier-looking options come up empty:

Analytics tools miss bots on purpose. Google Analytics and most similar tools work by running a small piece of JavaScript in the visitor's browser. Crawlers usually do not run that JavaScript, and the good analytics tools also filter out known bots by design, because their job is to measure people. So an AI crawler can hit every page on your site and your analytics dashboard will show nothing.

Search Console is Google-only. It will tell you how often Googlebot crawled you and what it found, which matters, because Googlebot is what feeds Google's AI Overviews and AI Mode. But it says nothing about GPTBot, ClaudeBot, PerplexityBot, or any of the others.

AI visibility tools measure the answer, not the visit. A tool that tracks whether Perplexity cites you works by asking Perplexity a list of questions and recording the replies. That is useful, but it happens entirely on the tool's side. It cannot see your server, so it cannot confirm a crawler ever arrived or what it got when it did.

Analytics also only counts people who clicked. Seeing visits that arrived from an AI answer is a real and separate thing worth doing, and tracking AI traffic in GA4 covers it. But that is humans arriving after the fact. It tells you nothing about whether the crawl that made the citation possible ever happened.

The server log sits underneath all of that. It records the raw arrival, before any filtering, for every visitor of any kind.

Getting hold of your logs

Where your log lives depends on how your site is hosted.

A traditional server or VPS. If you run your own server with Nginx or Apache, the access log is a file on disk, usually at /var/log/nginx/access.log or /var/log/apache2/access.log. Your host may rotate it daily into dated, compressed files in the same folder.

Shared hosting with cPanel or Plesk. Look for "Raw Access Logs" (cPanel) or the "Logs" section (Plesk). You download a compressed file and open it locally. Some hosts only keep the last few days unless you turn on archiving, so switch that on now if you want history later.

Behind Cloudflare or another CDN. This is the important one, because a CDN answers many requests before they ever reach your server, so your origin log is missing a chunk of the traffic. Cloudflare's own analytics has a "Crawlers" or bot section that shows verified AI crawler activity without any log wrangling, and on paid plans its Logpush feature streams the complete request log somewhere you can query it. If you use a CDN, its dashboard is usually a better starting point than your origin log.

Managed platforms like Vercel, Netlify, Squarespace, Wix, or Shopify. These often do not give you a raw access log at all on standard plans. Vercel and Netlify expose recent logs in their dashboards and as paid log-drain add-ons. The hosted site builders generally do not expose crawler-level logs, so on those platforms you lean on Cloudflare in front of the site, on Search Console for the Google side, and on the crawl view inside an AI readiness audit for a point-in-time check.

If you genuinely cannot get a log from anywhere, that is worth knowing on its own: it means you have no visibility into crawler behaviour and should put a CDN with bot analytics in front of the site.

Reading a single log line

A standard log line looks like this:

20.171.207.15 - - [07/Sep/2026:14:22:04 +0000] "GET /pricing/ HTTP/1.1" 200 18342 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"

Reading left to right, the parts that matter are:

• 20.171.207.15 is the visitor's IP address. You will use this later to confirm the visitor is genuine.
• [07/Sep/2026:14:22:04 +0000] is the date and time, here in UTC.
• "GET /pricing/ HTTP/1.1" is the request: the method, then the exact path asked for. This tells you which pages a crawler is actually reading.
• 200 is the status code your server returned. 200 means it worked. 403 means blocked, 404 not found, 429 rate-limited, 503 unavailable. For a crawler you want to see 200 here.
• 18342 is the number of bytes sent. A suspiciously tiny number on a page that should be large can mean the crawler got an error page or an empty shell.
• The last quoted string is the user agent: the visitor's self-reported identity. This is what you search for to find crawlers, and also the thing that can be faked.

The crawlers worth searching for

There are dozens of AI-related user agents. For the purpose of "is my site being read for AI answers," they fall into three groups, and the group matters more than the individual name.

Search and answer crawlers (these decide live citations)

These build the index an AI answer is assembled from right now. If one of these is not reaching you, you are not eligible to be cited by that system today.

• OAI-SearchBot, per OpenAI's crawler documentation, is "used to surface websites in search results in ChatGPT's search features." Full string: ... compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot.
• PerplexityBot, per Perplexity's crawler documentation, is the one that surfaces and links sites in Perplexity answers. String contains PerplexityBot/1.0; +https://perplexity.ai/perplexitybot.
• Claude-SearchBot, per Anthropic's crawler documentation, "navigates the web to improve search result quality."
• Googlebot and bingbot still matter here, because Google's AI Overviews and AI Mode run on the normal Google index, and ChatGPT's search has historically drawn on Bing's. A healthy Googlebot crawl is part of AI readiness, not just classic SEO.

User-triggered fetchers (someone handed the assistant your link)

These fire when a real person pastes your URL into an assistant or asks it to look at a specific page: ChatGPT-User, Perplexity-User, Claude-User. Seeing these means people are actively pointing AI tools at your pages. They tend to ignore robots.txt because the request is on behalf of a named human, not an automated sweep.

Training crawlers (no visitor, no citation, just load)

These collect pages to train future models. GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl, whose public dataset feeds many models), Bytespider (ByteDance), meta-externalagent (Meta), Applebot-Extended, Amazonbot. Whether to allow these is a real decision with points on both sides, covered in should you block AI crawlers. For now, just know which bucket they are in when you see them.

One name you will not see in your logs is Google-Extended. Google states plainly that it "doesn't have a separate HTTP request user agent string" and that the crawling "is done with existing Google user agent strings." It is a control word you put in robots.txt to opt out of Gemini training, nothing more. If you went looking for it in a log and found nothing, that is expected.

Practical searches

Open the log in a text editor and search, or use the command line. A few that do most of the work:

Every AI-related hit today, newest first:

grep -iE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider|Amazonbot|Applebot" access.log

A count of how many times each one showed up:

grep -ioE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|PerplexityBot|Perplexity-User|Googlebot|bingbot" access.log | sort | uniq -c | sort -rn

Which pages OAI-SearchBot actually requested:

grep "OAI-SearchBot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn

What status codes GPTBot is getting (you are hoping for 200, not a column of 403):

grep "GPTBot" access.log | awk '{print $9}' | sort | uniq -c

On Windows without these tools, open the file in an editor and use Find for each user agent, or load it into a spreadsheet and filter the user agent column. The logic is the same: count the bots, see which pages they touched, check the status codes.

If you would rather not touch the command line

A few tools do this reading for you:

• GoAccess is free and open source. Point it at a log file and it opens a live dashboard in your browser or terminal, broken down by visitor, status code, requested page, and user agent. It is the quickest way to go from a raw file to a readable picture.
• Screaming Frog Log File Analyser is a desktop app built specifically for this. You drag in a log, and it groups activity by bot, shows which URLs each one crawled, flags the error responses, and lets you compare that against a list of your real pages to see what is being missed.
• Your CDN or host dashboard. As noted above, Cloudflare's bot analytics already does the verification and grouping. Some hosts and security plugins keep their own crawler log with a filterable view.

Any of these gets you to the same three facts: which crawlers, how often, and what they received.

What the answers are telling you

Healthy looks like this. The search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot) show up regularly, over days and weeks rather than once. They request a spread of real pages, not just the homepage. The status codes next to them are almost all 200. You may also see occasional ChatGPT-User or Perplexity-User hits, which means real people are pointing assistants at you.

No AI crawlers at all over a few weeks of logs means something upstream is stopping them: a robots.txt rule, a firewall, a hosting-level block, or a site so new and unlinked that nothing has found it yet. Start with your robots.txt rules for AI crawlers.

AI crawlers getting 403, 429, or 503 is the most common real problem this exercise turns up. A web application firewall, a "bot fight" setting, or an over-eager rate limit is turning them away at the door. The crawler tried, your infrastructure said no, and you would never have known without the log. The fix is to allow the crawlers you want through that layer explicitly.

Crawlers only ever hitting the homepage means they arrive but cannot find their way deeper. That points at missing internal links or a sitemap that is absent or stale.

One crawler hammering you hundreds of times an hour, often Bytespider or an unfamiliar name, is a cost and performance issue. Rate-limit or block that specific agent without touching the others.

A crawler pulling tiny responses from pages that should be large suggests it is getting an error page or a near-empty shell, which is worth checking against what a browser sees.

All of this feeds the same underlying question the Discover pillar of an AI readiness check asks: can crawlers actually get to your pages. The log is how you check the real-world answer rather than the theoretical one.

Making sure the crawler is real

Here is the catch with everything above: the user agent is just text the visitor sends, and anyone can send any text. Scrapers routinely label themselves GPTBot or Googlebot to look trustworthy, and bad actors do it to get through filters that trust the name. Before you make a decision based on "GPTBot visited," confirm it was actually OpenAI.

There are two reliable ways.

Check the IP against the official published list. The major providers publish the exact address ranges their crawlers use, as files you can download:

• OpenAI: openai.com/searchbot.json, openai.com/gptbot.json, openai.com/chatgpt-user.json
• Anthropic: claude.com/crawling/bots.json
• Perplexity: perplexity.com/perplexitybot.json
• Google publishes its crawler ranges in its documentation, and Bing publishes theirs.

If the IP in your log line is inside the published range, the visit is genuine. If it is not, the user agent is lying, whatever it claims to be.

Do a forward-confirmed reverse DNS lookup. Take the IP, look up the hostname it points back to, and check that hostname belongs to the provider (for example a .googlebot.com or .google.com host for Googlebot). Then look that hostname back up and confirm it resolves to the same IP. Google documents this method for verifying Googlebot, and it works the same way for the others that use identifiable hostnames.

Note that Anthropic and some others specifically advise against blocking purely by IP, because their ranges change and an out-of-date block can also stop the crawler reading your robots.txt. Use the IP list to verify what you are seeing; use robots.txt and named user agents to control access.

Common mistakes

Judging it from one day of logs. Crawlers do not visit every site every day. A single day showing no GPTBot activity means nothing. Look at two to four weeks before concluding anything.

Trusting the user agent without checking the IP. "Googlebot crawled 4,000 pages" can easily be a scraper hiding behind the name. Verify against the published ranges before you act on a number, especially before you decide a crawler is abusive and block it.

Only checking the origin log when you sit behind a CDN. The CDN answers a large share of requests itself, so the origin log undercounts. Use the CDN's own analytics or its full log export.

Reading robots.txt as proof of what happened. robots.txt states your intent. The log states the outcome. A permissive robots.txt does not prove a firewall or rate limiter further in isn't quietly turning crawlers away.

Treating every AI bot the same. Blocking a training crawler like GPTBot is a defensible content decision. Accidentally blocking a search crawler like OAI-SearchBot cuts you out of live citations. The log lets you tell which is which; a blanket rule does not.

Panicking at a single 403. One stray error among hundreds of clean 200s is noise. A consistent wall of 403s to one crawler is the signal.