Your web server already keeps a diary of every visit, including every AI crawler that read your pages. That diary is the access log, and most site owners never open it because it looks like a wall of numbers. This guide shows you where to find it, how to read one line, and how to turn the whole file into a plain report with the free IndexPedia Log Analyzer.
What an access log is and why it matters for AI
Every time someone, or something, asks your server for a page, the server writes one line to a file. That file is the access log. It records the visitor's address, the time, the page, what the server answered, and the name the visitor gave.
This matters for AI search because AI crawlers never show up in Google Analytics. Analytics runs in the browser, and bots do not run it. The access log is written by the server itself, so nothing slips past it.
Hidden in that file is a full history of:
- Which AI and search crawlers visited, such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot and Bingbot
- Which pages they read, and which ones they never touched
- Where they ran into errors, redirects or blocks
- Which visitors were fake bots using a famous name
- Which real people arrived by clicking a link in an AI answer
Where to find your access log
The file lives in a different place depending on your host. Here are the most common spots.
Shared hosting with cPanel
Log in to cPanel and look under Metrics, then Raw Access. You can download the last few days as a compressed .gz file. Some hosts call it Access Logs or simply Logs.
Plesk
Open your domain, then go to Logs. You can view or download the access log from there.
Your own server (VPS or dedicated)
The log is usually at one of these paths:
/var/log/nginx/access.log /var/log/apache2/access.log
Older days are often rotated into files such as access.log.1 or access.log.2.gz.
Managed hosts
Many managed WordPress hosts offer a log download in their dashboard. If you cannot find it, ask support for "raw access logs in combined format". That wording gets you the right file.
Download one to seven days of log. That is usually enough to see every crawler that visits regularly, and it keeps the file under the 50 MB upload limit.
How to read one line of the log
Most servers use the standard "combined" format. Here is one real-looking line from Golden Crust Bakery's site at goldencrust.example:
66.249.66.1 - - [14/Oct/2026:09:12:44 -0400] "GET /menu/sourdough HTTP/1.1" 200 18342 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Read it from left to right:
66.249.66.1is the IP address of the visitor.[14/Oct/2026:09:12:44 -0400]is the date and time.GET /menu/sourdoughis the page that was asked for.200is the status code. 200 means the page was sent normally.18342is the size of the response in bytes."-"is the referrer, the page the visitor came from. A dash means none.- The last part in quotes is the user agent, the name the visitor gave. Here it says Googlebot.
Status codes worth knowing
| Status code | What it means | What to do |
|---|---|---|
| 200 | Page sent normally | Nothing, all good |
| 301 / 302 | Redirected somewhere else | List the final address in your sitemap |
| 404 | Page not found | Fix the link or redirect it to a live page |
| 403 / 429 / 503 | Blocked, rate limited or unavailable | Check your firewall, security plugin or host limits |
| 500 | Server error | Ask your host to check the error log |
A crawler that gets 403 again and again is being blocked by a firewall or security plugin. That is one of the most common reasons a site never appears in AI answers.
The problem with reading logs by hand
You can search a log for "GPTBot" with a text editor or a command like grep. For a quick look, that works. But it falls apart fast.
A busy site can write hundreds of thousands of lines a day. And a line that says GPTBot is not proof it was GPTBot. Here is a line from the same log:
203.0.113.45 - - [14/Oct/2026:09:13:02 -0400] "GET /wp-login.php HTTP/1.1" 403 512 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)"
The name says GPTBot. But the address, 203.0.113.45, is not on the list OpenAI publishes for GPTBot. And the real GPTBot has no reason to request a login page. This is a scraper wearing a famous name.
To tell real from fake by hand, you would need to look up every address against each company's official list. That is exactly the job the Log Analyzer does for you.
How the IndexPedia Log Analyzer works
The Log Analyzer reads your file line by line and builds a short report. It is free, and you do not need an account.
- 1Download your access log
Get it from cPanel, Plesk or your server, as a plain .log or a .gz file.
- 2Upload it
Drop the file on the Log Analyzer page. Files up to 50 MB work.
- 3Watch the progress
The analyzer reads the file and shows each step. Most logs take a few seconds.
- 4Read the report
See every crawler, real or fake, pages read, blocks and AI referrals.
What the report shows
- Visits per crawler, grouped by company: OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta and others
- Real, fake or not checkable, for every crawler visit
- What your server answered each crawler: normal pages, redirects, not found, blocked and errors
- Pages AI reads most, and the pages it skips
- An hourly or daily chart of crawler activity
- Visits sent by AI: people who clicked through from ChatGPT, Perplexity, Gemini, Copilot or Claude, found from the referrer
- A shareable report link, so you can send it to your web person
How fake bots are caught
Every visit that claims to be a known crawler is checked against the IP addresses its owner publishes. OpenAI, Perplexity, Google, Bing and Apple all publish these lists. For Googlebot, Bingbot and Applebot, a reverse and forward DNS check is also used, the method those companies document. If the address does not match, the visit is counted as fake. If the owner offers no way to check yet, the visit is marked not checkable instead of guessed.
Which formats work
The analyzer reads the standard combined format used by nginx, Apache, LiteSpeed and most hosting companies. Upload the plain file or the compressed .gz version.
See every AI crawler in your server log
Example: what Golden Crust Bakery found
Golden Crust Bakery downloaded seven days of log from cPanel and uploaded it. The report took a few seconds.
Three things stood out.
- A fake GPTBot was hitting the login page from 203.0.113.45 and a handful of 198.51.100.x addresses. None of them matched OpenAI's list.
- PerplexityBot was blocked. It received 403 on every visit. The owner's security plugin had a "block AI bots" setting turned on.
- Three ChatGPT referrals landed on the catering page. Real people had asked ChatGPT about catering in Raleigh and clicked through.
PerplexityBot blocked with 403, fake GPTBot scraping freely, no idea which pages AI read.
Security plugin set to allow real AI search crawlers, fake addresses blocked by IP, catering page linked from the home page.
What to do after you read the report
A report is only useful if it leads to a fix. Here are the common findings and the next step for each.
Real crawlers are blocked
Check your security plugin, your host's firewall and your robots.txt. Allow the AI search crawlers you want, such as OAI-SearchBot, PerplexityBot and Claude-SearchBot. The Fix Kit writes robots.txt and llms.txt files from your own site, and our guide to robots.txt, llms.txt and schema fixes explains each rule.
Fake bots are scraping
Block them by IP address in your firewall or hosting panel. Never block by user agent name, because the real crawler uses the same name. You can double-check any single address with the free Bot checker.
Important pages get no crawler visits
Link to them from your menu or home page and make sure they are in your sitemap. Then tell search engines about them with IndexNow for Bing, or request indexing in Google Search Console.
Crawlers hit lots of 404s
Old links point to pages that no longer exist. Add redirects to the right pages so crawlers stop wasting visits.
Run the Log Analyzer again a week after any fix. Comparing two reports is the easiest way to confirm that a blocked crawler is now getting through.
Is my log file safe to upload?
Yes. The file is read once to build your report and then deleted. Only the counts in the report are kept, so your report link keeps working. If you choose to, anonymous totals, with no addresses, pages or site name, are added to the public AI Crawler Radar.
From one-time log to daily tracking
Uploading a log gives you a snapshot of the past. If you want to see crawler visits every day without downloading anything, set up the AI crawler tracker. It records visits through automatic server log reading on supported servers, the WordPress plugin, a PHP snippet or a Cloudflare Worker. Read more in Which AI crawlers visit your site?.
The tracker is included in the Free plan for one site. Pro ($19/month) and Agency ($49/month) add more sites.
See every AI crawler in your server log
FAQ
How do I see bots in my server logs?
Download your access log from your hosting panel or server, then search it for crawler names such as Googlebot or GPTBot. For a full picture, upload the file to a log analyzer that groups visits by bot and checks each address against the official IP lists.
Where is the access log in cPanel?
Look under Metrics, then Raw Access. You can download recent days as .gz files. Some hosts label it Access Logs or Logs instead.
What log formats does the IndexPedia Log Analyzer support?
It reads the standard combined format used by nginx, Apache, LiteSpeed and most hosting companies. You can upload a plain .log file or a compressed .gz file, up to 50 MB.
How can I tell if a GPTBot visit in my log is fake?
Compare the visitor's IP address with the list OpenAI publishes for GPTBot. If the address is not on the list, it is a different bot using the GPTBot name. The Log Analyzer does this for every visit automatically.
Is the Log Analyzer free?
Yes. You can analyze a log without an account. Without an account there is a limit of a few logs per hour.
How much log should I upload?
One to seven days is usually enough to see every crawler that visits regularly. For busy sites, one day of log, or the .gz version, keeps the file under the 50 MB limit.