Which AI Crawlers Visit Your Site? How to See Every Visit

ChatGPT, Perplexity, Claude and Google's AI Overviews can only talk about your business if their crawlers have read your pages. Most owners have no idea whether that is happening, because the usual analytics tools never show bots. This guide explains which AI crawlers exist, what each one does, and how to see every visit to your own site.

Why Google Analytics never shows AI crawlers

Google Analytics and similar tools work by running a small piece of JavaScript in the visitor's browser. People run it. Bots do not.

AI crawlers fetch the raw HTML of your page and leave. They never load the analytics script, so they never show up in your reports. Analytics tools also filter out known bots on purpose, to keep human numbers clean.

That means a site could be read by GPTBot a hundred times a day, or not at all, and the analytics dashboard would look exactly the same.

The only place every visit is seen is your server. Each request arrives there before any page is sent, whether it comes from a person or a bot. That is why the IndexPedia AI crawler tracker records visits on the server side, not in the browser.

The AI crawlers you should know

Not every AI bot does the same job. Some read pages to answer a question right now. Some build a search index. Some collect text to train future models. It helps to know the difference before you decide what to allow.

Search and answer crawlers

These are the ones that decide whether you appear in AI answers today. OAI-SearchBot feeds ChatGPT search. PerplexityBot feeds Perplexity. Claude-SearchBot does the same job for Claude. Googlebot feeds both classic Google results and AI Overviews. Bingbot matters more than many people think, because Bing's index powers Copilot and is used by ChatGPT search.

User-triggered fetchers

ChatGPT-User, Perplexity-User and Claude-User visit a page because a real person asked a question and the AI went to read your page live. A visit from one of these often means someone was asking about your business, your product or your topic.

Training crawlers

GPTBot, ClaudeBot, CCBot (Common Crawl), Bytespider and meta-externalagent collect text that may be used to train models. Blocking them does not remove you from AI search results run by the same company. For example, blocking GPTBot does not block OAI-SearchBot.

Robots.txt-only tokens

Google-Extended and Applebot-Extended are not separate crawlers. They are names you use in robots.txt to say whether Google or Apple may use your content for AI training. They never show up in your logs, so a tracker cannot count them. You control them only with rules like this:

User-agent: Google-Extended
Disallow: /

If you want to understand how robots.txt rules affect AI tools, read Can AI crawlers read your website?.

What the IndexPedia AI crawler tracker records

The tracker recognises the crawlers listed above plus Applebot, Amazonbot, DuckAssistBot, MistralAI-User and others. For every visit it records:

• Which crawler it was, grouped by company
• Which page it read
• When it came, so you see a daily trend over 30 days
• What your server answered: a normal page, a redirect, a missing page, a block or an error
• Whether it was real, checked against official IP lists

It also counts AI referrals. These are real people who clicked a link inside ChatGPT, Perplexity, Gemini, Copilot or Claude and landed on your site. Put together, you see the whole path: AI crawled your page, an answer used it, and a person clicked through.

Real crawler or fake?

Anyone can send a request that says "I am GPTBot". The user agent is just a line of text, and scrapers copy famous names all the time because many sites let those names through.

So a count based on names alone is not trustworthy. IndexPedia checks each visit against the IP addresses the crawler's owner publishes. OpenAI publishes lists for GPTBot, OAI-SearchBot and ChatGPT-User. Perplexity, Google, Bing and Apple publish their own lists too. For crawlers that are verified by hostname, such as Googlebot and Bingbot, a reverse and forward DNS check is used.

Visits that use a famous name but come from other addresses are marked as not verified. If a crawler's owner does not offer a list IndexPedia can check yet, the visit is shown but labeled as not checkable, rather than guessed.

Want to check one address by hand? Paste it into the free Bot checker. For the full story on fake bots, see How to tell real Googlebot from fake bots.

How to start tracking AI crawlers

You do not need a developer. Pick the method that fits how your site is built.

Ways to record visits

• Server logs read automatically on supported servers, with nothing to install.
• The WordPress plugin, for WordPress sites.
• A PHP snippet added to the top of your site's main file.
• A Cloudflare Worker, if your site runs behind Cloudflare.
• Uploading a log file, to see visits from before you set anything up.

Each method records the visit on the server. The snippet sends its report after your page has gone to the visitor, using a short timeout, so neither people nor crawlers wait for it.

1. 1
   Add your site

   Create a free IndexPedia account and add your domain.

2. 2
   Choose a method

   Install the WordPress plugin, paste the PHP snippet, add the Cloudflare Worker, or use automatic log reading.

3. 3
   Upload a past log

   Optional: upload an access log to see the weeks before you started.

4. 4
   Watch the dashboard

   See visits per crawler, real or fake, pages read and AI referrals.

Example: Golden Crust Bakery in Raleigh

Golden Crust Bakery runs a small WordPress site at goldencrust.example. The owner wanted to know why ChatGPT kept recommending a competitor for "best sourdough in Raleigh".

She installed the WordPress plugin and uploaded the last week of her access log. Within a few minutes the picture was clear.

Googlebot and Bingbot came every day. GPTBot read plenty. But OAI-SearchBot, the crawler behind ChatGPT search, had visited only the home page and the contact page. It never reached the sourdough page, which sat three clicks deep with no internal links pointing to it.

The fix was not complicated. It just needed the data to know where to look.

What to do with your crawler data

The numbers are only useful if they lead to an action. Here are the most common patterns and what they mean.

Important pages get no AI search visits

Check that the page is linked from your menu or home page, listed in your sitemap, and not blocked in robots.txt. Run a Site Scan to catch missing titles, broken links and blocked pages in one go.

Crawlers get 403 or 429 answers

Your firewall, security plugin or host is blocking them. This is very common with WordPress security plugins and some hosting plans. Allow the real crawlers you want, and block by IP address rather than by name.

Lots of fake bot visits

Block the fake addresses in your firewall. Do not block the bot name, or you will lock out the real crawler too.

Few AI referrals

If crawlers visit but people never click through from AI answers, the problem is usually how your business is described, not access. The AI visibility check shows whether ChatGPT and other AI tools mention you at all. You can also read Does ChatGPT recommend your business?.

Tracker or Log Analyzer: which one?

They answer slightly different questions.

• The AI crawler tracker records visits from now on, every day, automatically. It is the long-term view.
• The Log Analyzer reads a log file you already have, so you can see the past without installing anything. It is free and needs no account.

Many owners start with the Log Analyzer to get a quick answer, then set up the tracker so they never have to download a log again. Our guide on how to read your server logs for AI bots walks through the log side step by step.

Crawler tracking is included in the Free plan for one site. Pro at $19/month and Agency at $49/month add more sites.

FAQ