How to Tell Real Googlebot From Fake Bots, and Block the Fakes

Your server log says Googlebot visited 400 times yesterday. Some of those visits may not be Google at all. Scrapers copy the names of famous crawlers because many sites wave those names through, and the only way to know is to check where each visit really came from.

Why fake bots pretend to be Googlebot or GPTBot

Every visitor to your site sends a short line of text called the user agent. A browser sends something like "Chrome on Windows". Google's crawler sends a line that includes "Googlebot". OpenAI's training crawler sends "GPTBot".

The problem is simple: anyone can type anything into that line. It takes one setting in a scraping tool to say "I am Googlebot".

Scrapers do this for a reason. Many sites, firewalls and security plugins treat known crawlers kindly. They skip rate limits for Googlebot, or they allow AI crawlers through a block that stops everyone else. By borrowing a trusted name, a scraper gets the same treatment.

Fake bots cause real trouble:

• They copy your content, prices and photos.
• They use your server's resources and can slow the site for real customers.
• They probe for weak spots like login pages and old plugins.
• They make your crawler numbers wrong, so you cannot trust what you see.

How the real crawlers can be checked

The big crawler owners know about this problem, so they tell site owners how to check their bots. There are two official methods.

Method 1: Published IP ranges

Google, Microsoft (Bing), OpenAI, Perplexity and Apple publish the IP address ranges their crawlers use, as files anyone can download. If a visit's address falls inside one of those ranges, it really came from that company.

OpenAI publishes separate lists for GPTBot, OAI-SearchBot and ChatGPT-User. Perplexity publishes lists for PerplexityBot and Perplexity-User. Google publishes a list for Googlebot. Bing publishes one for Bingbot, and Apple for Applebot.

Method 2: Reverse and forward DNS

Google, Bing and Apple also document a DNS check. It has two steps:

1. Look up the host name that belongs to the IP address (reverse DNS).
2. Check that the name ends in the owner's domain, then look up that name and confirm it points back to the same IP address (forward DNS).

The second step is the important one. A scraper can set up a fake reverse DNS name that looks like Google, but it cannot make Google's own DNS point back to its address.

Checking Googlebot by hand with reverse DNS

You can run the DNS check yourself on a Mac or Linux computer with the host command. Here is a real Googlebot address from the range Google publishes:

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The first lookup gives a name ending in googlebot.com. The second lookup points that name back to the same address. Both steps pass, so this is the real Googlebot.

Now a visit from Golden Crust Bakery's log at goldencrust.example that also claimed to be Googlebot:

$ host 203.0.113.45
Host 45.113.0.203.in-addr.arpa not found: 3(NXDOMAIN)

No Google host name at all. Whatever the user agent said, this was not Google.

Why checking by hand does not scale

Running two lookups for one address is fine. Doing it for every crawler visit in a week of log is not. Many AI crawlers, such as GPTBot and PerplexityBot, are checked against IP lists rather than DNS, so you would also need to download each company's file and compare addresses against ranges like 66.249.64.0/19. That is the part a tool should do for you.

How the IndexPedia Bot checker works

The free Bot checker does both checks the way each company says to do them. You do not need an account.

1. 1
   Paste an IP

   Copy an address from your server log, firewall or security plugin.

2. 2
   Pick the crawler

   Choose the bot it claims to be, such as Googlebot, GPTBot or Bingbot.

3. 3
   Get the verdict

   See real, fake or not checkable, plus how it was checked and the host name if there is one.

What the three verdicts mean

• Real: the address is on the owner's official list, or it passed the reverse and forward DNS check.
• Fake: the owner publishes a way to check, and this address failed it.
• Not checkable: the owner does not publish a method the checker can use yet. The visit may be real or not; there is no official way to say.

The official IP lists are reloaded regularly from each owner's published file, so new ranges are picked up without you doing anything.

Checking thousands of addresses at once

The Bot checker is perfect for one suspicious address. For a whole log, use the Log Analyzer. Upload your access log and every visit that claims to be a known crawler is checked against the same official lists. The report shows real, fake and not checkable counts for each bot, plus the pages they asked for.

Golden Crust Bakery did exactly that with one week of log.

Almost one in five visits calling itself GPTBot came from addresses OpenAI does not use. Most of them were asking for /wp-login.php and old plugin folders, which the real GPTBot has no reason to visit. For a full walk-through of reading the report, see How to read your server logs for AI bots.

What to do with a fake bot

Once you know an address is fake, block it. How you block it matters a lot.

Block by IP address, never by name

If you block the user agent "Googlebot", you also block the real Googlebot. That can drop your pages out of Google over time. Fake bots share the name, not the address, so block the address.

Use your firewall, not robots.txt

Robots.txt is a polite request. Real crawlers follow it. Fake bots usually ignore it. To stop a fake bot, block it in your firewall, your hosting panel or your security plugin. On an nginx server, a simple rule looks like this:

location / {
    deny 203.0.113.45;
    deny 198.51.100.0/24;
}

If you are not sure how to add a rule, most hosts and security plugins have an "IP blocklist" setting that does the same thing.

Will blocking fakes hurt my SEO?

No. Blocking addresses that are not on the official lists does not touch the real crawlers. Your visibility in Google, Bing and AI search stays the same, and your server has less junk traffic to deal with.

Make sure the real crawlers still get in

The opposite mistake is common too: a security setting blocks the real crawlers. Some WordPress security plugins and hosting firewalls have "block bots" or "block AI" switches that stop Googlebot, Bingbot, OAI-SearchBot or PerplexityBot without telling you.

Check two things:

1. Your crawler data. The AI crawler tracker shows every verified visit per bot and what your server answered. A real crawler getting 403 or 429 is being blocked.
2. Your robots.txt. A Site Scan tests whether search and AI crawlers are allowed in, and flags rules that block them.

Our article on which AI crawlers visit your site explains what each crawler does, so you can decide which ones to welcome.

A simple routine for small businesses

You do not need to watch your logs every day. A short routine is enough:

1. Once a month, upload a few days of log to the Log Analyzer, or check the tracker dashboard.
2. Look at the fake counts for Googlebot, GPTBot and Bingbot.
3. Paste any busy, suspicious address into the Bot checker to confirm.
4. Block confirmed fakes by IP in your firewall.
5. Make sure the real crawlers are getting normal 200 answers.

The Bot checker and Log Analyzer are free. Ongoing crawler tracking is included in the Free plan for one site, and Pro at $19/month or Agency at $49/month add more sites.

FAQ