Why bots pretend
Many scrapers send the user agent of a well-known crawler, such as Googlebot or GPTBot, because sites tend to let those through. The name alone proves nothing. The address the request comes from does.
How the check works
The major crawler owners publish the IP ranges their bots use: Google for Googlebot, Microsoft for Bingbot, OpenAI for GPTBot, OAI-SearchBot and ChatGPT-User, Perplexity for PerplexityBot, and Apple for Applebot. The Bot checker compares the address you enter against those official lists, and for crawlers without a list, uses a reverse and forward DNS check, the method the owners themselves recommend.
- Enter an IP address from your logs
- Pick the bot it claims to be, or let the checker test all of them
- See whether it matches an official range, and which one
What to do with a fake
Block it in your firewall by address, not by user agent. Blocking by user agent would also block the real crawler.
Key terms
- IP range
- A block of addresses written like 66.249.64.0/19.
- Reverse DNS check
- Looking up the host name of an address, then confirming that name points back to the same address.