Can AI Crawlers Read Your Website? How to Test and Fix It

AI assistants can only recommend what they can read. If ChatGPT, Perplexity or Claude cannot reach your pages, your business is left out of their answers, no matter how good your site is. This guide explains how AI crawlers work, the quiet ways sites block them by accident, and how to test and fix your own site in a few minutes.

How AI tools read websites

AI assistants use crawlers, just like search engines do. A crawler is a program that visits web pages and reads them. Each one sends a name with every visit, called a user agent, such as GPTBot or PerplexityBot.

The big AI companies run several crawlers, and each one has a different job:

• OpenAI: OAI-SearchBot crawls pages so they can appear in ChatGPT search results. ChatGPT-User visits a page when someone asks ChatGPT about it directly. GPTBot collects content that may be used to train OpenAI's models.
• Anthropic: ClaudeBot collects training data. Claude-SearchBot crawls for search results in Claude, and Claude-User fetches pages when a Claude user asks for them.
• Perplexity: PerplexityBot crawls pages for Perplexity's answers, and Perplexity-User fetches pages when a user asks.
• Google: Googlebot crawls for Google Search. Google-Extended is not a separate crawler. It is a token you use in robots.txt to control whether Google may use your content for its Gemini models. It does not change your Google Search rankings.
• Microsoft: Copilot relies on Bing's index, so Bingbot matters for Copilot answers.
• Apple: Applebot crawls for Siri and Spotlight. Applebot-Extended is a token that controls whether Apple may use your content to train its AI models.

Every one of these crawlers follows your robots.txt rules. That is good news, because it means you are in control. It is also the source of most problems.

Search crawlers vs training crawlers

This is the most important idea in this guide. AI crawlers fall into two groups.

Search and answer crawlers fetch your pages so an AI assistant can show or quote them in an answer, often with a link back to you. Blocking these keeps you out of AI answers.

Training crawlers collect content that may be used to train future AI models. Blocking these keeps your content out of model training, but does not remove you from AI search.

For example, blocking GPTBot does not remove you from ChatGPT search, because ChatGPT search uses OAI-SearchBot. Blocking OAI-SearchBot does.

Many sites choose to allow search crawlers and block training crawlers. Others allow everything. Either choice is fine, as long as you make it on purpose.

How sites block AI crawlers by accident

Most owners who block AI crawlers never meant to. Here are the usual causes.

An old line in robots.txt

A few years ago, some guides told site owners to block every bot they did not recognize. A line like User-agent: * followed by Disallow: / blocks every crawler that does not have its own rules, including new AI crawlers. Sometimes a developer adds it on a test site and it gets copied to the live site.

A plugin setting

Some security and SEO plugins have a "block bots" or "block AI" switch. It may have been turned on by default, or by someone who did not know it also blocks AI search.

A firewall or CDN rule

This is the hardest one to spot. Some hosts and Cloudflare settings answer AI crawlers with an error page or a "verify you are human" challenge. Your robots.txt says yes, but the crawler still cannot read anything. Looking at robots.txt alone will never show this.

Pages built with JavaScript

Some sites send an almost empty page and build the text in the browser with JavaScript. Many AI crawlers do not run JavaScript. To them, your menu, prices and opening hours simply do not exist.

What AI Reach tests

AI Reach tests whether the crawlers used by AI assistants can get into your site and read it. It runs for free as part of every site scan.

It checks:

• Your robots.txt rules for every major AI and search crawler, read the way the crawlers read them
• Whether your server or firewall answers each crawler, or blocks it
• Pages that only show their text after JavaScript runs
• Pages that return a challenge or "verify you are human" screen
• Structured data and clear text that AI can quote
• How fast your site answers crawlers

Each crawler is shown separately, because they do different jobs. You see exactly which rule blocks which crawler, so the fix is clear.

AI Reach requests your pages the way each crawler does. That is how it finds firewall blocks that a robots.txt check alone would miss.

1. 1
   Enter your domain

   Type your site address on the free scan page.

2. 2
   Test every AI crawler

   IndexPedia checks robots.txt, firewall answers and page content for each one.

3. 3
   Read your score

   See which crawlers get in, which are blocked and why.

4. 4
   Fix what blocks them

   Copy the corrected robots.txt from your Fix Kit, or ask your host to allow a blocked crawler.

Example: Golden Crust Bakery

Golden Crust Bakery in Raleigh, NC runs its site at goldencrust.example. The owner wonders why ChatGPT never mentions the bakery. A scan shows two problems.

First, the robots.txt file contains an old rule copied from a template:

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

Googlebot has its own group, so it gets in. Every other crawler falls under User-agent: * and is blocked from the whole site. That includes OAI-SearchBot, PerplexityBot, Claude-SearchBot and Bingbot.

Second, the bakery's menu page loads its prices with JavaScript, so crawlers that do not run JavaScript see an empty menu.

The fixed robots.txt

Here is a version that lets search and AI answer crawlers in, keeps a private folder private, and blocks training crawlers. Blocking training crawlers is optional. It is shown here only to illustrate the choice.

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Bingbot
Allow: /
Disallow: /admin/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://goldencrust.example/sitemap.xml

Two details matter. First, a crawler follows only the most specific group that names it, and ignores the rest. So any private folder must be repeated inside each named group. Second, robots.txt lives at the root of your site, at https://goldencrust.example/robots.txt. It does nothing anywhere else.

You do not have to write this by hand. If AI Reach finds a blocked crawler, the Fix Kit writes a corrected robots.txt from your current file. It keeps the private folders you already hide from everyone, lets search and AI answer crawlers in, adds your sitemap address and leaves the choice about training crawlers to you. Our guide on robots.txt, llms.txt and schema fixes walks through every file.

Fixing firewall and JavaScript problems

A robots.txt fix is a copy and paste. The other two blocks take a little more work.

Firewall blocks

If AI Reach shows that a crawler is refused even though robots.txt allows it, the cause is usually your host, your CDN or a security plugin. Check your plugin settings for a "block bots" option first. If you use Cloudflare, look at your bot settings. If you cannot find it, send your host the name of the blocked crawler from your report and ask them to allow it.

Be careful not to allow anything that only claims to be a crawler. Fake bots often borrow real names. Our Verify Bot tool and the guide on telling real Googlebot from fake bots explain how to check.

JavaScript-only pages

Make sure the important text is in the page when it is first sent: your name, services, prices, hours and contact details. Many website builders do this already. If yours does not, ask your developer about server-side rendering or a pre-rendered version of key pages.

Does llms.txt help AI crawlers?

You may have heard of llms.txt. It is a proposed standard: a plain text file that describes your site and links to its key pages, written for AI tools. No major AI engine has confirmed that it reads llms.txt yet, so AI Reach does not count it in your score.

It is harmless to add one, and the Fix Kit writes one for you from your own pages. But it does not replace robots.txt. If robots.txt or a firewall blocks a crawler, llms.txt will not help.

Can you see which crawlers actually visit?

AI Reach tells you whether crawlers can get in. To see which ones actually do, look at your server logs. The AI Crawler Tracker and Log Analyzer show real visits by AI bots. Our guide on which AI crawlers visit your site explains what the numbers mean.

FAQ