How AI tools read websites
AI assistants use crawlers, just like search engines. OpenAI uses OAI-SearchBot for ChatGPT search, ChatGPT-User when someone asks ChatGPT to open a page, and GPTBot for model training. Perplexity uses PerplexityBot, Anthropic uses ClaudeBot and Claude-SearchBot, and Microsoft Copilot relies on Bing's crawler.
Each crawler follows your robots.txt rules. Many sites block AI crawlers without meaning to, through an old robots.txt line, a security plugin or a firewall rule that treats bots as attacks.
What AI Reach tests
- robots.txt rules for every major AI and search crawler, read the way the crawlers read them
- Whether your server or firewall answers each crawler, or blocks it
- Pages that only show their text after JavaScript runs, which many AI crawlers cannot do
- Pages that return a challenge or "verify you are human" screen
- Structured data and clear text AI can quote
Search crawlers vs training crawlers
Blocking a training crawler such as GPTBot keeps your content out of future model training, but does not remove you from ChatGPT search, which uses OAI-SearchBot. Many sites choose to allow search crawlers and block training crawlers. AI Reach shows each one separately, so you can decide on purpose.
Key terms
- robots.txt
- A file at the root of your site that tells crawlers which pages they may visit.
- User agent
- The name a crawler sends with each visit, such as GPTBot or PerplexityBot.
- Client-side rendering
- Pages built by JavaScript in the browser. Crawlers that do not run JavaScript may see an empty page.