Your website can look perfect to you and still be invisible to Google, Bing and AI assistants like ChatGPT. The reasons are usually small and technical: a blocked crawler, a slow page, or pages that look too much alike. A site scan finds those problems on your real pages, so you know exactly what to fix first.
Why a good-looking site can still be invisible
A search engine or AI assistant can only show a page it can reach, read and trust. If any one of those fails, the page is left out, no matter how nice it looks.
Here is how that plays out for a real small business. Golden Crust Bakery in Raleigh, NC has a site at goldencrust.example with a homepage, a menu, a page for each cake and a few blog posts about baking. The owner searches for "custom birthday cakes Raleigh" and the cake pages never show up. Nothing looks broken. But under the hood:
- The cake pages share 90% of their text with each other, because the template repeats the same ordering info on every page.
- A security plugin blocks unknown bots, which includes some AI crawlers.
- The menu page has no meta description, so Google writes its own from random page text.
None of these show up when you click around your own site. They only show up when you look at the site the way a crawler does. That is what a site scan is for.
Google and AI search are two separate checks
Google uses Googlebot. Bing uses Bingbot, which also feeds Microsoft Copilot. AI assistants bring their own crawlers. OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for model training. Anthropic uses ClaudeBot. Perplexity uses PerplexityBot. Your robots.txt and your firewall can treat each one differently, so a page that Google reads fine can still be closed to ChatGPT search. If you want the full picture of these bots, read Can AI crawlers read your website?.
What the IndexPedia site scan checks
The IndexPedia site scan checks a sample of real pages from every part of your site, live. It is free, needs no account, and takes about 20 seconds for most sites.
Instead of looking at your homepage only, the scan reads every sitemap file on your site and picks pages from each kind: products, posts, locations, profiles and so on. Problems usually affect a whole page type at once. If one cake page has thin text, they all do. So a sample from every type finds the problem fast.
For each page, the scan records:
- HTTP status and redirects
- Load time, word count and unique words
- Title, meta description, headings and canonical tag
- noindex tags and robots.txt rules
- Text repeated from your template across many pages
- Structured data (schema) on the page
It also reads your robots.txt the way each crawler does and tests whether AI crawlers such as GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot can reach your pages. Every finding is tied to the page where it was found, so you never have to guess which page a warning is about.
How to run your first scan
You do not need an account or any technical setup. Just your domain.
- 1Enter your domain
Type goldencrust.example (or your own domain) on the site scan page, no account needed.
- 2Let it check live
IndexPedia reads your sitemaps, picks pages from each type and tests every AI crawler.
- 3Read the ranked report
Problems are sorted so the ones that hurt most come first, each tied to a page.
- 4Apply the fixes
Copy or download the ready-made files, or apply titles, descriptions and schema with Auto-fix.
The scan runs in the background, so the page never freezes. You can leave and come back to the report link. Very slow sites take a little longer, because each page is loaded for real.
Every report has its own link. You can send it to your web developer, a client or your team. Reports from free scans are kept for a short time, while reports in an account are kept with the site.
See what keeps your site out of Google and AI answers
How to read your site health score
At the top of the report you get one number: your site health score. It is useful because you can check exactly how it was made.
Every checked page gets a level: fine, warning or error. The score starts at 100 and drops based on the share of pages with problems. Pages with errors count three times as much as pages with warnings. That means one broken template counts more than a few long titles, which matches how much each problem really hurts.
The report lists exactly which pages made the score. Do not panic if you are not at 100. A real site rarely scores a perfect 100, and that is normal.
In this example, the cake pages drag the score down. That is the signal to fix the cake template first, not to rewrite the blog.
The most common problems a scan finds
Most pages that never show up in search have one of a few causes. They are blocked, they are slow or broken, or they look too much like other pages on the same site.
Template text and "Crawled, not indexed"
Template text is the wording repeated on many pages, such as menus, footers and boilerplate paragraphs. When most of a page is template text and only a few words are unique, search engines see little value in it. In Google Search Console this often shows up as "Crawled - currently not indexed". The scan measures unique words on every page it checks, so you can see which pages are mostly copy and paste. For more on this status, read Why Google is not indexing your pages.
Blocked crawlers
A single line in robots.txt can close your whole site to a crawler. So can a firewall rule or a bot-blocking plugin. The scan tests access for each crawler separately, so you see if Google can read a page that ChatGPT search cannot.
Titles and descriptions
Google usually shows about 50 to 60 characters of a title before cutting it off, and roughly 150 to 160 characters of a description. Titles that are too long get trimmed. Pages with no description get a snippet Google picks for you, which is often a menu line or a cookie notice.
Missing structured data
Structured data, also called schema, labels facts such as your address, hours and products so machines read them correctly. Without it, search engines and AI tools have to guess.
| Problem | What it looks like | Usual fix |
|---|---|---|
| Template text | Cake pages share most of their words | Add unique text to each page |
| Blocked AI crawler | robots.txt disallows OAI-SearchBot | Allow the crawler in robots.txt |
| Long title | Title over 60 characters | Shorten and lead with the key words |
| Missing description | Google picks random text | Write a 150 to 160 character description |
| No schema | No LocalBusiness markup on homepage | Add JSON-LD to the homepage |
From findings to fixes: the Fix Kit
A list of problems is only half the job. Each IndexPedia scan also includes a Fix Kit with the files your site needs:
- A corrected robots.txt
- An llms.txt built from your own pages
- Schema markup for your homepage
- Shorter titles and new meta descriptions for the pages that need them
- A clean list of sitemap URLs
You can copy or download each file. Or you can apply titles, descriptions and schema with Auto-fix, so you do not have to edit pages by hand.
Here is what a corrected robots.txt might look like for Golden Crust Bakery, if the owner wants to stay open to search and AI search crawlers:
User-agent: * Allow: / Disallow: /cart/ User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / Sitemap: https://goldencrust.example/sitemap.xml
Whether you allow training crawlers like GPTBot or ClaudeBot is your choice. Blocking them does not remove you from Google Search. If you want a line by line walkthrough, read robots.txt, llms.txt and schema fixes explained.
Cake pages share most of their text, OAI-SearchBot is blocked by a plugin, and the menu page has no description. The cake pages never show up for "custom birthday cakes Raleigh".
Each cake page has its own description and unique text, AI search crawlers are allowed, and the homepage has LocalBusiness schema. The pages are now readable and worth indexing.
What to fix first
You do not need to fix everything in one go. Work in this order:
- Access first. If a crawler cannot reach a page, nothing else matters. Fix robots.txt rules, firewall blocks and noindex tags you did not mean to set.
- Broken pages next. Pages that return errors or redirect in loops waste the crawler's visit.
- Template text. Add real, unique content to the page types with the fewest unique words.
- Titles and descriptions. Quick wins that change how your pages look in results.
- Schema. Add structured data to the homepage and key pages.
Fix one page type at a time, then re-run the scan. Watching the score for that type move tells you the fix worked before you spend time on the next one.
Once your pages are fixed, tell search engines about the change. You can submit updated pages to Bing and other IndexNow engines in minutes with IndexNow, and ask Google to recrawl key pages through Search Console.
Free scan vs an account
The free scan is a real scan, not a teaser. You get the full report and the ready-made fixes, and you can run up to 5 scans an hour without an account.
An account adds ongoing checks. A free account adds one site that IndexPedia checks automatically every week. Paid plans check more pages and more sites and run every day:
- Free: one site, checked weekly
- Pro: $19 a month
- Agency: $49 a month, for people who look after many sites
If you want to see which AI bots really visit your pages, not just whether they are allowed, the AI Crawler Tracker and the Log Analyzer read real visits from your server.
See what keeps your site out of Google and AI answers
FAQ
Is the IndexPedia site scan really free?
Yes. You can run up to 5 scans an hour without creating an account, and you get the full report and the ready-made fixes. A free account adds one site that IndexPedia checks every week, and paid plans check more pages, more sites and run every day.
Will a site scan slow down or change my website?
No. The scan only reads public pages, one at a time, the way a polite search engine crawler would. It never logs in, submits forms or changes anything on your site.
Why does the scan check a sample instead of every page?
The free scan reads every sitemap file and picks pages from each page type, such as products, posts and locations. Problems usually affect a whole page type at once, so a sample from every type finds them quickly. With an account, IndexPedia goes on to check your pages on a schedule.
Does the scan cover AI search or only Google?
Both. It covers Google and Bing, and it tests access for the crawlers behind ChatGPT search, Perplexity, Claude, Gemini and Copilot. A page that Google can read but an AI crawler cannot will show up in the report.
What does "Crawled - currently not indexed" mean?
It means Google visited the page but chose not to store it, so it cannot appear in results. A common cause is a page that is mostly template text with few unique words. The scan measures unique words on each page so you can find these pages.
How long does a scan take?
Most scans finish in about 20 seconds. The scan runs in the background, so you can leave and come back to the report link. Very slow sites take a little longer, because each page is loaded for real.