Straight answers about indexing, AI assistants, crawlers, IndexNow, fixes and your account. 36 questions, grouped by topic.
The most common reasons are that the pages look thin or too similar to other pages on your site, that Google has found more addresses than it wants to crawl, or that the pages are blocked, redirected or marked noindex. Google decides page by page, and it does not index everything it finds.
Learn more →It means Google visited and read the page, then chose not to add it to its index. This usually happens when Google sees little unique value in the page, for example when most of it is template text or it repeats other pages.
Learn more →It means Google knows the address exists but has not crawled it yet. On larger sites this often happens when there are many low-value addresses competing for Google's attention, so the important pages wait in line.
Learn more →No tool can, and claims that say otherwise are not true. What IndexPedia does is show why Google skips pages and what to change so Google sees them as worth indexing.
Learn more →No. Pages such as search results, filters, login screens and thin tag pages are often better left out. A smaller set of strong pages usually performs better than a large set of weak ones.
Learn more →Anywhere from a few hours to several weeks. Google crawls popular, frequently updated sites more often, and it may wait longer on pages that look similar to others. A clean sitemap, internal links to the new page and unique content all help Google find it and decide to keep it.
Learn more →It is a free test that shows whether AI assistants recommend your business when customers ask for what you offer. IndexPedia asks the questions a customer would ask, live and with web search, and reads every answer. You see whether you were named, in what position, which competitors were named instead, and which websites the answers relied on.
Learn more →AI assistants build answers from what they can find and trust online: business websites, listing sites, review sites and local directories. Competitors that are named usually have clear, consistent facts across those sources. The check shows which sources were used, so you can see where you are missing.
Learn more →No one can guarantee it, but the steps that help are well known. Get listed, with the same name, address, hours and services, on the sources the check shows. Make sure your own website states clearly what you do and where. Then give AI one clear, structured page about your business, which is what the AI Profile does.
Learn more →OAI-SearchBot crawls pages so they can appear in ChatGPT search results. ChatGPT-User visits a page when a ChatGPT user asks about it directly. GPTBot collects content that may be used to train OpenAI's models. Blocking GPTBot does not remove you from ChatGPT search; blocking OAI-SearchBot does.
Learn more →That is your decision, and AI Reach helps you make it on purpose. Blocking search crawlers keeps your pages out of AI answers, which usually costs you visibility. Blocking training crawlers only affects model training. Many sites allow search crawlers and block training crawlers.
Learn more →No. Google-Extended is a robots.txt token that controls whether Google may use your content for Gemini model training. It does not change how Googlebot crawls your site or how you rank in Google Search.
Learn more →SEO is about appearing in a list of search results. AI visibility is about being named in the answer an AI assistant writes. They overlap, because AI assistants read the web through crawlers and search indexes, but an AI answer has no page two: a business is either named or not. IndexPedia covers both.
Learn more →IndexNow is an open protocol that lets a website notify search engines the moment a page is added, updated or deleted. One notification is shared with all participating search engines, including Microsoft Bing, Yandex, Seznam.cz and Naver. Without it, search engines find changes only on their next crawl.
Learn more →No. Google does not take part in IndexNow. For Google, IndexPedia relies on your sitemap and, if you connect it, Search Console data to show which pages are indexed and why others are not.
Learn more →No. IndexNow makes sure search engines know about the page quickly. They still visit it and decide whether to index it based on its quality, the same as any other page.
Learn more →Microsoft Copilot answers with Bing's index, and ChatGPT search uses Bing among its sources. When your new pages reach Bing quickly, they can reach those AI answers sooner.
Learn more →The key is a short text file placed at the root of your site. It proves to search engines that the notifications really come from the owner of the site. IndexPedia gives you the file and checks that it is in place.
Learn more →Analytics tools run in the visitor's browser and ignore bots on purpose. AI crawlers do not run that code, so they never appear. The IndexPedia snippet runs on your server, where every request is seen.
Learn more →Each visit is checked against the IP addresses published by the crawler's owner, such as OpenAI, Perplexity, Google, Bing and Apple. Visits that use a famous name but come from other addresses are marked as not verified.
Learn more →Many sites let well-known crawlers through firewalls and rate limits. Scrapers copy their names to get the same treatment. The name in a request proves nothing; the address it comes from does.
Learn more →Block it by IP address in your firewall or hosting panel. Do not block by name, because that would also block the real crawler using the same name.
Learn more →In most hosting control panels, look for Logs, Raw Access or Access Logs. On your own server it is usually at /var/log/nginx/access.log or /var/log/apache2/access.log. Many hosts let you download the last few days as .gz files.
Learn more →The file is read once to build your report and then deleted. Only the report is kept. If you choose to, anonymous totals, with no addresses, pages or site name, are added to the public AI Crawler Radar.
Learn more →The Fix Kit contains the files and text your site needs, written from your own pages: a corrected robots.txt, an llms.txt guide for AI tools, schema markup for your homepage, shorter titles and new meta descriptions for pages that need them, and a clean list of sitemap URLs. Each item can be copied or downloaded.
Learn more →llms.txt is a proposed standard: a plain text file at the root of your site that tells AI tools what your site is and links to its most useful pages, in a format language models read easily. It sits beside robots.txt and your sitemap; it does not replace them.
Learn more →The Fix Kit keeps the rules that protect private areas of your site and opens access for search and AI crawlers, then points to your sitemap. Read it before you replace your current file, especially if your site has areas that must stay private.
Learn more →Auto-fix applies approved SEO fixes to your live pages without you editing code: shorter titles, new meta descriptions and schema markup. You add one small snippet to your site once, approve the fixes you want in your dashboard, and they appear on your pages as they are served.
Learn more →No. Your files and your database are not touched. The fix is applied to the page as it is sent to the visitor, and switching a fix off returns the original immediately.
Learn more →Every platform. One line, the IndexPedia tag, goes into your site header on Wix, Squarespace, GoDaddy, Shopify, Webflow, Square Online or through Google Tag Manager, and WordPress has a one-click plugin. The Connect page shows the steps for your builder, with no code to write.
Learn more →The free plan includes one site with up to 1,000 pages, automatic weekly checks of 100 pages, IndexNow submissions, AI crawler and AI visitor tracking, and the Fix Kit. The free site scan, AI Visibility Check, Log Analyzer and Bot checker can be used without an account.
Learn more →Pro is $19 a month for up to 5 sites and 50,000 pages per site, with daily checks of 1,000 pages, Google Search Console data and Auto-fix. Agency is $49 a month for up to 25 sites and 250,000 pages per site, with daily checks of 5,000 pages.
Learn more →Online payments are being finished. Until they are live, send a message from the contact page with your account email and the plan you want, and your account is moved the same day. You can cancel at any time and keep your history.
Learn more →It means you uploaded your site's key file, which proves the site is yours. IndexNow requires the same file before it accepts your pages, so verifying covers both.
Learn more →Yes. IndexPedia checks public pages one at a time with a pause between them and identifies itself as IndexPediaBot. It never logs in, submits forms or changes your site.
Learn more →No, and no honest service can. Search engines and AI assistants make their own decisions. IndexPedia shows what stands in the way, fixes what can be fixed, and measures whether things improve.
Learn more →Just your domain. We find the pages Google is likely to skip. About 10 seconds.
