Most SEO tools hand you a list of problems and leave you to figure out the fixes. That list is not much help if you do not know what a robots.txt file is, or where schema markup goes. This guide explains each file that affects how Google and AI tools read your site, shows what a good version looks like, and how the IndexPedia Fix Kit writes them for you from your own pages.
The five fixes that matter most
A handful of small files and tags decide a lot about how search engines and AI assistants see your site. None of them change how your site looks to visitors. They only change what crawlers read.
- robots.txt tells crawlers which parts of your site they may visit.
- llms.txt is a short guide to your site written for AI tools.
- Schema markup states your key business facts in a format machines read easily.
- A clean sitemap lists the pages you want indexed, and only those.
- Titles and meta descriptions are what people see in search results.
The Fix Kit covers all five. It is free with every scan, including the free one, and it works on any platform: WordPress, Shopify, Wix, Squarespace or a custom site.
robots.txt: who may come in
robots.txt is a plain text file at the root of your site, for example https://goldencrust.example/robots.txt. Search engines and AI crawlers read it before they visit your pages. It is made of groups. Each group names one or more crawlers with User-agent, then lists what they may or may not visit with Allow and Disallow.
A crawler follows the most specific group that names it, and ignores the others. If no group names it, it follows the User-agent: group. This is where many sites go wrong. One old Disallow: / under User-agent: can block every AI crawler at once.
What the Fix Kit does to robots.txt
The Fix Kit starts from your current robots.txt, not a template. It checks whether these search and AI answer crawlers can reach your site: Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Claude-User.
If any of them are blocked, it:
- Removes their old blocking groups, so the old rules no longer apply to them
- Adds a new group at the top that lets them in
- Copies into that group every private folder you already hide from everyone, so private stays private
- Adds your sitemap address if the file does not have one
It leaves training crawlers such as GPTBot and ClaudeBot alone. Whether those may collect your content is your choice. If you have no robots.txt at all, the kit gives you one to upload. If yours is already fine, it says so and changes nothing.
Here is what the fixed file can look like for Golden Crust Bakery in Raleigh, NC:
# Added by IndexPedia: let search and AI answer crawlers read the site User-agent: OAI-SearchBot User-agent: PerplexityBot User-agent: Claude-SearchBot Allow: / Disallow: /admin/ Disallow: /cart/ User-agent: * Disallow: /admin/ Disallow: /cart/ Sitemap: https://goldencrust.example/sitemap.xml
To learn which crawlers do what, and how to find firewall blocks that robots.txt cannot fix, read can AI crawlers read your website.
llms.txt: a short guide for AI tools
llms.txt is a proposed standard. It is a plain text file at the root of a site, written in Markdown, that tells AI tools what the site is and links to its most useful pages. It sits beside robots.txt and your sitemap. It does not replace them.
Be clear about one thing: no major AI engine has confirmed that it reads llms.txt. Adding one is cheap and harmless, but do not expect it to change your AI visibility on its own. That is why IndexPedia does not count it in your AI Reach score.
What the Fix Kit puts in llms.txt
The kit builds the file from your real pages:
- Your site name as the heading, and your homepage description as a short summary line
- A "Main pages" list, taken from the links in your site's navigation
- Further sections grouped by the parts of your site, such as Menu or Blog, each with a few pages and their descriptions
# Golden Crust Bakery > Fresh sourdough, pastries and custom cakes baked daily in Raleigh, NC. ## Main pages - [Menu](https://goldencrust.example/menu) - [Custom Cakes](https://goldencrust.example/cakes) - [Visit Us](https://goldencrust.example/visit) ## Blog - [How We Make Our Sourdough](https://goldencrust.example/blog/sourdough): Our 36-hour process, step by step.
Upload it to the root of your site as llms.txt, next to robots.txt.
Fixes you can copy, not just a list of problems
Schema markup: your facts, stated clearly
Schema markup is a small block of code, usually JSON-LD, that states facts about your business in the shared vocabulary from schema.org. Search engines and AI tools use it to read your name, logo, phone number and profiles correctly, instead of guessing from page text.
What the Fix Kit writes
The kit reads your homepage and builds Organization and WebSite markup from what it finds:
- Your business name, homepage address and description
- Your logo, from your sharing image or site icon
- Your phone number, from a click-to-call link
- Your social profiles (Facebook, Instagram, LinkedIn, YouTube and others) as
sameAslinks
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@graph": [
{
"@type": "Organization",
"@id": "https://goldencrust.example/#organization",
"name": "Golden Crust Bakery",
"url": "https://goldencrust.example/",
"logo": "https://goldencrust.example/logo.png",
"telephone": "+1 919 555 0142",
"sameAs": ["https://www.instagram.com/goldencrust.example"]
},
{
"@type": "WebSite",
"@id": "https://goldencrust.example/#website",
"name": "Golden Crust Bakery",
"url": "https://goldencrust.example/",
"publisher": { "@id": "https://goldencrust.example/#organization" }
}
]
}
</script>Paste it into the <head> of your homepage. If your homepage already has Organization or LocalBusiness markup, the kit tells you so, and you only need the new block if yours is missing details. Check every value before you use it. If the phone number on your site is old, the markup will be too.
For a fuller set of facts AI can use, such as hours, services and service area, see the AI Profile and our guide on getting your business into AI answers.
A clean sitemap: only pages worth indexing
A sitemap is an XML file listing the pages you want search engines to find. Over time many sitemaps fill up with redirects, broken pages and pages marked noindex. When a sitemap is full of pages that should not be there, search engines learn not to trust it.
For sites in your IndexPedia account, the kit provides a clean sitemap, hosted for you. It keeps the pages from your sitemap and leaves out pages found to be broken, marked noindex, redirected, pointing their canonical to another page, or too thin. Pages not checked yet stay in.
Titles and descriptions that fit
Your title and meta description are what people see in Google. A title that is too long gets cut off. A page with no description lets the search engine pick a random piece of text to show.
Shorter titles
The kit shortens titles to about 60 characters. It keeps whole sections, keeps a short brand at the end when it fits, and never cuts a word in half or ends on a little word like "in" or "and".
Golden Crust Bakery: Fresh Sourdough Bread, Pastries and Custom Cakes in Raleigh, North Carolina (96 characters, cut off in results)
Golden Crust Bakery: Fresh Sourdough Bread (42 characters, cut at a natural break and shown in full)
New descriptions
For pages with no description, the kit writes one from the page's own text. It uses whole sentences, skips menu fragments and stays at about 155 characters. You can edit every suggestion before you use it. For more on this, read fix titles and descriptions without code.
Read each suggested title out loud. If it sounds like something a customer would click, keep it. If not, edit it. The kit gives you a strong start, not the final word.
How to use the Fix Kit
Every scan writes the kit. You find it in your report under "Fixes ready for you."
- 1Scan your site
Run a free scan of your domain.
- 2Open the fixes
Go to "Fixes ready for you" in your report.
- 3Review each file
Read every file and suggestion and edit anything that is not right.
- 4Copy, download or auto-apply
Upload robots.txt and llms.txt, paste schema, or let Auto-fix apply titles, descriptions and schema.
| Fix | Where it goes | Do it yourself or Auto-fix |
|---|---|---|
| robots.txt | Root of your site | Upload the file |
| llms.txt | Root of your site | Upload the file |
| Schema | Homepage head | Paste the code or Auto-fix |
| Titles and descriptions | Each page | Edit in your CMS or Auto-fix |
| Clean sitemap | Hosted by IndexPedia | Submit its address |
Auto-fix: no code changes
If you would rather not touch files, Auto-fix can apply titles, descriptions and schema to your live pages. You add one small snippet to your site once, switch on the fixes you approve, and they go live as your pages are served. You can review before anything goes live and undo any time.
Rescan after big changes
Run a new scan after larger changes, such as a redesign or a new section. Sites in an account are re-checked automatically, and the kit is rebuilt from your latest pages. You can also use IndexNow to tell Bing about updated pages right away.
Before replacing your robots.txt, save a copy of the old one. If something looks wrong later, you can put it back in seconds.
Fixes you can copy, not just a list of problems
FAQ
Do I need a developer to use the Fix Kit?
Not always. Uploading robots.txt and llms.txt to the root of your site is usually possible from your hosting panel or CMS. For titles, descriptions and schema, Auto-fix can apply them without touching code.
Will the new robots.txt block anything I need?
It keeps the rules that protect private areas of your site, opens access for search and AI answer crawlers, and points to your sitemap. Read it before you replace your current file, especially if parts of your site must stay private.
What is llms.txt and do I need one?
llms.txt is a proposed standard: a plain text file at the root of your site that tells AI tools what your site is and links to its best pages. No major AI engine has confirmed reading it, so it is optional. It is harmless to add, and it never replaces robots.txt.
What schema markup does the Fix Kit include?
It includes Organization and WebSite markup for your homepage, filled with the name, logo, description, phone number and social profiles found on your site. If your homepage already has business markup, the kit tells you so.
Is the Fix Kit free?
Yes. Every site scan, including the free one, comes with its Fix Kit. Applying titles, descriptions and schema automatically on your live pages is what Auto-fix adds.
Will these files break my site?
No. They only change what search engines and AI tools read, not what visitors see. Still, review each file before you use it, and keep a copy of anything you replace.