Permasize AI Forge
    Back to Blog
    SEO, AEO and GEO

    Schema, llms.txt, and Crawlable Pages: Making Your Site Readable to AI Search

    Schema, llms.txt, and Crawlable Pages: Making Your Site Readable to AI Search

    There is a lot of advice about special files and markup that supposedly help you rank in AI search. Some of it is useful and some is hype. This article ranks the technical work by what the evidence supports, starting with the part that matters most.

    1. Crawlable pages come first

    An AI tool or search engine cannot use a page it cannot read. For most small business sites, these are the checks that matter.

    • The text must be in the page. If a page shows nothing until scripts run, crawlers that do not run scripts will see an empty page. Google's crawler can render scripts, but many other crawlers have been reported not to, so confirm in each vendor's documentation. To test, view the page source, not the inspector, and check that your headings, text, and prices are there. Search Console's URL Inspection tool also shows what Google saw.
    • The fix is usually server-side rendering or prerendering, so the finished HTML is delivered to every visitor and crawler.
    • Do not block crawlers by accident. Check robots.txt, and check any firewall or bot protection set up by your host or content delivery network.
    • Make pages indexable. No stray noindex tags, correct canonical tags, and an XML sitemap that lists the real pages you want found.
    • Give each service its own page and address. Sections that exist only as anchors on one long home page are hard to rank and hard to cite.
    • Link pages together. Google's own guidance lists making content easy to find through internal links as a basic requirement.
    • Keep pages fast and stable, which our October 12 article covers.

    2. Structured data: useful, not magic

    Google's guidance says no special schema markup is needed to appear in AI Overviews or AI Mode. Structured data is still worth having because it helps search engines understand who you are, and it can enable rich results in classic search.

    • Use Organization or LocalBusiness markup for your core details.
    • Use Service, and Offer where you show prices, so that your pricing is machine-readable as well as human-readable.
    • Use BreadcrumbList and Article markup where they fit.
    • Mark up only what is visible on the page, and only genuine reviews, following Google's rules.
    • For FAQ content, write the questions for your readers first. Since Google's 2023 change, FAQ rich results appear for only a limited set of sites, so the markup is optional.
    • Validate your markup with Google's Rich Results Test or the Schema Markup Validator.

    3. llms.txt: what the evidence says

    llms.txt is a proposed standard, introduced in 2024, for a Markdown file at the root of your site that lists your key pages for AI systems. It is often sold as a shortcut to AI visibility. The evidence so far does not support that.

    • Google says it does not use llms.txt for Search, including its generative AI features, and that creating one neither helps nor harms your visibility there.
    • A large study by SE Ranking of about 300,000 domains found no measurable relationship between having an llms.txt file and being cited by AI tools. A second study by Trakkr, covering nearly 38,000 domains that AI tools cite, reached the same conclusion.
    • Few crawlers ask for the file. In one analysis, 97% of domains with a valid file recorded no requests to it during the study month. The studies measured requests and citations, so they cannot show whether a fetched file ever influenced an answer.
    • One practitioner reports that Perplexity reads the file and that citations rose on a few sites after adding it. That is a small, unverified sample, and the same practitioner saw no change in ChatGPT.
    • Software documentation is the exception. Developer tools and coding agents do fetch the file, so it can be useful for documentation sites.

    Our view: an llms.txt file costs about an hour, does no harm, and has little demonstrated effect for local and B2B service businesses. Add it as a courtesy once the fundamentals are done, and do not pay for it as a visibility strategy or believe anyone who promises citations from it.

    4. What does move the needle

    The signals with the strongest support are the plain ones: clear and accurate content that answers real questions, a verified Google Business Profile, genuine reviews and third-party mentions, consistent business facts across the web, and pages that load quickly. These are the same things that have always helped, and they are what AI tools draw on.

    A one-afternoon checklist

    1. View the source of your home page and your top service pages. Is the main text there?
    2. Review robots.txt and your bot protection settings.
    3. Confirm the sitemap lists the right pages and that every service has its own address.
    4. Check Search Console for indexing problems.
    5. Add or validate Organization or LocalBusiness and Service markup that matches what is on the page.
    6. Fix missing or duplicate page titles and headings.
    7. If you want an llms.txt file, add it, and stop there.
    8. Retest after every change.

    Questions owners ask

    Do I need schema to appear in ChatGPT? There is no evidence that it is required. It helps machines understand your pages, which is useful, but it does not replace good content.

    Is llms.txt the new robots.txt? No. robots.txt controls crawler access and is widely honored. llms.txt is a suggestion file with no confirmed support from the major search platforms.

    Should I block AI crawlers? That is a business decision. Blocking search crawlers can reduce your chance of being cited, while blocking training crawlers is a separate choice. Check each vendor's documentation before changing anything.

    Permasize AI Forge's free AI Visibility Audit includes a website health check that looks at these basics: https://site.permasizeaiforge.com/free-audit

    Sources