How to Check If Your Site Is AI Ready

"AI ready" sounds like a property of your content. In practice the first half of it is a property of four files, three of which most people never look at, and the order they are read in is what decides whether your content is ever considered at all.

An AI system that wants your page resolves /robots.txt at your origin before it requests the page. If the rules there name its user-agent token and disallow the path, it stops, and nothing about your headings or your schema markup matters. If they do not, it fetches the page and reads the HTML. Separately, index-building crawlers look for /sitemap.xml to find pages nobody linked them to, and some coding agents look for /llms.txt.

That is exactly the fetch set our own AI-readiness analysis makes, and it is worth knowing precisely because it tells you where to look. A free check costs five requests: the page itself, two parallel HEAD requests for /sitemap.xml and /sitemap_index.xml, one for /robots.txt, and one for /llms.txt. In a full audit the page HTML is already in hand from another analyzer, so only the last four run.

The consequence people miss: three of those four are looked up at your domain root, not at the URL you are checking. Your robots.txt, your sitemap and your llms.txt describe the whole origin. Only the structured data and the heading outline belong to the individual page. A crawler map is therefore a fact about your domain, and you do not need to check it page by page.

A tool that answers "are you blocking AI?" with one verdict is worse than useless, because the most common deliberate posture in 2026 is to block some of it on purpose. Plenty of publishers Disallow GPTBot while allowing OAI-SearchBot: no content for training, full participation in AI search. That is a coherent content-policy position, and grading it as a defect would tell someone who made a considered decision that they had made a mistake.

So the 9 model-training crawlers are checked, reported, and never scored. Blocking one costs you nothing in our score and, as far as anyone can measure, nothing in AI search visibility. Our own AI recommendations never advise unblocking them either.

What is scored is the 8 points spread across 6 crawlers, inside a 20-point crawlability block that also gives 8 points for a reachable sitemap and 4 for a robots.txt that answers at all. Those 6 are the search indexers and on-demand fetchers whose operators are answering questions at scale today.

Two consequences of scoring per crawler rather than per category are worth stating plainly. First, a crawler can sit in a scored category and still be reported only: Amazonbot is checked and shown, and blocking it costs nothing, because Amazon moved the visibility-relevant work to Amzn-SearchBot and Amzn-User and documents Amazonbot itself as possibly feeding model training. Second, only an explicit block withholds points. A crawler your robots.txt has never heard of scores exactly the same as one you deliberately allowed, because under RFC 9309 those two states mean the same thing to the crawler.

/llms.txt is a proposed markdown file that tells a language model what a site contains and where its documentation lives. We check for it, we show whether you have one, and we do not score it. That last part is the useful bit, so here is the reasoning rather than just the position.

Adoption sits at roughly 10% of sites. Log studies covering hundreds of millions of AI bot visits found effectively no AI search crawler fetching the file, and Google has said Search does not support it. A checker that docked you points for not having one would be selling you a file that the systems you care about do not read.

It does get fetched, though, and by a specific audience: AI coding agents. Cursor, Claude Code, Copilot and their peers routinely pull llms.txt on documentation sites. If developers integrate with your product, that is a real reason to ship one — just not a ranking reason. Google's Lighthouse has since promoted its Agentic Browsing category, including an llms-txt audit, into PageSpeed Insights' default configuration; measured against a live run, that audit passes when the file exists and reports not-applicable when it does not, so even Google's own tooling does not penalise its absence.

If that changes, the stance changes. It is a judgement about current crawler behaviour, not a principle, and it is written down so it can be revisited rather than quietly assumed.

The overlap is real and worth saying first: a reachable sitemap, a clean heading outline, structured data and readable text serve both audiences. An AI-readiness check is not a replacement for an SEO audit, and a site that is genuinely good at one is usually not terrible at the other. Four things are genuinely different.

The gatekeeper file is read 22 times, not once. An SEO audit checks robots.txt for Googlebot and Bingbot, where the question is binary and the answer is almost always yes. AI readiness reads the same file for 22 tokens whose consequences differ by category, and where blocking some of them is correct. Most of the findings live in that difference.

There is no position to lose, and no console to check. A ranking problem shows up as a number that moved, in a report someone publishes. Blocking a search indexer is not a demotion, it is an absence: there is no AI equivalent of Search Console telling you your impressions went to zero, because there is no impression to count. The failure is silent by construction, which is the entire reason to check for it on purpose.

Structured data is read for meaning, not for eligibility. In an SEO audit, schema markup is largely a gate on rich results: either you qualify for the star ratings or you do not. For a model, the same block is a set of stated facts about the page, and it is useful whether or not any search feature is attached to it.

Some fetches happen at question time. Search crawling is a background process whose results are read later. On-demand assistant fetchers work differently: they request your page at the moment somebody asks about it, often having been handed your URL directly. Block one and you have broken a read that the user explicitly asked for. There is no SEO analogue to that failure at all.

What an AI-readiness check does not tell you is equally worth being clear about. It cannot count citations, it cannot show you traffic from AI answers, and nobody can, because the systems involved do not publish either. Being fetchable, labelled and readable is necessary for being cited. It has never been sufficient, and a tool that promised otherwise would be guessing.