AI Visibility Checker

Before an AI system reads a word of your content, it reads your robots.txt. That one file decides whether the rest happens at all, and it is where most sites accidentally decide their AI visibility — usually by writing a rule aimed at one crawler and catching several others with it.

The 22 crawlers this checker knows about are not interchangeable, and that is the part worth getting right. They fall into three groups by what their operator documents them as doing:

• AI search indexers — OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot and their peers. These build the retrieval indexes behind AI answers. Block one and you are removed from the answers and citations that index feeds.
• On-demand assistant fetchers — ChatGPT-User, Claude-User, Perplexity-User and their peers. These fetch a page live, at the moment a person asks an assistant about it. Block one and the assistant cannot read your page even when the user hands it your URL.
• Model-training crawlers — GPTBot, ClaudeBot, Google-Extended, CCBot and their peers. These collect content for training foundation models. Blocking them is a legitimate content-policy choice and it does not reduce your AI search visibility, so this checker reports their status and never scores it.

That distinction is the whole reason a single "are you blocking AI?" verdict is useless. The common posture in 2026 is to block training and allow search and on-demand fetching — for example Disallow GPTBot while allowing OAI-SearchBot. Scoring that as a defect would punish a deliberate, reasonable decision, so we do not.

Rules are applied the way RFC 9309 specifies rather than by string matching. A group naming the crawler shadows the User-agent: * group entirely — even an empty group, which means everything is allowed — and among matching paths the longest one wins. User-agent tokens match exactly, so Applebot and Applebot-Extended are two different crawlers with two different consequences, and a rule for one never silently applies to the other.

One more thing a checker owes you honestly: llms.txt is reported here and never scored. Adoption is around 10% of sites, log studies across hundreds of millions of AI bot visits found effectively no AI search crawler fetching it, and Google has said Search does not support it. AI coding agents do routinely fetch it on documentation sites, which is why its presence is worth showing as context — but a tool that scored you down for not having one would be selling you a file nobody reads.

A blocked search indexer is not a ranking penalty. It is an absence. Your pages are not demoted in an AI answer, they are simply not available to be cited in one, and there is no report anywhere that tells you it happened. That is what makes this failure mode worth a deliberate check: everything looks normal from the inside.

The costly version is rarely a decision anyone made. It is a Disallow: / left in a staging robots.txt that shipped to production, a security plugin's "block AI scrapers" toggle that turns out to cover the search indexers too, or a WAF rule written against one user-agent string that catches four. Each looks like a single line in a file, and each removes a site from the answers people are now asking for.

Blocking the training crawlers is a genuinely different question, and one where the honest answer is that it is yours to decide. It has no effect on whether you appear in AI search results, which is why this checker reports it plainly and leaves the score alone.

Structured data and heading outline are the other half. Once a crawler can reach the page, those are what tell it what the page is and which part answers which question. A page with no JSON-LD is not unreadable — it is just unlabelled, which means the model has to infer everything a schema block would have stated.