2026-08-01 · Jack Stovell

How to check whether AI can actually read your website

Most sites that are invisible to AI aren't being ignored. They're being blocked or served blanks — usually by accident, usually by a default nobody chose on purpose. We built a checker that runs these tests automatically, but every one of them can be done by hand. Here's the full list, with the manual method for each. Readability is one of five different things people mean by AI visibility; the other four, and what measures each, are separated at what AI visibility means.

1. Is your robots.txt turning AI crawlers away?

Fetch yoursite.com/robots.txt and look for user-agent groups naming the current AI crawler tokens. The ones that decide whether you appear in AI answers are the search-index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot) and the user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, DuckAssistBot, meta-externalfetcher). The training crawlers (GPTBot, ClaudeBot, CCBot, Applebot-Extended, meta-externalagent, Amazonbot, Bytespider) only shape future models, so blocking them is a policy choice, not a visibility loss — the checker reports them as informational. Google-Extended is the one training token with a live-answer cost: per Google's docs it also controls grounding in Gemini apps and Vertex AI, so treat a Google-Extended block as a deliberate trade, not a free opt-out. A `Disallow: /` under any group means you've asked that crawler to stay out — and a `User-agent: *` group with `Disallow: /` blocks every one of them at once, which is the case people miss most often because no AI name appears in the file. Older Anthropic tokens (Claude-Web, anthropic-ai) are no longer documented; if your file still names them, add the current three.

This is the check that surprises people most, because several hosts and CDNs now add AI-crawler blocks by default. We've been bitten ourselves: Cloudflare's newer zones default to a managed robots.txt that disallows AI crawlers, and it silently overrode our own file on three of our zones before we found the switch.

2. Does your content exist without JavaScript?

Crawlers largely read raw HTML; many never execute your scripts. Test: curl your homepage, or in the browser use View Source (not Inspect), and search for a sentence you know is on the page. If your HTML is a few hundred bytes of div-and-script with no visible words, AI systems see an empty shell — whatever the rendered page looks like.

Single-page apps fail this by default. The fixes are prerendering, server-side rendering, or serving bot-specific HTML at the edge.

3. Do error pages admit they're errors?

Visit yoursite.com/some-page-that-does-not-exist and check the HTTP status (browser dev tools, Network tab). If it returns 200 instead of 404, crawlers can index garbage URLs as real pages and trust your site less overall. SPAs with catch-all routing get this wrong constantly.

4. Is there identity markup?

Search your homepage source for application/ld+json. Schema.org markup — Organization or Person, with a correct url and sameAs links — is how you tell a machine which entity this site belongs to, rather than hoping it infers correctly. No markup means every mention of your name is a disambiguation puzzle.

5. Do sitemap.xml and llms.txt exist and respond?

Both should return their actual content type, not an HTML fallback page pretending to be there. A sitemap that returns your homepage HTML with a 200 counts as missing — worse, actually, because nothing flags it.

6. Does a model already know you?

Ask ChatGPT or Claude directly: "What is [your business]?" — ideally from a fresh session. You're testing two things: whether it can find and read your site live, and whether you exist in its trained knowledge at all. The second improves slowly, as a function of how much readable, consistent material about you exists in public.

The shortcut

Our checker at hilyt.it/tools/ai-visibility runs all six against any domain in about fifteen seconds, with the specific fix for each failure. It's free, no account. We built it because we kept finding these failures on our own sites — including this one, which once scored 41/100 on its own test.

Every weight, each check's pass/fail/unverified rule, and the limitations of checking a site with a plain fetcher are written out on how the AI readiness score works.

Run the free AI visibility check