2026-08-01 · Jack Stovell
How to check whether AI can actually read your website
Most sites that are invisible to AI aren't being ignored. They're being blocked or served blanks — usually by accident, usually by a default nobody chose on purpose. We built a checker that runs these tests automatically, but every one of them can be done by hand. Here's the full list, with the manual method for each.
1. Is your robots.txt turning AI crawlers away?
Fetch yoursite.com/robots.txt and look for user-agent blocks naming GPTBot, ClaudeBot, PerplexityBot, CCBot or Google-Extended. A Disallow under any of them means you've asked that system's crawler to stay out.
This is the check that surprises people most, because several hosts and CDNs now add AI-crawler blocks by default. We've been bitten ourselves: Cloudflare's newer zones default to a managed robots.txt that disallows AI crawlers, and it silently overrode our own file on three of our zones before we found the switch.
2. Does your content exist without JavaScript?
Crawlers largely read raw HTML; many never execute your scripts. Test: curl your homepage, or in the browser use View Source (not Inspect), and search for a sentence you know is on the page. If your HTML is a few hundred bytes of div-and-script with no visible words, AI systems see an empty shell — whatever the rendered page looks like.
Single-page apps fail this by default. The fixes are prerendering, server-side rendering, or serving bot-specific HTML at the edge.
3. Do error pages admit they're errors?
Visit yoursite.com/some-page-that-does-not-exist and check the HTTP status (browser dev tools, Network tab). If it returns 200 instead of 404, crawlers can index garbage URLs as real pages and trust your site less overall. SPAs with catch-all routing get this wrong constantly.
4. Is there identity markup?
Search your homepage source for application/ld+json. Schema.org markup — Organization or Person, with a correct url and sameAs links — is how you tell a machine which entity this site belongs to, rather than hoping it infers correctly. No markup means every mention of your name is a disambiguation puzzle.
5. Do sitemap.xml and llms.txt exist and respond?
Both should return their actual content type, not an HTML fallback page pretending to be there. A sitemap that returns your homepage HTML with a 200 counts as missing — worse, actually, because nothing flags it.
6. Does a model already know you?
Ask ChatGPT or Claude directly: "What is [your business]?" — ideally from a fresh session. You're testing two things: whether it can find and read your site live, and whether you exist in its trained knowledge at all. The second improves slowly, as a function of how much readable, consistent material about you exists in public.
The shortcut
Our checker at hilyt.it/tools/ai-visibility runs all six against any domain in about fifteen seconds, with the specific fix for each failure. It's free, no account. We built it because we kept finding these failures on our own sites — including this one, which once scored 41/100 on its own test.