2026-08-03 · Jack Stovell

Indexed by Google but invisible to AI? Here is why

Being indexed by Google proves almost nothing about whether AI systems can read your site. Googlebot executes JavaScript; most AI crawlers fetch raw HTML once and do not. AI crawlers also obey different robots.txt rules than Googlebot, and CDN bot protection frequently blocks them specifically. Your site can rank on page one while GPTBot sees an empty shell, an error page, or nothing at all.

Why can Google see your site when AI cannot?

Three separate mechanisms, and any of them alone is enough. The first is JavaScript. Googlebot renders pages in a headless browser, so a client-side-rendered site looks complete to it. Most AI crawlers — GPTBot, ClaudeBot, PerplexityBot and the rest — fetch the raw HTML and stop. If your server sends a root div and a script tag, that is the whole page as far as they are concerned. We know this failure mode first-hand: our own site served crawlers a 1.7-kilobyte shell while scoring fine everywhere humans looked.

The second is robots.txt. Rules are per user agent, so a file that welcomes Googlebot can simultaneously disallow every AI crawler — sometimes because a plugin or CDN added those rules without you noticing.

The third is bot protection. CDNs and firewalls score unfamiliar crawlers as threats, and some — Cloudflare in particular, since mid-2025 — block AI user agents by default at the edge, returning errors before your origin ever sees the request. Googlebot is almost always allowlisted; AI crawlers frequently are not.

How do you find out which problem you have?

Fetch your site the way an AI crawler does and compare. Three commands cover all three mechanisms:

# 1. Check robots.txt for AI-crawler blocks
curl -s https://yoursite.com/robots.txt
# Look for User-agent lines naming GPTBot, ClaudeBot, PerplexityBot,
# CCBot or Google-Extended, each followed by "Disallow: /".

# 2. Fetch the homepage as an AI crawler
curl -s -A "GPTBot" https://yoursite.com/ -o bot.html -w "%{http_code}\n"
# A 403 or 5xx here, when the site works in your browser,
# means something is blocking the AI user agent specifically.

# 3. Fetch it as a browser UA and compare
curl -s -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" https://yoursite.com/ -o browser.html
ls -l bot.html browser.html
# Both near-empty (a root div, script tags, no readable text):
# your site is a JavaScript shell — no crawler without a browser
# can read it. Browser file full but bot request errored:
# user-agent blocking. Both full: your HTML is fine.

Note that curl never executes JavaScript, so the browser-UA fetch shows you what any non-rendering crawler receives — the same thing you see with View Source in a browser, as opposed to Inspect Element, which shows the page after scripts have run. Which user agents matter and how to match them is covered in the AI crawler user agents list.

How do you fix it?

Fix whichever mechanism the checks exposed — and re-test, because sites often have more than one. If robots.txt is blocking, edit the rules for the agents you want to allow; if the live file contains rules you never wrote, your CDN is probably injecting them, which is a Cloudflare-managed robots.txt symptom. If AI user agents get errors while browsers get 200s, find the bot-protection layer doing it — usually a CDN security setting — and exempt the crawlers you want. If both fetches came back as an empty shell, the fix is bigger: server-side rendering, prerendering, or serving crawler-specific HTML built from the same content humans see. We took the edge-rendering route ourselves and documented it in the JS shells guide.

The free AI visibility checker runs these checks for you — robots.txt stance, JavaScript-shell detection, identity markup and more — in one pass. One honest caveat: making your site readable removes the reasons AI cannot cite you. It does not make citation happen; that still depends on retrieval, ranking and model behaviour.

Check your AI visibility — free

Related guides

All guides · Free AI visibility checker · hilyt.it