2026-08-01 · Jack Stovell
JS shells and soft 404s: how your site lies to crawlers
In July we ran hilyt.it through its own AI-visibility checker. It scored 41 out of 100. The site whose entire pitch is being readable by AI was serving crawlers a 1.7-kilobyte empty shell.
That embarrassment turned out to be the most useful bug report we ever got, because the two failure modes behind it are the same ones hiding most modern sites from AI. Here's the anatomy, first-hand.
Failure one: the JavaScript shell
hilyt is a single-page app. The server sends a stub of HTML — a root div, a script tag — and the browser assembles the actual page. Humans never notice. Crawlers mostly don't run the script, so they see the stub: no headings, no text, no links. To every AI retrieval system, the page was blank.
The insidious part is that nothing looks broken. The site renders, analytics fire, Lighthouse is happy. You only see the blank if you check what the server sends before JavaScript runs — View Source, not Inspect Element.
Failure two: the soft 404
Ask a single-page app for a URL that doesn't exist and the catch-all route cheerfully returns the shell with HTTP 200. The crawler records: page exists, contains nothing. Multiply by every dead link and typo'd URL ever pointed at you, and your site becomes a sea of indexable empty pages that erode how much crawlers trust the URLs that do matter.
We had this everywhere: nonexistent profile URLs returned 200, on every host.
How we fixed it without a rewrite
Rebuilding the SPA as a server-rendered app would have been the textbook fix and weeks of work. We fixed it at the CDN edge instead, in a Cloudflare worker that sits in front of the static files:
- Known crawler user-agents requesting a profile get real, server-built HTML — headings, text, identity markup — generated from the same data the React app uses. Humans still get the app.
- The pages that prove content exists — the homepage, the directory, the guides — get the same treatment.
- Unknown usernames now return an actual 404 to crawlers instead of an indexable blank.
- One header matters more than it looks: Vary: User-Agent. Without it, the CDN cache serves whichever variant it saw first to everyone — we watched bots receive the cached human shell and humans receive bot HTML until we set it.
If you serve different HTML to crawlers, it must be the same content humans get, just pre-rendered — cloaking different content is both against search guidelines and self-defeating. Ours is built from the identical data source.
The general lesson
Your site has two audiences that never see the same bytes: people with browsers, and machines without them. Modern frontend stacks quietly optimised the second audience out of existence. The machines are now the ones deciding whether you get cited — so check what they see. It takes one curl.