How the AI readiness score works

Scoring model version 2 · updated 2026-09-01

The AI readiness score is a technical measurement. It asks whether the page you submit can be fetched, read and understood by an AI system that visits it. It is a whole number from 0 to 100, and the denominator is always 100: points a check cannot earn are never removed from the total.

It is not a measure of whether any assistant cites, recommends or ranks you. Nothing in the score observes a model's answers. A separate, closed-book model observation is reported alongside the score and never counts towards it; it is described below.

The check fetches the submitted URL with its path preserved, plus robots.txt, llms.txt and sitemap.xml from the site's origin root — each once, plus, when /sitemap.xml is not a sitemap, the first sitemap robots.txt declares — following redirects, with a 12-second limit per fetch and bodies cut at 400,000 characters — identifying itself as HilytVisibilityCheck/1.0. Every rule on this page is applied to what those fetches returned. Private and internal hosts are refused.

The five dimensions and their weights

Each check is one of pass, fail or unverified. A pass earns the check's full weight; a fail and an unverified check both earn nothing. The dimension score is the sum of its checks, and the total is the sum of the dimensions. There is no partial credit and no rounding.

Access — 30 points

Can a crawler get in at all.

robots.txt lets AI search and user agents in — 20 points

Pass: Every search-tier and user-tier token in the registry below is allowed on the submitted path. Rules are evaluated as RFC 9309 describes: a token's own User-agent group wins over the * group; consecutive User-agent lines share one group; the longest matching Allow or Disallow pattern wins, with ties going to Allow; * and $ work inside patterns; an empty Disallow allows everything; Sitemap: lines belong to no group. A site with no robots.txt — HTTP 404, or any status other than 200 that is not one of the unverified statuses — passes, because crawlers treat that as allowed. Blocks on training-tier tokens, a Content-Signal ai-train=no line, legacy tokens, and a path that is allowed while the rest of the site is disallowed are reported as notes and do not fail the check.
Fail: Any search-tier or user-tier token is disallowed on the submitted path — through its own group or through User-agent: * — or the group that applies to it carries Content-Signal search=no or ai-input=no. Content-Signal is read per group, so a named crawler's search=no is not erased by a search=yes under *.
Unverified: robots.txt returned HTTP 403 or 429, a 5xx, or could not be fetched at all.

Page responds to a plain fetch — 10 points

Pass: The submitted URL returned an HTTP status below 400 to our fetcher, after following redirects.
Fail: It returned 400 or higher, other than the statuses treated as unverified — a 404 or 410 fails.
Unverified: HTTP 403 or 429, a 5xx, or no response (timeout, DNS or network error).

Extractability — 25 points

Is there readable content in the HTML itself, without running JavaScript.

Content readable without JavaScript — 15 points

Pass: After removing script, style and noscript blocks and every tag, at least 250 characters of text remain, and the page is not a soft 404.
Fail: Fewer than 250 characters of readable text — a JavaScript shell — or the readable text is a not-found page.
Unverified: The page itself was unreachable or returned an error status. The HTTP check carries that failure once; this check is reported as not evaluated rather than failing for the same reason.

Title and description present — 5 points

Pass: A <title> and a meta description (name=description, or og:description as the fallback) are both present.
Fail: Either one is missing.
Unverified: The page itself was unreachable or returned an error status. The HTTP check carries that failure once; this check is reported as not evaluated rather than failing for the same reason.

Not a soft 404 — 5 points

Pass: The page does not present itself as a not-found page.
Fail: The status was a success but the page says it was not found. A <title> or first h1 counts only when one of its separator-delimited segments is nothing but a not-found phrase — "404", "Page not found", "This page does not exist" — and a lone heading still needs corroboration: the other heading agrees, is absent, or the readable text is shorter than 600 characters. A page under 600 characters whose text says "page not found", "404 error" or "error 404" also counts. A real title that merely contains such words ("404 Media", "What is a soft 404?") does not.
Unverified: The page itself was unreachable or returned an error status. The HTTP check carries that failure once; this check is reported as not evaluated rather than failing for the same reason.

Entity clarity — 25 points

Does the page say, in a form a machine can parse, who or what it is about.

schema.org identity markup — 20 points

Pass: A JSON-LD block (script type application/ld+json — microdata and RDFa are not read) contains a Person or Organization node, or a recognised subtype: LocalBusiness, Corporation, ProfessionalService, NGO, EducationalOrganization, LegalService and the other schema.org business types, plus any type ending in Organization, Business, Agency, Store or Shop. It may sit at the top level, inside an @graph, or as the mainEntity of a ProfilePage, WebPage, AboutPage, ContactPage, CollectionPage or ItemPage. @type may be a string or an array; schema: and URL-prefixed types are normalised; a broken block is skipped and the valid ones still count. The detail also reports whether the entity's name contains the submitted name and how many sameAs links point off-site, but neither changes the pass.
Fail: No valid JSON-LD; JSON-LD with no identity type — a WebSite node alone does not count, and neither does a Service, which is an offering rather than the entity offering it; or an identity type that exists only as a property of something else (Article.author, WebSite.publisher, Service.provider), which the detail names so it can be promoted to a top-level node or a page's mainEntity.
Unverified: The page itself was unreachable or returned an error status. The HTTP check carries that failure once; this check is reported as not evaluated rather than failing for the same reason.

Name is consistent on the page — 5 points

Pass: The submitted name appears inside the <title>, the first h1, the JSON-LD entity's name, og:site_name or og:title. The comparison ignores case, punctuation and diacritics and works in any script.
Fail: It appears in none of them.
Unverified: No name was submitted, or the page was unavailable.

Discovery — 15 points

Can a crawler find the page, and is it allowed to keep it.

Sitemap present — 8 points

Pass: /sitemap.xml at the origin root returns 200 with a <urlset> or <sitemapindex>; failing that, the sitemap the first Sitemap: line in a 200 robots.txt declares is fetched and is one — later Sitemap: lines are not tried.
Fail: Neither: no /sitemap.xml and no Sitemap: line in a 200 robots.txt, or the first declared URL answered without sitemap XML.
Unverified: /sitemap.xml, or the first declared sitemap, returned HTTP 403 or 429, a 5xx, or nothing.

No noindex on this page — 7 points

Pass: No noindex or none directive in a robots, googlebot or bingbot meta tag, or in the X-Robots-Tag header.
Fail: One is present. Matching is whole-token — max-image-preview:none is not a noindex — and a scoped X-Robots-Tag counts only when it is unscoped or scoped to *, robots, googlebot, bingbot, or a search- or user-tier AI token. Commented-out meta tags are ignored.
Unverified: The page itself was unreachable or returned an error status. The HTTP check carries that failure once; this check is reported as not evaluated rather than failing for the same reason.

Optional agent aids — 5 points

Conventions some tools look for. Never worth more than 5 points.

llms.txt for AI readers — 5 points

Pass: /llms.txt returns 200 and its body does not begin with < (an HTML fallback page served at that path is not an llms.txt). llms.txt is an emerging convention with no vendor-confirmed consumption, which is why it is worth at most 5 points.
Fail: Anything else with a definite answer, including a 404.
Unverified: HTTP 403 or 429, a 5xx, or no response.

The crawler tokens the robots check evaluates

The robots check evaluates the submitted path for every token below, as that crawler would read the file. Only the search and user tiers decide the verdict; training and legacy tokens are reported as notes. Which token belongs to which product, and what each one does, is kept current in the AI crawler user-agents list.

Search tier — decides the verdict

Builds the index an answer engine queries at runtime. Blocking one of these keeps the page out of that engine's live answers.

OAI-SearchBot — OpenAI
Claude-SearchBot — Anthropic
PerplexityBot — Perplexity
Amzn-SearchBot — Amazon

User tier — decides the verdict

Fetches a page because a person asked about it right now. Blocking one of these turns away the request that most directly means someone is asking about you.

ChatGPT-User — OpenAI
Claude-User — Anthropic
Perplexity-User — Perplexity
MistralAI-User — Mistral
DuckAssistBot — DuckDuckGo
meta-externalfetcher — Meta

Training tier — informational only

Training corpora. Blocking these is a legitimate policy choice and is reported as a note; it never fails the check.

GPTBot — OpenAI
ClaudeBot — Anthropic
CCBot — Common Crawl
Google-Extended — Google
Applebot-Extended — Apple
Bytespider — ByteDance
meta-externalagent — Meta
Amazonbot — Amazon

Legacy tokens — recognised, never decisive

No longer documented by the vendor. A robots.txt that names only these does not block the current tokens; the result says so and names the current ones.

Claude-Web — Anthropic (legacy; the current token is ClaudeBot)
anthropic-ai — Anthropic (legacy; the current token is ClaudeBot)

What "unverified" means for the score

A check is reported as unverified when our fetcher got HTTP 403 or 429, a 5xx, or no response at all for the resource it needs. That usually means an edge or bot-management layer challenged an unknown fetcher, or the host failed. Either way we cannot tell what a crawler would get, and the result says so instead of guessing.

An unverified check earns nothing, and its weight stays in the denominator. Until 2026-09-01 the checker dropped unverifiable checks from the denominator and rescored the rest, so a site that refused the fetcher at the edge could score 100 on whatever was left. Each such row now reads "scored as not earned".

Page-level checks — readable text, title and description, soft 404, identity markup, name consistency, noindex — are reported as not evaluated when the page itself was unreachable or returned an error status. The HTTP check carries that failure once rather than six checks failing for one reason.

The model observation, outside the score

Alongside the score, the checker asks one model — Claude Haiku, claude-haiku-4-5-20251001 — with no web access and no tools: who is the submitted name, answered from its own knowledge only, with an instruction to reply UNKNOWN when it is not confident. An answer is reported as a pass with its first 180 characters quoted; UNKNOWN is reported as a fail. This is closed-book recall, not retrieval, and it says nothing about what the model would find if it searched.

It never changes the score. The API returns it as model_observation, and the result list shows it as a row of weight 0 labelled "Not counted in the score".

A confident answer can describe a different entity with the same name. When the submitted name does not appear in the answer the row says the model may be describing someone else, but the check cannot tell the two apart — read the quoted answer. If the probe is not configured, unavailable, or times out, the observation is reported as unverified.

Limitations

Our fetcher identifies itself as HilytVisibilityCheck/1.0. It cannot present itself as OAI-SearchBot, Claude-SearchBot or any vendor crawler, because vendors verify their crawlers by published IP range, not by user-agent string. Edge rules keyed on a verified crawler are therefore invisible to us: a site may serve those crawlers something different from what we saw, in either direction.
robots.txt is evaluated as each crawler would read it, token by token. The page fetch itself is what an unknown, generic fetcher gets — not what a specific crawler gets.
hilyt.it itself recognises HilytVisibilityCheck as a crawler in its own edge worker and serves it the same prerendered HTML real AI crawlers receive, which most sites will not do. hilyt.it's own score therefore reflects what AI crawlers receive here, not what an anonymous fetcher receives. This is the one condition the checker cannot equalise, and we state it rather than hide it.
llms.txt is an emerging convention. No answer engine has confirmed reading it, so its check is worth at most 5 points and passing it should not be read as evidence that any model uses the file.
Results are cached for 24 hours per domain, name (case-folded), path and query. A repeat check inside that window returns the earlier result and is marked as cached.
Uncached checks are capped at 10 per day per IP address. The API and MCP route adds its own cap of 3 per day per token.
Identity markup is read from JSON-LD only. Microdata and RDFa are not parsed, so a site that carries its Person or Organization markup in those forms will fail the check despite having it.
The score can change when the scoring model changes. Every response carries a version field (currently 2), the cache key includes it, and a result is a point-in-time reading, not a standing.
The submitted path is what gets scored. A different path on the same site can get a different number, which is why shared results carry the checked path.
The soft-404 and name-consistency rules are heuristics. They are written out above so that a call you disagree with can be checked against the rule that made it.

How to read your result

Each row is one check: pass, fail or unverified, with the detail of what our fetcher saw. A failed row says what was found, not only what was missing, so the fix is usually in the sentence.

Read the number as "what a plain fetcher could read on this date". 100 means every check passed as our fetcher saw it; it does not mean an assistant will mention you. Access is the dimension to fix first — nothing else matters while crawlers are turned away or the page does not answer.

The model observation is separate. A pass there with a low readiness score usually means the model knows the name from elsewhere; a good readiness score with a fail there is the ordinary state for most sites, and if it changes it changes slowly, as readable material about you accumulates.

Run the check at hilyt.it/tools/ai-visibility. For which token belongs to which product, see the AI crawler user-agents list.

Changelog

v2 — 2026-09-01: readiness split from model recall. The closed-book probe left the score and became the model observation; the denominator was fixed at 100; robots.txt is evaluated per RFC 9309 against the role-split registry (previously seven tokens with no wildcard handling); JSON-LD is walked as JSON rather than matched by regular expression; the submitted path is scored rather than the homepage; the cache key gained path, query and a case-folded name.
v1 — 2026-07-22: six checks (robots 25, readable text 25, identity JSON-LD 20, llms.txt 10, sitemap 5, model recognition 15), renormalised when a check was unavailable.

Run the AI readiness check · AI crawler user-agents list · Guides · hilyt.it