2026-08-03 · Jack Stovell
robots.txt rules for AI crawlers, explained
robots.txt controls AI crawlers the same way it controls any crawler: per-user-agent rules that allow or disallow paths. The part most advice misses is that there are two different kinds of AI agent — training crawlers that collect text for future models, and answer-time agents that fetch your pages when someone asks an assistant a question right now — and robots.txt lets you treat them separately. Blocking the first is a legitimate business choice; blocking the second makes you invisible at the exact moment you could be cited.
How do you let GPTBot crawl your site?
If your robots.txt does not disallow GPTBot, it may crawl; to be explicit, add a stanza naming the user-agent with an Allow rule, as in the example file below. Two things commonly override this without site owners realising:
- An edge or CDN block. Cloudflare has defaulted many zones to blocking AI crawlers since mid-2025 — a managed robots.txt plus bot rules under Security → Bots — so the robots.txt being served may not even be yours, and requests can be refused before your server sees them. We hit this on our own zones; the detail is in how Cloudflare blocks AI crawlers by default.
- A catch-all rule. A robots.txt whose only stanza is "User-agent: *" with "Disallow: /" blocks every AI agent that lacks a stanza of its own.
Check what is actually served at yourdomain/robots.txt from outside, not what sits in your repository.
Should you block or allow AI crawlers?
Treat it as two decisions, not one. Blocking training crawlers such as GPTBot or ClaudeBot is a reasonable choice when your content is the product: you give up long-term model familiarity in exchange for control, and for a content business that trade can be worth making. Blocking answer-time agents is a different act entirely. When someone asks an assistant about you and it tries to fetch your site, a block means the answer gets built from whatever else it can find — or not at all. For most people and small businesses, that visibility loss outweighs the control gained.
The stance we use ourselves: allow answer-time agents unconditionally, and make a deliberate, revisitable choice about training crawlers.
What does a complete robots.txt look like?
Here is a full robots.txt you can adapt. Exact user-agent tokens change as vendors ship new crawlers, so treat the stanzas as slots and fill them from our AI crawler user-agent list, which tracks the current ones:
# robots.txt — an explicit AI crawler policy
# Answer-time agents: fetch pages live to answer a question.
# Blocking these removes you from AI answers.
# More tokens: hilyt.it/guides/ai-crawler-user-agents-list
User-agent: PerplexityBot
Allow: /
# Training crawlers: collect text for future model training.
# This is your decision point — keep Allow for long-term model
# familiarity, or change to "Disallow: /" if your content is
# the product you sell.
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
# Everyone else, and where the site map lives
User-agent: *
Allow: /
Sitemap: https://samwright.example/sitemap.xmlIf you also publish an llms.txt file, the two are complementary: robots.txt says where agents may go, llms.txt proposes what matters once they arrive.
Does blocking AI crawlers protect your content?
Only as far as crawlers choose to comply. robots.txt is a published request, not an enforcement mechanism — nothing in the protocol stops a scraper that never reads the file. The major AI vendors state that their named crawlers obey robots.txt, and that makes the file genuinely useful for steering them, but it cannot bind an agent that ignores it. Actual enforcement happens at the network edge, through firewall and bot-management rules, and it carries its own cost: edge blocks are blunt instruments, and misconfigured ones are among the most common reasons a site turns out to be unreadable by AI without anyone having decided that.
What this won't do: robots.txt cannot recall content that has already been collected, and allowing crawlers does not earn you citations — being fetchable is necessary, not sufficient. Our free checker at /tools/ai-visibility reads your live robots.txt and reports how it treats the major AI agents, alongside the other checks that decide whether AI can read you at all.
Check your AI visibility — free
Related guides
- AI crawler user agents: the full list (GPTBot, ClaudeBot, PerplexityBot and more)
- Cloudflare is blocking AI crawlers on your site — how to check and fix it
- What is llms.txt — and how to write one (with a full example)