2026-08-03 · Jack Stovell

Cloudflare is blocking AI crawlers on your site — how to check and fix it

Since mid-2025, Cloudflare has defaulted many zones — especially newly added ones — to blocking AI crawlers in two separate ways: a managed robots.txt that appends AI-crawler disallow rules on top of your own file, and edge-level blocking that returns errors to AI user agents before the request ever reaches your origin. Both controls live under Security → Bots, each has its own toggle, and turning one off does not turn off the other. We run our sites on Cloudflare and hit this on multiple zones, including ones where the robots.txt we deployed was being silently overridden.

How do you check if Cloudflare is blocking AI crawlers?

Two comparisons, both from the outside. First, fetch your live robots.txt and compare it to the file you actually deployed — if Cloudflare's managed directive is on, the live version contains AI-crawler disallow rules you never wrote, typically appended after your own rules. Second, request the same page with an AI crawler's user agent and with a browser's, and compare the responses — if the browser gets a 200 and the AI agent gets an error, the edge is blocking by user agent.

# 1. Is Cloudflare rewriting your robots.txt?
curl -s https://yoursite.com/robots.txt
# Compare against the robots.txt file in your repo or CMS.
# Blocks for GPTBot, ClaudeBot, CCBot etc. that you did not
# write mean the managed robots.txt directive is on.

# 2. Is Cloudflare blocking AI user agents at the edge?
curl -s -o /dev/null -w "GPTBot:  %{http_code}\n" -A "GPTBot" https://yoursite.com/
curl -s -o /dev/null -w "Browser: %{http_code}\n" -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" https://yoursite.com/
# Same URL, two user agents. Browser 200 with GPTBot 403 (or
# another error) means requests are being stopped at the edge —
# your origin never sees them, so origin logs will show nothing.

Run both checks even if the first one comes back clean. The two blocks are independent, and a zone can have either, both, or neither. On our zones we found both states in the wild — and the failure is silent: nothing in your deploy pipeline warns you, because your origin is serving exactly what you told it to. The rewrite happens at the edge, on the way out.

Which two settings are involved?

Both sit in the Cloudflare dashboard under the zone's Security → Bots area, and they are separate controls. The first is the managed robots.txt directive: Cloudflare serves an edited robots.txt for your zone, appending disallow rules for known AI crawlers on top of whatever your origin serves. Your file is not modified — it is overridden in flight, which is why the live URL and your deployed file disagree. The second is the AI-crawler blocking control: an edge rule that matches known AI user agents and refuses them outright, regardless of what robots.txt says. Exact menu labels shift as Cloudflare iterates on the dashboard, so look for the two by purpose — one governs the robots.txt served for your zone, the other governs whether AI bots are blocked at the edge — rather than by precise wording. Fixing one does not fix the other; verify each with the commands above after changing anything.

Should you turn the blocking off?

It depends on whether you want AI systems to read your site — Cloudflare's default is a legitimate choice for people who do not. If you sell content and object to it being used for training, leaving the blocks on is coherent. But if you want to be visible in AI answers — cited, recommended, correctly described — the blocks are doing you active harm: a crawler that gets a 403 cannot read your pages, and being indexed by Google does not compensate, because Google's crawlers and AI crawlers are entirely separate.

The middle path is to turn off the blanket blocks and write your own per-agent rules, allowing the crawlers you want and disallowing the ones you do not — robots.txt rules for AI crawlers covers the trade-offs, and the AI crawler user agents list names the agents. Whatever you decide, decide it yourself: the failure mode we are warning about is not blocking AI, it is believing your robots.txt says one thing while the edge serves another. After any change, re-run the checks, or point the free AI visibility checker at your site — it tests the live robots.txt and crawler responses, not what your repo thinks is deployed.

Check your AI visibility — free

Related guides

All guides · Free AI visibility checker · hilyt.it