Yes — and the block that catches the most sites isn't in your robots.txt at all, which is why the tool you'd normally reach for comes back clean. Since July 2025 Cloudflare has blocked known AI crawlers by default on new domains, and it does this at the network edge: OAI-SearchBot requests your page, Cloudflare returns a 403 before the request ever reaches WordPress, and nothing inside your site logs that it happened. The 2-minute check is one command — fetch a live page while pretending to be the bot and read the status line. 200 means reachable; 403 or a returned challenge page means Cloudflare stopped it.
We build an AI-visibility plugin for WordPress, and this is the single most common false negative we see. An owner opens their SEO plugin, reads a robots.txt that plainly allows GPTBot, and concludes the crawlers are getting in. They aren't. The robots.txt is telling the truth about itself and lying about the outcome, because a completely separate system decided the request never gets that far. That same edge layer is where Cloudflare has since added a way to charge crawlers rather than only block them, which raises a separate question for shops — whether your store should charge AI companies to read your content at all.
Why can't your robots.txt editor see the block?
Because robots.txt and Cloudflare's block live in two different systems that never talk to each other — one is a text file your SEO plugin generates inside WordPress, the other is a rule that runs at Cloudflare's edge before any request reaches WordPress. Yoast, Rank Math or a hand-edited file can read and write that text file all day; none of them has a window into what Cloudflare does to a request that never arrives at your origin.
The distinction that matters is advisory versus enforcement. A robots.txt line is a polite note the crawler can choose to honour; a Cloudflare rule is a locked door. You can have User-agent: GPTBot with Allow: / sitting in a pristine robots.txt while Cloudflare's managed rule returns a 403 to that same GPTBot at the door. Both are true at once, and only one of them decides whether you get cited.
robots.txt — advisory
- A text file your SEO plugin generates inside WordPress
- States which bots you would prefer to allow or block
- A request the crawler can honour or ignore
- Fully visible and editable in Yoast, Rank Math or by hand
Cloudflare edge — enforcement
- A rule that runs before any request reaches WordPress
- Returns a 403 or a challenge to the bot's real fetch
- A locked door, not a request — the bot never gets inside
- Invisible to every plugin that lives inside WordPress
This is also why no mainstream SEO plugin flags it: they operate from inside WordPress, so they can only report the file they wrote, not the edge that overruled it. That structural blind spot is the whole argument in do you need an AI visibility tool if you already have Yoast or Rank Math — the short version is that the check has to run from outside the site, as the bot, or it can't see the block that matters.
How do you check in 2 minutes?
Fetch one of your real pages twice — once as a normal browser, once as an AI bot — and compare the status codes; if the browser gets 200 and the bot gets 403, Cloudflare is blocking at the edge. You don't need dashboard access, a plugin, or your host — just a terminal and one live URL.
- 1
Fetch as a browser
curl -A "Mozilla/5.0" -I on a real post — confirm you get 200
- 2
Fetch as the bot
curl -A "OAI-SearchBot" -I on the same URL
- 3
Read the status line
200 = reachable; 403 or a challenge = Cloudflare blocked it
- 4
Compare the two
browser 200 but bot 403 means the block is edge-level, not robots.txt
Two details decide whether the test is honest. Use a real deep page — a product or post, not the homepage — because some rules only trigger on specific paths, and the homepage is often the one page that stays open. And test the bot that actually feeds citations: OAI-SearchBot builds ChatGPT's index, PerplexityBot builds Perplexity's, ClaudeBot fetches for Claude. Checking GPTBot alone can pass while the search bot that quotes you is the one getting the door. And a clean 200 isn't the finish line — if the bot reaches the page but your text only renders after scripts run, it still reads an empty shell, the separate failure covered in why content that needs JavaScript is invisible to AI.
What does a Cloudflare block actually look like?
A Cloudflare edge block shows up as one of three responses to the bot — an HTTP 403, a challenge page, or a robots.txt that Cloudflare rewrote on the way out — and each points at a different setting. The tell that Cloudflare is even in the path is a Server: cloudflare header and a cf-ray value in the response; from there the status code tells you which lever tripped.
- 403 Forbidden
- The "Block AI bots" managed rule or a WAF custom rule — a hard edge block
- 503 + "Just a moment…"
- Bot Fight Mode or a Managed Challenge — permanent for non-JS AI bots
- Rewritten robots.txt
- Cloudflare's managed robots.txt editing the file at the edge
- Clean 200 with your real HTML
- No edge block — if AI still can't see you, look elsewhere
The challenge case is the sneakiest, because to a human it looks harmless. Bot Fight Mode and Managed Challenge answer with a 503 and a "Just a moment…" interstitial that a browser clears in a second by running JavaScript. AI crawlers don't run JavaScript as of mid-2026, so a challenge they can never solve is a permanent block dressed up as a temporary one — the page looks fine to you and is a dead end to the bot. Reachability is the first rung of AI visibility for exactly this reason, and it sits above every writing fix in why AI assistants never cite your blog: a flawless answer-first paragraph earns nothing while the crawler that would quote it is getting a 403.
Which AI crawlers does Cloudflare block by default?
Cloudflare's default block targets the AI crawlers it can verify by signature — GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended and a growing list of others — and since July 2025 new domains get this switched on at signup unless the owner opts out. Older domains that predate the change are more likely to have it off, which is why two stores on the same platform can behave completely differently.
The trap here is treating "AI bots" as one switch. Vendors split their crawlers by job, and Cloudflare lets you allow or block each category separately — the same per-bot nuance that makes robots.txt an allow/deny decision rather than a single on-off toggle. Allowing the training crawler (GPTBot) does nothing for citations if the search crawler (OAI-SearchBot) that builds the index is still blocked. If you want to appear in answers, it's the search-and-retrieval bots — OAI-SearchBot, PerplexityBot, Claude's search fetcher — that have to get a 200, and that list shifts as vendors rename and add bots, so treat any allow-list as something to re-check, not set once.
How do you unblock the right AI crawlers?
You fix it where the block lives — inside Cloudflare, not in robots.txt — by turning off the AI-bot block for the crawlers you want and clearing any challenge that catches them, then re-running the fetch to confirm a 200. In the dashboard this is AI Crawl Control on current accounts (older ones keep it under Security → Bots, sometimes labelled "Block AI bots" or "AI Audit"); allow the search-and-citation bots there, and if Bot Fight Mode is on, add a rule that skips the challenge for verified AI search crawlers.
This is the gap Contexta's AI Visibility test is built to close: it fetches each page as the individually named bots and reports the robots.txt rule and the Cloudflare edge response side by side, so you see the 403 the crawler sees rather than the clean robots.txt your SEO plugin reports. It names the exact block and the setting behind it instead of just telling you that you're absent from AI answers.
How do you know the fix actually worked?
Re-run the same two fetches: when the browser and the bot both return 200 on a real deep page, the edge block is gone. That confirmation matters because Cloudflare caches rule changes and robots.txt at the edge, so a setting you flipped can take a few minutes to take effect — a 403 right after the change isn't proof the fix failed. Nor is a 200 full proof it worked: the fetch comes from your address rather than a verified crawler prefix, so the definitive confirmation is a logged hit from OpenAI's published IP ranges.
The proof that lasts is downstream, not in the dashboard. Once the search bots get a 200 and your pages are readable, you start seeing real visitors arrive from ChatGPT, Perplexity and Gemini — cache-proof referrals you can watch in your analytics rather than infer. If that number stays flat after you've cleared the edge, the block was never the whole story, and it's worth walking back up the visibility ladder from reachability.
FAQ
Does Cloudflare block AI crawlers by default?
Yes for new domains — since July 2025 Cloudflare has switched on its AI-bot block at signup, so verified crawlers like GPTBot, OAI-SearchBot and PerplexityBot get a 403 at the edge unless the owner opts out. Domains that predate that change are more likely to have the block off, which is why two sites on the same host can behave differently. The block runs at Cloudflare's network edge, separate from your robots.txt, so it can be active even when your robots.txt allows every bot.
Can an AI bot be blocked even if my robots.txt allows it?
Yes, and this is the most common case people miss — robots.txt is advisory and Cloudflare's edge rule is enforcement, so a clean robots.txt that allows GPTBot means nothing if Cloudflare returns a 403 to that same bot before the request reaches your site. The two systems don't talk to each other. To know the real outcome you have to fetch a live page as the bot and read the status code, not just read the robots.txt file.
Why does my site load fine in a browser but AI can't see it?
Because Cloudflare can answer bots with a challenge page that a browser clears instantly and an AI crawler never can. Bot Fight Mode and Managed Challenge return a 503 with a 'Just a moment…' interstitial that runs JavaScript to verify a real visitor, and AI crawlers don't execute JavaScript as of mid-2026, so the challenge is a permanent block for them while looking harmless to you. Fetch the page as the bot with curl to see the challenge the crawler actually hits.
Which Cloudflare setting blocks AI crawlers and how do I turn it off?
The block usually comes from the 'Block AI bots' managed rule, a WAF custom rule, or Bot Fight Mode — all found under AI Crawl Control on current accounts, or Security → Bots on older ones. Allow the search-and-citation bots you want (OAI-SearchBot, PerplexityBot, Claude's search fetcher) there, and if Bot Fight Mode is on, add a rule that skips the challenge for verified AI search crawlers. Re-run your fetch afterward, because edge changes can take a few minutes to propagate.