Blocking AI training is one of the cheapest decisions a store owner can make and one of the least consequential: it saves you bandwidth, it costs you no AI shopping visibility, and it moves no revenue in either direction. Over two and a half million websites had chosen to completely disallow AI training by August 4, 2025 (Cloudflare's own figure, checked July 2026) — but that number counts blocks on training crawlers like GPTBot and ClaudeBot, which send you no visitors, not on the search bots that decide whether ChatGPT recommends your product.
That gap between the two is why Contexta fetches a store's pages as each individually named bot instead of testing for a generic "AI crawler" — because in practice the two jobs get collapsed into one switch, and the switch catches the wrong bot. The headline number gets quoted as "2.5 million sites blocked AI." What actually happened is narrower and more interesting, and it changes what you should do about it.
Does blocking AI training remove your store from ChatGPT answers?
No. Training and answering run on separate user-agents at every major AI company, and the crawler you block to stay out of a model is not the crawler that fetches your page when a shopper asks for a recommendation. OpenAI's own developer documentation is explicit about the split: GPTBot is "used to crawl content that may be used in training our generative AI foundation models," while OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features" (developers.openai.com, checked July 2026).
The consequence line in those same docs is the one worth pinning to the wall: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." Losing the answer while keeping a bare link is exactly the downgrade a store cannot afford — and there is no matching sentence anywhere for GPTBot, because disallowing GPTBot has no effect on whether ChatGPT can find and cite you.
Google sits in the overlap for a reason we come back to below. Every other major operator keeps the two cleanly apart, which means "should I block AI training?" and "should I block AI crawlers?" are not the same question — and answering yes to the first carries almost none of the risk of answering yes to the second.
Which crawlers train, and which ones answer?
Here is the split as each vendor states it in its own documentation, checked July 2026. Every cell below comes from the company's own crawler page, not from a round-up.
| Company | Training crawler or opt-out token | Answering / search crawler | What blocking the training one costs you |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot, ChatGPT-User | Nothing in ChatGPT search answers |
| Anthropic | ClaudeBot | Claude-SearchBot, Claude-User | Nothing in Claude's answers |
| Perplexity | None declared | PerplexityBot, Perplexity-User | Not applicable |
| Google-Extended (a token, not a fetcher) | Googlebot | Nothing in Google Search |
Two rows deserve a second look. Anthropic's support documentation describes ClaudeBot as collecting web content "that could potentially contribute to their training," while Claude-SearchBot and Claude-User exist to answer questions — so a store can disallow ClaudeBot and still be quoted by Claude. And Perplexity, unusually, documents no training crawler at all: its bot guide lists only two user-agents and says of PerplexityBot that "it is not used to crawl content for AI foundation models," with the same disclaimer on Perplexity-User. Take that as what the company declares rather than as settled fact — Cloudflare published evidence on August 4, 2025 of Perplexity using stealth, undeclared crawlers to get around no-crawl directives, so the declared list and the observed behaviour have come apart before. Either way, blocking Perplexity's named bots to stay out of training blocks nothing that was training you, and removes you from a shopping surface for no gain — the kind of self-inflicted invisibility the block, charge, or allow decision for a small store is meant to prevent.
What does blocking AI training actually save you?
Server load, mostly — and for a small store, not much of that either. Cloudflare's July 1, 2025 analysis put OpenAI's crawl-to-referral ratio at roughly 1,700 to 1 as of June 2025, and Anthropic's at about 73,000 to 1: tens of thousands of fetches for every visitor sent back. That asymmetry is the entire argument for blocking training, and it is a real argument — the classic crawl-for-traffic bargain genuinely does not hold for training crawlers.
2.5M
Sites fully disallowing AI training (Aug 4, 2025)
3.8M
Domains on Cloudflare's managed robots.txt (Sep 24, 2025)
1,700:1
OpenAI crawls per referral, June 2025
73,000:1
Anthropic crawls per referral, June 2025
What it does not save you is money, at storefront scale. A 400-product catalog crawled a few thousand times a month is noise on any modern host, and the fee side is no better: Cloudflare's pay per crawl lets a publisher set a flat per-request price across the site but publishes no rate itself, and whatever you pick, a small catalog's crawl volume is the multiplier that decides the total — which is why the sums stay trivial no matter how the price is set. We work through why that math flips for a publisher and not for a shop in should your store charge AI companies to read your content. Blocking training is a position about how your work gets used. It is not a performance fix, and any article selling it as one is selling you something.
What does blocking AI training actually cost you?
Nothing you can measure today, and possibly something you cannot. The measurable cost is zero: no referral traffic disappears, no citation is lost, no ranking moves, because training crawlers were never the source of any of those. That is the honest, boring answer, and it is why blocking training is a low-stakes choice rather than a brave one.
The unmeasurable cost is worth stating plainly as a hypothesis rather than a fact. A model that has never seen your brand in training has no prior about you — it can only reach you through retrieval, when a search crawler pulls your page in mid-answer. A model that has seen you can name you unprompted, before any fetch happens. Whether that difference is worth anything to a small store is genuinely unproven as of mid-2026; we have no clean way to test it, and neither does anyone claiming otherwise. If you sell a commodity product, retrieval is almost certainly all that matters. If you are building a brand people ask for by name, the case for staying in the training set is at least arguable.
The cost that is unambiguously real, and does show up in analytics, is collateral damage — and it is the reason most stores that lose AI visibility lose it.
How do you block training without blocking the bots that sell?
Name the training crawlers individually in robots.txt and leave every answering crawler alone — never use a one-click "block AI bots" toggle, which is where the accidents come from. A store's disallow list should contain the training and bulk-collection user-agents by name (GPTBot, ClaudeBot, CCBot, Google-Extended) and nothing else, while OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User stay allowed.
There is now a formal vocabulary for exactly this. Cloudflare's Content Signals Policy, published September 24, 2025, adds a machine-readable line to robots.txt that separates three uses of a page: search (building an index and returning links and excerpts), ai-input (feeding content into a model in real time for a generated answer), and ai-train (training or fine-tuning a model). A store's honest setting is search=yes, ai-input=yes, ai-train=no — a preference signal, not an enforcement mechanism, which is the whole point of stating it in words a crawler operator can parse.
Google is the one exception you cannot engineer around. Its Search Central documentation on AI features states that "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search," and points to Google-Extended only for limiting "AI training and grounding in some of Google's other systems" (developers.google.com, checked July 2026). In practice: you can opt out of Gemini's training with Google-Extended and keep Google Search, but you cannot be in Google Search and out of AI Overviews. That trade is not on offer.
How do you check you did not block the wrong bot by accident?
Fetch your own pages as each named user-agent and see what comes back, because robots.txt is only half the story — the other half is enforced at your CDN, where no SEO plugin can see it. A store can have a perfectly correct robots.txt allowing OAI-SearchBot and still be returning a 403 to it from Cloudflare's edge, which is the failure mode behind Cloudflare's September 15, 2026 default block on mixed-use crawlers and behind most of the "we did everything right and still vanished" cases we see.
This is the gap Contexta's AI Visibility check was built for: it reads the robots.txt rules, then performs a live fetch as GPTBot, OAI-SearchBot and PerplexityBot separately and reports which ones actually got the page, including when the block is coming from Cloudflare rather than from your rules. The point is not the score — it is being able to say which specific bot is being turned away before you decide whether you meant to turn it away.
Run it after any change to a security plugin, a CDN setting, or a host-level bot rule. Those three are how most stores join the 2.5 million without ever deciding to.
FAQ
Is it too late to unblock AI training crawlers if I already blocked them?
No — training crawlers re-read your robots.txt on subsequent visits, so removing a disallow line restores access from the next crawl onward, typically within days to weeks depending on how often that bot visits. What you cannot recover is the specific model version already trained without your content, since a released model's weights do not change when your rules do. Unblocking an answering crawler like OAI-SearchBot works faster and matters far more, because that one affects live citations rather than a future training run.
Does blocking GPTBot stop ChatGPT from quoting my product pages?
No. GPTBot is OpenAI's training crawler, and ChatGPT's live answers are served by OAI-SearchBot's index plus on-demand fetches from ChatGPT-User, both separate user-agents with separate robots.txt rules. Disallowing GPTBot removes your content from potential future training and changes nothing about whether ChatGPT can find, read, or cite your store today.
Should a WooCommerce store block AI training crawlers at all?
It depends on why you would, and the honest answer for most stores is that it barely matters either way. Blocking training saves a modest amount of crawl bandwidth and makes a statement about consent; allowing it keeps your brand in the pool a model can recall unprompted, a benefit nobody has yet measured for a small store. The decision that actually moves revenue is the answering-crawler one, so make that call carefully and treat the training call as a preference.
Does robots.txt actually stop AI training crawlers?
Only for the operators that choose to honor it — robots.txt is an advisory standard with no enforcement, so it works on declared, well-behaved crawlers like GPTBot and ClaudeBot and does nothing against undeclared ones. Cloudflare documented Perplexity using stealth, undeclared crawlers to evade no-crawl directives in its August 4, 2025 post, which is the clearest published example of the limit. If you need a block that holds rather than a preference that is usually respected, it has to be enforced at the network edge or your WAF, not in a text file.