Block, charge, or allow: the 2026 AI-crawler decision for a small WooCommerce store

For a small WooCommerce store in 2026, the right call is to allow the AI bots that shop and cite — OAI-SearchBot, ChatGPT-User, PerplexityBot — and reserve blocking or charging for the training-only crawlers that send zero buyers. The one choice that quietly costs you money is "block everything," the single toggle most security plugins and Cloudflare offer, because it sweeps up the search bots that place your products in ChatGPT and Perplexity answers, where referred sessions convert several times higher than organic search (Semrush and Similarweb, 2026). The decision is per-bot, not one switch.

We watch this exact mistake land on stores while building Contexta, a WordPress plugin that reads a store's own Google Search Console data — and the pattern is almost always accidental. Nobody decides "make my store invisible to ChatGPT shopping"; a well-meaning admin flips a "Block AI Scrapers" switch, a CDN rule catches an unfamiliar user-agent, and the store drops out of AI answers with nothing in the normal dashboards to flag it. So here is the actual three-way decision — block, charge, or allow — sized for a small catalog, and the specific bots where getting it wrong is expensive.

Should a small WooCommerce store block, charge, or allow AI crawlers?

Allow the search-and-shopping bots, and block or charge only the training and scraper bots — because for a storefront the three levers are not equal in value. Your revenue from AI comes from being cited and bought, so the bots that read a page to answer a shopper get you paid, while the bots that bulk-collect text for model training send no traffic at all. Treating those two groups the same is the root error behind almost every bad crawler policy.

The grouping only works if you can tell the bots apart, and the trap is that they share company names while doing opposite jobs. OpenAI alone runs GPTBot for training, OAI-SearchBot for its search index, and ChatGPT-User for live on-demand fetches — block the first and you lose nothing, block the second and you vanish from ChatGPT. Sorting each named crawler into "sends buyers" or "doesn't" is the whole exercise, and the full per-bot breakdown lives in which AI crawlers to allow and why.

Why does "block everything" quietly cost a store AI shopping visibility?

Because a blanket block catches OAI-SearchBot and PerplexityBot — the bots that build the indexes ChatGPT and Perplexity shop from — not just the scrapers you meant to stop. Block GPTBot and you keep ChatGPT Shopping, since GPTBot only feeds training; block OAI-SearchBot and you lose ChatGPT Shopping entirely, because that is the crawler whose index the assistant reads when a shopper asks it to compare products. The two look almost identical in a robots.txt file and have opposite consequences.

The cost is easy to underrate because the traffic it removes was never large in the first place. AI referrals are still a low single-digit share of most stores' visits in 2026, so losing them doesn't show up as a cliff in your analytics — it shows up as a channel that simply never grows. But those sessions convert several times higher than organic search, and the shopper who asked ChatGPT for "a waterproof jacket under $150" arrives pre-qualified, at the moment of decision. Blocking the bot that would have surfaced you there is exactly the kind of self-inflicted invisibility we catalogue in why AI assistants won't cite your blog: the page is fine, the writing is fine, but the crawler that would quote it got a 403 before it read a word.

There is a second-order cost, too. Once a competitor's page is the one OAI-SearchBot could reach and yours wasn't, the assistant learns to cite them for that query, and clawing that back later is harder than never losing it. A blanket block trades a compounding channel for a bandwidth saving that a small store never needed.

When does charging AI crawlers make sense for a small store?

Rarely — for most small WooCommerce stores, charging your storefront pages earns pennies and loses the sale that would have followed the citation. Cloudflare's pay-per-crawl returns a 402 "payment required" instead of a 200, priced around $0.01 to $0.05 per request, and it only works if your site sits behind Cloudflare and is enrolled, which was still gated as of mid-2026. At those rates the fee never adds up at a small catalog's crawl volume, and every page you toll is a page a free competitor gets cited for instead.

Where charging can be reasonable is a bulk endpoint that costs you real bandwidth and sends no buyers — a full-catalog data feed, an export API, an archive a training crawler would hoover in one pass. Putting a price on that is defensible; putting a price on the product pages you want an assistant to recommend is not. The fuller math on when the toll pays and when it backfires is worked through in pay per crawl: should your store charge AI companies.

Which bots are safe to block, and which must stay free?

Block or charge the training-and-scraper bots freely, and keep the search-and-retrieval bots free without exception — that single split resolves most of the decision. Training crawlers (GPTBot, CCBot, Google-Extended, Meta-ExternalAgent) collect text for models and send no visitors, so blocking them protects future-training opt-out and costs you no traffic. Retrieval bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Googlebot and the Gemini rendering it feeds) decide whether you appear in a live answer, so blocking any of them is a direct hit to visibility.

Worth being clear-eyed about the upside of that left-hand branch, though: blocking the training crawlers saves bandwidth and states a position, but it does not move a store's revenue in either direction — the honest accounting is in what blocking AI training actually costs and saves.

Two honest caveats keep this from being a set-and-forget rule. Google-Extended is a training opt-out token, not a crawler with its own fetch, so blocking it leaves your Search rankings and AI Overviews untouched while opting you out of Gemini training — a free, consequence-light choice. And a robots.txt rule is advisory: some crawlers ignore a block, which is why the practical way to block only the bots you mean is at the edge rather than in a text file, the subject of blocking AI crawlers with Cloudflare.

How do you confirm your block or allow rules actually took effect?

Fetch each page as the specific bot and read the status code, because a rule you wrote is not the same as a rule that's enforced — a 200 means the bot gets through, a 403 or a challenge page means it's blocked, whatever your robots.txt claims. This is the step almost everyone skips, and it's where accidental blocks hide: a security plugin, a firewall rule, or a Cloudflare managed setting can override your intent, and the only way to know is to look from the crawler's side.

This is the check Contexta's AI Visibility test is built around: it fetches your pages as the individually named bots — GPTBot, OAI-SearchBot, PerplexityBot — reads your robots.txt, and flags a Cloudflare block sitting in the way, so you can confirm the shopping bots are actually allowed and only the training crawlers are stopped. Verifying the outcome rather than trusting the config is the difference between a decision you made and a decision that's live. Re-check it after any security-plugin update or CDN change, since those are exactly the events that flip a bot from allowed to blocked without telling you. Cloudflare's Sept 15 2026 default block on mixed-use crawlers over ad pages is one more such event — worth a re-test on that date if any of your pages carry ads.