For most WooCommerce stores, no — charging AI crawlers to read your product and category pages is the wrong move, because those are the exact pages that get you into AI shopping answers, and a price tag turns the crawler away from the recommendation you were trying to win. Cloudflare's pay per crawl, live in private beta since July 1, 2025, gives every site a per-crawler choice — allow, charge, or block — enforced at the network edge with an HTTP 402 Payment Required response. The real question isn't how much you can charge; it's which of your URLs you can afford to put behind a paywall without vanishing from the answers that sell.
We build Contexta, an AI-visibility plugin for WooCommerce, so we spend our days on the reachability side of this — checking whether the bots that feed ChatGPT, Perplexity and Gemini can actually fetch a store's pages. Pay per crawl turns that technical question into a business one: you now decide, per bot, whether reaching you costs money. For a publisher whose article is the product, charging is the whole point. For a store whose product sits behind the page, the same lever can quietly cut off demand.
How does pay per crawl actually work, and what does charging look like?
Pay per crawl adds a third setting next to allow and block: charge, where Cloudflare answers the bot with an HTTP 402 and a crawler-price header instead of your page, and only serves the content once the crawler presents payment. You set a flat, per-request price across your whole domain in the dashboard, and Cloudflare acts as the merchant of record — it verifies the crawler, aggregates the requests, bills the AI company, and pays you.
There are two ways a crawler meets that price. A well-behaved bot runs in proactive mode: it sends a crawler-max-price header up front — effectively "I'll pay up to this much per page" — and gets a 200 immediately if your rate fits its budget. A simpler bot runs reactive: it requests the page, hits the 402, then retries with a payment confirmation. Either way the money moves over ordinary HTTP headers, no checkout page, no redirect.
The part worth internalising is where this runs. Charging happens at the same network edge where Cloudflare already returns a 403 to bots it blocks — the layer your robots.txt file can't see and your SEO plugin never reports. That blind spot is why a store can already be turning crawlers away without knowing it, the failure mode we walk through in is Cloudflare silently blocking AI crawlers from your site. Pay per crawl lives in that same enforcement layer, one setting over.
Should a WooCommerce store charge AI crawlers for its product pages?
No — keep the storefront free. A publisher sells the words on the page, so a crawl that copies them is a straight loss; a store sells a product the page only describes, so a crawl that lands you in an AI shopping answer is free distribution you'd otherwise pay Google Ads for. Putting a 402 in front of a product page to earn a fraction of a cent, at the cost of the recommendation that drives a $60 sale, is a trade no store should take.
The clean way to hold the whole decision in your head is by page type, not by bot alone:
| Allow free | Charge (402) | Block | |
|---|---|---|---|
| Product & category pages | |||
| Blog posts & buying guides | |||
| Full-catalog feed / bulk export | |||
| Scraper that never cites or sells |
Pay per crawl is really the allow-versus-deny choice from robots.txt for AI with a third option bolted on, and the same discipline carries over: it's a per-bot, per-URL decision, not one master switch labelled "AI". Treat it as a switch and you'll either paywall the pages that sell or leave the endpoints that leak value wide open.
What happens if you charge the bot that feeds AI shopping answers?
You disappear from those answers. The crawlers that decide whether ChatGPT or Perplexity recommends your product — OAI-SearchBot, PerplexityBot, ChatGPT-User — are separate user-agents from the training crawlers, and if you put a price on the search-and-retrieval ones, they hit the 402, skip your page, and cite a competitor whose page was free. The bill you hoped to send never gets paid; you just quietly lose the citation.
The split is the whole trap. The bot you might actually want to charge — GPTBot, OpenAI's pure training crawler — sends you no traffic either way, so charging it changes little. The bot you must keep free is the search crawler that sends buyers. Confuse the two and you've priced out your customers while inconveniencing no one, the exact distinction laid out in which AI crawlers to allow and why. It's also why the 2.5 million sites that have blocked AI training outright gave up far less than the headline suggests: training access was never the thing paying them.
This is why we built Contexta's AI Visibility test to fetch each page as the individually named bots — OAI-SearchBot, PerplexityBot, GPTBot — and report which ones actually reach you. Before you attach a price to any crawler, you need to see which of them is feeding the answers you'd lose, not guess from a bot's name.
Which of your store's URLs are actually worth putting a price on?
The only pages worth charging for are the bulk, high-cost endpoints a crawler drains wholesale with no chance of a sale — a full-catalog CSV or XML feed, a product API, a dataset export — not the storefront a shopper's assistant reads to recommend you. That's where the lopsided economics pay per crawl was built for actually apply to a store.
The imbalance is real at the top of the funnel: Cloudflare reported OpenAI's crawlers pulling as many as 3,700 pages for every referral they sent back in late March 2025, and Perplexity peaking above 700:1 before settling below 200:1 later in the year. Those ratios are the argument for charging a firehose of bulk data. But a single product page that a buyer's agent reads is the opposite trade — one crawl against a possible sale — so the ratio that damns a bulk feed makes the storefront a bargain to give away.
None of this matters until the free pages are reachable in the first place. A product page a crawler can't fetch earns nothing whether it's marked free or paid, and reachability is the floor under every other AI-visibility fix, the point we keep returning to in why AI assistants never cite your blog. Sort out who can reach what before you start pricing access.
Is pay per crawl turning into pay per use — and what does that change for stores?
Yes — on July 1, 2026 Cloudflare announced the Monetization Gateway, which moves the charge from the crawl to the use: instead of billing a bot for fetching a page, you can charge any caller for any resource — web pages, datasets, APIs, or MCP tools — over the open x402 payment protocol, named for that same 402 status code.
Jul 2025
Cloudflare launches pay per crawl in private beta — allow, charge or block each bot, enforced with an HTTP 402
2026
Cloudflare reports its network sending more than 1 billion 402 "Payment Required" responses to AI crawlers a day
Jul 1 2026
Monetization Gateway extends the model to pay per use via x402 — charge any caller for pages, datasets, APIs or MCP tools
For a store this is the more interesting shift, because a store's genuinely monetizable asset was never its marketing pages — it's the machine endpoints an agent calls to check stock, confirm a price, or place an order. As commerce moves toward agents transacting through MCP tools and product APIs, "charge per use" describes a real service being consumed, not a marketing page being read. That's the version of this to watch.
Be honest about the timeline, though: pay per use is freshly announced, settled in stablecoins, and most WooCommerce stores have nothing to charge for yet. As of mid-2026 the practical decision is still the plain one — keep your storefront and its search bots free, block the scrapers that never cite or buy, and reserve any charge for bulk data most stores don't even expose. The pricing engine is here early; the reason to use it, for a typical store, mostly isn't. The access side of the same push is likelier to reach you first: Cloudflare's Sept 15 2026 default block on mixed-use crawlers over ad pages decides whether a bot is stopped, where pay per use only decides whether it pays.
FAQ
Will charging AI crawlers actually make my store money?
For most stores, no meaningful revenue — pay per crawl earns a flat per-request fee that only adds up at publisher-scale crawl volumes, and Cloudflare sets a domain-wide price with itself as merchant of record. A store's real return from AI crawlers is the referral and the sale that follows a citation, not the fraction of a cent per fetch. Charging your product pages trades that sale for pennies, which is why the math rarely works for a storefront even though it can for a large media site.
Do I need Cloudflare to use pay per crawl?
Yes — pay per crawl and the newer Monetization Gateway are Cloudflare features enforced at its network edge, so your site has to sit behind Cloudflare and be enrolled, which was still private beta as of mid-2026. Without Cloudflare your only levers are allow or block through robots.txt, which is advisory and easily ignored, plus whatever your own WAF or host can enforce. There is no charge option outside an edge that can return a 402 and settle the payment for you.
If I charge GPTBot, will I disappear from ChatGPT?
Charging GPTBot alone won't remove you from ChatGPT's answers, because GPTBot is OpenAI's training crawler and ChatGPT's live citations come from OAI-SearchBot and ChatGPT-User instead. Put a price on those two and you do lose the citation, since they hit the 402 and cite a competitor whose page was free. Keep the search-and-retrieval bots free and charge only the training crawler if you want any charge at all — it sends no traffic either way.
Is blocking AI crawlers better than charging them for a store?
It depends on the specific bot, not on a single policy — block the scrapers that ignore your rules and never send buyers, keep the search-and-citation bots free, and reserve charging for bulk data endpoints like a full-catalog feed or API. Blocking your storefront's search bots costs you the AI shopping traffic you are trying to grow, so a blanket block is usually as wrong as a blanket charge. The right answer is per bot and per URL, decided by whether that page can lead to a sale.