robots.txt for AI: allowing assistants without opening the door to scrapers

You cannot use robots.txt to keep scrapers off your site — it is a voluntary request that only well-behaved crawlers obey, and the scrapers you most want gone are precisely the ones that ignore it or fake their identity. What robots.txt is genuinely good at is a narrower job: telling the compliant AI companies — OpenAI, Anthropic, Perplexity, Google — whether they may use your pages for live answers, for model training, or neither. Get those two facts straight and the "allow assistants, block scrapers" question mostly dissolves, because the two halves live on two different layers of your stack.

We build an AI-visibility checker into our WordPress plugin, Contexta, so we have read a lot of real robots.txt files that were quietly doing the opposite of what their owners intended — blocking the search crawlers that drive citations while leaving the training crawlers and the rogue scrapers untouched. The binary advice you see everywhere ("block these bots, allow those") hides the part that actually bites: robots.txt is the wrong tool for half of what people ask it to do, and knowing which half is the whole skill.

Can robots.txt actually keep scrapers off your site?

No — robots.txt is advisory, not enforcement, so it stops only the crawlers that choose to read and honor it. The protocol that defines it, RFC 9309, is explicit that following the rules is voluntary; a compliant crawler like GPTBot, PerplexityBot or Googlebot reads your file and obeys, while a scraper built to harvest content ignores it entirely or spoofs an ordinary browser user-agent so your rules never match it. There is nothing in robots.txt that can compel anyone.

That flips the title's premise on its head. A Disallow: / written to keep "scrapers" out lands only on the polite bots — often the exact search crawlers you wanted citing you — while the impolite ones carry on regardless. The door to scrapers was never wired to robots.txt in the first place, which is also the most common reason a site owner tightens the file, sees no drop in bot load, and can't work out why. If you have accidentally shut out the wrong crawlers, that self-inflicted block is one of the fixable causes behind why AI assistants never cite your blog.

So what is robots.txt genuinely good for with AI?

Making the training-versus-retrieval decision for the AI companies that honor it — that is the job robots.txt does well and nothing else does at all. Each major AI company runs at least two separate crawlers with two separate jobs, and robots.txt lets you answer them independently by name. Allowing the retrieval bot while disallowing the training bot is a legitimate, fully supported configuration, not a hack.

The pairs matter more than any single line of a template:

• OpenAI: GPTBot gathers training data; OAI-SearchBot and ChatGPT-User fetch pages for live answers and citations. Blocking GPTBot does not remove you from ChatGPT search.
• Anthropic: ClaudeBot is the training scraper; Claude-Web / anthropic-ai retrieve pages to answer a user's live question.
• Google: Google-Extended is a training opt-out token — not even a crawler — while Googlebot keeps indexing you for search as normal.

Search crawling and training crawling are separate decisions controlled by separate user-agents, so you can allow citation while refusing training, or the reverse. The full bot-by-bot breakdown of what each one feeds and what you lose by blocking it is its own article — which AI crawlers to allow and why — so this piece stays on the strategy rather than the roster.

Why doesn't "allow assistants, block scrapers" map onto robots.txt?

Because robots.txt sorts crawlers by declared user-agent and stated purpose, not by whether they are friendly or hostile — and "assistant versus scraper" is a good-versus-bad axis the file simply cannot see. You are trying to give it an intent it has no field for. What it understands is "which named bot is this" and "am I allowing this path," and those questions do not line up with the one you actually care about.

Lay the two real axes side by side and the strategy becomes obvious: whether a bot obeys robots.txt at all, and what it wants your content for.

↑ Honors robots.txt

Retrieval + compliant

OAI-SearchBot, PerplexityBot, Claude-Web — allow here to earn citations. robots.txt works.

Training + compliant

GPTBot, ClaudeBot, Google-Extended — your call, and robots.txt is exactly how you make it.

Retrieval + rogue

Rare, but robots.txt is ignored — only edge rate-limits apply. Weigh against lost citations.

Training / scraping + rogue

The actual scrapers. robots.txt does nothing here — a WAF or rate limit is the only lever.

Wants: answers vs training →

The line runs horizontally, not diagonally. robots.txt is the right and only tool for the top row, and useless for the bottom one. Trying to solve the bottom row inside robots.txt is the binary-advice trap — you write ever-longer block lists that the rogue crawlers never read. The positive inverse of blocking is worth knowing too: instead of only telling AI what it may not touch, you can hand it a curated map of what it should read, which is the point of an llms.txt file on WordPress.

What's the most common way an AI robots.txt backfires?

The most common backfire is a User-agent: * rule the owner assumes covers everything, placed above a specific bot group — because robots.txt groups do not inherit from each other. The most-specific matching user-agent group wins and takes over completely; the * group is ignored for any bot that has its own named group. That single spec detail is behind most of the misfires we see:

• A named User-agent: GPTBot group with Allow: / silently drops every Disallow you wrote under *, because the two groups don't combine — GPTBot now sees only its own group and ignores the rest.
• A broad Disallow: / under *, added to stop "scrapers," is dutifully obeyed by the compliant search crawlers, so you delist yourself from AI answers while the scrapers you were aiming at never even notice.
• Blocking GPTBot to "get out of AI training" while leaving ChatGPT-User allowed — which is often exactly right, but only when you did it on purpose, knowing the two are different bots, rather than by mistake.

The catch is that reading the file tells you what you wrote, not what happens. The only reliable check is a live fetch as each bot's user-agent from outside your network, past your CDN. That test is what Contexta's AI Visibility check runs: it fetches your pages as GPTBot, OAI-SearchBot and PerplexityBot, compares that against your robots.txt and any Cloudflare rule in front of it, and flags where your intent and your actual reachability disagree — the group-inheritance mistakes especially, which no amount of re-reading the file will surface.

Where does robots.txt end and real scraper defense begin?

At the edge — a rate limit, a WAF, or Cloudflare's bot rules — because robots.txt has no enforcement and no throttle. It is allow-or-deny only; there is no "allow, but slowly," which matters because the training crawlers are the actual load problem, not the retrieval ones. Cloudflare reported that as of 2026 training drives roughly 80% of AI bot activity, up from about 72% a year earlier, and that Anthropic's training crawlers made on the order of 38,000 fetches per referral visit they sent back, against roughly 14 for Google. robots.txt can politely ask those bots to stop; only the edge can make them. And the edge now offers a third option beyond stop-or-allow — with Cloudflare's pay per crawl you can make a crawler pay for access, which turns the block/allow line into a pricing decision most stores should think twice about.

There is an emerging middle option: Content Signals, the machine-readable search / ai-input / ai-train categories Cloudflare added to robots.txt, which express purpose rather than naming each bot. It is a cleaner way to say "index me, answer with me, but don't train on me" — but it is still advisory, honored only by crawlers that choose to, and adoption is thin as of mid-2026, so treat it as a signal you send, not a wall you build. When a bot ignores robots.txt, the block that finally stops it usually lives at Cloudflare — which is also why a Cloudflare rule can silently block the AI crawlers you meant to allow without a single line changing in your robots.txt. Two layers, two jobs: use robots.txt to make the training and citation decision for the bots that listen, and use the edge for everyone who doesn't.