How to test whether ChatGPT's bot can actually read your site (and what the curl test can't prove)

Fetching your homepage with OAI-SearchBot's User-Agent string does not prove ChatGPT can read your site — it proves that one unverified request wearing a bot's name got a status code. The real crawler arrives from one of the 35 IPv4 prefixes OpenAI publishes at openai.com/searchbot.json (checked 25 July 2026), and bot-management systems decide what to do with it by checking that list, a reverse DNS lookup or a cryptographic signature, none of which your terminal can produce. The test that actually settles the question is your own access log.

The distance between what a spoofed fetch reports and what the crawler experiences is a problem we own rather than one we can dodge: Contexta ships that fetch test, and the honest thing to say about it — about every tool that ships one, ours included — is that a green result is evidence, not proof. What follows is what the fetch test genuinely catches, the two ways it invents an answer, and the check that ends the argument.

Does fetching your site as OAI-SearchBot prove ChatGPT can read you?

No — it proves that a single request, from your IP address, at one moment, on one URL, received a particular status code. That is useful evidence and it catches the crudest blocks, but it is not the same event as the crawler's request, because the thing modern bot management inspects is not the User-Agent header.

Cloudflare states it plainly: a verified bot declares identity "through a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS" (developers.cloudflare.com, checked July 2026). Your curl satisfies none of those. It carries a well-known crawler name from an address that is not on any crawler's published list — which, to a bot-management rule, is the signature of an impersonator rather than a crawler. The two-minute edge-block check is still worth running; it just answers a narrower question than its output suggests.

How can the fetch test be wrong in both directions?

It can report a block that does not exist and clear a block that does. Both failure modes are common enough that we treat a single fetch result as a hypothesis, not a finding.

A false negative — your test says 403, the real bot gets through:

• Your request comes from a residential or office IP that appears on no published crawler list, so a "known bot name from an unknown address" rule challenges it while the genuine crawler, arriving from a verified prefix, is allowed.
• Some WAF rules and managed rulesets score UA-only bot claims as more suspicious than ordinary browser traffic, which means your test is treated worse than the crawler it imitates.

A false positive — your test says 200, the real bot is blocked:

• Rate-based and behavioural rules fire on crawl patterns, not on one homepage hit. A crawler pulling hundreds of product URLs can trip a limit that your single request never approaches.
• Blocks are frequently path-scoped. A store can serve / happily and challenge /product/ or /?s=, which is exactly where the pages you want cited live.
• Your fetch may be answered from edge cache while the crawler's request, arriving with different headers, goes to origin and meets a rule the cached response never touched.
• The UA string itself drifts. OpenAI currently documents OAI-SearchBot as version 1.4; a rule written against an older literal string, or a test script carrying one, tests something that no longer exists.

Which OpenAI crawlers should you be testing for in 2026?

Four, and most site owners have heard of two. OpenAI documents OAI-SearchBot, ChatGPT-User, GPTBot and OAI-AdsBot, each with its own robots.txt token and its own published IP range file — figures below read from OpenAI's crawler documentation and its range files on 25 July 2026.

The dates carry the practical warning. ChatGPT-User's range was rewritten two days before we checked and holds more than ten times as many prefixes as OAI-SearchBot's, which has not moved since January. Any IP allowlist you paste into a firewall rule is a snapshot that starts rotting immediately; the files have to be re-fetched, not copied once. Which of these four you should keep open is a separate decision from whether they can reach you — the allow-and-block reasoning per crawler is its own call.

How do you prove it from your own server logs?

Search your access logs for the crawler tokens over a 30-day window, then confirm each hit's source IP falls inside the range file for that bot. A logged request from a published prefix is the only evidence that survives scrutiny, because it is the crawler's own request rather than an imitation of it.

grep -Ei 'OAI-SearchBot|ChatGPT-User|GPTBot|OAI-AdsBot' access.log | awk '{print $1, $9}' | sort | uniq -c | sort -rn

Read two columns: the source IP and the status code it received. Hits from a prefix listed in the matching JSON file and returning 200 are proof of reachability. Hits returning 403, 429 or 503 are proof of a block — a far stronger finding than any simulated fetch, because there is no question about whether the requester was genuine.

Running that grep against four bots, cross-checking every address against four range files that change on their own schedule, and repeating it monthly is the part nobody sustains. Contexta's AI Visibility check automates the front half — it fetches as the real documented User-Agents, reports the status code each bot receives, detects Cloudflare and Sucuri challenge pages, and confirms your text is present in the raw HTML without JavaScript — and the same caveat applies to it as to the manual version: it is a fast, honest simulation, and your logs are the verdict.

What does it mean when no OpenAI crawler shows up in your logs at all?

It means one of three things, and they are not equally bad. The crawler may be blocked at your CDN, in which case your origin log holds nothing because the edge answered before the request ever reached your server. The crawler may be permitted but has not visited, which is normal for a small site nobody has asked ChatGPT about. Or your logging may simply exclude bot traffic, which some managed hosts do by default.

Separating the first case from the second is why the CDN's own analytics matter more than the origin log here: Cloudflare, Sucuri and similar products record the requests they refuse, and a wall of blocked events with an OpenAI User-Agent is the finding you are looking for. If you see nothing anywhere and your site is new or low-traffic, absence is not evidence of a block.

One more failure sits behind a clean 200: a crawler that reaches the page and finds nothing on it. OpenAI's crawlers fetch HTML, so a theme or page builder that assembles the text client-side leaves them an empty shell — the JavaScript-rendering trap is a different problem from access, and it looks identical in a status-code test.

Which test should you run first?

Start at the cheapest rung and stop climbing the moment you get a definite answer. Each level costs more effort and buys more certainty, and most sites resolve the question at level two or four.

robots.txt review

States your intent. Proves nothing about what a request receives.

Fetch as the bot

Catches crude blocks in seconds. Wrong in both directions, so treat a result as a hypothesis.

Raw-HTML check

Confirms the text exists without JavaScript. A 200 on an empty shell is still a failure.

CDN event log

Shows requests your edge refused before they reached the origin — where most real blocks live.

Access log + IP verification

The crawler's own request, matched to OpenAI's published prefixes. This is proof.

The reason to bother with the whole ladder rather than the first rung is that access is the precondition for everything else, and it fails silently. A site can publish flawless answer-first content for a year and never be quoted because a managed WAF rule enabled by default in 2025 has been returning 403 to OAI-SearchBot the entire time — one of the more common reasons AI assistants never cite a site, and one of the few that a single log query can confirm or rule out this afternoon.