In August 2026 we requested 54 marketing, SEO and AI-visibility websites the way GPTBot does — one HTTP request, no JavaScript — and counted what came back. Not one of them blocks AI crawlers in robots.txt. Nine of them gave a crawler nothing usable anyway. One of the nine is ours.
The headline numbers: zero of 51 robots.txt files disallowed GPTBot, ClaudeBot, PerplexityBot, Google-Extended or CCBot from the site root. Nine of 54 sites still failed — four blocked or throttled a crawler by name at the CDN, five served a page with almost no readable HTML. The widest gap was 4,623 words to a browser against nothing at all to a bot. Ten of the 50 sites that answered carried no structured data of any kind.
Every figure below came from two scripts we ran ourselves on 7–8 August 2026, with the method and its limits stated in full, because an article arguing that unverified claims about crawler behaviour are the problem cannot rest on unverified claims.
Three years of advice about AI crawlers has been advice about a text file: check robots.txt, allow GPTBot, done. That advice describes intent. It does not describe what happens when a crawler knocks, because a CDN and a bot-management layer answer first, and they are configured in an infrastructure console that no content or SEO review ever opens.
All nine failures in this audit sit on sites whose robots.txt says the crawler is welcome. An audit that reads robots.txt and stops gives every one of them a pass.
The one content restriction we did find aimed at an AI crawler is precise and revealing: moz.com groups GPTBot with rogerbot and AhrefsBot, then disallows that group from /blog/ and /learn/seo/ — the two libraries Moz is known for. GPTBot has been filed with the competitor scrapers rather than with the audience. A site-root check reports it as allowed.
These four answered a browser normally and refused a request that differed only in what it called itself. To tell a deliberate exclusion apart from bot protection distrusting an unverified request, we sent a sixth user agent we invented for the test — FooBot/1.0, belonging to nobody — from the same machine, seconds apart. On neilpatel.com and marketmuse.com, FooBot was served the complete page while the named AI crawlers were turned away. Those rules match on the name: an unknown scraper has an easier time reading those sites than ChatGPT does.
3 of 3 trials · Cloudflare · refused GPTBot, ClaudeBot, PerplexityBot, Googlebot · served Chrome, FooBot (invented) · 4,623 words to a browser
The clearest result in the audit. A browser gets 4,623 words. Every user agent naming a known crawler gets 403 — including Googlebot. A user agent we made up on the spot, "FooBot/1.0", from the same machine and the same address, gets the full 4,623 words.
A block keyed on the crawler's name, not on whether the request looks like a robot. Anything unrecognised walks straight in.
3 of 3 trials · nginx · refused GPTBot, ClaudeBot · served Chrome, Googlebot, PerplexityBot, FooBot (invented) · 958 words to a browser
Two AI crawlers get 429 on every attempt. PerplexityBot, Googlebot, a browser and our invented user agent all get 200 and 958 words. A rate limit that only ever fires for two specific names is not a rate limit.
A content-optimisation platform, invisible to two of the three assistants its customers are trying to appear in.
3 of 3 trials · nginx · refused GPTBot, ClaudeBot · served Chrome, Googlebot, PerplexityBot · 3,824 words to a browser
GPTBot and ClaudeBot do not get a status code at all — the connection is closed before a response arrives. Our invented user agent got a 403. A browser, Googlebot and PerplexityBot each got 3,824 words.
The harshest failure mode to diagnose: no response means no log line, no status, and nothing for a monitoring tool to report.
2 of 3 trials · LiteSpeed · refused GPTBot (intermittent) · served Chrome, ClaudeBot, PerplexityBot, Googlebot, FooBot (invented) · 701 words to a browser
The first GPTBot request succeeded and the next two returned 429. Every other user agent was served throughout. Read this as genuine throttling rather than a deliberate block — but a crawler on a budget still leaves with nothing.
Intermittent is the worst kind to catch. It passes any test you happen to run once.
These returned a clean 200 to every crawler and had almost nothing in the HTML to read. The page exists — it is assembled in the browser, after the crawler has already gone.
byword.ai — 0 visible words in raw HTML (Cloudflare). Zero words of body text to every user agent we tried, a browser included. The page is assembled entirely in JavaScript, so the HTML that leaves the server is an empty shell.
altheo.ai — 0 visible words in raw HTML (Cloudflare). Ours. Our homepage serves head tags and structured data to a crawler and no body prose at all — a deliberate trade we had documented and then stopped questioning. It is still a zero, and it sits in this table for the same reason everyone else does.
evertune.ai — 67 visible words in raw HTML (Cloudflare). Sixty-seven words is a navigation bar and a cookie notice. An assistant asked what this company does has nothing to work from but the domain name.
goodie.ai — 94 visible words in raw HTML (openresty). Ninety-four words in the raw HTML against a full marketing site once JavaScript runs. The gap between the two is the entire pitch.
tareno.co — 322 visible words in raw HTML (Vercel). The most interesting near-miss in the sample. Its robots.txt is the most AI-aware we found — a hand-built allow-list naming GPTBot, ClaudeBot, PerplexityBot and CCBot, pointing each at markdown copies of the site. The homepage those crawlers land on still carries 322 words.
For scale: the 45 sites that did serve real HTML carried a median of 1,632 visible words, from 637 at the low end to 4,001 at the high. Nothing in that range is a target to aim for, but the distance between 1,632 and 67 is not a content-quality problem that better writing fixes. It is a delivery problem, and no amount of writing reaches a crawler that never receives it.
8 of the 25 URLs in our own sitemap served GPTBot 1,203 bytes of empty shell under the generic homepage title: /what-is-eeat, /instagram-script-generator, /tiktok-script-generator, /x-thread-generator, /reviews/conversionai, /reviews/amandaai, /reviews/scripai, /reviews/tareno.
Our edge Worker rewrites each page for crawlers that do not run JavaScript. The content for all 25 URLs existed and was correct. But the Worker only runs on paths listed in one deploy config, and eight routes had been added to the site without being added to that list. Every one of them served GPTBot 1,203 bytes of empty shell under the generic homepage title.
The content was written, reviewed and deployed. It was one list in one config file away from ever being read. Nothing in a page-level SEO audit would have caught it, because the pages themselves were perfect.
All eight are fixed, in the same change that published this article. Our homepage remains a deliberate zero — it serves head tags and structured data to crawlers but no body prose, because injected text flashes on screen for a moment before the app mounts and the homepage is where that costs most. That is a real trade-off with a real price, and it stays in the table as a zero.
Request your page with the crawler's user agent and count the words in what comes back, with no JavaScript executed. One curl command does it: fetch the URL with GPTBot's user agent string, strip the script and style blocks and the tags, and count what is left.
A 403 or 429 means you are blocked at the edge — the rule is in your CDN or WAF console, not your repository, usually a bot-management category named something like "AI crawlers" or "scrapers". A 200 with under a few hundred words means your page is built in the browser and the crawler left with your navigation bar. A 200 with a few thousand words means delivery is not your problem, and the content itself now has to be worth citing.
Run it against your homepage, your pricing page and your two best articles. Those four URLs are what an assistant reads when someone asks about your category, and they are the four most likely to be handled by different parts of your stack.
The good news in this dataset is larger than the bad. Forty-five of the 54 sites served a real page to an AI crawler with no JavaScript at all. The "everything is an empty React shell" version of this story is out of date, and every site in the sample that served under 400 words was either an AI-visibility startup or us.
Twenty-five of the 54 publish an llms.txt, which is real adoption for a convention this young. Size discipline is another matter: against a median of roughly 11 KB we found files of 2.0 MB, 324 KB and 102 KB, which are site dumps rather than summaries and defeat the point of a format whose value is being short enough to read whole.
The most forward-looking practice in the sample is the markdown twin. Tareno allow-lists /*.md for six AI crawlers and genuinely serves the files; Rankscale allow-lists /llms.txt and /llms-full.txt in one group naming seventeen AI crawlers, and its 13 KB llms-full.txt is live. A clean, parseable copy for machines next to the styled one for people. Do check the files exist, though — Seobility points four AI crawlers at /docs/ai/, and that URL returns 403.
This audit measures whether an AI crawler can read a site at all. The question of which crawlers there are, and which ones you should actually let in, is covered in the companion reference at https://altheo.ai/ai-crawlers — all 15 AI crawlers and robots.txt tokens with their exact user agents, robots.txt behaviour and live IP ranges, taken from operator documentation.
The distinction that matters most: GPTBot trains, OAI-SearchBot answers. Blocking GPTBot keeps your writing out of the next model while leaving you visible in ChatGPT today; blocking OAI-SearchBot is what removes you from the answers. Anthropic and Perplexity split the same way. A blanket rule against "AI bots" catches both halves.
The four edge blocks above get their own treatment at https://altheo.ai/cdn-blocking-ai-crawlers, which fingerprints the CDN in front of every site in this corpus. The short version, because it contradicts the common assumption: 31 of the 54 sites sit behind Cloudflare and only one of them blocked, while three of the four blocks had no CDN in front of them at all. That article also covers Cloudflare's 15 September 2026 change to its default crawler handling.
Fifty-seven domains, each requested at the site root with five user agents — Chrome, GPTBot, ClaudeBot, PerplexityBot and Googlebot — plus an invented sixth as a control. Three consecutive trials per user agent on the anomalies. Redirects were followed by hand so each hop could be attributed to the right host; ubersuggest.com redirects to neilpatel.com, and the 403 is recorded against the latter. robots.txt was fetched separately and parsed by group, so a rule naming a specific crawler takes precedence over the wildcard group, as the standard requires. Word counts are visible text in the raw response after removing script, style and comment blocks. No JavaScript was executed anywhere.
Every request came from one residential IP address in one location. OpenAI, Anthropic and Google publish the ranges their crawlers use, and Cloudflare and Akamai can verify a bot by address and reverse DNS, so a site that verifies could legitimately serve the real GPTBot while refusing an unverified request carrying its name. What this audit measures is what happens to anything claiming to be GPTBot from an address the site has not vetted. The FooBot control limits how far that caveat stretches: on two sites, an invented user agent from the same unverified address was served in full while the named crawlers were refused.
Two of the 57 domains did not resolve for any user agent and one turned out to be a parked domain for sale, which is why 54 sites are reported. One page per site, on 7–8 August 2026. Rate limits are stateful and edge rules change, so amandaai.com is reported as throttling rather than a block and any of these results can differ next week. Every site is named with its measurement, its trial count and the user agents that were and were not served, so the test can be repeated and contradicted.
GPTBot, ClaudeBot, PerplexityBot and CCBot fetch a URL and read the HTML that comes back. They do not run a browser, so anything your page assembles in JavaScript after load is not part of what they retrieve. Googlebot does render JavaScript, which is why a site can rank perfectly well in Google search and still be a blank page to an assistant. That difference is the single most common reason a page that "works fine in search" is missing from AI answers.
Request your page with the crawler's user agent and count the words in what comes back, without running any JavaScript. One curl command does it: fetch the URL with GPTBot's user agent string, strip the script and style blocks and the tags, and count what is left. Under a few hundred words on a page that looks full in a browser means the content is being assembled client-side and the crawler is not seeing it. A 403 or 429 means you are being blocked at the edge. Both are invisible in a normal browser and in most SEO tools.
No, and that is the main finding of this audit. Across 54 marketing sites in August 2026, zero of the 51 published robots.txt files disallowed GPTBot, ClaudeBot, PerplexityBot, Google-Extended or CCBot from the site root — yet nine of those sites still gave an AI crawler nothing usable. Four blocked or throttled the crawler by name at the CDN or WAF layer, which robots.txt knows nothing about, and five served it a page with almost no readable HTML. Reading robots.txt tells you what a site intends. Only fetching the page tells you what it does.
Usually because nobody decided to. Bot-management rules on Cloudflare, Akamai and similar services ship with categories like "AI crawlers" or "scrapers" that can be switched on by an engineer worried about bandwidth or content scraping, months before anyone in marketing has an opinion about AI visibility. The rule lives in an infrastructure console, not in the repository, so it never appears in a content or SEO review. In this audit one site refused GPTBot, ClaudeBot, PerplexityBot and Googlebot alike while serving a user agent we invented on the spot — a signature of a default blocklist rather than a considered decision.
There is no threshold that makes you citable, but there is one below which nothing else you do matters. Across the 45 sites in this audit that served real HTML, the median homepage carried 1,632 visible words in raw HTML, ranging from 637 to 4,001. The sites that failed carried 0, 67, 94 and 322. The useful reading is not "aim for 1,600 words" — it is that anything under a few hundred means your page is being built in the browser and the crawler is leaving with your navigation bar.
It is cheap and about half the market now does it — 25 of the 54 sites in this audit served one. Treat it as a summary, not an archive: against a median size of roughly 11 KB, we found files of 2.0 MB, 324 KB and 102 KB, which are site dumps rather than descriptions and defeat the purpose of a format whose value is being short enough to read whole. What it will not do is rescue a site an AI crawler cannot reach — an llms.txt on a domain that returns 403 to GPTBot is a note pinned to a locked door.
That is a real decision with a real trade-off, and this audit does not argue against making it. A publisher whose business is selling access to writing has a defensible reason to refuse crawlers that train on it. What it argues against is making that decision by accident, in an infrastructure console, on a site whose whole purpose is to be found — which is what nine of these 54 sites appear to have done. If you have chosen to block, block clearly in robots.txt so the intent is on the record. If you have not chosen, check, because the default may have chosen for you.
No. Our homepage served zero words of body text to every crawler we tested, and eight of the 25 URLs in our sitemap — including a 14-minute guide and four product reviews — returned 1,203 bytes of empty shell under the wrong page title. The cause was one list in one deploy config that eight newer routes had never been added to; the content itself existed and was correct the whole time. We found it by running the script in this article against our own domain, and we have published the result in the same table as everyone else's because a benchmark you exempt yourself from is marketing rather than evidence.