Four of the 54 sites in our August 2026 crawler audit refused an AI crawler while serving a browser normally. Every one of them publishes a robots.txt saying crawlers are welcome. We fingerprinted the edge in front of all 54 to find out whether this is the Cloudflare story everyone assumes it is. It is not.
31 of the 54 sites sit behind Cloudflare — well over half the sample — and exactly 1 of them blocked an AI crawler. The other three blocks were on plain nginx twice and LiteSpeed once, origin servers with no CDN detected in front of them at all. Edge blocking is a configuration story, not a vendor story.
The timing matters as well. On 15 September 2026, Cloudflare changes its default handling of AI crawler traffic by category: Training and Agent blocked by default on pages that display ads, Search still allowed. If you have been meaning to form an opinion about which AI crawlers reach your site, the deadline for having one is now on a calendar.
We re-fetched all 54 domains from the audit and fingerprinted the edge in front of each: a cf-ray header for Cloudflare, x-vercel-id for Vercel, x-amz-cf-id for CloudFront, x-fastly-request-id for Fastly, x-nf-request-id for Netlify, and the Server header for everything else. We expected the blocks to cluster on one provider; they did not.
Distribution across the corpus: Cloudflare 31, Vercel 9, nginx (no CDN detected) 5, LiteSpeed 1, CloudFront 1, Fastly 1, Netlify 1, Apache 1, openresty 1, Other or unidentified 3.
Thirty of the thirty-one Cloudflare sites served AI crawlers normally. Three of the four blocks had no CDN in front of them at all. You do not need a CDN to lock out an AI crawler; you need a config file and a reason. The rule gets switched on by whoever last worried about scraping or bandwidth, and the provider logo above the console has very little to do with it.
None of these is a robots.txt rule. All four sites tell crawlers in writing that they are welcome, then answer their requests with a refusal.
Edge: Cloudflare
An invented user agent was served the full 4,623-word page from the same address, seconds later. The rule matches on the crawler's name, not on whether the request looks automated.
Edge: nginx, no CDN detected
PerplexityBot, Googlebot, a browser and our invented agent all got 200. A rate limit that only ever fires for two specific names is not a rate limit — and there is no CDN here to blame it on.
Edge: nginx, no CDN detected
No status code at all, so nothing appears in a monitoring tool that only records responses. Our invented agent got a 403, which means there are at least two rules here, not one.
Edge: LiteSpeed
Intermittent, so it passes any test run once. Read as genuine throttling rather than a deliberate block — but a crawler on a budget still leaves with nothing.
It is not in the repository. A bot rule lives in a vendor console. It does not appear in a pull request, a deploy log or a site audit, so no code review will ever surface it and no SEO tool will read it.
It was switched on for a good reason. Someone was watching bandwidth climb, or worried about scraping, and there was a toggle labelled something reasonable. That decision was defensible at the time and was never revisited against a marketing goal that did not exist yet.
robots.txt says everything is fine. All four sites we found refusing crawlers publish a robots.txt that welcomes them. The file and the firewall disagree, and only one of them is answering requests.
A browser cannot see it. Every one of these sites looks perfect in Chrome, because the rule keys on the crawler's name. You cannot find this by visiting your own website, which is exactly why it survives.
Until now, an edge rule blocking AI crawlers was something a person switched on. Cloudflare is about to make a version of it the starting position, and it has split crawlers into exactly the three categories that matter.
Search — Cloudflare defines this as "any behavior that collects or indexes your content, so it can answer questions about it later". After the change: Allowed. Stays allowed by default. This is the category that decides whether an assistant can cite you, so the default is the right way round.
Agent — Cloudflare defines this as "automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now". After the change: Blocked on ad pages. This is the category that fires when a customer pastes your URL into an assistant and asks about it. On a page carrying ads, the new default turns that away.
Training — Cloudflare defines this as "a crawler taking your content to train or fine-tune a model". After the change: Blocked on ad pages. The category most people mean when they say they want to block AI. Blocking it has the least effect on whether you appear in answers.
Cloudflare sets new defaults for its three crawler categories. Training and Agent are blocked by default on pages that display ads. Search remains allowed by default.
Cloudflare's own post says the automatic defaults apply to all new domains onboarding to Cloudflare. Several press write-ups described a wider scope that also swept in existing free-tier customers.
We link Cloudflare's post below and quote only what it says. Given that reputable coverage disagrees about who is affected, the sensible move is not to settle the argument from the outside but to open your own dashboard and look.
Cloudflare says owners can record a different preference in their security settings before the date, and that it will notify customers beforehand.
Note which category stays allowed. Search remains on by default, and that is the right way round: it is the category that decides whether an assistant can cite you at all. It also confirms, from the infrastructure side, the split we make on editorial grounds at https://altheo.ai/ai-crawlers — training, search and acting on a person's behalf are three different things with three different costs.
Two steps, in this order. Request your own homepage with several user agents — a browser, GPTBot, ClaudeBot, PerplexityBot, and one you invent as a control — and compare the status codes. A browser getting 200 while the named crawlers get 403 or 429 is an edge rule. If the control agent is served while the real crawlers are refused, the rule matches on the name rather than on anything about the request.
Use each client's full user agent string rather than building them from a template. A bare "(compatible; …)" user agent is itself a bot signature, and against managed bot rules it returns 403 for every agent including the browser, which makes a site that serves everyone look like a site that blocks everyone.
Expect this test to get noisier the more you run it. Re-running our own examples later the same day, two things drifted. neilpatel.com/ubersuggest/ returned 403 to a browser as well — while neilpatel.com itself and our control sites still answered normally from the same machine, which points at our own address being rate-limited on that path after a day of repeated testing rather than a change at the site. And marketmuse.com, which returned 429 to GPTBot on three consecutive trials in the morning, served it a clean 200 in the afternoon while still refusing ClaudeBot. Neither result contradicts the measurements above; both are what stateful rate limiting looks like from outside. But they are a warning about method. Run the loop more than once, space the requests, test from a network you have not been hammering, and treat an external probe as a smoke alarm rather than a verdict. The only authoritative answer to which crawlers your site is refusing is in your own edge dashboard, which is the second half of this page.
Then find it in the console. On Cloudflare: Log in to the Cloudflare dashboard and select your account and domain. Open AI Crawl Control, then the Security tab. The Crawlers tab lists every AI crawler requesting your content, with its operator, category, request counts and any robots.txt violations. Each row has an Actions column with Allow and Block. On paid plans, Settings → Block response → Edit controls the status code and message a blocked crawler receives.
On other providers the path differs but the shape does not: look for bot management, managed rules, or a firewall category named for AI or scrapers. The rule will not be in your repository, which is why no code review or site audit has ever surfaced it.
Finally, make the two layers agree. robots.txt states a policy that well-behaved crawlers honour; the edge enforces it against everything. Wanting both is reasonable. What is not reasonable is what we measured on all four sites — a robots.txt inviting crawlers in and a rule at the edge turning them away. Whichever you decide, write it in both places so your site says what it does.
Each of the 54 domains from the crawler audit was re-fetched on 8 August 2026 with a browser user agent and its response headers fingerprinted. Header fingerprinting identifies what answered the request, which is not always the whole stack — a site can sit behind a CDN that strips its own headers, so "nginx with no CDN detected" means exactly that rather than proof none exists.
The block classifications come from the audit's three-trial user-agent tests, not from this pass. Fifty-four sites in one industry is a sample, not a census, and the shares should be read as the shape of a market rather than a measurement of the web.
Cloudflare's category definitions, the 15 September 2026 change and the dashboard steps are quoted from Cloudflare's own blog post at https://blog.cloudflare.com/content-independence-day-ai-options/ and its documentation at https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/, both read on 8 August 2026. Where press coverage described a wider scope than Cloudflare's own post does, this page says so instead of choosing a side.
For new domains, from 15 September 2026, partly. Cloudflare's own announcement says that from that date its Training and Agent crawler categories are blocked by default on pages that display ads, while the Search category stays allowed, and that these automatic defaults apply to new domains onboarding to Cloudflare. Some press coverage described a broader scope including existing free-tier customers. Rather than settle that from the outside, open AI Crawl Control in your dashboard and read your own settings. Our own measurement is a useful counterweight to the panic: of 31 sites behind Cloudflare in our August 2026 audit, 30 served AI crawlers normally.
It is possible, and you cannot tell from a browser. Request your own page with a crawler's user agent and compare the status code to the one a browser gets: a 403 or 429 that only appears for named AI crawlers is an edge rule, not a coincidence. In our audit of 54 marketing sites, four refused or throttled an AI crawler while serving a browser normally, and all four publish a robots.txt that says AI crawlers are welcome. The rule lives in a vendor console, not the codebase, which is why nothing in a normal site audit finds it.
Our data says no, and we checked specifically because that is the assumption. We fingerprinted the edge in front of all 54 sites in our August 2026 audit: 31 sat behind Cloudflare, and exactly one of them blocked an AI crawler. The other three blocks were on plain nginx twice and LiteSpeed once — origin servers with no CDN detected in front of them at all. Blocking is a configuration decision available on every stack, and it is made by whoever last worried about scraping. The platform with the most sites in the sample accounted for the fewest blocks per site.
Cloudflare defines Search as "any behavior that collects or indexes your content, so it can answer questions about it later", Agent as "automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now", and Training as "a crawler taking your content to train or fine-tune a model". The distinction matters because the consequences differ enormously: blocking Training keeps your writing out of future models, blocking Agent turns away a person who pasted your URL into an assistant, and blocking Search removes you from AI answers altogether. From 15 September 2026 the new defaults block Training and Agent on ad-bearing pages while leaving Search allowed.
Log in to the Cloudflare dashboard, select your account and domain, open AI Crawl Control and go to the Security tab. The Crawlers tab lists every AI crawler requesting your content along with its operator, category, request counts and any robots.txt violations, and each row has an Allow or Block action. On paid plans you can also set the status code and message that blocked crawlers receive under Settings → Block response. Do this before changing anything else: the report tells you what is actually reaching your site, which is usually more surprising than the settings.
They do different jobs and the honest answer is that you may want both. robots.txt is a request that well-behaved crawlers honour and everything else ignores; a CDN rule is enforcement that applies regardless of intent. If your goal is to state a policy to legitimate operators, robots.txt is the right instrument and all the major AI crawlers say they follow it. If your goal is to stop traffic that is costing you money or ignoring your stated rules, the edge is where that happens. What causes the damage we measured is neither of those: a rule at the edge that contradicts the robots.txt in front of it, so the site says one thing and does another.
Cloudflare changes the default handling of AI crawler traffic by category. Training and Agent crawlers are blocked by default on pages that display ads; Search crawlers remain allowed. Cloudflare states the automatic defaults apply to new domains onboarding to the platform, that owners can record a different preference in their security settings before the date, and that it will notify customers beforehand. The broader significance is not the specific setting but the direction: crawler policy is becoming an infrastructure default rather than something each site writes for itself, and defaults are exactly the kind of decision that gets made for you.