A daily census · reading 2026-10-12
Every day, 2,110 fixed origins are asked two questions: do you publish anything written for agents, and do you let agents read you at all. Published adoption estimates for these files disagree by two orders of magnitude. This one states its method, lists its hits, and repeats itself forever.
26.15% of the top 1,000 block CCBot
Origins publishing files written for machines rather than people. Measured against a frozen cohort, so a change in the number means a change in adoption.
| Signal | Web head Tranco top 1,000 n=1,000 | Web tail sampled to rank 100k n=1,000 | AI-native model & dev platforms n=110 |
|---|---|---|---|
| Any agent-facing signal | 12.3% (123) | 4.7% (47) | 74.55% (82) |
| llms.txt (apex domain) | 9.9% (99) | 3.9% (39) | 60% (66) |
| llms.txt (docs subdomain) | 4.1% (41) | 1% (10) | 46.36% (51) |
| .well-known/mcp.json | 0.6% (6) | 0.2% (2) | 9.09% (10) |
| .well-known/agents.txt | 0.3% (3) | 0.1% (1) | 0% (0) |
| ai.txt | 0.2% (2) | 0.1% (1) | 0% (0) |
| ai-plugin.json (deprecated) | 0.7% (7) | 0.1% (1) | 0.91% (1) |
Share of origins whose robots.txt disallows each crawler at the site root. Training crawlers and answer-engine fetchers are separated, because refusing to be trained on is a different decision from refusing to be cited.
| Crawler | Operator | Purpose | Web head Tranco top 1,000 | Web tail sampled to rank 100k | AI-native model & dev platforms |
|---|---|---|---|---|---|
| GPTBot | OpenAI | training | 23.55% (118) | 13.31% (41) | 3.13% (3) |
| ClaudeBot | Anthropic | training | 23.35% (117) | 11.36% (35) | 3.13% (3) |
| anthropic-ai | Anthropic | training | 18.76% (94) | 9.42% (29) | 2.08% (2) |
| CCBot | Common Crawl | training | 26.15% (131) | 11.69% (36) | 3.13% (3) |
| Google-Extended | training | 19.96% (100) | 10.39% (32) | 4.17% (4) | |
| Applebot-Extended | Apple | training | 19.96% (100) | 9.42% (29) | 2.08% (2) |
| Bytespider | ByteDance | training | 25.75% (129) | 12.99% (40) | 6.25% (6) |
| meta-externalagent | Meta | training | 21.76% (109) | 10.06% (31) | 3.13% (3) |
| AI2Bot | AI2 | training | 16.77% (84) | 7.79% (24) | 2.08% (2) |
| cohere-ai | Cohere | training | 20.76% (104) | 9.42% (29) | 3.13% (3) |
| Diffbot | Diffbot | training | 21.16% (106) | 9.09% (28) | 3.13% (3) |
| omgili | Webz.io | training | 21.96% (110) | 9.74% (30) | 4.17% (4) |
| ImagesiftBot | ImageSift | training | 19.16% (96) | 9.42% (29) | 3.13% (3) |
| Timpibot | Timpi | training | 18.16% (91) | 9.09% (28) | 3.13% (3) |
| PanguBot | Huawei | training | 16.17% (81) | 7.14% (22) | 2.08% (2) |
| Webzio-Extended | Webz.io | training | 17.37% (87) | 7.79% (24) | 2.08% (2) |
| ChatGPT-User | OpenAI | user-fetch | 15.77% (79) | 7.79% (24) | 2.08% (2) |
| Claude-User | Anthropic | user-fetch | 15.97% (80) | 7.47% (23) | 2.08% (2) |
| Meta-ExternalFetcher | Meta | user-fetch | 14.57% (73) | 7.14% (22) | 2.08% (2) |
| DuckAssistBot | DuckDuckGo | user-fetch | 15.37% (77) | 6.82% (21) | 2.08% (2) |
| Perplexity-User | Perplexity | user-fetch | 16.77% (84) | 8.12% (25) | 2.08% (2) |
| OAI-SearchBot | OpenAI | search | 13.77% (69) | 7.47% (23) | 2.08% (2) |
| PerplexityBot | Perplexity | search | 18.96% (95) | 9.09% (28) | 2.08% (2) |
| Amazonbot | Amazon | search | 18.16% (91) | 11.04% (34) | 3.13% (3) |
| GoogleOther | search | 14.17% (71) | 6.17% (19) | 2.08% (2) | |
| YouBot | You.com | search | 18.56% (93) | 8.77% (27) | 2.08% (2) |
Denominator is origins that served a parseable robots.txt: head 501/1,000 · tail 308/1,000 · ai_native 96/110. A site counts as blocking when the group that applies to that crawler disallows the root, with an exact user-agent match taking precedence over the wildcard group.
Roughly half the ranked strata return no robots.txt, and that is expected rather than
a gap: Tranco ranks by DNS resolution volume, so its head contains a great many CDN and infrastructure
hostnames like akamaized.net, googleapis.com,
apple-dns.net and gtld-servers.net, which are not
browsable websites and correctly serve nothing at the apex. Rates are computed over origins that
answered, never over the whole stratum, so infrastructure hostnames cannot dilute the result.
Who you sample decides the answer.
The head of the web, its long tail, and the AI-native platforms are three different worlds. A survey of one, reported as a fact about the web, is how you get figures an order of magnitude apart. This census keeps the strata separate and never blends them into one headline.
These files usually live on the docs subdomain.
Probing apex domains alone undercounts adoption. On this reading anthropic.com, perplexity.ai, fireworks.ai, deepinfra.com, anyscale.com and 11 others publish an llms.txt that an apex-only survey scores as absent. This census probes both, and counts a 200 only when the body validates as the file it claims to be; an HTML page served with status 200 is not an llms.txt.
Repeated daily against a frozen cohort, which is what turns single readings into a trend.
| Date | Panel | head | tail | ai_native | Top-1k blocking GPTBot |
|---|---|---|---|---|---|
| 2026-10-12 | v2 | 12.3% | 4.7% | 74.55% | 23.55% |
| 2026-10-11 | v2 | 12.3% | 5.3% | 73.64% | 23.49% |
| 2026-10-10 | v2 | 12.2% | 5.2% | 74.55% | 23.54% |
| 2026-10-09 | v2 | 13% | 5.1% | 74.55% | 23.4% |
| 2026-10-08 | v2 | 13.1% | 5.9% | 73.64% | 23.67% |
| 2026-10-07 | v2 | 12.9% | 5.1% | 72.73% | 23.31% |
| 2026-10-06 | v2 | 12.5% | 6.4% | 74.55% | 23.66% |
| 2026-10-05 | v2 | 12.7% | 6% | 74.55% | 23.66% |
| 2026-10-04 | v2 | 12.7% | 5.1% | 74.55% | 23.66% |
| 2026-10-03 | v2 | 11.3% | 4% | 74.55% | 24.28% |
| 2026-10-02 | v2 | 11.8% | 5.3% | 74.55% | 24.03% |
| 2026-10-01 | v2 | 12.8% | 5.9% | 74.55% | 23.95% |
| 2026-09-30 | v2 | 11.8% | 4% | 74.55% | 23.91% |
| 2026-09-29 | v2 | 12.1% | 5.2% | 74.55% | 23.58% |
| 2026-09-28 | v2 | 11.4% | 4.9% | 74.55% | 24.09% |
| 2026-09-27 | v2 | 11.9% | 5.9% | 74.55% | 23.8% |
| 2026-09-26 | v2 | 11.9% | 4.2% | 74.55% | 24.19% |
| 2026-09-25 | v2 | 12% | 4.4% | 74.55% | 24.03% |
| 2026-09-24 | v2 | 12.2% | 5.4% | 74.55% | 23.74% |
| 2026-09-23 | v2 | 11.6% | 7.3% | 74.55% | 23.16% |
| 2026-09-22 | v2 | 11.8% | 4.1% | 72.73% | 24.03% |
| 2026-09-21 | v2 | 11.5% | 5.7% | 74.55% | 23.2% |
| 2026-09-20 | v2 | 11.6% | 4.5% | 73.64% | 23.52% |
| 2026-09-19 | v2 | 12.1% | 5.8% | 74.55% | 23.55% |
| 2026-09-18 | v2 | 12.1% | 7.1% | 73.64% | 23.61% |
| 2026-09-17 | v2 | 12.2% | 5.6% | 73.64% | 23.48% |
| 2026-09-16 | v2 | 11.6% | 5.4% | 75.45% | 23.17% |
| 2026-09-15 | v2 | 12.1% | 6.6% | 75.45% | 23.6% |
| 2026-09-14 | v2 | 12% | 6% | 75.45% | 24.06% |
| 2026-09-13 | v2 | 11.9% | 4.9% | 75.45% | 25.26% |
| 2026-09-12 | v2 | 11.5% | 4.7% | 75.45% | 23.99% |
| 2026-09-11 | v2 | 11.5% | 6.6% | 75.45% | 24.16% |
| 2026-09-10 | v2 | 11.3% | 3.9% | 74.55% | 24.39% |
| 2026-09-09 | v2 | 10.9% | 4.1% | 75.45% | 24.07% |
| 2026-09-08 | v2 | 11.8% | 4.6% | 75.45% | 24.33% |
| 2026-09-07 | v2 | 9.5% | 3.7% | 75.45% | 23.83% |
| 2026-09-06 | v2 | 10.5% | 2.6% | 75.45% | 24.2% |
| 2026-09-05 | v2 | 12.1% | 3.9% | 74.55% | 24.16% |
| 2026-09-04 | v2 | 11.7% | 4.3% | 75.45% | 24.79% |
| 2026-09-03 | v2 | 11.4% | 3.2% | 75.45% | 24.6% |
| 2026-09-02 | v2 | 11.6% | 3.3% | 74.55% | 24.36% |
| 2026-09-01 | v2 | 10.9% | 2.9% | 74.55% | 24.59% |
| 2026-08-31 | v2 | 11.2% | 4.4% | 75.45% | 24.3% |
| 2026-08-30 | v2 | 10.8% | 5.4% | 75.45% | 24.03% |
| 2026-08-29 | v2 | 9.4% | 3.6% | 73.64% | 23.99% |
| 2026-08-28 | v2 | 10% | 5.5% | 75.45% | 24.06% |
| 2026-08-27 | v2 | 11.6% | 5.1% | 72.73% | 23.73% |
| 2026-08-25 | v2 | 10.8% | 5.8% | 74.55% | 23.85% |
The panel is versioned. When the cohort changes, the old readings are kept and labeled rather than rewritten; a stratum that did not exist in an earlier panel is shown as absent, never back-filled with a guess.
Several larger projects already measure parts of this, and pretending otherwise would be the fastest way to lose a reader who knows the field.
| Who | What they publish | Scale |
|---|---|---|
| Cloudflare Radar | Aggregate AI-crawler robots.txt policy and agent-readiness signals, weekly, free | ~200k domains |
| HTTP Archive | Per-origin robots.txt, monthly since Feb 2022, free on BigQuery | ~12.9M origins |
| Common Crawl | Raw robots.txt captures in WARC archives since 2016 | web scale |
| MIT Data Provenance Initiative | Reconstructed robots.txt history, 2016 to 2024 | ~14k domains |
So this is not an unrepeatable measurement, and it is not presented as one. Four things here are genuinely not published elsewhere:
The decision, not the directive count.
The largest public dataset stores how many Allow and
Disallow lines each user agent has, and discards the paths. That makes
Disallow: / indistinguishable from Disallow: /admin/,
so it cannot tell you whether a crawler is actually blocked. This resolves group specificity and
Allow-overrides and reports a decision per origin and crawler.
Crawlers labeled by purpose.
Blocking a training crawler and blocking a fetch made on a person's behalf are different decisions with different consequences. Separating them is what surfaces the gap between the two rates, and that axis is precisely what the IETF working group on AI preferences is currently arguing about.
Paths nothing links to.
Archival crawlers find pages by following links. Nothing links to
/.well-known/mcp.json, or to an llms.txt sitting on a
docs subdomain, so those were never captured and cannot be recovered. This is the one part of the reading
that really is only available going forward.
Who wrote the policy.
Cloudflare ships a managed robots.txt to millions of zones
whose template permits search and denies AI training. If most blocking origins sat behind it, then
“the web blocks training harder” would be measuring one vendor's default rather than what
publishers chose. So every origin's CDN is recorded from headers already received, and the share is
published below rather than assumed away.
Of the 164 origins in the top 1,000 that block at least one AI crawler, this is what served their robots.txt. Read it as a check on the headline: the more concentrated this is behind one vendor, the less the blocking reflects individual publisher intent.
| Served by | Blocking origins | Share |
|---|---|---|
| other | 43 | 26.2% |
| cloudflare | 36 | 22% |
| cloudfront | 36 | 22% |
| unknown | 27 | 16.5% |
| fastly | 19 | 11.6% |
| azure | 1 | 0.6% |
| vercel | 1 | 0.6% |
| akamai | 1 | 0.6% |
Panel v2 · 14,770 probes · 4,178 errors · 2,983 hosts that do not exist.
Fixed stratified cohort. Each domain is probed for six agent-facing files, on both the apex and the docs subdomain where applicable, plus robots.txt. A 200 counts only when the body validates as the declared type; an HTML page returned with status 200 is not an llms.txt. A subdomain that does not resolve is recorded as absent, not as an error. robots.txt is parsed with exact user-agent precedence over the wildcard group; a group counts as blocking when it disallows the root.
curl https://28081999.me/api/census/latest.json curl https://28081999.me/api/census/series.json curl https://28081999.me/api/census/2026-10-12.json