A daily census of the agentic web · reading 2026-10-12
Every morning, 2,110 fixed origins are asked two questions: do you publish anything written for an AI to read, and do you let AI read you at all. The answers are resolved to a decision per crawler, recorded with who served the policy, and frozen.
The finding
That gap is a choice, and it is the axis the IETF working group on AI preferences is currently arguing about. See the full reading, the method, and the prior work this builds on.
Origins publishing files written for machines rather than people. Measured against a frozen cohort, so a change in the number means a change in adoption.
| Signal | Web head Tranco top 1,000 n=1,000 | Web tail sampled to rank 100k n=1,000 | AI-native model & dev platforms n=110 |
|---|---|---|---|
| Any agent-facing signal | 12.3% (123) | 4.7% (47) | 74.55% (82) |
| llms.txt (apex domain) | 9.9% (99) | 3.9% (39) | 60% (66) |
| llms.txt (docs subdomain) | 4.1% (41) | 1% (10) | 46.36% (51) |
| .well-known/mcp.json | 0.6% (6) | 0.2% (2) | 9.09% (10) |
| .well-known/agents.txt | 0.3% (3) | 0.1% (1) | 0% (0) |
| ai.txt | 0.2% (2) | 0.1% (1) | 0% (0) |
| ai-plugin.json (deprecated) | 0.7% (7) | 0.1% (1) | 0.91% (1) |
Share of origins whose robots.txt disallows each crawler at the site root. Training crawlers and answer-engine fetchers are separated, because refusing to be trained on is a different decision from refusing to be cited.
| Crawler | Operator | Purpose | Web head Tranco top 1,000 | Web tail sampled to rank 100k | AI-native model & dev platforms |
|---|---|---|---|---|---|
| GPTBot | OpenAI | training | 23.55% (118) | 13.31% (41) | 3.13% (3) |
| ClaudeBot | Anthropic | training | 23.35% (117) | 11.36% (35) | 3.13% (3) |
| anthropic-ai | Anthropic | training | 18.76% (94) | 9.42% (29) | 2.08% (2) |
| CCBot | Common Crawl | training | 26.15% (131) | 11.69% (36) | 3.13% (3) |
| Google-Extended | training | 19.96% (100) | 10.39% (32) | 4.17% (4) | |
| Applebot-Extended | Apple | training | 19.96% (100) | 9.42% (29) | 2.08% (2) |
| Bytespider | ByteDance | training | 25.75% (129) | 12.99% (40) | 6.25% (6) |
| meta-externalagent | Meta | training | 21.76% (109) | 10.06% (31) | 3.13% (3) |
| AI2Bot | AI2 | training | 16.77% (84) | 7.79% (24) | 2.08% (2) |
| cohere-ai | Cohere | training | 20.76% (104) | 9.42% (29) | 3.13% (3) |
| Diffbot | Diffbot | training | 21.16% (106) | 9.09% (28) | 3.13% (3) |
| omgili | Webz.io | training | 21.96% (110) | 9.74% (30) | 4.17% (4) |
| ImagesiftBot | ImageSift | training | 19.16% (96) | 9.42% (29) | 3.13% (3) |
| Timpibot | Timpi | training | 18.16% (91) | 9.09% (28) | 3.13% (3) |
| PanguBot | Huawei | training | 16.17% (81) | 7.14% (22) | 2.08% (2) |
| Webzio-Extended | Webz.io | training | 17.37% (87) | 7.79% (24) | 2.08% (2) |
| ChatGPT-User | OpenAI | user-fetch | 15.77% (79) | 7.79% (24) | 2.08% (2) |
| Claude-User | Anthropic | user-fetch | 15.97% (80) | 7.47% (23) | 2.08% (2) |
| Meta-ExternalFetcher | Meta | user-fetch | 14.57% (73) | 7.14% (22) | 2.08% (2) |
| DuckAssistBot | DuckDuckGo | user-fetch | 15.37% (77) | 6.82% (21) | 2.08% (2) |
| Perplexity-User | Perplexity | user-fetch | 16.77% (84) | 8.12% (25) | 2.08% (2) |
| OAI-SearchBot | OpenAI | search | 13.77% (69) | 7.47% (23) | 2.08% (2) |
| PerplexityBot | Perplexity | search | 18.96% (95) | 9.09% (28) | 2.08% (2) |
| Amazonbot | Amazon | search | 18.16% (91) | 11.04% (34) | 3.13% (3) |
| GoogleOther | search | 14.17% (71) | 6.17% (19) | 2.08% (2) | |
| YouBot | You.com | search | 18.56% (93) | 8.77% (27) | 2.08% (2) |
Denominator is origins that served a parseable robots.txt: head 501/1,000 · tail 308/1,000 · ai_native 96/110. A site counts as blocking when the group that applies to that crawler disallows the root, with an exact user-agent match taking precedence over the wildcard group.
Roughly half the ranked strata return no robots.txt, and that is expected rather than
a gap: Tranco ranks by DNS resolution volume, so its head contains a great many CDN and infrastructure
hostnames like akamaized.net, googleapis.com,
apple-dns.net and gtld-servers.net, which are not
browsable websites and correctly serve nothing at the apex. Rates are computed over origins that
answered, never over the whole stratum, so infrastructure hostnames cannot dilute the result.
Larger projects already count these directives. Three choices make this reading different, and each one is a thing the bigger datasets cannot give you.
A decision, not a count of directives.
The largest public dataset records how many Allow and
Disallow lines each crawler has and discards the paths, which makes
Disallow: / indistinguishable from Disallow: /admin/.
This resolves precedence and reports allow, partial, or full block.
Crawlers separated by what they are for.
Refusing to be trained on is a different decision from refusing to be read on a person's behalf. Blending the two hides the only interesting thing in the data.
Who wrote the policy is recorded too.
One CDN ships a managed robots.txt to millions of zones
that permits search and denies training. If most blockers sat behind it, the headline would be measuring a
vendor default. So every origin's CDN is recorded, and the share is published rather than assumed away.
No key, no account, no rate limit. Every endpoint is a static file on a CDN.
curl https://28081999.me/api/census/latest.json curl https://28081999.me/api/census/series.json
Also llms.txt, openapi.json, .well-known/mcp.json and .well-known/agents.txt, served at the paths this census probes.
A second, quieter thing this machine does, kept because it costs nothing and the record is unbroken.
Alongside the census it records what the most people looked up, what developers argued about, and what was published, once a day, frozen. 346 days so far with no gaps, 2025-11-01 to 2026-10-12. Unlike the census it is an aggregation of public APIs and anyone could rebuild it, which is why it is not the point of this site.
Browse the archive Compute a model's knowledge gap Latest day