Measurement release · AI Crawler & llms.txt Adoption Index
llms.txt adoption has roughly doubled in five months — and the web is blocking training, not AI
November 19, 2026 · AnswerIndex
On November 19, 2026, AnswerIndex measured the Tranco top 10,000 websites: their robots.txt policy toward 14 AI agents, and whether they publish an llms.txt or llms-full.txt file. 1,123 sites (11.23%) now publish llms.txt, roughly double the June 2026 baseline. At the same time, sites draw a sharp line between kinds of AI traffic: training crawlers are blocked by up to a quarter of sites, while search and user-fetch agents are blocked far less.
Key findings
- llms.txt roughly doubled in about five months. 1,123 of the top 10,000 sites (11.23%) serve an llms.txt file, against 556 in Rankability’s June 2026 measurement of the same Tranco tier. llms-full.txt doubled too, from 112 to 222 sites (2.22%). The comparison is directional — the Tranco list changes daily, so the two samples are not identical domain sets — but the direction and scale are unambiguous.
- The web is not blocking AI — it is blocking training. Every training and data-collection crawler is blocked by 22.6–25.8% of sites with a determinable robots policy. Every search and user-fetch agent sits at 18.6% or below. The gap between the two groups is consistent across all 14 agents measured.
- Blocking is still a minority position. Even CCBot, the most-blocked agent, is allowed by 74.2% of sites with a determinable policy. Most of the top of the web has set no AI-specific robots restriction at all.
- llms-full.txt almost never appears alone. Of the 222 sites with llms-full.txt, 219 also publish llms.txt — only 3 sites serve the full file without the index file. Publishers treat llms-full.txt as an extension of llms.txt, not an alternative.
Who gets blocked: all 14 AI agents
Sites discriminate by purpose. Agents that collect content for model training and datasets cluster at the top of the blocking ranking; agents that power AI search results or fetch a page at a user’s request cluster at the bottom.
llms.txt adoption: June 2026 vs November 2026
In June 2026, Rankability measured llms.txt adoption across the same Tranco top-10,000 tier. Five months later, both files have roughly doubled:
| File | June 2026 (Rankability) | November 19, 2026 (AnswerIndex) | Change |
|---|---|---|---|
llms.txt | 556 | 1,123 | +567 |
llms-full.txt | 112 | 222 | +110 |
| Either file | 558 | 1,126 | +568 |
| Both files | ~110 (implied) | 219 | — |
Today, 1,123 sites (11.23% of the top 10,000) publish llms.txt and 222 (2.22%) publish llms-full.txt; 1,126 publish at least one of the two. The comparison is directional, not exact: the Tranco list changes daily, so the two samples are not identical domain sets, and counting rules may differ slightly (this crawl follows redirects and rejects HTML soft-404 bodies).
Full results: AI crawler access in robots.txt
Of the 10,000 domains measured, robots.txt was retrieved for 6,397 (64.0%); 694 (6.9%) refused with HTTP 401/403 and are treated as fully disallowed; for 2,909 (29.1%) robots.txt was missing or unreachable. Blocked percentages below are computed over the 7,091 sites with a determinable policy.
| User-agent | Blocked | Blocked % | Allowed |
|---|---|---|---|
| GPTBot | 1,764 | 24.9% | 5,327 |
| OAI-SearchBot | 1,127 | 15.9% | 5,964 |
| ChatGPT-User | 1,245 | 17.6% | 5,846 |
| ClaudeBot | 1,711 | 24.1% | 5,380 |
| Claude-SearchBot | 1,131 | 15.9% | 5,960 |
| Claude-User | 1,150 | 16.2% | 5,941 |
| PerplexityBot | 1,321 | 18.6% | 5,770 |
| Perplexity-User | 1,175 | 16.6% | 5,916 |
| Google-Extended | 1,611 | 22.7% | 5,480 |
| CCBot | 1,828 | 25.8% | 5,263 |
| Bytespider | 1,821 | 25.7% | 5,270 |
| Meta-ExternalAgent | 1,637 | 23.1% | 5,454 |
| Amazonbot | 1,602 | 22.6% | 5,489 |
| Applebot-Extended | 1,601 | 22.6% | 5,490 |
Methodology & limitations
How the crawl worked
On November 19, 2026 we took the first 10,000 domains of the Tranco top-1M list (snapshot downloaded the same day) and probed each domain for three files: /robots.txt, /llms.txt, and /llms-full.txt. An llms file counts as present only when the probe returns HTTP 200 with a non-HTML body — an HTML response is treated as a soft-404 and counted as absent; redirects are followed. Each retrieved robots.txt was evaluated for 14 AI user-agents using RFC 9309 group selection: an agent-specific group beats the * group, the longest matching rule for / wins, and Allow wins ties. A robots.txt refused with HTTP 401/403 is treated as fully disallowed for every agent; other non-200 responses and transport failures are recorded as no determinable policy.
Limitations — read these before citing
- The unreachable share is an upper bound, so llms.txt adoption is a slight undercount. 2,168 domains produced no HTTP response at all for robots.txt (DNS/connect failure or a hanging server); for those, the llms probes were skipped and counted as absent. Reachability was measured from our crawler’s network egress, which drops some connections under parallel load: a low-concurrency second-chance pass recovered 707 domains that then answered normally. A residual share of the unreachable set is therefore likely to be egress-related false negatives rather than genuinely dead sites.
- The June 2026 comparison is directional. The Tranco list changes daily, so the two samples are not identical domain sets, and Rankability’s counting rules may differ slightly from ours. Treat the doubling as a well-supported direction, not a precise growth rate.
- This is one snapshot in time. robots.txt and llms.txt files change without notice; a site’s status here reflects November 19, 2026 only.
- Download caps do not affect the counts. llms bodies were capped during download (2 MB / 5 MB); presence needs only the status code and the first bytes, so the cap matters only in pathological cases.
Dataset
The complete per-domain dataset behind this report — all 10,000 domains, robots.txt status, the verdict for each of the 14 agents, and llms.txt / llms-full.txt presence — is published as a CSV:
Download results-2026-11-19.csv
Cite this: AnswerIndex, AI Crawler & llms.txt Adoption Index, November 2026. https://answerindex.org/