About AnswerIndex
AnswerIndex is an independent measurement publication. We run our own crawls and measurements of how the web treats AI crawlers — and how AI search engines answer — and we publish the data itself, not commentary about other people’s data.
The first release, the AI Crawler & llms.txt Adoption Index (November 2026), measured the Tranco top 10,000 websites: their robots.txt policy toward 14 AI agents and their adoption of llms.txt. The full per-domain dataset is published with the report.
Data series
- AI Crawler & llms.txt Adoption Index — a quarterly census of robots.txt policy toward AI agents and llms.txt / llms-full.txt adoption across the top of the web. First edition: November 2026; next edition Q1 2027.
- Cross-Engine Answer Lab (planned) — the same questions put to AI search and answer engines, with the answers recorded and compared over time.
- Field Measurement Notes (planned) — shorter, single-site or single-question measurements, published with the same standards as the index.
Methodology principles
- Primary data only. Every number we publish comes from a measurement we ran ourselves, with the crawl date and the sample stated on the page.
- Caveats disclosed. Known limitations — coverage gaps, counting rules, comparison baselines — are published alongside the findings, in the report itself, not in fine print.
- Datasets published. The underlying per-domain data behind each report is released in full, so anyone can check our arithmetic or reuse the numbers.