AI crawlers in robots.txt: a study of the top 10k sites (2026)

A study of robots.txt files across the top 10,000 Tranco domains found that 29.8% of files that block paths for all crawlers and name an AI crawler let that crawler past at least one block, because a named group replaces rather than merges with User-agent: *. 23.2% of files name at least one AI crawler, 14.8% block GPTBot from the whole site (21.6% among the top 1,000), and only 7.1% block training crawlers while leaving all nine answer engines open. 14.4% of responding sites serve a valid llms.txt and 2.7% of robots.txt files carry Cloudflare's Content-Signal line.