Which AI crawlers do news sites block most? CCBot tops 75%

Our own report on live 403 rates tracks how often an AI crawler request is actually refused at the edge, and cites a robots.txt scan that ranks crawlers by how often they are named in a disallow rule. BuzzStream ran a separate scan in the same spirit but narrowed to one publisher category: the top 100 UK and US news sites, individually checked against 11 named AI bots rather than a broad web-wide sample. That narrower lens surfaces a per-crawler ranking and a UK-versus-US split that a general sample would average away.

Which crawlers are blocked?#

Across the 100 sites, CCBot is blocked by 75%, the highest rate BuzzStream recorded. Anthropic-ai follows at 72%, then ClaudeBot at 69%, PerplexityBot at 67%, Claude-Web at 66%, GPTBot at 62% and Applebot-Extended at 61%. The rest trail behind: OAI-SearchBot is blocked by 49% of sites, Google-Extended by 46%, ChatGPT-User by 40%, and Perplexity-User by just 17%, the lowest figure in the study.

Why Google-Extended less?#

Google-Extended’s 46% overall rate splits sharply by country: US publishers block it at 58% against 29% for UK publishers, the biggest country gap BuzzStream reported. BuzzStream says it remains unclear why. It notes that Google-Extended trains Gemini and is not used for Google’s search index, and speculates that publishers go easier on Google because blocking anything Google-related risks organic traffic.

Training bots vs search bots#

BuzzStream groups the 11 crawlers into training, retrieval and indexing bots, and the gap between training and retrieval is clear: 79% of the 100 sites block at least one training bot, against 71% for live-retrieval bots. Within Perplexity’s two bots, the split is starker still: PerplexityBot, its indexing bot, at 67% against Perplexity-User at 17%, so many sites disallow the background crawl in robots.txt while naming no rule against the live, user-initiated fetch.

Why we care#

A crawler-by-crawler ranking, not just an aggregate block rate, tells a publisher which specific bot to check first. If CCBot is blocked by three in four of the largest news sites, a smaller publisher weighing the same decision is looking at what has become close to a category norm rather than a fringe move, while Google-Extended’s 46% overall rate shows the opposite is true for it.

The UK-US gap on Google-Extended is the most actionable single number in the study. A US publisher blocking that crawler is following the regional majority at 58%; a UK publisher doing the same is in the 29% minority, and either one should know which group they are joining before setting the rule, since our earlier robots.txt research found that publishers frequently copy a competitor’s policy without checking what it actually blocks.