AI training now drives 52% of crawler requests, Cloudflare says

Most crawling headlines count bots. Cloudflare’s July 2026 report does something more useful for an SEO: it splits crawler requests by what the crawler is for. On that split, AI search visibility is the smaller half of the story, because most requests are not asking to send anyone back.

How much is AI training?#

Cloudflare writes that “52% of crawler requests are now for AI training as of June 2026, up from 22% in Spring 2025.” It also says mixed-use crawlers, those “blending search, agent use, and training”, represent over 36% of activity, and that pure search crawling is now “a small and declining share of overall crawler activity, despite remaining critical for publisher visibility.”

Read together, the two named categories cover at least 88% of identified crawler requests. Cloudflare does not publish the remaining split on the page, so this report does not estimate it.

Why mixed-use bots blur it#

Cloudflare says more than one-third of crawler activity on its network still comes from mixed-use bots. A crawler that fetches a page for both training and answer retrieval cannot be counted as either, so the 52% training figure is a floor for training-related fetching, not a clean ceiling. For technical SEO work, that means a single robots.txt rule for such a bot blocks or allows training and retrieval together.

How crawl-to-refer is measured#

The ratio comes from a separate Radar feature. Cloudflare divides “the total number of requests from relevant user agents associated with a given search or AI platform where the response was of Content-type: text/html” by the number of HTML requests whose Referer header names that platform, normalized to a single referral. Cloudflare notes that traffic referred by Claude’s native app carries no Referer header, and believes the same holds for other native apps, so ratios can read higher than reality by an amount it calls unclear. For the week of June 19 to 26, 2025, the ratios ran from Anthropic at 70,900:1 down to Mistral at 0.1:1.

How big is the network?#

The split is only as broad as Cloudflare’s view. The same post says more than 20% of the web sits behind its network, including 36% of the world’s most-visited websites, and that more than 50% of traffic on the Internet is now non-human. It also recalls the default that sits under these figures: on new Cloudflare domains, AI training crawlers are blocked unless the owner chooses otherwise. Sites outside Cloudflare, or with different settings, may see a different mix.

What to do with the split#

Decide per crawler, not per company. Google shows why that matters: its Google-Extended token controls whether content Google crawls may be used to train future Gemini models, and Google says it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.” It has no separate user agent; crawling uses Google’s existing agents and the token only acts in robots.txt. Blocking a single-purpose training token therefore leaves search crawling alone, while a mixed-use bot gives no such choice. Our robots.txt and llms.txt generator lists what each AI crawler does, and our report on AI crawler block rates covers how often a block actually stops the bot.

What this report does not show#

The 2026 post does not give a crawl-to-refer ratio, an absolute request count or a sample size for the purpose split, so none appear here. Cloudflare also reports that some of the most heavily crawled categories have seen human traffic decline as much as 40% in less than one year, but the page does not tie that figure to the purpose split, and this report does not either.