---
title: "AI training now drives 52% of crawler requests, Cloudflare says"
url: https://surfil.com/blog/cloudflare-ai-training-crawler-share-2026
section: reports
language: en
published: 2026-09-18T00:00:00.000Z
modified: 2026-09-25T00:00:00.000Z
author: Adam Hafez
topics: ["AI search", "Technical SEO"]
---

# AI training now drives 52% of crawler requests, Cloudflare says

## The short answer

Cloudflare reports that 52% of crawler requests it identifies by purpose were for AI training as of June 2026, up from 22% in Spring 2025. Mixed-use crawlers add over 36% more. Pure search crawling, the kind that can send readers back to a site, is now a small and declining share of crawler activity.

## Key takeaways

- Cloudflare says 52% of crawler requests were for AI training as of June 2026, up from 22% in Spring 2025.
- Mixed-use crawlers, which blend search, agent use and training, account for over 36% of crawler activity.
- Cloudflare describes pure search crawling as a small and declining share, despite remaining critical for publisher visibility.
- More than one-third of crawler activity still comes from mixed-use bots, so purpose labels cannot fully separate training from search.
- Cloudflare defines its crawl-to-refer ratio as HTML requests from a platform's crawlers divided by HTML requests referred from that platform, normalized to one referral.

Most crawling headlines count bots. Cloudflare's July 2026 report does something more useful for
an SEO: it splits crawler requests by what the crawler is for. On that split, [AI search](/topics/ai-search)
visibility is the smaller half of the story, because most requests are not asking to send anyone
back.

## How much is AI training?

Cloudflare writes that "52% of crawler requests are now for AI training as of June 2026, up from
22% in Spring 2025." It also says mixed-use crawlers, those "blending search, agent use, and
training", represent over 36% of activity, and that pure search crawling is now "a small and
declining share of overall crawler activity, despite remaining critical for publisher visibility."

Read together, the two named categories cover at least 88% of identified crawler requests. Cloudflare
does not publish the remaining split on the page, so this report does not estimate it.

## Why mixed-use bots blur it

Cloudflare says more than one-third of crawler activity on its network still comes from mixed-use
bots. A crawler that fetches a page for both training and answer retrieval cannot be counted as
either, so the 52% training figure is a floor for training-related fetching, not a clean ceiling.
For [technical SEO](/topics/technical-seo) work, that means a single robots.txt rule for such a
bot blocks or allows training and retrieval together.

## How crawl-to-refer is measured

The ratio comes from a separate Radar feature. Cloudflare divides "the total number of requests from
relevant user agents associated with a given search or AI platform where the response was of
Content-type: text/html" by the number of HTML requests whose Referer header names that platform,
normalized to a single referral. Cloudflare notes that traffic referred by Claude's native app
carries no Referer header, and believes the same holds for other native apps, so ratios can read
higher than reality by an amount it calls unclear. For the week of June 19 to 26, 2025, the
ratios ran from Anthropic at 70,900:1 down to Mistral at 0.1:1.

## How big is the network?

The split is only as broad as Cloudflare's view. The same post says more than 20% of the web sits
behind its network, including 36% of the world's most-visited websites, and that more than 50% of
traffic on the Internet is now non-human. It also recalls the default that sits under these
figures: on new Cloudflare domains, AI training crawlers are blocked unless the owner chooses
otherwise. Sites outside Cloudflare, or with different settings, may see a different mix.

## What to do with the split

Decide per crawler, not per company. Google shows why that matters: its Google-Extended token
controls whether content Google crawls may be used to train future Gemini models, and Google
says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
It has no separate user agent; crawling uses Google's existing agents and the token only acts in
robots.txt. Blocking a single-purpose training token therefore leaves search crawling alone,
while a mixed-use bot gives no such choice. Our [robots.txt and llms.txt generator](/tools/robots-and-llms-txt-generator)
lists what each AI crawler does, and our report on [AI crawler block rates](/blog/ai-crawler-blocking-rate-trend-2026)
covers how often a block actually stops the bot.

## What this report does not show

The 2026 post does not give a crawl-to-refer ratio, an absolute request count or a sample size for
the purpose split, so none appear here. Cloudflare also reports that some of the most heavily
crawled categories have seen human traffic decline as much as 40% in less than one year, but the
page does not tie that figure to the purpose split, and this report does not either.

## Sources

1. [Content Independence Day, one year on: building the business model for the agentic Internet](https://blog.cloudflare.com/agentic-internet-bot-report/) - Cloudflare Blog (primary)
2. [The crawl before the fall... of referrals: understanding AI's impact on content providers](https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar/) - Cloudflare Blog (primary)
3. [Google's common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) - Google Search Central (primary)