Cloudflare's imperfect truce between search engines and AI training

On 15 September 2026 Cloudflare switched on its "Disallow AI Training" option, configurable per domain in the security settings. The stated aim is to break a trade-off that has so far been forced on anyone publishing on the web: handing your own text over to generative model training just to avoid disappearing from ordinary search results. Crawler activity is now sorted into three categories: "Search" for indexing, "Training" for model training and "Agent" for real-time assistants. With the change of defaults for new domains and for customers on free plans, sites that carry advertising will have Training and Agent blocked automatically, while Search stays on. For sites without advertising, by contrast, all three remain allowed by default.

To handle mixed-use crawlers such as Googlebot, Applebot and Bingbot, which account for 36.6% of verified crawler traffic on Cloudflare's network, the company has introduced the status of "Accountable" operator. Cloudflare lists Apple, Google and Microsoft as Accountable operators, meaning parties that meet — or have committed to meeting by stated deadlines — four transparency requirements, among them an opt-out from training and inspection tools at the level of the individual URL. A technical reading of these concessions, however, leaves wide margins of uncertainty. The big technology companies have issued no direct confirmation: the list is a unilateral attribution by Cloudflare. On top of that, the main promised tools sit in the future — the URL-level inspection tool Apple has announced for 2027, the support for a "no training" preference in robots.txt that Microsoft/Bing is targeting for early 2027, or the unspecified "few weeks" mentioned by Google.

The picture is also shaped by figures released by Cloudflare, according to which fewer than 1% of sites block classic indexing, while 17% switch on measures to prevent the training of artificial intelligence. The company further claims that more than half of users read AI summaries in search engines and that, having read them, a user is over 40% more likely to end the search there without going on to the source sites. These, though, are measurements internal to Cloudflare's network, released with no public methodology and impossible to verify independently. Then there is the gap between setting a preference and having it honoured: a PPC Land investigation in August 2026 found that in Europe 15% of AI crawler fetches concerned URLs for which a refusal had been expressed, in a context where bot traffic made up 57.4% of all HTML traffic.

Cloudflare's move, summed up by chief executive Matthew Prince as a way to preserve the openness of the network by giving control back to content producers, looks like an attempt to regulate lawless territory single-handedly. But as long as the block rests on promises of future compliance and on a technical courtesy that crawlers can bypass, publishers' sovereignty over their own work remains a temporary concession rather than a right protected by technology.

— Olya