AI bots and Europe’s robots.txt: in TollBit’s “The Bad Bots” report, 15% of agents reach disallowed pages

On 14 August 2026 the findings from the “The Bad Bots” edition of TollBit’s State of the Bots report were published, and Digiday reported on them after reviewing the document. The report covers the first half of 2026 and draws on a sample of 3,906 publisher sites, 456 of them European. The source is a company that sells publishers tools for detecting and monetising bot traffic, so the sample reflects its own client network rather than the web as a whole (its representativeness is uncertain).

According to the report, around 15% of the AI agents identified as “page fetchers” on European sites reached URLs marked disallow in robots.txt files. ChatGPT-User (OpenAI), Bytespider (ByteDance) and Youbot in particular reached pages marked as off-limits on almost half of the European sites that had explicitly listed them, with ChatGPT-User present on the largest number of sites. OpenAI’s official documentation states that, because ChatGPT-User’s actions are initiated by a user, robots.txt rules “may not apply” (OpenAI, developers.openai.com).

The metrics show a sharp gap between Europe and North America: robots.txt instructions on European sites are ignored nearly three times as often; the median number of AI fetches per site is four times higher in Europe; the ratio of bot fetches to human visits referred back to the site worsened from 150:1 in the first quarter to 227:1 in the second, and across the whole half-year the European average is one human visit for every 179 bot fetches, more than three times worse than North America. AI apps also account for 0.05% of external referrals for European publishers, against 0.16% for North American ones.

Part of the gap, though, comes down to publishers themselves: only 13% of European sites block Perplexity-User in robots.txt, against 26% of North American ones (figures cited by PPC Land). In absolute terms, Lai Yi Ohlsen of Cloudflare Radar told Digiday, North America receives more bot requests than Europe does.

TollBit’s full report has not been made public; the figures come from Digiday, Search Engine Journal and PPC Land, which reviewed the document. TollBit has a commercial interest in the result, and its “bypass” methodology counts any access to a disallowed URL without distinguishing a deliberate choice by the operator from a configuration error. Those limits mean the data has to be read with care, but they point clearly to one thing: the voluntary convention of robots.txt is under growing strain from AI agents.