The AI text tax: what it costs to train models on a web written by machines

A preprint posted on arXiv by researchers at the University of Maryland and Pangram Labs measures how much of the web is now AI-generated text, and what that means for model pre-training. Analysing the FineWeb corpus and extending it through August 2026, the study finds that the share of AI-generated tokens rose from under 0.1% in 2021 to 31.1% in summer 2026. The authors trained 800 models with fewer than one billion parameters and derived a scaling law from them. By that law, a 268-million-parameter model trained at 20 tokens per parameter on the unfiltered web at the August 2026 share (31.1%) needs 1.6 times the compute required on the human-only subset. The gap widens as the human-data budget grows. The inefficiency is also set to increase: according to the authors' projections as reported by Digital Applied, the multiplier would climb to 2.1 and then to 3.0 at the shares expected for 2027-2028, with AI-generated text reaching 51% by the end of 2028. These are forecasts, not measurements.

The pattern is clear. When human data is plentiful, adding AI-generated tokens makes the loss worse almost immediately. When human data is scarce, AI text gives an early boost that soon levels off and then reverses, while fresh human data keeps improving performance. That is why the authors recommend filtering out machine-generated text when the goal is performance on human text. They even suggest repeating human data already used rather than padding the dataset with AI text scraped from the web. The project has released the WildAI dataset (96.04 million documents, 83.31 billion labelled tokens), all 800 trained models and the code.

These results should still be read with care. The study is a preprint that has not yet been peer-reviewed. It covers English only, measures loss alone rather than downstream capabilities, and uses small models, so any conclusion about frontier systems is an extrapolation. The measurement also rests on a single commercial detector, Pangram 3.3.2, built by the company that funded the research and that employs four of the authors. The authors report an extremely low false-positive rate. Independent tests by PCMag, however, showed that text "humanizer" tools can get past the detector and that it has trouble spotting AI-assisted writing. The real share of AI-generated text on the web could therefore be different, or higher.

If training tomorrow's models means first cleaning the web of what today's models have produced, we are looking at a curious industrial paradox. Gains in computational efficiency may come to depend less on chip power and more on whether we can still pick out our own voice from the background noise we have only just started to make. — Olya