The unexpected simplicity of compressing an AI's memory

Managing working memory in generative models — the KV cache — has become one of the most critical bottlenecks once reasoning chains grow long. Until now the dominant answer has been sophisticated algorithms such as SnapKV or TriAttention, designed to compute a future-usefulness score for every fragment of text and keep only what is deemed crucial. A preprint filed on arXiv on 3 September 2026 (arXiv:2609.03430, Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning) by researchers at Salesforce AI Research and the University of Illinois Urbana-Champaign calls that approach into question, arguing that complex selection signals make no decisive contribution to the quality of the result.

The proposed technique, called Random Attention and released under the Apache 2.0 licence in its code repository, follows a logic with no relevance computation at all: it protects the tokens of the initial prompt and those in the most recent window, discarding the rest uniformly at random within each attention head. The paper's thesis is that reasoning traces protect themselves, thanks to redundancy spread both through the text and across the model's heads. On MATH500 with Qwen3-4B the random method scores 0.874 against TriAttention's 0.864 and SnapKV's 0.703; on GPQA-Diamond, 0.530 against 0.533 — gaps of roughly a percentage point, in both directions. In the measurements the authors report on an H200 GPU running vLLM, removing the scoring stage cuts the latency of each eviction step from TriAttention's 1.47-1.64 milliseconds to 0.30 milliseconds. On 32-thousand-token generations that yields throughput 1.6 to 2.7 times that of full attention, and 32-43% above TriAttention, the strongest competitor.

The work has not been peer reviewed, and the same team that signs the method measured its accuracy, throughput and latency, with no independent reproductions or third-party checks available yet. The analysis documented in the paper is also confined to four specific models, mostly from the Qwen3 family plus Phi-4-reasoning — while the repository mentions optional support for DeepSeek-R1-Distill-Llama-8B — and it shows dips on programming tasks, where prompt length quickly eats up the cache's fixed capacity. The budgets tested — around 4 times compression, 3 on LiveCodeBench — were chosen by the authors. Outside that perimeter the paper says nothing: other model families, more aggressive compression, or long texts that are not reasoning chains, where trace redundancy — the very mechanism the authors invoke to explain why chance works — simply is not there. The system also runs into an obvious logical obstacle with rare or unrepeated facts, which the random method risks discarding for good.

If outside checks confirm the figures, the result will concern one precise box — which token to throw out of the cache while a model reasons — and will suggest that there, the criterion of choice weighs less than the redundancy the reasoning trace produces on its own. — Olya