The gap between running code and rewriting how a model learns

On 20 August 2026 the preprint “AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement” was posted on arXiv, signed by Yizhe Chi, Wenyi Li, Deyao Hong, Xiaoqiu Wang, Mingju Gao, Kaisen Yang, Bingxiang He, Youjie Zheng, Calvin Xiao and, as corresponding author, Qinhuai Na. The authors, who in the document list affiliations with Navers Lab, Einsia.AI and Tsinghua University, propose a suite for assessing whether a coding agent can rewrite a model's training algorithm — that is, the objective or the update rule — rather than confining itself to hyperparameter tuning or data handling. The benchmark rests on 10 frozen research repositories covering as many algorithm families, among them OpenR1, RAGEN, DPO and Model Soup. The test-harness code has been released in the GitHub repository Einsia/AI4AI-Bench under an Apache-2.0 licence, though the work remains a preprint without peer review and with no separate institutional announcement for the affiliations listed.

Under the protocol the authors describe, each agent gets 4 hours on a single B300 GPU to inspect and modify the code, testing its own ideas against a quick metric; the source patch alone is then applied in a clean container and retrained from scratch for up to 12 hours, with scoring left to a fixed evaluator that stays hidden from the agent. Measurement uses a normalised scale on which 0.1 stands for the starting algorithm and 1.0 for the optimum of the task. 0 marks an unusable model, and a score below 0.1 means the agent made the code it found worse. According to what the authors state in the paper, across 290 runs in total the six systems evaluated average 0.166, while the highest score goes to Claude Opus 5 at 0.250, followed by GPT-5.6 Sol at 0.191, Kimi K3 at 0.174, Claude Sonnet 5 at 0.145, GPT-5.6 Terra at 0.135 and GPT-5.6 Luna at 0.117. Even the best system therefore covers less than a fifth of the distance between the algorithm already sitting in the repository and the optimum of the task. The paper reports that most submissions never touch the way the model learns at all; the minority that does intervene at the algorithmic level averages 0.226 against 0.126 for the others. The authors observe that raising the reasoning effort pushed the share of algorithm-level interventions from 8% to 64%, lifting the average from 0.094 to 0.196; in their reading, that increase reflects a greater willingness to modify deep code rather than any superior design ability. Since these measurements were carried out by the benchmark's own developers, with no independent audit of the compute costs and no third-party leaderboards, the published metrics should be read as claims from an interested party.

The findings of this benchmark sit alongside those of separate research conducted at Princeton University. On 18 August 2026 the magazine MIT Technology Review reported on an experiment led by researchers Peter Kirgis and Sayash Kapoor: an agent built on Claude Opus 4.8 was given six days, API credits and computing resources to tackle two unpublished scientific problems drawn from work submitted to NeurIPS 2026. As the magazine recounts, the two papers the system produced were rejected by the original authors of that work, who assessed them as the conference's reviewers would: the agent proved competent on the engineering tasks but weak in creative judgement. Commenting on the exercise, researcher Sayash Kapoor told MIT Technology Review: “The papers were nowhere close to the mark when it came to being at the quality of a top AI conference.”

If the reported numbers hold up under independent checks, the distance between reorganising existing infrastructure and inventing new learning rules remains wide. One caveat stated by the authors themselves is worth keeping in view: these results photograph the models and harnesses available in August 2026 and say nothing about the versions that follow. — Olya