The judge that changes its mind: Ai2 tests the models that rate research novelty

Many systems billed as “AI scientists”, or that generate research ideas automatically, hand a second language model the job of deciding whether what they produce is actually new. The improvements they claim then rest on that verdict. On 27 October 2026 Ai2 (the Allen Institute for AI) published a study testing how reliable that yardstick really is. The paper, “Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation”, is on arXiv (2610.02022). The first version dates from 1 October, the third from 26 October. It is a preprint and does not appear to have been peer reviewed yet. The benchmark is available on Hugging Face under an ODC-BY licence, and the code is on GitHub.

The authors did not use paper acceptance as their reference. Instead, they pulled from the ICLR 2026 reviews on OpenReview the passages where reviewers confirm or dispute a paper's originality. That produced two scenarios. The first, Human-Only, pits 154 ideas praised for their novelty, taken from the top 10% of accepted papers, against 145 ideas criticised on that front, taken from the bottom 10% of rejected ones. The second, Human+Generated, pairs the same 154 human ideas with 154 ideas from a simple LLM generator. The authors treat the generated ideas as less novel, but concede this is a weak label that was not checked idea by idea. So part of the numbers rests on an assumption.

The judges under test are GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6, plus GPT-5.6 Sol in agentic mode and two specialised evaluators: Idea Novelty Checker and the AI Scientist reviewer from Sakana AI. The most striking result concerns GPT-5.4 on the Human+Generated pairs. When the prompt states that one of the two ideas is novel, accuracy is 93.5%. When it simply asks which of the two is novel, accuracy drops to 40.9%, below chance, and the verdict flips on more than half the pairs. According to the abstract, this single change can shift pairwise accuracy by more than 50 points. On the Human-Only set the gap shrinks to about 7 points, though for Claude Opus 4.6 it reaches 21. When ideas from two AI generators are compared, between a quarter and a third of verdicts change as the prompt varies. Nor is there a wording that works for everyone: removing an instruction against trivial ideas improves some judges and hurts others. Ai2's blog sums it up in a sentence worth quoting: “Judges that appear reliable on human-authored research may behave differently on AI-generated ideas”. Reliability measured on human work, in other words, does not automatically carry over to ideas produced by a machine.

The cost section is counterintuitive too. The agentic GPT-5.6 Sol judge, which can run searches, scores 0.78 macro-F1. Claude Opus 4.6 reaches 0.89 (the blog also reports 0.854 for the same model in a different configuration; the paper has the details) at roughly a tenth of the cost per decision. A simple prompt on Opus 4.6, with low reasoning and a majority vote over three calls, hits 0.854 for a total of $2.59. Idea Novelty Checker stops at 0.642 after spending $30.66; Sakana's evaluator reaches 0.772 for $82.05. Mindverse also reports a validation step: according to the outlet, an annotator confirmed the review signal in 47 of 50 checks, and an expert re-examined 30 pairs where the best judge got it wrong, preferring the human idea in all 30. From these results the authors draw doubts about the novelty gains claimed by automated ideation systems and call for more robust evaluation methods. One limitation is worth keeping in mind: the models tested are not the latest ones, and the findings may not hold for newer models.

What I find instructive is that the problem lies not only in the model that generates, but also in the one meant to check it. If removing one sentence from a prompt can turn a near-perfect judgment into a rigged coin toss, every chart promising “more original” ideas should also disclose the exact words used to ask the question. Novelty, it seems, also depends on how you ask about it. — Olya