Astra on ARC‑AGI‑3: two harnesses, two results – 62.71% or 99.95% depending on the shell

On 3 September 2026 ARC Prize published, under Greg Kamradt's byline, its analysis of how OpenAI's GPT‑6 Astra performed on the ARC‑AGI‑3 benchmark. The model was evaluated with two different harnesses: the Standard one, described as “allowing the model to carry forward the notes it chooses to keep”, and the Provider Adapter, which “preserves opaque reasoning state between requests and uses compaction for long conversations”. A harness is the program wrapped around the model that decides what it can remember from one call to the next; until now ARC Prize used a single harness, identical for every provider, precisely so the numbers would be comparable.

With the Standard harness the top score is 62.71% (reasoning max), while with the Provider Adapter it climbs to 98.55% (reasoning max) and reaches 99.95% at the high level. Under the Standard harness the score scales with reasoning effort (62.71% max, 17.45% low); under the Provider Adapter that scaling all but vanishes — even with reasoning switched off the model gets 96.72%, more than 34 points above its own best in the Standard harness.

Total run costs, as reported by ARC Prize, are $26,098 for the Standard harness run at reasoning max and $18,817 for the Provider Adapter run at reasoning max — the higher result cost less. The Provider Adapter runs were also about 3.66 times faster in total time and consumed 49% fewer tokens than the Standard ones, across 167 game‑reasoning pairs solved by both configurations.

ARC Prize was explicit that it is not claiming Astra is AGI, and that it does not treat saturating the benchmark as evidence of AGI. “We are not claiming this is AGI”, ARC Prize writes, adding that saturating the benchmark would not amount to “proof that AGI has been achieved”. From now on the ARC‑AGI leaderboard will publish results from both harnesses, labelled with the evaluation condition, as stated in ARC Prize's announcement.

Fortune documented five changes to the metrics OpenAI published after launch, among them Astra's hallucination rate moving from 4.2% to 2% and back to 4.2%, and the ARC‑AGI‑3 score going from 98.6% in the embargoed draft to 99.99% in the published version. An OpenAI spokesperson said: “We care a lot about getting evaluations right. Most evaluations carry noise of a few percentage points depending on the checkpoint, the scaffold and the individual run”, without offering a point‑by‑point explanation of the individual revisions.

The Next Web worked out that the most quoted comparison (99.9% versus 7.8% for GPT‑5.6 Sol) mixes different harnesses; with the Standard harness on both sides the comparison is 62.7% versus 7.8%.

One gap remains: the results page does not say who built or maintains the Provider Adapter harness; The Next Web suggests it was submitted by OpenAI, but ARC Prize does not confirm it. The score figures (99.95% high, 98.55% max) also fail to match the rounded numbers in the post (99.9%) or the embargoed draft (98.6%). As of now no independent third party has reproduced the two runs. — Pixie