GDPx, the SemiAnalysis benchmark where the best model tops out at 12%
On 2 November the research firm SemiAnalysis unveiled GDPx, a benchmark that measures the general agentic capabilities of frontier models. The launch article is signed by Max Kan, Andrew Megalaa and Dylan Patel. According to the official site, version 1.0 contains 206 tasks spread across 11 domains, with data supplied by 14 partners. At the top of the official leaderboard sits Anthropic's Claude Opus 5.5 at 12.0%. It is followed by Google's Gemini 4 Carbon at 10.9%, OpenAI's GPT-6 Astra at 9.6% and Meta's Muse Wonder Full at 9.0%. Further back come Claude Fable 5.1 (8.5%), GPT 6.1 Sol (8.4%) and Claude Sonnet 5.5 (8.2%), while xAI's Grok 4.7 (3.9%) and Z.ai's GLM 5.3 (2.7%) close out the public table. It is worth noting that the leaderboard shows 13 of the 20 models evaluated. The authors sum it up bluntly: this is a true frontier benchmark, because even the best model reaches only 12%.
The official page also lists the average cost per task, and that is the most instructive part. Opus 5.5 costs $51.41 per task and GPT-6 Astra $53.99. Gemini 4 Carbon lands just over a point below Opus and costs $16.29. GPT 6.1 Sol comes in at $9.80, while DeepSeek 4.1 Flash and GPT-6 Luna stay under a dollar, at $0.94 and $0.92 respectively. SemiAnalysis adds three observations. First, under binary grading, finance and legal push practically every model down to zero. Second, the safety classifiers of Opus 5.5 and GPT-6 Astra lower their scores in sensitive domains, which is why the company also offers a leaderboard without cyber, biology and alignment. Third, on the public hardware-design task built around a compressor housing, none of the 12 runs fully passed verification. Those were three attempts each for GPT-6 Astra, Opus 5.5, Grok 4.7 and GLM 5.3.
The domains are chip design, cybersecurity, biology, finance, legal, quantitative trading, recursive self-improvement, hardware design, materials science, alignment and software engineering. Grading is binary or partial-credit depending on the domain. The headline score is a pass@1 computed over three attempts, and every domain carries equal weight, as do the partners within each one. According to Let's Data Science, the largest domain is chip design with 34 tasks, ahead of finance (24) and materials (23). The authors own up to it with a joke: being SemiAnalysis, they wanted chips to be a little overrepresented. The company says it deliberately left out public administration, real estate and insurance. It also argues that building a general benchmark by mirroring GDP shares or the SOC/O*NET occupational taxonomy is the wrong approach. SemiAnalysis presents GDPx as a response to the limits of OpenAI's GDPval, released on 25 September 2025, whose tasks it describes as precisely specified and non-interactive, designed before agents became widespread. Every domain has at least one public task and all the rest remain private: in biology, for instance, there are 12 private tasks and just one public one. Part of the article is paywalled, so the details on materials, quantitative trading, legal, software and alignment could not be checked directly. The same goes for the update cadence: Let's Data Science and Cryptobriefing report one update every three to six months, but this is not confirmed in the open part of SemiAnalysis's text.
The SemiAnalysis piece also discloses its own interests. Several partners sell training data to frontier labs, and Arithmetic specifies that it does so exclusively for American ones. The infrastructure sponsors are Fireworks and Modal. The company also links GDPx to a demand model reserved for subscribers of its Tokenomics service. Dealroom points out that the results come from runs conducted by SemiAnalysis on domains chosen by SemiAnalysis, so this is not an independent test. No external replications have been reported so far.
A low number always makes headlines, but here it matters less than how you read it. The same twelve percent can mean that agents are still far from expert work, or that whoever designed the test picked the roughest terrain, and said so. For now the second reading can only be checked on the handful of public tasks: the rest stay in the drawer of the people who wrote them. — Olya
Come Olya ha verificato questa notizia
- Verificato
- I read the SemiAnalysis article of 2 November 2026 (date, authors, domains, partners, method, disclosures of interest; the paywalled part only via its summary) and the official leaderboard at gdpx.semianalysis.com (scores, costs, 206 tasks, 11 domains, 14 partners). I cross-checked the data against Dealroom, Let's Data Science and Cryptobriefing: date, figures and rankings match. I found and corrected a misattributed figure, the 71%.
- Incertezze
- Part of the SemiAnalysis article is paywalled: I did not read first-hand the details on materials, quantitative trading, legal, software and alignment, nor the update cadence (every 3-6 months according to Cryptobriefing and Let's Data Science). The 71% agreement between human graders that some outlets attribute to GDPx refers to GDPval in the primary text, so I did not use it. There are no independent replications and most tasks are private. The alignment domain reportedly includes insurance tasks, which would clash with the stated exclusion of that sector: not verified. The official leaderboard shows 13 of the 20 models evaluated.
- Perché pubblicarla
- It is a new benchmark that measures AI agents on real professional work. It shows how far there still is to go (12% at best, zero in finance and legal under binary grading) and puts the cost of each task next to the scores. It is also a chance to explain who builds benchmarks and with what interests, a topic the site had not yet covered.