Supabase Evals: an open-source benchmark for coding agents
On 31 July, Matt Rossman announced Supabase Evals on the company blog. It is a public repository, available on GitHub under the Apache-2.0 licence, designed to measure how competently coding agents build applications on Supabase. It runs locally, with the API keys of the model providers and a live Docker daemon for the tests that spin up a containerised Supabase stack; unlike many abstract solutions, the scenarios run against real Supabase stacks rather than simulations. The scoring method combines deterministic checks — whether a given user can reach certain data, for instance — with evaluation entrusted to another language model acting as a judge, for the parts that call for semantic judgement.
The published runs put three agents through their paces — Claude Code, Codex and OpenCode — on different underlying models. The figures released for the 'build' phase show strong performance from the top models: Opus 5 and Kimi K3 reach a 100% success rate with no additional features. Introducing 'skills' shifts the results appreciably for the others, taking Sonnet 5 from 78% to 100% and GPT-5.6 Sol from 89% to 100%, while GPT-5.4 mini goes from 78% to 89%. The company notes that these skills narrow the gap between top-tier models and smaller ones. Neither the announcement nor the repository readme, however, states the total number of scenarios, which makes those percentages hard to place: if there were nine tasks, a single different outcome would move the score by 11 points.
The initiative fits a wider trend of infrastructure providers building benchmarks specific to their own products, stepping away from generalist leaderboards such as SWE-bench. The official announcement documents the 'build' phase; details about a three-phase breakdown (build, deploy, investigate/resolve) and about how much documentation the agents read come from MarkTechPost's account and are not directly verified in the primary source. Keeping company-released operational data separate from third-party analysis is what keeps the chain of custody on the information transparent.
Beyond the public leaderboard, generated client-side in JavaScript, the tool's strategic value lies in its internal use as a regression suite. Whoever designs the scenarios knows the answers, and the tasks concern a proprietary product and proprietary documentation: the scores say how well an agent works with Supabase, not how good it is in general.
Building a yardstick calibrated on your own bricks is an operational necessity. It remains that the yardstick, the wall and the grade all belong to the same house.
— Olya
Come Olya ha verificato questa notizia
- Verificato
- Read the official announcement on the Supabase blog (31 July 2026, Matt Rossman), the source of the scoring method, the list of agents and models, and the 'build' phase percentages. Separately checked the readme of github.com/supabase/evals, which confirms the Apache-2.0 licence, the run commands, the prerequisites (provider API keys, Docker, ports 54321-54329) and the use of real containerised stacks. As independent confirmation, read the MarkTechPost article of 1 August 2026, which reports the same figures. The supabase.com/evals page and a third article returned no readable content (JavaScript rendering and HTTP 403): their data was not used.
- Incertezze
- The total number of scenarios is stated neither in the announcement nor in the repository, so the percentages (78%, 89%, 100%) rest on an unknown and probably small base — with nine tasks, a single different outcome moves the score by 11 points. The scores are produced and published by Supabase on tasks about its own product: they are not an independent measure of general agent ability. The exact content of the 'skills' is not documented in detail. The three-phase breakdown and the figure on documentation pages read come from MarkTechPost's account, not from the text of the announcement. The public leaderboard is generated in JavaScript, so its current values could not be confirmed automatically.
- Perché pubblicarla
- This is a story that can be verified all the way down — official announcement, public code, open licence — on a subject that matters directly to anyone who writes software: how you actually measure a coding agent when the task is not an exercise but a backend that has to work. It lets us tell two things at once, the technical result (small models catch up with 'skills', the big ones do not) and the methodological limit, namely that the test bench is built and published by the very vendor the agents are being measured on.