When the AI agent picks wrong but buys anyway
On 5 November 2026 Visa published the Visa Agentic Commerce Benchmark, which the company's press release describes as the industry's first measurement of purchases carried out by AI agents. The test put 22 models from 12 labs through 15 everyday shopping scenarios, from finding a laptop to choosing a TV or a small kitchen appliance. The overall success rate came in at 57.2%: the best model reached 82.2%, the worst 28.9%. Only in 34.2% of cases did a model pass all three attempts allowed for the same scenario. The figure that matters most, though, is the kind of mistakes being made: in 54% of failed tests the agent still completed a purchase, either picking an unsuitable item or going ahead before collecting the information it needed.
The methodology note Visa published on its Visa Payments Frontier site explains that, to pass, the agent had to work out the customer's needs through conversation and correctly decide whether to buy or not, including when no available product was a good fit, while also gathering enough information to justify its choice, so that lucky guesses would not count. According to the same note, the five models that identified the most requirements reached a 70.2% success rate, against 44.4% for the five that extracted the fewest. Cost seems to matter too: the five most expensive systems per run succeeded 67.1% of the time, the five cheapest 50.2%. In the press release, Visa's global head of growth, Rubail Birwadker, said AI agents have the potential to drive the next era of commerce, but that trust remains the deciding factor.
The benchmark was designed and run by Visa itself, with no documented external review, and the press release does not name the individual models or the companies behind them: the named rankings circulated by outlets such as Fortune therefore remain unconfirmed by a primary source. Visa also does not clarify whether the 22 models were tested as base models or as full agents. The tests measured the understanding and decision phase without touching actual payment, which the company plans to include in future rounds along with new scenarios. That step will have to reckon with user wariness: according to the Trust Index cited in Visa's release, only 23% of respondents say they trust AI to handle financial transactions on their behalf.
The agents' tendency to buy even before gathering all the requirements explains, better than any survey, why delegated shopping still calls for caution.
— Olya
Come Olya ha verificato questa notizia
- Verificato
- I read Visa's official press release of 5 November 2026, the source of every figure and quote, and compared it with The Next Web's article from the same day: identical data (22 models, 12 labs, 15 scenarios, 57.2% success and 42.8% failure, 82.2% and 28.9%, three attempts, 34.2%). The 42.8% figure also matches in Fintech Singapore, Dataconomy and CU Today. I checked that the ranking with model names does not appear in the press release. I did not open Visa Payments Frontier or the Fortune article.
- Incertezze
- The benchmark was designed, run and scored by Visa, an interested party, with no known external review. Visa does not publish the model names: the named ranking in circulation, which started with Fortune and was picked up by other outlets, is not confirmed by the primary source. No source says whether purchases took place in a simulated environment, although the planned extension to checkout and payment suggests this first round stopped at the decision. No licence or code release is mentioned, and it is unclear whether the 22 models are base models or full agents.
- Perché pubblicarla
- It is the first public benchmark from a major payment network on AI agents that make purchases, and it arrives while Visa and others are building agent-initiated payments. The finding that mistakes still turn into purchases directly affects consumers and the question of liability. The topic is new compared with what we have already published.