Visa AI shopping benchmark: agents fail 42.8% of runs

Visa just put AI shopping agents through a real test, and they flunked almost half of it. The new Visa Agentic Commerce Benchmark, released November 5, ran 22 AI models from 12 labs through 15 shopping tasks. They failed 42.8% of the runs. The number that should worry a merchant is the next one: about half of those failed runs still ended in a purchase.

What actually happened

Each task reads like a real customer. Buy a TV, a laptop, a personal care appliance. The shopper drops a few hints and leaves the rest unsaid. In Visa's TV example, the agent has to dig out the budget, format compatibility, HDR10+ support and gaming needs, and it has to decline to buy if nothing fits. A run passes only if the agent finds every requirement and makes the right buy-or-don't call.

The results, per Visa:

• 57.2% overall pass rate. The best models passed 82.2% of attempts. The worst passed 28.9%.
• 54% of failed runs still bought something. Either the agent recommended a product that missed the customer's stated needs, or it never found all the needs in the first place.
• Poor repeatability. Every model got three tries per scenario. All three passed only 34.2% of the time.
• Asking questions pays. The five models that uncovered the most hidden requirements averaged 70.2%. The five that uncovered the fewest averaged 44.4%.
• Price helps, a bit. The five most expensive models passed 67.1% of runs. The five cheapest passed 50.2%.

Visa's release doesn't name models. Fortune reports Google and Meta agents tied at the top near 82%, a Claude agent third at 76%, and an OpenAI agent sixth at 64%. Results and replays are public on Visa Payments Frontier. Visa says it will extend testing to checkout, payment and order placement next.

Why the AI shopping benchmark matters for your business

A failed run that buys the wrong thing isn't a lost sale. It's a return, a support ticket and a customer who now trusts your brand less, even though an agent made the call. We've seen this coming since Visa's own Trust Index found only 23% of consumers trust AI with a payment.

You can't fix the model. You can fix what it reads. Agents fail on the details they can't find, so put the details where they can:

1. Spec every attribute as structured data. Dimensions, ports, HDR formats, voltage, compatibility. Not buried in a paragraph or a JPEG.
2. State what a product is not. "Not compatible with X" helps an agent rule you out correctly instead of guessing you in.
3. Run the test yourself. Ask two or three assistants to shop your top SKUs with a vague brief and see what they get wrong.

Sources: Visa press release, Fortune, Visa Payments Frontier.