Thinkingbox: agents hit 65% once, 25% every time

Every agent demo you have ever watched was a pass@1. Somebody ran the task once, it worked, and that clip became the pitch. A new benchmark runs the same business workflows twenty times each, and it surfaces the number nobody demos: agent reliability under repetition. The strongest model tested finished a task correctly on its first attempt 65.36% of the time. It finished correctly on all twenty attempts 25.25% of the time.

What actually happened

Thinkingbox went up on arXiv on August 20, from a team spanning Microsoft, the University of Pittsburgh, Northwestern, and UC Irvine, with the code at microsoft/thinkingbox.

The sandbox part gives an agent isolated, MCP-compatible tool sessions and records the complete execution trace. The important design choice is the grading: it evaluates the terminal backend state, not the agent's summary of what it did. Task-specific executable checks reject wrong effects, missing effects, and extra effects.

On top of that sits Thinkingbox-bench — 507 policy-conditioned workflows across five domains: retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support. Multi-turn work, with policies to follow and information the agent has to go get.

Three numbers from the paper:

• 65.36% pass@1 for the strongest model — correct on the first try
• 91.12% of tasks solved at least once across twenty attempts
• 25.25% pass^20 — solved on every one of twenty attempts

And the one we keep coming back to. Across 121,680 valid recorded trials over 12 models, 79,853 failed. Of those failures, 67,763 — 84.86% — terminated cleanly: well-formed final response, no dangling question, agent reporting done.

Why agent reliability matters for your business

Sit with that 84.86%. Five out of six failures looked like successes from the outside. The run completed, the transcript read fine, and the record was wrong. If your monitoring watches for errors, timeouts, and exceptions, it caught none of them.

Then look at the spread between 91.12% and 25.25%. That is the distance between "an agent can do this" and "an agent does this." Nine in ten of these tasks are solvable at least once — which is exactly what a demo proves. One in four is solvable every time, which is what a business needs. A refund workflow that works 19 times in 20 isn't 95% good. It's one wrong refund a week, and you hear about it from the customer.

The method this suggests is cheap and nobody does it: before an agent touches a system of record, run it twenty times on the same input and diff the resulting state after each run. Not the output. The state.

Sources: arXiv:2608.19741, microsoft/thinkingbox on GitHub.