Coding agent harness study: 3x the cost, same accuracy

The coding agent harness you picked may be tripling your bill without buying you a single extra fix. A study posted to arXiv this month, What Does a Harness Buy? Tokens, Mostly, ran the same models through Claude Code, OpenCode and a 100-line harness called mini-SWE-agent. Accuracy came out roughly the same. Cost per task did not — it varied by up to 3x. VentureBeat picked it up on October 23.

What actually happened

Researchers Yangze Liu and Zhongyi Han ran 492 offline SWE-bench Verified tasks in isolated containers. Two locally served Qwen models ran a 447-task pool. Five models — including DeepSeek-V4-Flash and GLM-5.3-Flash over API — ran a 45-task hard subset, with Claude Opus 5 inside Claude Code as an anchor.

The headline results, from the paper's abstract and VentureBeat's write-up:

• On the 447-task pool, the heaviest and the lightest harness landed within five points of each other. Claude Code and mini-SWE-agent were within about 2 points.
• OpenCode was the one harness with a measurable effect, trailing by up to 9 points. On one model, about half that gap came from runs hitting the 32,000-token output limit.
• On the hard subset, swapping harnesses flipped 13% of tasks between pass and fail — the same rate as simply rerunning an unchanged setup.
• Fixed context before the agent does anything: about 16.6k tokens for Claude Code, 7k for OpenCode, 829 for mini-SWE-agent.

The authors also flag their own limits. Forty-five tasks only detect a 13-point gap about half the time. Small benchmark runs are mostly noise.

Why it matters for your business

Most small teams choose a coding agent the way they choose a text editor: whatever the last demo used. This study says that choice is mostly a cost decision, not a quality decision — at least on bug-fix work like SWE-bench. A harness with a 16k-token system prompt and a big tool schema charges you that overhead on every call, and prompt caching (96–99% hit rates in the study) softens it but does not erase it.

Two practical moves. First, stop trusting leaderboard deltas of a few points; this paper shows rerun noise is that large. Second, log cost per completed task on your own repo, across two harnesses, for a week. That number tells you more than any benchmark. We covered the same lesson from the vendor side when Writer cut cost 41% by changing only the harness. Now there is independent data pointing the same way.

One caveat: this is SWE-bench-style bug fixing. Long, multi-step feature work may reward a heavier harness. Test on the work you actually do.

Sources: arXiv: What Does a Harness Buy? Tokens, Mostly, VentureBeat.