Programmatic tool calling beat JSON in 11 of 14 models

Almost every agent in production today calls tools the same way: the model emits a JSON blob, your harness parses it, runs one function, hands back the result, and the model goes again. A paper posted to arXiv on August 6 tested the alternative — let the model write a short script instead — and the numbers favor the script. Programmatic tool calling matched or beat native JSON tool calling in 11 of 14 models evaluated. If you build agents for a living, this is a harness change, not a model change.

What actually happened

Ishan Patel, Sahil Sen, Elias Lumer and Vamse Kumar Subbiah published The Bitter Lesson of Tool Calling and ran the comparison across 14 language models on BFCL v4.

The setup is the interesting part. In programmatic tool calling (PTC), tools are exposed to the model as typed Python stubs. The model invokes them by writing code, and execution plus results happen inside a single agent turn — no round trip per call. Chaining and parallelism come free, because that is what code does.

The results the authors report:

• PTC matched or exceeded native JSON tool calling in 11 of 14 models
• The GPT-5.6 family gained 10.6% over the JSON baseline
• Under parallel fan-out, PTC matched or beat the baseline in 13 of 14 models
• Under context rot, PTC stayed stable while the JSON baseline degraded 2.3% on average

The authors also note the effect tracks model capability across release generations — newer, more code-fluent models benefit more. That is the "bitter lesson" in the title: the rigid schema was scaffolding for models that could not write code well. They can now.

Why programmatic tool calling matters for your business

Look at where your agent spends money and time. A JSON-calling agent that touches six tools burns six round trips, and every one of them replays the whole conversation into the context window. Six turns of prompt tokens, six chances for the model to lose the thread, six places for a retry to fire. A PTC agent writes one script, runs it, reads one result.

That is a token bill and a latency profile, not an academic score. The parallel fan-out result is where it shows up hardest — checking twelve SKUs against three suppliers is a for loop, not twelve negotiated turns.

Three honest caveats before anybody rewrites a harness this week. BFCL v4 is a benchmark, not your workload — the only number that decides this is your own eval on your own tools. Letting a model write and execute code raises the blast radius, so the sandbox, the egress rules, and the allowlist stop being optional; the labs have already demonstrated what happens when those are wrong. And MCP is not the loser here — MCP describes what a tool is and where it lives. How your harness invokes it is a separate decision, and this paper is about that second layer.

The practical move is small. Pick the one agent in your stack that makes the most tool calls per task. Instrument it: calls per task, tokens per task, wall-clock per task. Then build a code-execution path for the same tools and run both against fifty real tasks from your logs. You will know inside a day whether the 10% is yours.

Sources: The Bitter Lesson of Tool Calling (arXiv:2608.06370), arXiv HTML full text.