An 8B agent hit 96.9% — read the unseen split

The most expensive line in your agent bill is the model, so every operator running an AI agent wants the same thing: a smaller model that does the same work. A paper from Meta AI and the University of Illinois Urbana–Champaign, EvoHarness-RL, took an 8B open model from 47.9% to 96.9% on a long-horizon benchmark without touching model size. The trick was training the agent harness — the scaffolding around the model — instead of the model's reasoning.

That is a real result. It is also being reported one number short.

What actually happened

The setup: Qwen3-8B running ALFWorld, a text environment of multi-step household tasks that punishes agents for losing track of state. Baseline ReAct scored 47.9%. EvoHarness-RL scored 96.9% — a 49-point jump.

The method is the interesting part. The authors expose three pieces of external state to the model as actions it can take: Belief (a world-state tracker), Progress (a committed subgoal plan), and Experience (a skill bank carried across episodes). Supervised fine-tuning teaches the model that action space. Then cost-aware GRPO teaches it when reading, writing, and consolidating that state is worth the tokens.

The behavior that emerges is worth stealing even if you never train anything. Harness calls per episode fall toward roughly one as training proceeds — the model internalizes the recurring patterns and stops asking the scratchpad questions it already knows the answer to.

Now the number the headlines skip. Claude Opus 4.5 running plain ReAct scored 96.4%, so 96.9% reads as parity. But on ALFWorld's unseen split, EvoHarness-RL scored 86.6% — a ten-point drop on tasks it was not trained against. And Opus 4.5 with the untrained harness scored 98.5%. The 8B model matches the frontier model on the distribution it was trained for, and loses ground off it.

Why the harness matters more than the model for your business

Most small-business agent work is narrow and repetitive by nature: reconcile these invoices, triage these tickets, update these listings. That is a seen distribution. It is exactly where a trained-harness 8B model is a defensible cost decision, and where a 10-point generalization gap costs you nothing.

The transferable lesson is cheaper than fine-tuning. Almost every failing agent we get called in on fails the same way — it re-reads context it already processed, forgets a subgoal four steps back, and burns its budget re-deriving facts. The paper's answer is structural: give the agent explicit, writable state, and make it decide when to touch it. You can implement Belief/Progress/Experience as three files and a rule about when to write to them. No RL required.

What you should not do is read "8B matches Opus" and re-platform onto a small model for open-ended work. The unseen split is the honest preview of what happens when your workflow changes and your training data does not.

Sources: arXiv — EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents, VentureBeat.