An 8B agent scored like Opus 4.5 — the harness did it

Every time we scope an agent build, someone asks which model. It is almost never the right first question. A paper making the rounds today puts a number on that: an 8B agent trained with a learned runtime harness hit 96.9% on ALFWorld, while Claude Opus 4.5 out of the box scored 96.4% on the same benchmark. Same task, roughly the same result, two very different invoices.

What actually happened

The paper is EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents, submitted to arXiv on August 5 and covered by VentureBeat on August 28, which attributes the work to researchers at Meta and the University of Illinois Urbana-Champaign.

The idea is small and specific. Most agent scaffolding — memory, scratchpads, skill libraries — is bolted on by the framework and fires on every turn whether it helps or not. EvoHarness-RL instead exposes the scaffold to the model as four meta-actions it can choose to call: track (read environment state), commit (write a subgoal update), recall (pull cross-episode knowledge), and note (record a new insight). The model learns when to reach for its own memory.

Training runs in two stages: supervised fine-tuning to teach the meta-action semantics, then cost-aware GRPO that rewards task success while penalizing wasted harness calls. That penalty is the point — harness actions burn interaction budget exactly like environment actions do.

The numbers from the paper's main table:

• Qwen3-8B with plain ReAct: 47.9% → with EvoHarness-RL: 96.9%
• SkillRL (Qwen2.5-7B): 89.9%; SkillOS (Qwen3-8B): 80.2%
• GPT-4.1: 47.9% → 70.0% with the harness (+22.1)
• GPT-5: 60.7% → 85.0% (+25.7)
• Claude Opus 4.5: 96.4% baseline → 98.5% with the harness

Read that last line before you cancel anything. The frontier model still wins when it gets the same scaffolding. And every number here comes from one benchmark — ALFWorld, a text-based household-task environment. No coding, no tool APIs, no messy real inputs. Treat it as a signal, not a result.

Why the harness matters for your business

Small teams shopping for agents get sold on model quality because that is what has a price tag and a leaderboard. Then the thing goes to production and fails on turn nine of a twelve-step workflow — not because the model was dumb, but because it lost track of what it had already done.

The paper's contribution is evidence that the fix lives in the layer you actually control. You cannot retrain Opus. You can absolutely change how state gets written, when it gets read back, and what it costs your agent to look something up. That layer is code. It is yours, it is testable, and it does not reprice in November.

Two things this changes in how we scope work. First, before anyone upgrades a model tier to fix reliability, instrument the failure: is it reasoning or is it amnesia? Those have different fixes and wildly different bills. Second, if your workflow is narrow and repetitive — order triage, returns, intake, reconciliation — a small open-weight model with a well-built harness is a real candidate, and one you can host.

The honest catch is the training. Getting an 8B model to that number took SFT plus RL against a benchmark environment, not a prompt. Most operators will not run GRPO. What most operators can copy is the design: make memory an explicit, budgeted action instead of an always-on middleware tax.

Sources: arXiv 2608.05446, VentureBeat.