Real-SWE: best coding agent scores 38.8% on real repos

Specific Labs published Real-SWE this month, a coding-agent benchmark built from tasks in private production codebases instead of public GitHub issues. Eight frontier models ran it. The best score was 38.8%. If you have been sizing an AI engineering budget off public benchmark numbers, this is the correction.

What actually happened

Real-SWE, from authors Snagnik Das, Siddhant Paliwal, and Janak Sunil, sources its tasks from actual engineering work at real companies — billing logic, migrations, infrastructure changes, and business rules that only exist inside one company's head. The sample tasks come from codebases including a consumer events platform with 200K+ users, a consumer fintech handling 100K+ bank statements, and enterprise sales tooling.

The published results, in order: Fable 5.1 at 38.8%, GPT-6 Astra at 33.8%, Gemini 3.8 Flash at 31.2%, GLM 5.3 at 28.8%, Grok 4.6 and Muse Spark 1.3 tied at 23.8%, Kimi K3 at 18.8%, and GPT-5.6 Sol at 16.2%.

One methodological choice matters more than the ranking. Real-SWE evaluates model-and-harness combinations, using each vendor's native agent harness rather than a neutral scaffold. That is the honest way to measure it, because that is how the tool arrives on your laptop. It also means a score is a verdict on the whole product, not on the weights.

Why it matters for your business

Public coding benchmarks are built from public repositories, which means well-documented projects with visible issue threads, readable test suites, and conventions a model has already seen in training. Your repo is not that. Your repo has a payments module nobody has touched since the contractor left, a migration path with three live exceptions, and business rules that live in a Slack thread from 2024.

The gap between a headline benchmark score and 38.8% is the cost of that context — and it is the part you pay for.

Stop treating benchmark scores as procurement inputs. They rank tools. They do not size your work. A model that leads a public leaderboard and one that trails it may both fail on the same ambiguous ticket in your codebase.

Build a ten-task eval from your own closed tickets. Pull ten merged PRs from the last quarter, revert them, hand the original ticket to each agent you are considering, and count passes. It costs an afternoon and it is the only number that describes your repo.

Budget for the 60% the agent misses. Agents are useful at a 39% pass rate — that is real leverage. But a plan that assumes the agent closes the ticket is a plan built on a public benchmark. Assume it drafts, and staff the review.

Sources: Real-SWE, Specific Labs.