Repo-To-Skill: 5,000 skills lifted an agent 134%

The cheapest upgrade to an AI agent is usually not a better model. A paper posted to arXiv on September 2 puts a number on that: hold the model, the harness, and the compute budget fixed, hand the agent a library of distilled skills, and its MLE-bench score goes up 134.3%. The work comes from the Beijing Academy of Artificial Intelligence with co-authors at USTC, Renmin University, and Hong Kong Polytechnic.

What actually happened

Repo-To-Skill names the gap it is filling: operational knowledge. The authors describe it as "the know-how that separates knowing a method from making it work" — the kind of thing that lives in a repository's setup scripts and default hyperparameters, written for humans and far too long to paste into a context window.

Their system, DisCo, distills that knowledge two ways. Task-agnostic distillation compresses widely used repositories into reusable skills ahead of time. Task-oriented distillation generates the skills a specific job needs on demand. Run across the open ecosystem, the first mode produced the AREX-Skill Library: 5,000+ verified skills from 1,000 ML repositories, organized into 20 areas and 178 capability families, published on GitHub.

The evaluation is the useful part. With a GPT-5.5 backbone, the same research harness, and the same downstream execution budget, the skill-equipped agent scored 134.3% higher on MLE-bench, 34.4% higher on PaperBench, 9.2% higher on FrontierCS, and 14.0% higher on PassNet than the identical agent without skills. This is a preprint with self-reported numbers and no independent replication yet — read it as a strong signal, not a settled result.

Why agent skills matter for your business

We have written before that the harness around a model moves outcomes as much as the model does. This paper points at the layer above the harness: the accumulated, verified know-how of doing your specific job, in a form an agent can load.

You already own that raw material and it is probably rotting in the wrong format. Your deployment runbook. The three gotchas in your Stripe webhook handling. The reason invoices from one vendor need a different parser. Today that lives in a Notion page nobody opens and in one senior person's head. Distilled into short, verified, task-scoped skill files, it becomes the thing that makes an agent competent at your work rather than generically capable.

Start small and concrete. Take the five tasks your team explains most often to new hires, write each as a compact procedure with the failure modes named, and put them where your coding agent reads context. Verify them the way the paper does — a skill that has not been executed successfully is a guess. Then measure: same model, same prompt, skills on versus off, on a task you actually run. If the delta is real on your work, you have found a cheaper lever than the next model upgrade.

Sources: arXiv:2609.02749, AREX-Skill on GitHub.