Microsoft's Orchard is open — read the agent pass rates, not the pitch

Microsoft Research published its writeup of Orchard on August 3 — an open framework for training and evaluating AI agents, MIT-licensed on GitHub. The interesting thing for anyone deploying agents isn't the framework. It's that Microsoft published the pass rates, and they're a useful reality check against every vendor demo you've watched this year.

What actually happened

Orchard's core is Orchard Env, a Kubernetes-native sandbox service that handles the unglamorous parts of agent work: sandbox lifecycle, command execution, file I/O, network policy, a REST API. On top of it sit three research recipes, each with published results (paper, submitted May 14 and last revised July 30):

• Orchard-SWE (software engineering, Qwen3.5-35B-A3B backbone): 69.7% on SWE-bench Verified with RL, 73.0% with value-model reranking
• Orchard-GUI (a 4B vision-language model for browser navigation): 68.4% average across WebVoyager, Online-Mind2Web, and DeepShop
• Orchard-Claw (personal assistant tasks — email, calendar): 59.6% pass@3 on Claw-Eval, rising to 73.9% with a stronger harness

The training data is public too: 107,000 software-engineering trajectories and 3,070 browser-navigation rollouts on Hugging Face. Deployment scripts stand up a cluster on Azure AKS in roughly 20 minutes; other Kubernetes clusters work as well.

Why agent pass rates matter for your business

Look hard at that last number. 59.6% pass@3 means: on everyday assistant tasks, with three attempts allowed and credit given if any one of them works, the agent still fails four times in ten. That's a research team optimizing for the benchmark, publishing their best result. Your inbox is messier than Claw-Eval.

That's not a reason to skip agents. It's a reason to design for the failure rate instead of the demo. Concretely: an agent that drafts a reply and queues it for one-click approval is a fine bet at 60%. An agent that sends the reply is not. Route the confident cases and escalate the rest — the value is in the 60%, and the damage is all in the other 40% if nobody's watching.

The second takeaway is architectural. Microsoft's contribution here is mostly the environment, not the model — sandboxing, execution, network policy, reusable across domains. That maps to what we keep seeing in real deployments: the model is the part you swap, and the harness around it is the part you own. If your agent stack has the vendor's SDK holding the sandbox, the tool definitions, and the eval loop, you own none of it.

And build an eval set out of your own tasks. Orchard is free proof that you don't need a vendor's benchmark — you need fifty examples of work your business actually does, scored honestly.

Sources: Microsoft Research, arXiv:2605.15040, microsoft/Orchard on GitHub.