NVIDIA NeMo Switchyard: route agent work, cut cost

The expensive part of an agent is not the hard step. It is the four hundred cheap steps around it — tool calls, result checks, subagent handoffs — all billed at frontier prices because that is the one model your code knows how to call. NVIDIA shipped two things yesterday aimed at exactly that: a small open-weight model built for the cheap steps, and an open-source model routing library that decides which model gets which request.

What actually happened

NVIDIA announced Nemotron 3.5 Lightning and NeMo Switchyard on August 11.

Nemotron 3.5 Lightning is a 30-billion-parameter mixture-of-experts model with open weights, published on Hugging Face, ModelScope, OpenRouter, and build.nvidia.com as an NVIDIA NIM microservice. NVIDIA claims up to 4x faster output speed against models in its class, which it translates to roughly 30% faster completion on agentic tasks. It runs on RTX PCs, DGX Spark and Station, Jetson, workstations, data centers, and cloud — and it is post-trainable on your own data through NeMo.

NeMo Switchyard is the more interesting half. It is an open-source routing library, published on GitHub, that sits between your agent and the models and keeps routing logic separate from providers. It routes on three signals: which model can actually do the task, what the task costs in latency and dollars, and what the infrastructure is doing right now.

The numbers NVIDIA published, in its developer writeup:

• A LangChain benchmark showed a 74% cost reduction using escalation routing between Lightning and Claude Opus 4.8
• Cognition, running it on FrontierCode, reported 50.6% accuracy at $3.11 mean cost — within 2.8 percentage points of Opus 5, at about 28% lower mean cost

Read those as vendor benchmarks, because they are. Named integrations include LangChain, Kong, LiteLLM, and Boomi.

Why model routing matters for your business

Escalation routing is the pattern worth stealing even if you never install Switchyard. Try the cheap model first, check the result, and pay for the expensive one only when the cheap one fails the check. That is not an NVIDIA idea — it is the same shape as Alibaba's classify-first support system — and it works because your agent's workload is not uniformly hard. Most of it is plumbing.

The catch is the check step. Routing only saves money if you can tell a bad cheap answer from a good one without asking the expensive model. For a code agent that is a test suite. For a data extraction agent that is a schema validator. For a support agent that is a confidence threshold. If you cannot name your check, routing just moves the failures downstream where they cost more.

The other reason to care: a router is an abstraction layer you own. Providers retire aliases, reprice tiers, and gate capacity. A routing layer with providers behind an interface turns each of those events into a config change instead of a sprint.

Sources: NVIDIA Blog, NVIDIA Developer Blog.