Nvidia maps KV cache across models: skip the re-prefill tax

Everybody selling you an AI gateway makes the same pitch: route the easy work to a cheap model, escalate the hard work to an expensive one. The pitch skips the bill. Switching models mid-conversation means the new model re-reads everything said so far, from scratch. Nvidia researchers just published a way around that, and the method is old-fashioned linear regression rather than anything exotic.

What actually happened

The paper is Cross-Model KV Cache Transfer in LLM Families, posted to arXiv on August 4 by a team of Nvidia researchers and picked up in VentureBeat's coverage on August 21.

The mechanism: when a small model and a large model come from the same family — Qwen3 14B and Qwen3 32B, say — the internal key/value state one builds while reading a prompt is close enough to the other's that you can map between them with a fitted linear function. Not a trained neural network. Per-head ridge regression, calibrated on 500 text sequences of 1,024 tokens.

The abstract's numbers are modest and honest. On Qwen3 14B→32B, one source layer explains 56% of the variance in the target's keys and 32% in values; using several source layers pushes that to 79% and 65%. Across four model pairs, the linear mapper retains 73–98% of the accuracy the receiving model would get if it read the prompt itself. The mapping runs 2.7–25x faster than re-prefilling. VentureBeat reports one example at 278 milliseconds against roughly 7 seconds for a 32,768-token cache.

Why cross-model KV cache transfer matters for your business

It is research, not a product. There is no API, no config flag, no vendor shipping this next week. Do not replatform around it.

It tells you where your cost curve is heading. Right now the honest advice on model routing is: decide which model handles a task before the task starts, because switching partway means paying to re-read the context. If cache transfer lands in serving stacks, that constraint loosens — escalate at the point you actually discover the work is hard, not at the point you guessed.

Same-family only. Every result here is within a model family. It does not move state from Claude to Gemini. Portability between vendors is still an application-layer problem you solve by keeping your prompts and tools model-agnostic.

73% accuracy retention is a real loss. The low end of that band means one in four answers drifts. Anything touching money, inventory, or a customer record needs the full read.

Sources: Cross-Model KV Cache Transfer in LLM Families (arXiv), VentureBeat.