AKV cache engineering: cut AI agent costs 27.7% vs summarizing

Long AI agent runs get expensive because the agent keeps re-reading its own history. A team led by researchers at Tsinghua University has open-sourced AKV, a serving engine for what they call cache engineering. The agent removes old tool outputs from its KV cache instead of summarizing the conversation. In the authors' tests, that cut average cost by 27.7% compared with summarization, and benchmark scores went up by 1.7 points. Those are the authors' own numbers. The habit behind them is useful anyway.

What actually happened

The paper, Cache Engineering for Agents, was posted to arXiv on October 20. The code is on GitHub under Apache-2.0.

The problem is familiar. An agent calls a tool, gets 4,000 tokens of JSON back, and carries that JSON through every later step. The usual fix is context engineering: trim or summarize the history. But a rewritten history breaks the cached state. The server must re-prefill everything after the edit, and you pay for that compute again.

AKV gives the agent two direct operations on its cache:

• Drop removes the KV entries of chosen messages from future attention. It does not change the trajectory text.
• Reposition compacts the position encodings of the tokens that stay.

AKV is built on SGLang and uses its OpenAI-compatible /v1/chat/completions endpoint, with extra drop_message and reposition fields in the request. The README lists support for Qwen2/Qwen3 (including MoE), GPT-OSS and the MiniMax-M2 family. The quick start uses GPT-OSS-120B.

The paper reports up to 1.2x output throughput and a 1.5x P95 latency speedup. A rolling policy that drops old tool responses gave the 27.7% cost cut. None of this has been reproduced by a third party yet.

Why it matters for your business

You cannot run AKV against OpenAI or Anthropic. It needs a self-hosted open-weight model, which most small teams do not run. Take the lesson instead: tool outputs are the expensive part of an agent's context, and most of them are dead after one step.

What we do on hosted APIs:

• Return less from tools. A tool that returns 40 fields when the agent needs 3 makes every later call cost more. Shape the response in the tool, not in the prompt.
• Keep the prefix stable. Put system prompt, tool schemas and static docs first so provider prompt caching holds. Do not rewrite early messages mid-run.
• Expire tool results on purpose. After the agent has used a result, swap it for a one-line note like "order 1182 fetched, status shipped".
• Measure cost per finished task, not per call. That is the number this paper improves.

Sources: arXiv: Cache Engineering for Agents, AKV on GitHub.