Prefix Sliding: 3x faster long reasoning, no retraining
A new paper caps memory during long reasoning by keeping only the prefix and a recent window — 3x faster thinking time on existing models, no training required.
Every long agent run you pay for has the same hidden cost: the model is carrying its entire chain of thought in attention, forever, whether or not any of it still matters. A paper posted to arXiv on August 26 says most of it does not — and that throwing it away makes existing models 3x faster on long reasoning with no retraining at all. The method is called Prefix Sliding, and it is the rare research result that maps directly onto a line item in your bill.
What actually happened
Prefix Sliding for efficient test-time scaling (arXiv:2608.26070) comes from a group including Niklas Muennighoff, Percy Liang, Andrew Y. Ng, Luke Zettlemoyer, Yejin Choi, Jason Wei, and Mike Lewis.
The observation is simple: "most intermediate reasoning tokens lose importance as the model continues reasoning." So Prefix Sliding discards tokens that are not part of either the prefix or a window of the last few thousand tokens. The prefix holds the instructions and the available tools. The recent window holds what the model is actually working on right now. Everything in the middle gets dropped.
That caps total memory regardless of how long the model reasons — which is the whole point. Two results:
- Training-free: existing models run 3x faster while maintaining performance. The paper measures this as average thinking time per sample in seconds, which the authors argue is what users actually experience — not tokens per second.
- With reinforcement-learning training on top, it scales to reasoning traces beyond a hundred thousand tokens with better performance than not doing it.
Evaluation runs on Qwen3-1.7B across GPQA, MATH500, AIME25, LiveCodeBench, and HealthBench, with scaling to 7B for the training work. Ablations compare against keeping only the last k tokens, summarizing the middle into a fresh context, and a plain sliding window with no prefix retention. Prefix Sliding beats all three.
Why long-reasoning cost matters for your business
If you run agents on real work — a nightly reconciliation, a support triage loop, a code migration — you have already met the failure this addresses. The run gets long, latency climbs superlinearly, and the invoice climbs with it. The naive fix teams reach for is summarizing the middle of the context. This paper says the dumber approach beats it: just drop the middle and keep the instructions.
Nothing here is a product you can install this week. Qwen3-1.7B is a small model and one paper is not a deployment. But three things are worth doing regardless:
Measure thinking time per task, not tokens per second. The paper is right about which number matters. If your agent dashboards report throughput, you are watching a metric that does not predict your invoice or your users' patience.
Stop paying for context you are not using. Whatever your framework does with long agent traces, know it. Many keep everything by default because that is the safe engineering choice, and safe is expensive at 100,000 tokens.
Keep the prefix disciplined. Prefix Sliding works because the prefix is where the instructions and tools live and stays cheap to retain. That is also good practice in a system prompt you wrote by hand: stable instructions at the top, churn at the bottom.
Expect this to arrive in inference stacks quietly, as a flag. When your provider ships it, the win shows up as lower latency on your longest runs — which are the ones that were costing you the most anyway.
Key takeaways
- Prefix Sliding (arXiv:2608.26070, August 26, 2026) discards reasoning tokens outside the prefix and a recent few-thousand-token window
- Training-free, it makes existing models 3x faster while maintaining performance, measured as average thinking time per sample
- With RL training it enables reasoning traces beyond 100,000 tokens
- Evaluated on Qwen3-1.7B across GPQA, MATH500, AIME25, LiveCodeBench, and HealthBench
- It beats summarizing the intermediate tokens and beats a vanilla sliding window in ablations
- Operator takeaway: measure thinking time per completed task, and know what your agent framework retains by default
Long agent runs are where automation quietly gets expensive. We instrument the loops we build — cost and thinking time per completed task, not tokens per second — so you can see which runs are worth the money before the invoice tells you. See what we've built and what it costs to run, or estimate the payback on automating one recurring workflow.
Sources: arXiv:2608.26070, Prefix Sliding for efficient test-time scaling, full paper HTML.
- #inference-costs
- #reasoning-models
- #test-time-scaling
- #agents
- #research
Tommy Rush — Founder, Rush Commerce
Operator turned builder. 15+ years running operations — now shipping the systems businesses run on. More
Get The Rush Report weekly — one email, zero fluff.
Keep reading
OpenAI retires whisper-1 and the gpt-4o-transcribe line
Four OpenAI transcription models shut off February 26, 2027. If your call logs, voicemails, or meeting notes run through whisper-1, you have a migration and a re-benchmark to schedule.
Read itA third of firms skipped a software purchase to build it
McKinsey's State of AI 2026 survey: nearly a third of organizations skipped buying software because agentic coding tools let them build it. The ROI numbers say be selective.
Read it