Kog's 3,000 tokens/sec claim: read what was measured

A French startup with eleven people says it can make the GPUs you already rent run large language models dramatically faster — no exotic silicon, no new hardware order. Kog demonstrated 3,000 output tokens per second on a single request, on a standard eight-GPU node. That is genuinely in the speed class people usually associate with dedicated inference chips. It is also a much narrower result than the headline number suggests, and the gap between the two is the whole story.

What actually happened

Kog raised roughly $5 million in seed funding, co-led by Varsity VC, with backing from Bpifrance and the French Tech 2030 programme and support from cloud provider Scaleway (TechCrunch). Founder and CEO Gaël Delalleau came out of solid-state physics and offensive security — which shows in the approach. The Kog Inference Engine attacks the microsecond-level overhead in conventional inference stacks rather than betting on a new chip.

The measured result, from Kog's own writeup: 3,000 output tokens/sec per request on 8× AMD MI300X, and 2,100 on 8× NVIDIA H200. Batch size 1. No quantization, no speculative decoding. Impressive engineering, cleanly reported.

Now the caveats, which Kog does not hide. The model in that benchmark is Laneformer 2B — their own small coding model, two billion parameters. The numbers for frontier mixture-of-experts models are presented as projections, in a 1,000–5,000 tokens/sec band, not measurements. And there are no published comparisons against vLLM, SGLang or TensorRT-LLM, the stacks you would actually be replacing. Per TechCrunch, the company's own Series A gate is demonstrating 10x on a major model, with a September milestone in view. The "30x faster" framing is a promise about work not yet finished.

Why inference speed matters for your business

Speed on the inference layer is the closest thing to a free lunch in this stack, because it cuts two bills at once: fewer GPU-seconds per task, and shorter wall-clock time in workflows where a human is waiting. For agentic work — a coding agent chewing through a repo, a support agent chaining tool calls — latency compounds across every step in the loop.

But this is exactly the class of claim to evaluate on your own workload, not on a vendor's chart. A 2B model at batch size 1 is the friendliest possible case: it fits comfortably, and there is no contention. Your production reality is a bigger model, concurrent users, and a batching policy. Those are the conditions that decide your actual cost per task.

So: interesting company, honest benchmark, wrong number to plan around. Put it on the watch list, and check back when a frontier model runs on it. In the meantime, the leverage available to you today is measuring cost per completed task on your current stack, and keeping the model layer swappable so a real 10x — whoever ships it — is a config change and not a rewrite.

Sources: TechCrunch, Kog engineering blog.