Groq 3 LPX hits 3,400 tokens/sec: latency is now a purchase
Nvidia's Groq 3 LPX rack benchmarked at ~3,400 tokens/sec, 4x the next-fastest endpoint. What token speed actually buys you in an agent workflow.
Speed is the one AI variable your customers actually feel. Nobody notices which model wrote the reply; everybody notices a four-second pause. At Hot Chips 2026 Nvidia put numbers on the fastest end of that spectrum, and Groq 3 LPX now benchmarks at roughly four times the next-quickest public endpoint.
What actually happened
Nvidia presented the Groq 3 LPX architecture and said the racks are in full production, about eight months after the $20 billion Groq deal. Nebius is among the first datacenters deploying LPX alongside Nvidia's Vera CPUs and Rubin GPUs, with racks online this year.
The benchmark, as The Register breaks it down: about 3,400 output tokens per second on Gemma 4 31B with a 100,000-token input, against 882 tok/s for the next-fastest platform under the same conditions. The hardware is a deliberate inversion of a GPU. Each LPX rack packs up to 256 LPUs with 128 GB of SRAM total — roughly 500 MB per chip, against 288 GB on a top-spec Rubin — and pushes 150 TB/s of memory bandwidth. Tiny memory, absurd bandwidth.
Read the caveats, because they are structural. Hitting that number takes at least 64 chips for a 31B model. A 671B model like DeepSeek V3 would need roughly 1,342 accelerators, just over five racks. This is a best-case configuration on a mid-size model, not a general-purpose result.
Why token speed matters for your business
Most operators buy inference by the token and think about the bill. The thing they should also price is wall-clock time inside a loop. A single 800-token response at 100 tok/s is eight seconds — annoying. An agent that makes nine sequential model calls to resolve one support ticket is over a minute of dead air, and that is where automation quietly stops feeling automatic.
You do not need an LPX rack. You need to know which of your calls are on the customer's critical path and which are not. Batch work — nightly reconciliation, catalog enrichment, report generation — should run on whatever is cheapest. Interactive work should run on whatever is fastest, and you should be measuring that per call, not per month.
The strategic read on Nvidia buying Groq's team and shipping this: fast inference is becoming a product tier you select, not a constraint you accept. Build so you can point different parts of your workflow at different endpoints.
Key takeaways
- Groq 3 LPX benchmarked at ~3,400 tok/s on Gemma 4 31B at 100K context, ~4x the 882 tok/s runner-up
- Up to 256 LPUs per rack, 128 GB total SRAM, 150 TB/s bandwidth — small memory, huge bandwidth
- Caveat: 64+ chips for a 31B model; a 671B model needs roughly five racks
- In full production, with Nebius among the first deployments this year
- Split your workload: cheapest endpoint for batch, fastest for anything a customer waits on
We instrument latency before we optimize it. Rush Commerce builds agent workflows with per-call timing so you can see which step is costing you the customer, then route that step to the right endpoint. Run the numbers on your automation or tell us where your workflow stalls.
Sources: The Register, CNBC.
- #nvidia
- #groq
- #inference
- #ai-agents
- #latency
Tommy Rush — Founder, Rush Commerce
Operator turned builder. 15+ years running operations — now shipping the systems businesses run on. More
Get The Rush Report weekly — one email, zero fluff.
Keep reading
Radar podcast search: your agents are blind to audio
Particle launched Radar, a podcast search API and MCP server over 130,000 shows. The lesson for operators: agents can only use media somebody indexed first.
Read itGLM-5.3-Flash: MIT open weights that read your documents
Z.ai shipped GLM-5.3-Flash under MIT — 320B total, 18B active, native vision, 1M context. The first cheap open-weight model aimed at document and screenshot work.
Read it