Edge0 runs a 35B model in 3GB of RAM by streaming off SSD

The reason you rent inference is RAM. A model that fits in memory runs on your hardware; a model that does not runs on somebody's rented H100 at their margin. Edge0 attacks that constraint directly: a paper posted September 16 and an Apache 2.0 preview checkpoint that keep a 35B mixture-of-experts model on the SSD and stream the experts in as the model asks for them. Peak active memory: under 3GB.

What actually happened

The paper — "The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction", submitted September 16, 2026 — names the problem that makes naive SSD offloading useless. In a mixture-of-experts model, layer N+1's experts cannot be chosen until layer N produces output. So the disk reads start too late to hide behind compute, and you sit there waiting on I/O.

Edge0's fix is a prerouter: a small per-layer head that predicts the next layer's routing one token ahead. The prediction is not a hint that gets checked — it is the routing. Nothing gets dropped, and the reads start early enough to overlap with the math. An unmerged recovery LoRA, trained on the student path, pays back the quality lost to int4 quantization and to replacing the real router. The paper reports accuracy within a few points of the fp16 teacher across five public benchmarks.

The published preview checkpoint is where the numbers get concrete. Edge0-35B-A3B-preview sits on a Qwen3.6-35B-A3B base — 256 experts, 4 active per token — ships at 4-bit with LoRA and prerouter adapters, and is Apache 2.0. On a Mac mini M4 Pro with 24GB, the card reports 14.9–17.7 tok/s decode, 113 tok/s prefill cold and 140 warm, and 2.9 GiB peak active memory, at an average 3.9-point drop against the fp16 base. The paper's headline figure is ~20 tok/s on a 24GB consumer machine. Believe the card's measured range — that is the one you will see.

Why cheap local inference matters for your business

Fifteen tokens a second is not a chatbot. It is roughly reading speed, which makes it wrong for live customer chat and right for everything that runs while nobody is watching: classifying an inbox overnight, tagging support tickets, extracting fields from a stack of PDFs, drafting first passes a human will edit in the morning. That is most of the AI work a small business actually needs, and it is the work whose API bill scales with volume rather than with headcount.

The architectural point is that the bottleneck moves. You are no longer buying RAM, you are buying sustained random-read bandwidth — the drive matters as much as the chip. Price a machine accordingly before you assume the cheap box works.

Two honest caveats. This is a preview checkpoint against a research paper, not a supported product; the Apache 2.0 license means you can run it, not that anyone will fix it. And a prerouter is a prediction — a mispredicted route is an accepted approximation baked into the design, not a bug you tune away. Combined with the ~4-point accuracy gap, that means you do not put this where a wrong answer costs money. You put it where a wrong answer costs a minute of somebody's review.

Sources: arXiv:2609.18063, Edge0-35B-A3B-preview on Hugging Face.