Edge0: The MoE Engine That Hides SSD Latency in Plain Sight

A deep dive into the prerouter, streaming expert offload, and Recover-LoRA, the three ideas that let a 35B-class model behave like it belongs on a MacBook.

10 min read • View on GitHub • More from Edge0-AI

A MacBook sits beside a slim SSD tower while a narrow stream of expert drawers moves toward the laptop in carefully timed bursts. A small clipboard-wielding figure stands between them, coordinating what gets fetched next. The scene explains Edge0’s core trick: it does not load everything into memory, it schedules the right weights so decoding never waits on disk longer than necessary.
Edge0 treats SSD access like a planned handoff, not a stall.
Key Takeaways

Edge0’s best trick is easy to miss because it looks like an inference engine. Under the hood, it behaves more like a storage scheduler for a sparse model: resident layers keep moving, expert weights arrive just in time, and a prerouter tries to keep the SSD one token ahead of the decoder. That shifts the question from “can this model fit?” to “can the next expert get here before the forward pass needs it?”

Why Edge0 Feels Like a Storage Engine, Not Just an Inference Library

The project is built around a hard constraint: the whole model does not live in memory. Edge0 splits execution into resident and streaming paths, then overlaps the expensive part, loading expert weights, with useful work already happening on the current token. That is why the repo feels unusually disciplined. It is not trying to win by brute force. It is trying to make latency disappear.

The README puts the thesis plainly: “edge0 is an open-source streaming MoE inference framework. It generalizes the production-proven recipe — SSD expert offload + Recover-LoRA + prerouter routing prediction — into an extensible framework.” That combination is the point. Each piece solves a different failure mode of local inference.

The Prerouter Is the Trick

The prerouter is a one-token-ahead scheduler. Its job is to make storage behave like a pipeline, not a wall.

Two adjacent routes inside a token-generation machine overlap like interlocking gears. On the left, the current token is decoding. On the right, a small prerouter head points toward the next expert path while arrows feed a disk fetch that arrives just before the next forward pass needs it. The image explains why prediction matters more than raw bandwidth in Edge0.
Edge0’s prerouter is not a cache. It is a learned look-ahead system.

The key detail is that the prerouter is not a dumb cache miss predictor. It uses the current hidden state and expert history to guess the next routing decision, then starts the fetch early enough to cover most of the wait. If the guess is right, the decoder never sees the disk. If it misses, the system still falls back to the normal path. That is a practical trade-off, not a theoretical flex.

How the Engine Stages a Token

At the code level, Edge0Engine is the orchestrator. It handles prefill, then decode, and uses hooks like _step_pre and _step_post to trigger early-submit loads before the next token needs them. That is a small detail with big consequences. It means the engine is always preparing the next move while finishing the current one.

# Pseudocode for the staged token loop
for token in tokens:
    _step_pre(state)          # predict and stage next experts
    resident_forward(state)   # attention and already-loaded experts
    stream_in_experts(state)  # overlap SSD fetch with compute
    _step_post(state)         # commit routing, update history

The backend boundary matters just as much. The repo uses a TensorStore protocol so the engine does not care whether tensors come from MLX today or something else tomorrow. That is how a project starts on Apple Silicon without trapping itself there. The abstraction is the roadmap.

Recover-LoRA Changes the Quality Math

If streaming only made the model small, Edge0 would still be incomplete. Recover-LoRA is the other half of the story. It lets a frozen, read-only quantized base model regain quality through small distilled adapters, so the system can stay lightweight without throwing away too much capability.

ApproachWhat it optimizesBase modelAdaptersWhy it matters
Standard quantizationFootprintUsually mergedMerged or ignoredSmaller, but quality loss is accepted
Recover-LoRAQuality recovery after quantizationFrozen, read-onlySmall unmerged LoRA adaptersOne base model can serve multiple specialized behaviors
Full fine-tuneTask fitModified in placeNot separateHigh quality, but expensive to maintain and swap

That operational advantage is easy to underestimate. One base model can support multiple personalities or deployment targets without rewriting the whole artifact. Swapping a tiny adapter is a much cleaner commercial story than shipping several large merged checkpoints. It is also easier to govern when the base stays immutable.

How Edge0 Differs From Nearby Projects

ProjectPrimary optimizationHardware focusExpert predictionStreaming offloadAdapter strategyMain bottleneck addressed
Edge0MoE expert schedulingApple Silicon firstYesYesRecover-LoRA, unmergedSSD latency during decode
llama.cppBroad local inferenceCPU and many backendsNoPartial mmap supportStandard quantization and LoRA workflowsGeneral memory and portability
MLXFast Apple Silicon tensor runtimeApple SiliconNoNoExternal to the runtimeLow-level tensor execution
PowerInferPredictor-style sparse loadingConsumer GPUs and local systemsYes, but different targetSelective loadingNot centered on adapter recoveryKeeping sparse models efficient

Edge0 is not trying to be the most general local inference stack. It is narrower and more opinionated. The novelty is the combination: MoE streaming, one-step-ahead prediction, and adapter recovery as a single pipeline. That is different from simply shrinking weights or relying on mmap alone.

A 35B language model running on an iPhone using only 1–2.5 GB of peak memory. No cloud. No remote server. No desktop GPU. Today, we're open-sourcing Edge0 — a framework for running large AI models fully on-device.

Samuel Zeng, Founder / Maintainer · Samuel Zeng (@SamuelZengML) on X

The Team, the Backend Roadmap, and the Discipline

The engineering discipline shows up in the boring details, which is usually where the real signal lives. The project pins MLX tightly, warns against incompatible versions, and keeps a placeholder for CUDA instead of pretending the future is already solved. That says the team understands how fragile backend assumptions can be when asynchronous loading is part of the design.

That is why Edge0 reads like a platform. It has a clear first backend, but it also has a clean seam where another backend can slot in later. If the abstraction holds, the idea scales beyond Apple Silicon. If it does not, the whole latency-hiding story collapses back into a demo.