mac-code: The Mac SSD as RAM for Giant Local Models
A local coding agent that treats model weights, expert routing, and conversation memory as separate layers. The result is a small-Mac system that can run surprisingly large MoE models without surrendering to swap death.
- mac-code's real invention is not compression, but a memory hierarchy that lets SSDs feed inference without forcing the whole model into RAM.
- Mixture-of-Experts models make that hierarchy practical because each token only needs a subset of experts, not the full weight set.
- The repo pairs flash streaming with agent routing, so the same system handles both large-model inference and day-to-day coding workflows.
- Its trade-off is control over convenience: explicit I/O beats mmap, but the approach stays experimental and hardware-sensitive.
mac-code is not trying to win a purity contest about what counts as local inference. It is trying to make a 16GB Mac do something that usually sounds impossible: keep a large MoE model usable without collapsing into swap death.
That is the useful frame for the whole repo. The cleverness is not in squeezing everything smaller. It is in deciding what must stay resident, what can be streamed, and what can be forgotten on purpose.
The Mac That Refuses to Swell
mac code — Claude Code, but it runs on your Mac for free. 35B AI agent at 30 tok/s via Apple Silicon flash-paging. $0/month.
That quote is the whole pitch in one breath. The repo treats consumer Apple Silicon as a platform for out-of-core AI, not as a device that must wait for more RAM before it can participate.
The crucial move is philosophical as much as technical. Instead of asking how to fit the model into memory, mac-code asks how to make memory hierarchical enough that the model no longer needs to fit all at once.
Flash Streaming, In Plain English
The repo splits model state into two tiers. Pinned weights, like attention and embeddings, stay hot. Streamed weights, like FFN and expert blocks, come off SSD only when the current token needs them, then get dropped again.
That changes the storage problem from a blunt capacity issue into a scheduling problem. If the right blocks are fetched at the right time, the SSD stops behaving like overflow storage and starts acting like a controlled feeder for inference.
Why MoE Changes the Game
MoE is the enabling trick. A dense model wants much of itself active on every token. An MoE model only activates a subset of experts, so the system can load far less per step.
That is why the repo can talk about a 35B model without sounding naive. The model is large on paper, but the active slice per token is much smaller, which makes selective streaming plausible on small hardware.
| Approach | Memory behavior | I/O behavior | Failure mode |
|---|---|---|---|
| Traditional mmap loading | Depends on the OS page cache and often bloats unpredictably | Pages in and out automatically | Swap death when the working set explodes |
| Naive streaming | Keeps less in RAM, but still reads too much too often | Repeated reads for overlapping work | SSD thrash and sluggish token throughput |
| mac-code flash streaming | Pins the urgent weights and streams the rest by design | Loads only the needed blocks, then discards them | Experimental complexity, but far better control over pressure |
The Trick Behind Union-of-Experts
The sharpest idea in the research folder is not just streaming. It is batching the need.
In batched_moe.py, the system looks at a batch of tokens, collects the experts each one wants, and loads the union instead of loading each request independently. That reduces redundant SSD reads and turns overlapping demand into a single I/O event.
# Conceptual sketch of the union-of-experts idea
batch_requests = [route(token) for token in tokens]
unique_experts = sorted(set().union(*batch_requests))
loaded = load_from_ssd(unique_experts)
outputs = run_batch(tokens, loaded)
That is a deceptively small idea with large consequences. It says the bottleneck is not just how much model you can hold. It is how cleverly you can collapse repeated storage work across a batch.
An Agent That Routes Before It Thinks
The user-facing side of the repo is practical, not decorative. The agent layer routes intent first, then chooses between search, shell, or chat, which keeps the interaction fast and predictable.
That structure matters because it keeps the expensive model work pointed at the right task. A quick classifier can save a lot of unnecessary inference, especially when the underlying system already spends effort managing memory and storage.
The point is not that mac-code is a full IDE replacement. The point is that it behaves like an opinionated local assistant that knows when to search, when to execute, and when to just answer.
Memory That Escapes RAM
The same hierarchy shows up again in the conversation cache. Hot KV state stays close, older prompt state can spill to SSD, and colder memory can be pushed even further out to R2.
That extends the repo's thesis beyond model size. It is also about persistence. The assistant can keep more context alive without forcing all of it to occupy the same expensive tier at the same time.
| Layer | What stays there | Why it matters |
|---|---|---|
| GPU hot cache | Current KV cache | Keeps the active conversation fast |
| SSD warm cache | Older prompt segments | Lets the system page history without crowding RAM |
| R2 cold store | Very old memory | Extends persistence across devices or reboots |
The Trade-Offs Are the Point
This repo makes a deliberate trade. It rejects mmap, uses explicit I/O, and embraces a tighter relationship between model architecture and storage behavior. That gives it control, but not simplicity.
It is also experimental. Much of the code lives in research folders, and that is a clue. The project is a working system and a moving target at the same time.
Compared with cloud assistants like Claude Code, the appeal is obvious: local, private, and potentially free to run. Compared with polished local tools like Cursor or OpenCode, the difference is narrower but deeper. mac-code is trying to change the boundary conditions of what a small Mac can host.
That is why it feels interesting. It is not just another agent wrapper. It is a hardware argument disguised as a coding assistant.
