AirLLM: The LLM Engine That Lives on Disk, Not in VRAM
A layer-streaming inference library that makes 70B to 405B models run on consumer hardware by treating GPU memory as a transient workspace, not a home.
- AirLLM makes giant models runnable by moving the model off the GPU and turning VRAM into a temporary workbench.
- Its real bottleneck is disk throughput and orchestration, not FLOPS, which is why it can be useful and still feel slow.
- The project is strongest for batch jobs and offline experimentation, where patience buys access to models that would otherwise be out of reach.
- Quantization, prefetching, and Apple Silicon support widen the footprint, but the core idea stays the same: stream layers, do not store them.
The GPU becomes a temporary workspace
Most local inference stacks assume the model should live in VRAM. AirLLM rejects that assumption. It keeps the weights on disk as sharded layer files, loads one layer into the GPU, runs it, and discards it before the next layer arrives.
That shift sounds small until you feel the consequences. The GPU stops being a warehouse and becomes a workbench. AirLLM is not trying to make the model smaller. It is trying to make memory stop being the reason a model cannot run at all.
AirLLM optimizes inference memory usage, allowing 70B large language models to run inference on a single 4GB GPU card without quantization, distillation and pruning. And you can run 405B Llama3.1 on 8GB vram now.
Why AirLLM exists
AirLLM is aimed at people who hit the VRAM wall first. That includes researchers, independent developers, and prosumers who want access to large models without renting a datacenter GPU just to get started.
The project’s bet is simple. Model size should not be gated entirely by storage in the GPU. If disk and orchestration can carry part of the burden, the hardware conversation changes. Capability becomes less about owning the biggest card and more about tolerating a slower pipeline.
How the layer-streaming engine works
The heart of the library is `AirLLMBaseModel`. It builds an empty model skeleton with `init_empty_weights`, then iterates through layer names in sequence. For each layer, it loads the weights, moves them into VRAM, executes the forward pass, and clears the workspace before the next layer starts.
The clever part is prefetching. A `ThreadPoolExecutor` can load the next layer while the current one is still computing, which overlaps disk I/O with GPU work just enough to keep the relay moving. The model is still serialized by layer, but the pipeline is not entirely idle between steps.
for layer_name in self.layer_names:
next_layer = self.prefetch_executor.submit(load_layer, next_layer_name)
layer = load_layer(layer_name)
output = layer(hidden_states)
del layer
hidden_states = outputWhy the bottleneck moves to disk
AirLLM wins on memory ceiling and loses on throughput. That is not an implementation flaw. It is the trade. Once the GPU stops holding the whole model, the system becomes sensitive to NVMe speed, layer size, and prefetch timing.
That is why AirLLM is compelling for batch work and awkward for chat. If you want low-latency conversation, the repeated relay is expensive. If you want to process a dataset overnight, or run an otherwise impossible model on modest hardware, the slowdown is acceptable.
This repo lets you run LLMs without a lot of memory by just-in-time loading a single layer for inference in the VRAM. It's extremely slow because you're bottlenecked by Disk -> RAM -> VRAM bandwidth. There's no solution other than to not use this repo and only offload a portion of the model layers that fit within your VRAM limit (using, for example, ollama + layer offloading).
Compression is an enabler, not the point
Quantization helps AirLLM, but it does not define it. The architecture still works because the model is streamed layer by layer. Compression mainly reduces disk pressure and loading time, which makes the relay less painful.
That distinction matters. In many inference stacks, quantization is the headline. In AirLLM, it is an assist. The real innovation is that the library can keep a giant model off the GPU almost all the time and still complete inference.
Why the project fans out across platforms
The Apple Silicon path shows the same philosophy adapted to a different memory model. On Mac, the project leans on MLX and unified memory rather than pretending every machine looks like a CUDA box. The target stays the same: make huge models usable on hardware that would normally be ruled out.
That makes the codebase feel broader than a one-off hack. It is a portability play built around a single idea. Wherever memory is scarce, AirLLM asks what can be streamed instead of stored.
AirLLM versus the usual suspects
AirLLM is not trying to beat the mainstream engines at their own game. Ollama and llama.cpp focus on usability and speed through quantization and practical model offloading. vLLM focuses on throughput and serving efficiency. AirLLM is for the cases where the first question is not performance, but feasibility.
| Tool | Primary goal | Memory strategy | Speed profile | Best fit |
|---|---|---|---|---|
| AirLLM | Run very large models on minimal VRAM | Streams one layer at a time from disk | Slow, especially for interactive chat | Batch jobs, experiments, extreme hardware limits |
| Ollama / llama.cpp | Usable local inference | Quantization and partial offloading | Fast enough for interactive use | General local chat and desktop deployment |
| vLLM | High-throughput serving | PagedAttention and efficient runtime memory use | Optimized for serving performance | Production endpoints and concurrent workloads |
The table is the point. AirLLM solves a different problem. It does not ask how to make a model fast once it fits. It asks how to make the model run when it plainly does not fit.
The project’s shape tells you what kind of tool it is
The repository looks like a tool built by someone who cared more about getting the model to run than packaging it as a polished platform. That shows up in the modular adapters, the persistence layer, the notebook-heavy examples, and the strong dependence on a small set of maintainers.
That shape is a strength and a warning. AirLLM is elegant because the core idea is so clean. It is fragile because the whole system hangs on a narrow architectural bet: keep the GPU free, stream aggressively, and accept that time is the thing you will spend instead of memory.
