esp32-ai: The LLM That Treats Flash Like Memory
How slvDev/esp32-ai runs a 28.9M-parameter language model on an ESP32-S3 by splitting weights across SRAM, PSRAM, and Flash, then reading only what each token actually needs.
- esp32-ai makes Flash part of the model’s architecture by reserving SRAM for hot weights and sampling the rest sparsely.
- Per-Layer Embeddings turn a large table into a low-bandwidth lookup problem instead of a RAM problem.
- The repository is split cleanly between Python research tooling and C firmware, with binary export and golden-logit verification bridging the two.
- The result is not just a clever demo. It is a pattern for offline AI on very cheap embedded hardware.
The model that redefined the bottleneck
A 28.9 million parameter language model on an $8 ESP32-S3 sounds like a stunt until you look at the memory map. The trick is not that the model got smaller in some abstract sense. It is that the bottleneck changed. Once Flash becomes a sparse weight store, the usual “fit everything in RAM” assumption stops being the only game in town.
That is why this repo matters. It does not just prove that an LLM can run on a microcontroller. It shows that on tiny hardware, the real design problem is deciding which weights must stay hot, which can stream, and which can live far away until a token actually needs them.
Why Flash, not RAM, is the real breakthrough
The repo’s memory strategy has three tiers. SRAM holds the weights touched every token. PSRAM or a stream path handles sequential access. Flash stores the large Per-Layer Embedding table, which is queried sparsely. That is the whole idea in one sentence: put the expensive-to-touch weights in fast memory, and let the rest sit where they are cheap to store.
The important nuance is access pattern, not just capacity. Flash is slow if you read it like RAM. It is much less slow if you only pull a few rows per token. That is why the project can keep the bulk of its parameters out of SRAM without turning inference into a dead stop.
| Project | Target | Scale | Memory strategy | Distinctive idea |
|---|---|---|---|---|
| llama2.c | General C inference | Small to medium models | Plain weights, standard loading | Made tiny LLM inference practical in C |
| esp32-llm | ESP32-S3 | ~260K parameters | Fits most weights into constrained memory | Proved the chip could host a local model |
| esp32-ai | ESP32-S3 | 28.9M parameters | SRAM for hot weights, Flash for sparse PLE table | Treats Flash as a model tier, not just storage |
| Bouija | ESP32-S3 hardware demo | Depends on build | Uses prior ESP32 inference core | Creative device form factor more than architecture |
Per-Layer Embeddings: the trick that makes the model possible
PLE is the centerpiece because it changes what “model capacity” means on this chip. Instead of storing all useful detail in the fast path, the block mixes hidden state with a layer-specific embedding that can be fetched sparsely from Flash. The hidden state gets a gate, then a projection, then a merge. Capacity lives in the table, not in the hot path.
That matters because the model is not relying on a naive compression trick. It is making a structural bet: a lot of the model’s expressive power can sit in a lookup table if the network only touches a few rows per token. On a microcontroller, that is the difference between impossible and merely expensive.
From Python lab to firmware
The repository is split into two worlds. In `src/`, Python handles training, model design, quantization, export, and memory accounting. In `firmware/`, C and C++ turn that research artifact into something the ESP32 can execute. The handoff is not fuzzy. It is explicit, binary, and verified.
src/
model.py # Transformer with PLE blocks
budget.py # Three-tier memory accounting
quantize.py # Groupwise 4-bit quantization
export.py # Packs weights into PLE1 binary
firmware/
esp32_llm/
common/llm.h
host_verify/
`quantize.py` compresses weights with groupwise 4-bit quantization and FP16 scales. `export.py` writes a custom `PLE1` binary format and golden logits so the firmware can be checked against the PyTorch reference. That is what makes this feel engineered rather than improvised.
What the ESP32 firmware is doing differently
The firmware is tuned for the chip, not for portability. One core runs the main inference loop while the other helps with the output head, so the device can overlap work instead of serializing everything. The output head also uses `dot_i8`, which fits the ESP32-S3’s int8-friendly strengths better than a generic floating-point path.
That split matters because the last layer is often where cheap ports get clumsy. Here, it is a first-class execution plan. Core 0 can carry the head worker while Core 1 keeps the block stack moving, which is exactly the sort of small architectural choice that turns a demo into a usable system.
How this compares to earlier ESP32 LLMs
The right comparison is not “embedded AI versus normal AI.” It is `esp32-ai` versus the projects that proved the category in smaller form. `llama2.c` showed that tiny language models could live in plain C. `esp32-llm` showed that the ESP32-S3 could host a local model. `esp32-ai` changes the scale by making Flash part of the weight hierarchy, not just a dumping ground for static data.
That is why the tradeoff shift matters. Earlier projects asked how much model could fit. This one asks how much model can be served by the chip if memory is arranged by access pattern instead of by convenience.
| Axis | llama2.c | esp32-llm | esp32-ai |
|---|---|---|---|
| Primary contribution | Plain C inference | ESP32-S3 port | Flash-backed memory hierarchy |
| Scale | Small models | Hundreds of thousands of parameters | 28.9M parameters |
| Main bottleneck | General portability | Memory fit | Access pattern and placement |
| Defining move | Simplify runtime | Run on the chip | Treat Flash as sparse model storage |
| Why it matters | Proof of concept | Category proof | New design pattern |
The comparison also explains the project’s value. `esp32-ai` is not just a larger version of the same thing. It is a different answer to the same constraint.
Why this matters beyond a clever demo
This repo points toward a class of products that do not want cloud dependence at all. Offline toys. Privacy-first devices. Battery-powered sensors. Cheap assistants that can speak, think, and stay local. The headline is a model on an $8 chip, but the deeper lesson is that edge AI gets interesting when memory is treated as a system design problem, not a footnote.
Slava Serkov puts that philosophy plainly in the README: “As the AI industry obsesses over scaling up, this project is a compelling proof that scaling down may be equally transformative.” slvDev/esp32-ai README





