esp32-ai: The LLM That Treats Flash Like Memory

How slvDev/esp32-ai runs a 28.9M-parameter language model on an ESP32-S3 by splitting weights across SRAM, PSRAM, and Flash, then reading only what each token actually needs.

9 min read • View on GitHub • More from slvDev

A small ESP32-S3 board sits at the center of a machine-room metaphor. Tight SRAM gears cluster closest to the chip, a conveyor belt carries PSRAM loads, and a distant wall of Flash drawers feeds only a few selected cards into the system while tokens move in and text comes out. The scene explains that the project wins by placing each kind of weight in the right memory tier.
The breakthrough is not just compression. It is memory placement, with Flash promoted from storage to a sparse weight tier.
Key Takeaways

The model that redefined the bottleneck

A 28.9 million parameter language model on an $8 ESP32-S3 sounds like a stunt until you look at the memory map. The trick is not that the model got smaller in some abstract sense. It is that the bottleneck changed. Once Flash becomes a sparse weight store, the usual “fit everything in RAM” assumption stops being the only game in town.

That is why this repo matters. It does not just prove that an LLM can run on a microcontroller. It shows that on tiny hardware, the real design problem is deciding which weights must stay hot, which can stream, and which can live far away until a token actually needs them.

A single token makes the case for the whole repo. Most weights stay hot, a few rows are fetched sparsely, and the output head can be split off to another core.

Why Flash, not RAM, is the real breakthrough

The repo’s memory strategy has three tiers. SRAM holds the weights touched every token. PSRAM or a stream path handles sequential access. Flash stores the large Per-Layer Embedding table, which is queried sparsely. That is the whole idea in one sentence: put the expensive-to-touch weights in fast memory, and let the rest sit where they are cheap to store.

The important nuance is access pattern, not just capacity. Flash is slow if you read it like RAM. It is much less slow if you only pull a few rows per token. That is why the project can keep the bulk of its parameters out of SRAM without turning inference into a dead stop.

ProjectTargetScaleMemory strategyDistinctive idea
llama2.cGeneral C inferenceSmall to medium modelsPlain weights, standard loadingMade tiny LLM inference practical in C
esp32-llmESP32-S3~260K parametersFits most weights into constrained memoryProved the chip could host a local model
esp32-aiESP32-S328.9M parametersSRAM for hot weights, Flash for sparse PLE tableTreats Flash as a model tier, not just storage
BouijaESP32-S3 hardware demoDepends on buildUses prior ESP32 inference coreCreative device form factor more than architecture
A close-up mechanical scene shows a transformer block standing in front of a huge card catalog behind glass. A small gate opens, pulls only a few cards from the catalog, and feeds them into the block while most of the catalog remains untouched in Flash behind the wall. The image explains how Per-Layer Embeddings keep most of the table out of fast memory.
Per-Layer Embeddings are the project’s core trick. The model consults a giant table, but only a tiny slice of it moves for each token.

Per-Layer Embeddings: the trick that makes the model possible

PLE is the centerpiece because it changes what “model capacity” means on this chip. Instead of storing all useful detail in the fast path, the block mixes hidden state with a layer-specific embedding that can be fetched sparsely from Flash. The hidden state gets a gate, then a projection, then a merge. Capacity lives in the table, not in the hot path.

That matters because the model is not relying on a naive compression trick. It is making a structural bet: a lot of the model’s expressive power can sit in a lookup table if the network only touches a few rows per token. On a microcontroller, that is the difference between impossible and merely expensive.

From Python lab to firmware

The repository is split into two worlds. In `src/`, Python handles training, model design, quantization, export, and memory accounting. In `firmware/`, C and C++ turn that research artifact into something the ESP32 can execute. The handoff is not fuzzy. It is explicit, binary, and verified.

src/
  model.py       # Transformer with PLE blocks
  budget.py      # Three-tier memory accounting
  quantize.py    # Groupwise 4-bit quantization
  export.py      # Packs weights into PLE1 binary

firmware/
  esp32_llm/
  common/llm.h
  host_verify/

`quantize.py` compresses weights with groupwise 4-bit quantization and FP16 scales. `export.py` writes a custom `PLE1` binary format and golden logits so the firmware can be checked against the PyTorch reference. That is what makes this feel engineered rather than improvised.

What the ESP32 firmware is doing differently

The firmware is tuned for the chip, not for portability. One core runs the main inference loop while the other helps with the output head, so the device can overlap work instead of serializing everything. The output head also uses `dot_i8`, which fits the ESP32-S3’s int8-friendly strengths better than a generic floating-point path.

That split matters because the last layer is often where cheap ports get clumsy. Here, it is a first-class execution plan. Core 0 can carry the head worker while Core 1 keeps the block stack moving, which is exactly the sort of small architectural choice that turns a demo into a usable system.

How this compares to earlier ESP32 LLMs

The right comparison is not “embedded AI versus normal AI.” It is `esp32-ai` versus the projects that proved the category in smaller form. `llama2.c` showed that tiny language models could live in plain C. `esp32-llm` showed that the ESP32-S3 could host a local model. `esp32-ai` changes the scale by making Flash part of the weight hierarchy, not just a dumping ground for static data.

That is why the tradeoff shift matters. Earlier projects asked how much model could fit. This one asks how much model can be served by the chip if memory is arranged by access pattern instead of by convenience.

Axisllama2.cesp32-llmesp32-ai
Primary contributionPlain C inferenceESP32-S3 portFlash-backed memory hierarchy
ScaleSmall modelsHundreds of thousands of parameters28.9M parameters
Main bottleneckGeneral portabilityMemory fitAccess pattern and placement
Defining moveSimplify runtimeRun on the chipTreat Flash as sparse model storage
Why it mattersProof of conceptCategory proofNew design pattern

The comparison also explains the project’s value. `esp32-ai` is not just a larger version of the same thing. It is a different answer to the same constraint.

Why this matters beyond a clever demo

This repo points toward a class of products that do not want cloud dependence at all. Offline toys. Privacy-first devices. Battery-powered sensors. Cheap assistants that can speak, think, and stay local. The headline is a model on an $8 chip, but the deeper lesson is that edge AI gets interesting when memory is treated as a system design problem, not a footnote.

Slava Serkov puts that philosophy plainly in the README: “As the AI industry obsesses over scaling up, this project is a compelling proof that scaling down may be equally transformative.” slvDev/esp32-ai README