The 300-Line LLM Engine: Inside nanoGPT

This is a brief guide to my new art project microgpt, a single file of 200 lines of pure Python with no dependencies that trains and inferences a GPT... This script is the culmination of multiple projects (micrograd, makemore, nanogpt, etc.) and a decade-long obsession to simplify LLMs to their bare essentials, and I think it is beautiful 🥹.

Andrej Karpathy, Creator · microgpt

Teeth Over Education

Most modern artificial intelligence frameworks bury the actual math under layers of enterprise abstractions. They rely on complex class hierarchies and endless YAML configurations. nanoGPT proves that production-grade large language model training does not require a bloated codebase. Andrej Karpathy stripped the GPT-2 architecture down to its absolute studs. The core logic spans just two files and roughly 600 lines of code.

This is not merely an educational toy. Karpathy built minGPT years prior to teach the basics, but it was too slow for real work. nanoGPT is the high-performance rewrite. It utilizes PyTorch 2.0 and Flash Attention to train a 124-million parameter model in four days on a single node. It is small enough to read in an afternoon but fast enough to do real damage.

The Memory-Mapped Data Pipeline

In large language model training, the bottleneck is often the data loader. Reading millions of small text files or parsing JSON on the fly is too slow for modern GPUs. The nanoGPT pipeline bypasses this entirely.

The preparation script converts gigabytes of text into a flat, memory-mapped array of unsigned 16-bit integers. This allows the training script to slice batches of tokens directly from disk into video RAM with zero overhead. The GPU never waits for the CPU to parse text.

Flat Architecture and Flash Attention

The entire model definition lives in a single file. There are no deep abstractions or hidden library calls. The code implements the Pre-Norm architecture, meaning Layer Normalization occurs before the attention and feed-forward blocks.

The implementation features manual weight tying between the input embedding matrix and the output projection matrix. It also natively leverages PyTorch fused attention kernels. This avoids materializing the massive attention matrix entirely, which is the primary driver of the repository's speed.

The Configuration Hack

Instead of using standard argument parsers or YAML files, nanoGPT uses pure Python scripts injected via execution commands. The configurator pattern dynamically updates the global namespace based on the provided configuration file.

Treating configuration as pure code is a dangerous pattern for production web applications. For research code, however, it is a brilliantly simple solution. It allows users to write inline math for batch size calculations and keeps the training script entirely flat.

The Minimalist Legacy

The repository sits perfectly between purely educational toys and heavy production frameworks. It remains the definitive clean-room implementation of a Transformer. It shows exactly what is required to make neural networks learn, and nothing more.

Sources:

• nanoGPT GitHub Repository
• microgpt Blog Post
• minGPT GitHub Repository