hyunwoongko/transformer: The Transformer Repo That Reads Like the Paper
A from-scratch PyTorch build that turns multi-head attention, masking, and positional encoding into a file tree you can actually follow.
I implemented the Transformer model of Google Brain using Pytorch. It was specially written together in very detailed and easy explanatory comments. If you're a beginner who wants to implement Transformer, look at my code and try it!
- This repository turns the Transformer paper into a navigable code tour, so the structure itself becomes the explanation.
- Its biggest teaching win is not novelty, but visibility: masking, head splitting, and positional encoding are laid out where readers can inspect them.
- The code is intentionally pedagogical, which makes its rough edges useful evidence of how Transformer understanding matured in public.
- Modern implementations hide more machinery, but this one exposes the parts that help a learner build a mental model first.
Why this repo still works as a Transformer tutorial
Most Transformer repos teach by abstraction. This one teaches by exposure. It does not wrap the model in a polished API and call it a day. It walks you through the architecture in the same order the paper does, which makes the code feel less like a framework and more like a diagram you can run.
That matters because the Transformer is easy to name and hard to internalize. The real lesson is not that attention replaced recurrence. It is how the pieces fit together: embeddings, positional signals, masked decoder attention, feed-forward blocks, and the final projection back to tokens.
A codebase that mirrors the paper
The repository is organized the way a careful teacher would outline the model. `models/embedding` handles token and positional encoding. `models/layers` contains the atomic math, including multi-head attention and layer normalization. `models/blocks` assembles encoder and decoder layers, and `models/model` ties the whole thing together.
That hierarchy matters more than it sounds like it should. When the file tree matches the conceptual stack, you stop hunting for the implementation and start reading the architecture as a sequence of decisions.
| Layer | What it does | Why it helps learners |
|---|---|---|
| embedding | Token and positional input | Keeps the first step visible |
| layers | Attention, normalization, math primitives | Shows the core equations directly |
| blocks | Encoder and decoder layer assembly | Explains how parts compose |
| model | Full Transformer orchestration | Makes end-to-end flow obvious |
The most important trick is hidden in the tensors
The key move is not conceptual. It is tensor reshaping. The code projects inputs into Q, K, and V, then splits those matrices across heads with `view` and `transpose`, so every head can run in parallel on the GPU. After attention is computed, the heads are concatenated back into one stream and projected again.
That is the part many readers miss when they first learn attention. The innovation is mathematically elegant, but the implementation story is brutally practical: reshape, split, mask, merge.
# Conceptual shape flow in multi-head attention
Q = linear_q(x) # [batch, seq, d_model]
K = linear_k(x)
V = linear_v(x)
Q = split_heads(Q) # [batch, heads, seq, d_k]
K = split_heads(K)
V = split_heads(V)
scores = (Q @ K.transpose(-2, -1)) / math.sqrt(d_k)
scores = scores.masked_fill(triangular_mask == 0, float('-inf'))
weights = softmax(scores, dim=-1)
context = weights @ V
context = concat_heads(context) # [batch, seq, d_model]
out = linear_out(context)The target mask is the quiet hero here. `torch.tril` builds the lower triangular pattern that prevents the decoder from seeing future tokens during training. Without that gate, the model would cheat. With it, the training signal matches the real task: predict the next token from the ones already available.
Pedagogical purity comes with rough edges
That README warning is not a disclaimer you skip past. It is the point. The repo is a learning artifact from the period when Transformer implementation was still being actively decoded by individual developers, and you can feel that in the code.
Some choices are intentionally didactic. The custom layer norm makes the normalization math visible. The fixed sinusoidal positional encoding sticks close to the paper. The training stack leans on legacy `torchtext`, which is less convenient today but historically explains how many people first got these examples running.
| Choice | What it buys | What it costs |
|---|---|---|
| Custom LayerNorm | Shows the math directly | Less optimized than native PyTorch |
| Fixed sinusoidal positions | Matches the paper closely | Less flexible than learned embeddings |
| Legacy torchtext | Fits the era of the project | Harder to run on modern setups |
| Simple training script | Easy to follow | Less production-ready |
What modern Transformer code usually abstracts away
| Dimension | hyunwoongko/transformer | Modern higher-level implementation |
|---|---|---|
| Level of abstraction | Low enough to inspect every step | Higher, with many internals hidden |
| Readability for learners | Very strong | Often better for users than learners |
| Fidelity to the original paper | Close and explicit | Usually adapted to newer conventions |
| Ease of running today | May need refactoring | Usually smoother |
| Masking and head reshaping | Exposed directly | Often tucked inside helpers |
| Maintenance burden | Higher | Lower |
This is where the repo earns its status as a reference point rather than a production template. It exposes exactly the things modern libraries tend to hide: the masks, the head reshaping, the positional signal, the layer assembly. That makes it slower to use, but faster to understand.
So the comparison is not about winners. It is about intent. Modern implementations optimize for ergonomics and deployment. This one optimizes for learning, and it succeeds because those goals are not the same.
Why this repository became a reference point
The repo became memorable because it did something rare: it made a breakthrough architecture feel approachable without pretending the details did not matter. It is both a tutorial and a timestamp. A tutorial, because it teaches the moving parts cleanly. A timestamp, because it captures a moment when the Transformer was still new enough that people were learning it by rebuilding it from scratch.
That is why the code still matters. Not because it is the fastest or cleanest implementation you can find, but because it shows how a generation of developers learned to think in attention.