minor-project: What a Transformer Looks Like in Raw C

Most from-scratch neural network demos stop once they can fit a line or classify a toy dataset. minor-project does something more revealing. It reaches into attention, class tokens, positional embeddings, convolution, and backprop, then rebuilds those pieces with raw pointers and loops instead of a deep learning runtime.

That is why the repo is interesting. It is not trying to be fast or fashionable. It is trying to show what modern machine learning looks like when the abstractions are stripped away and the machinery is left in plain sight.

Why this repo matters

The central surprise is scope. This is not just an MLP exercise with a few matrix multiplies. The code reaches toward transformer-era building blocks, including attention and token-style structure, which pushes it far beyond the usual educational C project.

That matters because transformers are usually consumed as APIs. Here, they are assembled. The difference is not cosmetic. It changes the reader’s mental model from “call the layer” to “wire the layer.”

A transformer stack without the usual magic

The most revealing part of the code is that it appears to implement attention the hard way. That means explicit Q, K, and V projections, manual score computation, row-wise softmax, scaling by the square root of the key dimension, and the kind of numerical stability trick that usually disappears behind a library function.

Once you see those pieces, the project starts to read like a first-principles Vision Transformer attempt. The presence of class token logic and positional embeddings matters because it signals intent. This is not just feature extraction. It is sequence modeling, built from scratch, in a language that gives you almost no help.

What the code is really doing

// Simplified shape of the attention path
Q = linear_proj_per_row(X, Wq, bq);
K = linear_proj_per_row(X, Wk, bk);
V = linear_proj_per_row(X, Wv, bv);
S = matmul(Q, transpose(K));
S = S / sqrtf((float)d_k);
S = softmax_rowwise_stable(S);
O = matmul(S, V);

That snippet is the whole story. The hard part is not the syntax. It is the fact that each intermediate result must exist as a real buffer with real dimensions and real memory ownership. There is no autograd engine to rescue you if you forget a shape or leak an allocation.

The codebase is a manual training machine

The repository is organized like a small machine shop. forward.c handles inference operations, backward.c carries the gradient logic, utils.c provides matrix and image helpers, config.c manages learning-rate policy, and try.c wires the whole thing into a training demo.

That separation matters because it makes the learning loop legible. You can see where activations are created, where they are reused, where gradients are accumulated, and where parameters are updated. In a framework, those boundaries blur. Here, they stay visible.

That is the essential trade-off. The code is educational because it refuses to automate away the uncomfortable parts. It is also slow for the same reason. Matrix multiplies are naïve, convolution is nested-loop heavy, and memory safety depends on discipline rather than a runtime.

What backprop looks like when every buffer is yours

In a modern stack, backprop is an invisible service. In this repo, it is work. Intermediate activations are stored, gradients are computed explicitly, and updates happen through handwritten functions like a tiny optimizer room staffed by one person.

That has a clear pedagogical benefit. You can trace how a loss becomes a derivative, how a derivative becomes a weight update, and how that update depends on every earlier shape check. The model does not just learn. It consumes memory, step by step, in ways that are easy to ignore in higher-level tools.

The readme and code suggest a project built to understand first principles, not to ship a production training stack. That is the right ambition here. The value is not throughput. The value is making the hidden graph visible.

Transparency over speed

This is not a fair contest if the metric is raw performance. It is a very fair contest if the metric is understanding. Frameworks let you move quickly. This repo lets you see what you are moving through.

How it compares to mainstream ML stacks

That comparison is the real point of the repo. It is less a competitor to PyTorch than a microscope for anyone who uses it. If you already know the abstractions, this code shows the cost of having them. If you do not, it shows why they exist.