Pause Anim
Reset Model
Model Depth (Layers)
12
Adjusting depth automatically scales width and heads to maintain optimal aspect ratio (~64).
Hyperparameter Config
Hidden Width (n_embed)
768
Attention Heads (n_head)
12
Peak Learning Rate
1.5e-3
Depth × 64
Width ÷ 64
Scaled by 1/√N
Token IDs
Dense Vectors
Hidden States
Output Projection
TRANSFORMER LAYERS
Embedding
Token Input