recurrent-looped-tranformer: Recurrent Looped Transformer: The Transformer That Refuses to Forget
How RLT turns token-by-token hidden state into a continuous reasoning loop, and why that changes the way we think about memory, depth, and long-context computation.
- RLT’s real break from standard Transformers is not more depth, but state continuity across tokens and across the prompt-response boundary.
- The architecture combines a causal encoder, global KV memory, local sliding-window cache, and a recurrent merge step into one persistent loop.
- The 8+0 ablation matters because it isolates the feedback path from decoder depth and shows that recurrence itself is doing work.
- RLT is best understood as a bet on process continuity for reasoning, replay, and long-horizon state tracking.
A Transformer That Carries State Forward
Standard Transformers are powerful because they can look back, not because they remember the way a recurrent system does. RLT changes that contract. The state at token t is not discarded before token t+1; it is merged forward, so the next token begins from the last computation rather than a fresh pass.
In most decoder-only LLMs, nothing computed at the last layer of token t feeds the first layer of token t+1; positions communicate only through attention over cached keys and values. RLT changes this.
That sounds like a small implementation detail. It is not. If the hidden state survives the step boundary, the model behaves less like a lookup engine and more like a running process.
Why the Reset Assumption Matters
The usual Transformer stack is stateless at the token level. Yes, KV cache preserves prior keys and values, but the computation at the new token still starts from a fresh hidden trajectory. RLT argues that this reset is the wrong abstraction for tasks where the model must keep track of a rule, a parity bit, or a latent plan.
| Model | State persistence | Boundary behavior | Best fit | Main limit |
|---|---|---|---|---|
| Standard Transformer | No persistent hidden state across tokens | Prompt and response are separate computation steps | General language modeling and retrieval | Weak at carrying algorithmic state forward |
| Universal Transformer | Shared weights across depth | Still computes each token in a fixed recurrent-like depth pattern | Iterative refinement | Does not hand off final token state to the next token |
| Recurrent Memory Transformer | Segment-level memory tokens | State moves between chunks, not every token | Long documents and memory compression | Coarser than per-token recurrence |
| RLT | Persistent hidden state plus cache | Prompt-response boundary is not a reset point | Long-horizon reasoning and replay | Still a research claim, not a settled winner |
The distinction matters most when the task is not about remembering facts, but about preserving a latent computation. Parity, modular arithmetic, and sequential control are all unforgiving about that distinction.
Inside the Loop: Global Memory, Local Cache, and Hidden State
RLT splits the job into three moving parts. The causal encoder builds a global memory over the input. The decoder keeps a local sliding-window cache for high-resolution recent context. Then a merge step folds the previous hidden state into the next token’s computation.
The important move is the merge. The model does not merely append new context. It combines the new token with the previous state, then uses that updated state as the seed for the next computation. That is the continuity claim in its most concrete form.
(s_t, C^D_t) = D_φ(Merge(e_t, s_{t-1}); M_{≤t}, C^D_{t-1}, t)
Read that as a pipeline, not as ornament. The encoder contributes e_t, the previous decoder state enters through s_{t-1}, and the result is a new state plus cache. The architecture is trying to preserve process, not just store context.
The 8+0 Ablation Is the Tell
The most revealing variant is RLT 8+0. There are no decoder blocks there, but the recurrent merge still exists. That makes it a clean test of whether the feedback path itself matters, independent of decoder depth.
| Variant | Decoder blocks | Recurrence | What it isolates |
|---|---|---|---|
| Standard 8 | 8 | No | Baseline Transformer depth |
| RLT 4+4 | 4 encoder, 4 decoder | Yes | Balanced hybrid behavior |
| RLT 6+2 | 6 encoder, 2 decoder | Yes | State tracking with less decoder depth |
| RLT 8+0 | 8 encoder, 0 decoder | Yes | The value of recurrence without decoder stack depth |
That matters because a lot of architectural claims blur two separate things. Is the model better because it is deeper, or because it carries state forward? 8+0 is designed to answer that question.
The Recurrent Looped Transformer shipped without experiments. So we ran them. Same data. Same order. Same recipe. 500M tokens. At 140M params RLT is the worst model we trained. And it needed ~20× the GPU-hours.
What the Depth-Eight Experiments Are Actually Testing
The depth-eight experiments are not just a benchmark dump. They probe whether recurrence helps the model hold onto algorithmic state under controlled conditions: parity, mod-5 arithmetic, seed variance, and training stability. That is a good stress test for a model that claims to preserve a continuous thought stream.
The notation matters too. A split like 4+4 or 6+2 says how much of the depth lives in the encoder versus the decoder. If RLT is really about continuity, those splits should change how the model learns state, not just how many parameters it has.
How RLT Fits Next to Universal Transformer, RMT, and YOCO
RLT lives near several familiar ideas, but it draws a narrower line than most of them. Universal Transformer reuses weights across depth. RMT moves memory across segments. YOCO motivates the global-memory-plus-local-cache shape. RLT’s claim is more specific: the model keeps a per-token hidden-state contract alive across the prompt-response boundary.
| Project | What persists | Granularity | Distinctive move |
|---|---|---|---|
| Universal Transformer | Weights across iterations | Token step, fixed-depth loop | Iterative refinement with shared parameters |
| Recurrent Memory Transformer | Memory tokens | Segment level | Carries state between chunks |
| Block-Recurrent Transformer | Block state | Block level | Recurrent processing over token blocks |
| YOCO | Global KV once, local cache later | Attention layout | Efficient memory partitioning |
| RLT | Hidden state plus cache | Per token | No-reset recurrence at the boundary |
That is why the project reads less like a full replacement for Transformers and more like a deliberate change in the computation contract. It says that some tasks want continuity first, and lookup second.
For the source repo and paper, see yifanzhang-pro/recurrent-looped-tranformer, the project page at yifanzhang-pro.github.io/recurrent-looped-tranformer, and the independent evaluation at AlphaXiv.





