recurrent-looped-tranformer: Recurrent Looped Transformer: The Transformer That Refuses to Forget

How RLT turns token-by-token hidden state into a continuous reasoning loop, and why that changes the way we think about memory, depth, and long-context computation.

8 min read • View on GitHub • More from yifanzhang-pro

A long mechanical relay line where a small token-sized object moves from chamber to chamber, changing slightly at each stop before feeding into the next chamber. The loop is visible as a continuous circuit, showing that state is handed forward instead of reset.
RLT’s central claim is simple to say and easy to miss: the model keeps moving the same internal process forward instead of starting over at every token.
Key Takeaways

A Transformer That Carries State Forward

Standard Transformers are powerful because they can look back, not because they remember the way a recurrent system does. RLT changes that contract. The state at token t is not discarded before token t+1; it is merged forward, so the next token begins from the last computation rather than a fresh pass.

In most decoder-only LLMs, nothing computed at the last layer of token t feeds the first layer of token t+1; positions communicate only through attention over cached keys and values. RLT changes this.

Yifan Zhang, Project Creator · Recurrent Looped Transformer Project Page

That sounds like a small implementation detail. It is not. If the hidden state survives the step boundary, the model behaves less like a lookup engine and more like a running process.

Why the Reset Assumption Matters

The usual Transformer stack is stateless at the token level. Yes, KV cache preserves prior keys and values, but the computation at the new token still starts from a fresh hidden trajectory. RLT argues that this reset is the wrong abstraction for tasks where the model must keep track of a rule, a parity bit, or a latent plan.

ModelState persistenceBoundary behaviorBest fitMain limit
Standard TransformerNo persistent hidden state across tokensPrompt and response are separate computation stepsGeneral language modeling and retrievalWeak at carrying algorithmic state forward
Universal TransformerShared weights across depthStill computes each token in a fixed recurrent-like depth patternIterative refinementDoes not hand off final token state to the next token
Recurrent Memory TransformerSegment-level memory tokensState moves between chunks, not every tokenLong documents and memory compressionCoarser than per-token recurrence
RLTPersistent hidden state plus cachePrompt-response boundary is not a reset pointLong-horizon reasoning and replayStill a research claim, not a settled winner

The distinction matters most when the task is not about remembering facts, but about preserving a latent computation. Parity, modular arithmetic, and sequential control are all unforgiving about that distinction.

Inside the Loop: Global Memory, Local Cache, and Hidden State

RLT splits the job into three moving parts. The causal encoder builds a global memory over the input. The decoder keeps a local sliding-window cache for high-resolution recent context. Then a merge step folds the previous hidden state into the next token’s computation.

This diagram makes the hidden contract explicit. The boundary between prompt and response is a hop, not a wipe.

The important move is the merge. The model does not merely append new context. It combines the new token with the previous state, then uses that updated state as the seed for the next computation. That is the continuity claim in its most concrete form.

(s_t, C^D_t) = D_φ(Merge(e_t, s_{t-1}); M_{≤t}, C^D_{t-1}, t)

Read that as a pipeline, not as ornament. The encoder contributes e_t, the previous decoder state enters through s_{t-1}, and the result is a new state plus cache. The architecture is trying to preserve process, not just store context.

A split technical scene compares two workflows. On the left, a standard Transformer passes a token through a fixed stack and the state dissolves at the end. On the right, RLT carries the final state across a seam marked as the prompt-response boundary and threads it into the next token.
The boundary is the tell. One system restarts at each step. The other keeps the computation alive.

The 8+0 Ablation Is the Tell

The most revealing variant is RLT 8+0. There are no decoder blocks there, but the recurrent merge still exists. That makes it a clean test of whether the feedback path itself matters, independent of decoder depth.

VariantDecoder blocksRecurrenceWhat it isolates
Standard 88NoBaseline Transformer depth
RLT 4+44 encoder, 4 decoderYesBalanced hybrid behavior
RLT 6+26 encoder, 2 decoderYesState tracking with less decoder depth
RLT 8+08 encoder, 0 decoderYesThe value of recurrence without decoder stack depth

That matters because a lot of architectural claims blur two separate things. Is the model better because it is deeper, or because it carries state forward? 8+0 is designed to answer that question.

The Recurrent Looped Transformer shipped without experiments. So we ran them. Same data. Same order. Same recipe. 500M tokens. At 140M params RLT is the worst model we trained. And it needed ~20× the GPU-hours.

Empero AI, Independent Research Organization · Does Recurrence Pay? An Evaluation of RLT

What the Depth-Eight Experiments Are Actually Testing

The depth-eight experiments are not just a benchmark dump. They probe whether recurrence helps the model hold onto algorithmic state under controlled conditions: parity, mod-5 arithmetic, seed variance, and training stability. That is a good stress test for a model that claims to preserve a continuous thought stream.

The notation matters too. A split like 4+4 or 6+2 says how much of the depth lives in the encoder versus the decoder. If RLT is really about continuity, those splits should change how the model learns state, not just how many parameters it has.

How RLT Fits Next to Universal Transformer, RMT, and YOCO

RLT lives near several familiar ideas, but it draws a narrower line than most of them. Universal Transformer reuses weights across depth. RMT moves memory across segments. YOCO motivates the global-memory-plus-local-cache shape. RLT’s claim is more specific: the model keeps a per-token hidden-state contract alive across the prompt-response boundary.

ProjectWhat persistsGranularityDistinctive move
Universal TransformerWeights across iterationsToken step, fixed-depth loopIterative refinement with shared parameters
Recurrent Memory TransformerMemory tokensSegment levelCarries state between chunks
Block-Recurrent TransformerBlock stateBlock levelRecurrent processing over token blocks
YOCOGlobal KV once, local cache laterAttention layoutEfficient memory partitioning
RLTHidden state plus cachePer tokenNo-reset recurrence at the boundary

That is why the project reads less like a full replacement for Transformers and more like a deliberate change in the computation contract. It says that some tasks want continuity first, and lookup second.

For the source repo and paper, see yifanzhang-pro/recurrent-looped-tranformer, the project page at yifanzhang-pro.github.io/recurrent-looped-tranformer, and the independent evaluation at AlphaXiv.