OpenMythos: The Transformer That Thinks by Looping the Same Brain

A theoretical recurrent-depth architecture that treats depth as a runtime choice, stabilizes recurrence with control-theory logic, and uses halting plus shared weights to make inference feel like adjustable thought.

9 min read • View on GitHub • More from kyegomez

A mechanical brain is pulled through the same gear chamber multiple times, with the same core engine reused on each pass. The image explains OpenMythos as recycled depth, where computation is revisited instead of simply stacked taller.
OpenMythos treats depth as something you can reuse. The same block moves through the loop again and again, so inference budget becomes a knob rather than a fixed tower of layers.
Key Takeaways

OpenMythos is not trying to win by piling on more layers. Its pitch is sharper: reuse the same block, let the model loop, and decide at runtime how much internal work a prompt deserves.

That shift matters because it changes the unit of capability. In a normal transformer, depth is mostly baked in. In OpenMythos, depth becomes elastic, and that makes inference-time scaling the central idea rather than a side effect.


Why this model cares about loops, not layers

The first thing to notice is that OpenMythos does not equate depth with a longer feed-forward stack. It separates the model into a prelude, a shared recurrent block, and a coda, then lets the middle section run multiple times.

That is a different mental model from standard transformers. Instead of asking how many layers the model has, OpenMythos asks how many passes the model should take before it stops.

A close-up control panel shows a stable hidden-state reservoir, a safety dial, and a gate that opens only when the update stays within bounds. The image explains why recurrent depth needs stability checks before looping can be trusted.
The loop only works if the state stays bounded. OpenMythos treats that as an engineering constraint, not a philosophical one.

Inside the recurrent block

The architecture reads like a pipeline, but the interesting part is the repeated center. The prelude shapes the input, the recurrent block processes it again and again, and the coda turns the final hidden state into logits.

# Conceptual shape of the loop
h = prelude(x)
for t in range(n_loops):
    h = recurrent_block(h, loop_index=t)
    if halting_gate(h):
        break
logits = coda(h)

The repo’s technical choices reinforce that structure. LTI injection keeps the hidden state update within a controlled linear system, while the spectral-radius check acts like a guardrail against runaway recurrence.

Open source is better than closed source in every way.

Kye Gomez, Creator, Swarms Framework; Open Source Advocate · Open Source > Closed Source - Medium

That quote is blunt, but the code is more interesting than the slogan. OpenMythos is really arguing that architectural leverage, not scale alone, can make a smaller system feel larger at inference time.

How the model decides to stop thinking

ACT halting is the product idea hiding inside the architecture. It means the model can spend more latent work on hard prompts and less on easy ones, without changing the outer interface.

That is why this project feels less like a static model sketch and more like a runtime policy. It is not just about what the network can represent. It is about how much of that representation it chooses to use.

Why MoE and attention variants matter here

ComponentWhy it matters in a looped modelPractical effect
Standard FFNDense compute every passSimple, but expensive when loops increase
MoE FFNSparse experts can stay selectiveMore capacity without paying full cost on every iteration
MLACompresses KV memory for long-context workHelps when recurrence and context both demand memory discipline
GQAA familiar efficiency trade-offSimpler than MLA, easier to reason about in some deployments

In a looped architecture, efficiency is not a polish layer. Every extra pass multiplies memory pressure and compute cost, so sparse experts and attention choice become part of the architecture’s core economics.

That is why the repo’s MLA versus GQA split matters. It is not a footnote. It is the difference between a loop that is elegant on paper and one that is actually usable.

The stability trick most readers will miss

The quiet insight in OpenMythos is that recurrence is dangerous unless the hidden state stays bounded. That is where the control-theory flavor shows up most clearly.

Spectral-radius checks are not decorative math. They tell you whether repeated application of the update rule will settle into a stable orbit or spiral out of control as the loops accumulate.

What OpenMythos is really arguing about

QuestionStandard transformer answerOpenMythos answer
What is depth?A fixed stack of layersA runtime budget of passes
Can the model think more at inference?Only indirectly, with larger models or longer promptsYes, by looping the same block more times
How do you keep recurrence safe?Usually by architectural conventionBy explicitly checking stability
What makes it smarter?More parameters and more layersBetter control over when and how computation repeats

This is the deeper thesis. OpenMythos suggests that smaller systems can sometimes think deeper if they are allowed to reuse computation intelligently.

That does not eliminate the value of scale. It does challenge the reflex that more parameters are always the cleanest answer to harder reasoning.

Where this sits in the landscape

Model shapeDepth representationEarly haltingStability riskMain benefit
Standard transformer stackFixed layersNoLowStraightforward training and inference
Fixed-depth recurrent modelRepeated block with fixed passesUsually noModerateWeight reuse and parameter efficiency
Dynamic halting modelRepeated block with learned stoppingYesModerate to highCompute adapts to difficulty
OpenMythos style looped inferenceShared block, loop index embeddings, bounded recurrenceYesManaged explicitlyRuntime depth becomes a control knob

The cleanest way to read OpenMythos is as a thesis about inference-time control. It is not just a smaller model. It is a model that tries to make reasoning budget visible, adjustable, and stable.