Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers

Sajad Movahedi; Vera Milovanović; Shlomo Libo Feigin; Alexander Theus; Thomas Hofmann; Valentina Boeva; T. Konstantin Rusch; Antonio Orvieto

cs.AIJun 16, 2026

Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers

Authors: Sajad Movahedi, Vera Milovanović, Shlomo Libo Feigin, Alexander Theus, Thomas Hofmann, Valentina Boeva, T. Konstantin Rusch, Antonio Orvieto

Organizations: ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center · ETH Zurich · Swiss Institute of Bioinformatics · Université Paris Cité · Liquid AI

Abstract

Looped architectures provide an inductive bias toward learning step-by-step procedures for tasks that require compositional reasoning. The number of effective layers reached by looping determines the quality of the solution these models find. Like deep architectures, looped architectures are prone to a signal propagation problem induced by depth as the halting decision is postponed. In this paper, we address this signal propagation issue using pre-norm layers and residual scaling. Building on these architectural modifications, we propose FPRM, a Transformer-based Fixed-Point Reasoning Model that uses fixed-point convergence as an end-to-end halting mechanism in a looped architecture. We show that fixed-point halting allows FPRM to adapt its compute to task difficulty. FPRM is effective on common reasoning benchmarks, namely Sudoku, Maze, state-tracking, and ARC-AGI.

Explore similar work

Solve the Loop: Attractor Models for Language and Reasoning
May 12, 2026Jacob Fein-Ashley, Paria RashidinejadLarge Language Model Pretraining Attractors
Chain-of-Thought and Compressed Looped Transformers: A Memory-Budget Separation
May 29, 2026Haozhou ZhangEfficient Chain-Of-Thought Reasoning Chain-of-Thought Reasoning
Stability and Generalization in Looped Transformers
Apr 16, 2026Asher LabovichTransformer Architectures Fixed-Point Iteration

May 12, 2026 · cs.LGJ/K move · Enter open · S save

Solve the Loop: Attractor Models for Language and Reasoning

Jacob Fein-Ashley, Paria Rashidinejad

University of Southern California

Looped Transformers offer a promising alternative to purely feed-forward computation by iteratively refining latent representations, improving language modeling and reasoning. Yet recurrent architectures remain unstable to train, costly to optimize and deploy, and constrained to small, fixed recurrence depths. We introduce Attractor Models, in which a backbone module first proposes output embeddings, then an attractor module refines them by solving for the fixed point, with gradients obtained through implicit differentiation. Thus, training memory remains constant in effective depth, and iterations are chosen adaptively by convergence. Empirically, Attractor Models outperform existing models across two regimes, large-scale language-model pretraining and reasoning with tiny models. In language modeling, Attractor Models deliver a Pareto improvement over standard Transformers and stable looped models across sizes, improving perplexity by up to 46.6% and downstream accuracy by up to 19.7% while reducing training cost. Notably, a 770M Attractor Model outperforms a 1.3B Transformer trained on twice as many tokens. On challenging reasoning tasks, we show that our model with only 27M parameters and approximately 1000 examples achieves 91.4% accuracy on Sudoku-Extreme and 93.1% on Maze-Hard, scaling favorably where frontier models like Claude and GPT o3, fail completely, and specialized recursive reasoners collapse at larger sizes. Lastly, we show that Attractor Models exhibit a novel phenomenon, which we call equilibrium internalization: fixed-point training places the model's initial output embedding near equilibrium, allowing the solver to be removed at inference time with little degradation. Together, these results suggest that Attractor Models make iterative refinement scalable by turning recurrence into a computation the model can learn to internalize.

Large Language Model Pretraining Attractors Loop

Open paper Ask about it PDF

Abstract

Explore similar work

Solve the Loop: Attractor Models for Language and Reasoning

Chain-of-Thought and Compressed Looped Transformers: A Memory-Budget Separation

Stability and Generalization in Looped Transformers