cs.LGMay 11, 2026

Simply Stabilizing the Loop via Fully Looped Transformer

Authors: Rao Fu, Zixuan Yang, Jiankun Zhang, Jing Ma, Hechang Chen, Yu Li, Yi Chang

Organizations: Hong Kong Baptist University · Jilin University

Abstract

Scaling model performance typically requires increasing model size. Looped Transformer offers a compelling alternative by iteratively reusing the same Transformer blocks, trading additional computation for improved performance without increasing parameter count or context length. Because the number of loop iterations can be adjusted at inference, it also provides a natural mechanism for balancing performance and test-time compute. However, Looped Transformer still suffers from training instability when the number of loop iterations increases. Our analysis reveals that this instability stems from two sources: gradient oscillation and residual explosion. To address these two problems, we propose the Fully Looped Transformer, which introduces two parameter-free modifications: (1) Fully Looped Architecture, which distributes inter-loop signals across all layers to mitigate residual explosion; (2) Attention Injection, which reuses the existing attention block to suppress gradient oscillation. These modifications stabilize training dynamics, enabling the Fully Looped Transformer to be trained stably up to 12 loop iterations, whereas other baseline looped models collapse in this regime. In milder settings where Looped Transformer does not collapse, Fully Looped Transformer still improves average downstream-task performance by up to 13.2%. Overall, our experiments demonstrate that Fully Looped Transformer improves training stability, enhances downstream performance, and provides preliminary adaptability under different test-time compute budgets by varying loop iterations at inference.

Explore similar work

CardsList
Apr 16, 2026cs.LG

Stability and Generalization in Looped Transformers

Looped transformers promise test-time compute scaling by spending more iterations on harder problems, but it remains unclear which architectural choices let them extrapolate to harder problems at test time rather than memorize training-specific solutions. We introduce a fixed-point based framework for analyzing looped architectures along three axes of stability -- reachability, input-dependence, and geometry -- and use it to characterize when fixed-point iteration yields meaningful predictions. Theoretically, we prove that looped networks without recall have countable fixed points and cannot achieve strong input-dependence at any spectral regime, while recall combined with outer normalization reliably produces a regime in which fixed points are simultaneously reachable, locally smooth in the input, and supported by stable backpropagation. Empirically, we train single-layer looped transformers on chess, sudoku, and prefix-sums and find that downstream performance tracks the framework's predictions across tasks and architectural configurations. We additionally introduce internal recall, a novel recall placement variant, and show that it becomes competitive with -- and on sudoku, substantially better than -- standard recall placement once outer normalization is applied.
Asher Labovich
Fixed-Point IterationTest-Time ScalingSemantic Recall
AskPDF
Jun 29, 2026cs.LG

Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping

Looped Transformers, which repeatedly apply a shared transformer block, are an architecturally natural fit for variable-length algorithmic tasks. Although they can exhibit strong length generalization beyond the length of training sequences, this behavior is brittle, yielding high out-of-distribution (OOD) variance, even across well-performing in-distribution solutions. We trace this variance to the spurious correlation in simple algorithmic tasks between sequence length and number of loops. Introducing stochasticity into the number of loops during training sharply reduces OOD variance and stabilizes predictions across inference-time loop counts. To improve upon heuristic randomization schemes, we further analyze RL-Halting as a learned stochastic schedule and find that it generally improves the accuracy-stability trade-off. Across binary addition, Dyck-1, Unique Set, and Copy, learned stochastic stopping often improves this trade-off but can also stabilize a suboptimal computation. Our work suggests that "when to stop" should be treated as a training-time design choice, not merely an inference-time computation-allocation rule.
Hsun-Yu Kuo, El Mahdi Chayti, Patrik Reizinger +2
Early StoppingLow-Precision TrainingStochasticOptimal StoppingExtrapolationVariance Reduction+2
AskPDF
Date pendingcs.LG

On the Residual Scaling of Looped Transformers: Stability and Transferability

Looped (weight-tied) Transformers apply a shared residual block NN times (h←h+ε f(h)h \leftarrow h + \varepsilon\,f(h), same ff at each step), increasing effective depth without adding parameters. Prior depth-scaling analyses prescribe ε=1/ ⁣L\varepsilon = 1/\!\sqrt{L} for depth-LL residual networks. We show that this is insufficient for looped architectures: weight sharing makes residual updates correlated across iterations, requiring the stronger scaling ε=1/N\varepsilon = 1/N. For multi-layer blocks (LL unique layers looped NN times), we derive a factored parameterization ε=λ/(N ⁣L)\varepsilon = \lambda/(N\!\sqrt{L}) that separates the two sources of growth: 1/N1/N controls the within-layer loop correlation, and 1/ ⁣L1/\!\sqrt{L} controls the across-layer variance. A key consequence is that the optimal learning rate depends only on the number of unique layers LL, not on the loop count NN, enabling direct hyperparameter transfer from small to large NN without retuning. Experiments on looped Transformers confirm that 1/N1/N scaling improves trainability and yields better loss than 1/ ⁣N1/\!\sqrt{N} scaling across loop counts.
Shaowen Wang, Bingrui Li, Ge Zhang +3
Residual ConnectionsLearning RatesHyperparameter
AskPDF