Looping, reusing a block of layers across depth, and depth growing, training shallow-to-deep models by duplicating middle layers, have both been linked to stronger reasoning, but their relationship remains unclear. We provide a mechanistic unification: looped and depth-grown models exhibit convergent depth-wise signatures, including increased reliance on late layers and recurring patterns aligned with the looped or grown block. These shared signatures support the view that their gains stem from a common form of iterative computation. Building on this connection, we show that the two techniques are adaptable and composable: applying inference-time looping to the middle blocks of a depth-grown model improves accuracy on some reasoning primitives by up to 2×, despite the model never being trained to loop. Both approaches also adapt better than the baseline when given more in-context examples or additional supervised fine-tuning data. Additionally, depth-grown models achieve the largest reasoning gains when using higher-quality, math-heavy cooldown mixtures, which can be further boosted by adapting a middle block to loop. Overall, our results position depth growth and looping as complementary, practical methods for inducing and scaling iterative computation to improve reasoning.
Figures & tables
Figure 1: Trade-offs for looped and depth-grown models. Each point corresponds to a model in Section 1 (up to 1.7B parameters), plotted by average Reasoning Primitives accuracy versus unique parameters (left), inference FLOPs (middle), and training FLOPs (right). Looped and depth-grown models improve accuracy in reasoning primitives over standard baselines, suggesting a shared inductive bias toward better reasoning. Looped models improve reasoning under fixed parameter budgets and can be competitive under fixed inference budgets, while depth-grown models reach similar or better reasoning with less training compute. Fig. 10 shows additional benchmark categories.
Standard cooldown
Params
Open-book
Closed-book
Math Word
Reasoning
/
Holdout Set
Q&A
Q&A
Lambada
HellaSwag
Problems
Primitives
FLOPs
(NLL ↓ )
(F1 ↑ )
(F1 ↑ )
(Acc ↑ )
(Acc ↑ )
(Acc ↑ )
(Acc ↑ )
360M
Baseline
32 / 32
2.18
22.89
14.50
43.35
39.97
3.69
30.04
MIDAS
32 / 32
2.18
24.57
13.75
43.26
40.36
4.39
28.42
LIDAS
32 / 32
2.16
26.63
14.57
44.03
40.58
4.36
31.58
Table 1: Performance comparison of standard transformer baselines, looped models, and two depth-grown models MIDAS [ 33 ] and LIDAS [ 20 ] at 360M and 1.7B base model sizes . Looped models often outperform iso-param baselines and are competitive with iso-inference baselines, especially for reasoning-heavy task categories such as Open-book Q&A, Math Word Problems and Reasoning Primitives. Depth-grown models match the baselines across most task categories with roughly 80% of the pre-training compute, while outperforming them on reasoning. This suggests a shared inductive bias toward reasoning for looped and depth-grown models. Best performance per model size in bold and looped model rows in gray.
Figure 2: Looped and depth-grown models use later layers more. We compare Baseline, LIDAS , \LoopOp(4×6) and \LoopOp(4-4×4-4) on (A) depth score, (B) top-5 vocabulary overlap on GSM8K and (C) Tuned Lens early-exit normalized accuracy on the Variable Assignment Math reasoning primitive. All three diagnostics imply higher usage of later layers for the grown LIDAS and looped models.
Figure 3: Looped and depth-grown models exhibit similar (sub)layer usage. LIDAS , \LoopOp(4×6) and \LoopOp(4-4×4-4) share a slower residual norm growth than the baseline and exhibit periodic attention-sublayer contributions (ratio of norms for attention sublayer output over residual) with a 4-layer cycle, matching the block size of LIDAS and the size of the recurrent block.
Figure 4: Local effect of skipping a layer on downstream layer contributions for future tokens. We intervene in the residual stream by removing the contribution of a layer from all subsequent layers separately ( local effect ) and measuring the relative change on the representations of all future tokens. LIDAS and \LoopOp(4-4×4-4) both show a characteristic phenomenon. Every four layers, there is a layer that depends directly on most of the previous inputs (vertical pattern): removing the contribution of a previous layer results in relatively large changes in the representation of future tokens for this ”aggregation” layer.
Figure 5: Fully looped models are less robust to interventions. Swapping a single layer degrades the performance of the fully looped model \LoopOp(4×6) considerably on Lambada and Variable Assignment Math compared to the other models. Using unique encoder–decoder layers lets \LoopOp(4-4×4-4) recover the robustness of the baseline and LIDAS .
Figure 6: Grown models benefit from looping at inference time. Repeating a block of four layers in the middle of the network during inference increases the accuracy of grown models ( MIDAS , LIDAS ) on the Copy Real Words reasoning primitive up to 2× compared to the original network. In contrast, the baseline rarely benefits from additional repetitions.
Figure 7: Looped and depth-grown models use in-context examples better. As we increase the number of in-context examples up to the maximum context length, the grown—and especially the looped—models improve, while the baseline often does not.
Figure 8: Looped and grown models benefit substantially more from supervised fine-tuning than the baseline. Shaded regions indicate ± one standard deviation over three random seeds (varying the fine-tuning dataset).
Table 10
Figure 9: LIDAS benefits the most from additional inference repetitions. Repeating a fixed 4-layer block at inference increases accuracy on the reasoning primitive Variable Assignment Basic . LIDAS and the baseline make the best use of additional blocks, up to 3 repetitions, after adapting the models to loop over that block once. The looped model does not benefit from further repetition of the recurrent block, but it also degrades less than the baseline, especially before the baseline is adapted to looping.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Category-wise trade-offs for looped and depth-grown models. Each point corresponds to a model in Section 1 (up to 1.7B parameters). For each benchmark category, we plot the average metric versus unique parameters (left), inference FLOPs (middle), and training FLOPs (right), complementing Fig. 1 .
Figure 11: Swapping consecutive 2-layer subblocks
Figure 12: Repeat-block ablations across reasoning primitive tasks without further training.
Figure 13: Future Local Effects
Figure 14: Mean Relative Layer Contribution
Figure 15: Supervised Fine Tuning Experiments
Figure 16: In-context learning behavior across all reasoning primitive tasks.
Figure 17: Retrofitted-recurrence block position ablation. We sweep which 4-layer block is looped once during cooldown adaptation for the baseline and LIDAS . Dashed curves show performance before adaptation, and solid curves show performance after adaptation of the respective block. Middle blocks, in particular the third (layers 8–11), tend to be the strongest candidates, especially for reasoning.
Figure 18: Repeating blocks without further training (NMT finetuning setting) on copying tasks and GSM8K.
Figure 19: Repeating blocks without further training (NMT finetuning setting) on Variable Assignment tasks.
Looped computation shows promise in improving the reasoning-oriented performance of LLMs by scaling test-time compute. However, existing approaches typically require either training recurrent models from scratch or applying disruptive retrofits, which involve substantial computational costs and may compromise pretrained capabilities. To address these limitations, we introduce Looped Depth Up-Scaling (LoopUS), a post-training framework that converts a standard pretrained LLM into a looped architecture. As a key technical contribution, LoopUS recasts the pretrained LLM into an encoder, a looped reasoning block, and a decoder. It operationalizes this latent-refinement architecture through four core components: (1) block decomposition, guided by staged representation dynamics; (2) an input-dependent selective gate to mitigate hidden-state drift; (3) random deep supervision for memory-efficient learning over long recursive horizons; and (4) a confidence head for adaptive early exiting. Collectively, these mechanisms transform a standard non-looped model into a looped form while stabilizing it against both computational bottlenecks and representation collapse. Through stable latent looping, LoopUS improves reasoning-oriented performance without extending the generated traces or requiring recurrent training from scratch. For more details, see https://thrillcrazyer.github.io/LoopUS
Taekhyun Park, Yongjae Lee, Dohee Kim +1
Department of Data Science Pusan National University Busan, Republic of Korea · Department of Industrial Engineering Pusan National University Busan, Republic of Korea · Department of Artificial Intelligence Engineering Changwon National University Changwon, Republic of Korea
Repeating a small block of middle layers increases a language model's effective inference depth without adding parameters or generating extra tokens, and recent work shows that this latent recurrence improves reasoning. However, two design choices limit these gains. Each iteration sees only the previous output and cannot directly access earlier computations. Moreover, a fixed loop count wastes depth on easy inputs while leaving hard ones with too little computation. We introduce RecurTrace, which addresses both limitations using the loop's own trajectory. Specifically, Loop Memory Attention lets each looped layer attend to its own states from previous iterations along the loop-time axis, so the model can revisit earlier computations instead of relying on the latest state alone. A halting head then reads the loop state and predicts whether to continue, with supervision from an oracle that identifies when additional depth still reduces loss. In a controlled MathQA comparison on the same looped backbone, RecurTrace achieves 56.9% accuracy with an average of 2.0 loops, exceeding the best fixed loop depth by 2.2 points at matched compute. By comparison, ACT and PonderNet collapse to one loop, and CALM reaches only 54.1% with 5.6 loops, while the stronger LoopUS-Conf and TaH-Mismatch baselines reach 55.3% at 3.2 loops and 55.7% at 2.1 loops. Finally, RecurTrace improves generation accuracy over same-budget fine-tuned baselines at 0.6B, 1.7B, 4B, and 8B, with the gain growing with model size from 0.6 to 3.4 points.
Yuxiang Wang, Kunyu Feng, Yingda Shen +3
The Chinese University of Hong Kong, Shenzhen · The Chinese University of Hong Kong · Tianjin University +2
Looped language models increase effective depth by repeatedly applying a shared block of layers, but existing large-scale recipes require multi-stage training over trillions of tokens, while the benefits of recurrence remain difficult to separate from differences in data and training. In this work, we establish practical training recipes for looped language models, with three main results. (1) We develop a compute-efficient from-scratch pipeline that reduces the training budget from 7.7T tokens in Ouro to 310B tokens while retaining strong reasoning performance. Pretraining followed by high-quality mid-training, together with learning-rate warmup and stronger exit-gate regularization, enables stable recurrent training without prior multi-stage schedules. (2) Under controlled comparisons, our 1.4B LoopLM outperforms a parameter-matched dense model trained on the same data and token budget on all 12 evaluated benchmarks, including +14 points on GSM8K, +10 on MATH, and +22 on DROP. At matched inference compute, it approaches a 3.9B dense model on mathematical reasoning and reading comprehension while using only 36% as many parameters. (3) We introduce a minimal recipe for converting pretrained dense models into looped ones: a single learned input-mixing scalar and a smoothed exit loss, with no step-specific parameters. Applied to Qwen3-1.7B-Base, Looped Qwen improves over an identically continued dense baseline on every evaluated benchmark across two data regimes, with statistically clear gains on GSM8K, MATH, and MMLU-Pro on the curated mixture. Together, these results make looped language models substantially cheaper to train from scratch and practical to introduce into existing pretrained checkpoints, while isolating the gains due to recurrence itself.
Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin +5