cs.CLFeb 18, 2026
SaveFrom Growing to Looping: A Unified View of Iterative Computation in LLMs
Organizations: Technical University of Munich · Helmholtz AI, Munich · Google, Paradigms of Intelligence Team
Abstract
Looping, reusing a block of layers across depth, and depth growing, training shallow-to-deep models by duplicating middle layers, have both been linked to stronger reasoning, but their relationship remains unclear. We provide a mechanistic unification: looped and depth-grown models exhibit convergent depth-wise signatures, including increased reliance on late layers and recurring patterns aligned with the looped or grown block. These shared signatures support the view that their gains stem from a common form of iterative computation. Building on this connection, we show that the two techniques are adaptable and composable: applying inference-time looping to the middle blocks of a depth-grown model improves accuracy on some reasoning primitives by up to , despite the model never being trained to loop. Both approaches also adapt better than the baseline when given more in-context examples or additional supervised fine-tuning data. Additionally, depth-grown models achieve the largest reasoning gains when using higher-quality, math-heavy cooldown mixtures, which can be further boosted by adapting a middle block to loop. Overall, our results position depth growth and looping as complementary, practical methods for inducing and scaling iterative computation to improve reasoning.
Figures & tables
Figure 1: Trade-offs for looped and depth-grown models. Each point corresponds to a model in Section 1 (up to 1.7B parameters), plotted by average Reasoning Primitives accuracy versus unique parameters (left), inference FLOPs (middle), and training FLOPs (right). Looped and depth-grown models improve accuracy in reasoning primitives over standard baselines, suggesting a shared inductive bias toward better reasoning. Looped models improve reasoning under fixed parameter budgets and can be competitive under fixed inference budgets, while depth-grown models reach similar or better reasoning with less training compute. Fig. 10 shows additional benchmark categories.
| Standard cooldown | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Params | Open-book | Closed-book | Math Word | Reasoning | |||||
| / | Holdout Set | Q&A | Q&A | Lambada | HellaSwag | Problems | Primitives | ||
| FLOPs | (NLL ) | (F1 ) | (F1 ) | (Acc ) | (Acc ) | (Acc ) | (Acc ) | ||
| 360M | Baseline | 32 / 32 | 2.18 | 22.89 | 14.50 | 43.35 | 39.97 | 3.69 | 30.04 |
| MIDAS | 32 / 32 | 2.18 | 24.57 | 13.75 | 43.26 | 40.36 | 4.39 | 28.42 | |
| LIDAS | 32 / 32 | 2.16 | 26.63 | 14.57 | 44.03 | 40.58 | 4.36 | 31.58 | |
Table 1: Performance comparison of standard transformer baselines, looped models, and two depth-grown models MIDAS [ 33 ] and LIDAS [ 20 ] at 360M and 1.7B base model sizes . Looped models often outperform iso-param baselines and are competitive with iso-inference baselines, especially for reasoning-heavy task categories such as Open-book Q&A, Math Word Problems and Reasoning Primitives. Depth-grown models match the baselines across most task categories with roughly of the pre-training compute, while outperforming them on reasoning. This suggests a shared inductive bias toward reasoning for looped and depth-grown models. Best performance per model size in bold and looped model rows in gray.
Figure 2: Looped and depth-grown models use later layers more. We compare Baseline, LIDAS , and on (A) depth score, (B) top-5 vocabulary overlap on GSM8K and (C) Tuned Lens early-exit normalized accuracy on the Variable Assignment Math reasoning primitive. All three diagnostics imply higher usage of later layers for the grown LIDAS and looped models.
Figure 3: Looped and depth-grown models exhibit similar (sub)layer usage. LIDAS , and share a slower residual norm growth than the baseline and exhibit periodic attention-sublayer contributions (ratio of norms for attention sublayer output over residual) with a 4-layer cycle, matching the block size of LIDAS and the size of the recurrent block.
Figure 4: Local effect of skipping a layer on downstream layer contributions for future tokens. We intervene in the residual stream by removing the contribution of a layer from all subsequent layers separately ( local effect ) and measuring the relative change on the representations of all future tokens. LIDAS and both show a characteristic phenomenon. Every four layers, there is a layer that depends directly on most of the previous inputs (vertical pattern): removing the contribution of a previous layer results in relatively large changes in the representation of future tokens for this ”aggregation” layer.
Figure 5: Fully looped models are less robust to interventions. Swapping a single layer degrades the performance of the fully looped model considerably on Lambada and Variable Assignment Math compared to the other models. Using unique encoder–decoder layers lets recover the robustness of the baseline and LIDAS .
Figure 6: Grown models benefit from looping at inference time. Repeating a block of four layers in the middle of the network during inference increases the accuracy of grown models ( MIDAS , LIDAS ) on the Copy Real Words reasoning primitive up to compared to the original network. In contrast, the baseline rarely benefits from additional repetitions.
Figure 7: Looped and depth-grown models use in-context examples better. As we increase the number of in-context examples up to the maximum context length, the grown—and especially the looped—models improve, while the baseline often does not.
Figure 8: Looped and grown models benefit substantially more from supervised fine-tuning than the baseline. Shaded regions indicate one standard deviation over three random seeds (varying the fine-tuning dataset).
Figure 9: LIDAS benefits the most from additional inference repetitions. Repeating a fixed 4-layer block at inference increases accuracy on the reasoning primitive Variable Assignment Basic . LIDAS and the baseline make the best use of additional blocks, up to 3 repetitions, after adapting the models to loop over that block once. The looped model does not benefit from further repetition of the recurrent block, but it also degrades less than the baseline, especially before the baseline is adapted to looping.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Category-wise trade-offs for looped and depth-grown models. Each point corresponds to a model in Section 1 (up to 1.7B parameters). For each benchmark category, we plot the average metric versus unique parameters (left), inference FLOPs (middle), and training FLOPs (right), complementing Fig. 1 .
Figure 11: Swapping consecutive 2-layer subblocks
Figure 12: Repeat-block ablations across reasoning primitive tasks without further training.
Figure 13: Future Local Effects
Figure 14: Mean Relative Layer Contribution
Figure 15: Supervised Fine Tuning Experiments
Figure 16: In-context learning behavior across all reasoning primitive tasks.
Figure 17: Retrofitted-recurrence block position ablation. We sweep which 4-layer block is looped once during cooldown adaptation for the baseline and LIDAS . Dashed curves show performance before adaptation, and solid curves show performance after adaptation of the respective block. Middle blocks, in particular the third (layers 8–11), tend to be the strongest candidates, especially for reasoning.
Figure 18: Repeating blocks without further training (NMT finetuning setting) on copying tasks and GSM8K.
Figure 19: Repeating blocks without further training (NMT finetuning setting) on Variable Assignment tasks.