What Makes Recurrence Effective in Looped Language Models?

Xinlin Zhuang; Siyuan Wang; Imran Razzak; Weiyang Liu

cs.LGSep 29, 2026

What Makes Recurrence Effective in Looped Language Models?

Authors: Xinlin Zhuang, Siyuan Wang, Imran Razzak, Weiyang Liu

Organizations: The Chinese University of Hong Kong · MBZUAI

Abstract

Looped language models (LoopLMs) increase computational depth through parameter sharing, offering a path to scale inference computation without adding parameters. However, it remains unclear when additional recurrence is beneficial and how architectural choices affect its effectiveness. Through controlled experiments, we systematically examine (1) when recurrence helps, (2) where it should be applied, and (3) how its conditioning affects performance. Our evaluation covers inference budgets below, within, and beyond the training horizon under knowledge and reasoning tasks. (1) We find that recurrence can improve reasoning beyond the training horizon while degrading knowledge performance, but harder reasoning instances do not consistently benefit more. (2) Performance also depends on how distinct layers and recurrent iterations are allocated, showing that effective depth alone is insufficient to predict behavior. Non-recurrent output layers improve robustness to under-unrolling, while the preferred placement of input and output layers varies with inference budget. (3) Finally, we find that conventional initial-state injection offers limited robustness to varying recurrence depth. We therefore propose history-state injection as an alternative, and show that channel-wise history-state injection combined with timestep conditioning offers a low-cost and more effective design, better preserving knowledge under extended unrolling while improving robustness across inference budgets. Overall, our results clarify when recurrent computation helps, where it fails, and offer practical guidelines for designing LoopLMs across variable inference budgets.

Figures & tables

Figure 1: Task- and architecture-dependent benefits of recurrence. (a) Score changes relative to $L(r)=20$ for BaseLoop $4\times 5$ across knowledge and reasoning splits. (b,c) Absolute accuracy (%) for BaseLoop $2\times 10$ across inference depths, stratified by ProofWriter proof depth and CLUTRR chain length. (d) Reasoning scores across varying configurations of physical depth and loop count. In (a,d) , dashed vertical lines mark the training depth, and shaded parts indicate the depth extrapolation.

Figure 2: BaseLoop and CoreLoop performance based on the Llama3.1-1B architecture configuration on Overall , Knowledge , and Reasoning benchmarks across inference depths. Dashed vertical lines mark the effective training depth

L(K)=20

, and shaded regions denote loop extrapolation.

Figure 3: Geometric dynamics of BaseLoop and CoreLoop models with 4 physical layers from the Llama3.1-1B architecture. The horizontal axes in (b) - (d) are normalized by each configuration’s training iterations. (a) Angular distance between consecutive layer states, with crosses marking Coda layers. (b) Angular distance between recurrent states before and after each complete loop iteration. (c) Per-loop update magnitude relative to the preceding state. (d) Recurrent-state variance at the training horizon. (c-d) use logarithmic scales.

Figure 4: Initial-state injection results on Qwen3-0.6B BaseLoop configurations. BaseLoop provides the reference without state conditioning, and NonLoop

28\times 1

provides a non-recurrent reference.

Figure 5: History-state injection on the Qwen3-0.6B BaseLoop

4

\times

7

configuration. BaseLoop gives the reference without injection, and NonLoop

28

\times

1

is a non-recurrent reference. The dashed vertical line marks the effective training depth

L(K)=28

, and shading region indicates loop extrapolated up to

3\times

the training depth.

Figure 6: Timestep conditioning results of the Qwen3-0.6B backbone. All timestep variants are compared using rescaled time grids at inference. NonLoop

28\times 1

provides an unshared reference.

Figure 7: Performance of combined conditioning mechanisms for Qwen3-0.6B BaseLoop

4\times 7

. All state conditioning uses the channel-wise parameterization, with history window

w=2

. In the legends, I and H denote initial-state and history-state injection, LG denotes loop gating, and BG denotes branch gating.

Figure 8: Results of relative update responses to skipping an earlier block occurrence. (a) Llama BaseLoop/CoreLoop at

D=40

: responses averaged over separate skips at positions 21-24, divided by each model’s mean response in its final recurrent loop. Crosses mark coda blocks. (b,c) Qwen BaseLoop

4\times 7

: mean

C

grouped by loop lag for initial-state injection and pairwise conditioning. (d) Three-mechanism responses divided by the corresponding H+T responses, with the timestep and history settings matched. All state injections in (c,d) are channel-wise, with

w=2

for history. In (b-d) , dashed curves with open circles denote

D=56

and solid curves with filled triangles denote

D=84

; all measured lags are shown.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Architecture	$\bm{L_{\mathrm{ref}}}$	Parameters	$\bm{\|\mathcal{V}\|}$	$\bm{d_{\mathrm{model}}}$	$\bm{d_{\mathrm{ff}}}$	Q/KV heads	$\bm{d_{\mathrm{head}}}$
Llama3.1-1B	20	1,007,482,368	128,256	1,536	5,376	12/3	128
Qwen3-0.6B	28	596,049,920	151,936	1,024	3,072	16/8	128

Appendix

Table 1: Non-recurrent reference architectures.

L_{\mathrm{ref}}

denotes the number of Transformer blocks in the corresponding reference configuration. Parameter counts include token embeddings, the final RMSNorm, and the language-model head.

Backbone	$\bm{L_{\mathrm{eff}}}$	$\bm{L_{\mathrm{phys}}}$	Parameters	BaseLoop	CoreLoop
Llama3.1-1B	20	2	455,351,808	$2\times 10$	$0+1\times 19+1$ , $1+1\times 19+0$
Llama3.1-1B	20	4	516,699,648	$4\times 5$	$0+2\times 9+2$ , $1+2\times 9+1$ , $2+2\times 9+0$
Llama3.1-1B	20	5	547,373,568	$5\times 4$	$0+3\times 6+2$ , $1+3\times 6+1$ , $2+3\times 6+0$
Llama3.1-1B	20	10	700,743,168	$10\times 2$	$0+5\times 3+5$ , $1+5\times 3+4$ , $2+5\times 3+3$ , $3+5\times 3+2$ , $4+5\times 3+1$ , $5+5\times 3+0$
Qwen3-0.6B	28	4	218,507,264	$4\times 7$	$0+2\times 13+2$ , $1+2\times 13+1$ , $2+2\times 13+0$

Appendix

Table 2: Details of BaseLoop and CoreLoop models. An allocation is written as

p+s\times K+c

. Architectures within a row have the same physical depth, total parameter count, and training effective depth. The reported parameter counts correspond to the unconditioned naive-loop models, without adding any input-injection or timestep-conditioning parameters.

Models	$\bm{\eta_{\max}}$	Global batch	Steps	W/S/D updates	Tokens
Llama3.1-1B BaseLoop and CoreLoop variants	$1.33847\times 10^{-3}$	1,824	5,394	270/4,585/539	20.15B
Qwen3-0.6B BaseLoop and CoreLoop variants	$1.80111\times 10^{-3}$	2,048	2,860	143/2,431/286	12.00B

Appendix

Table 3: Training settings for LoopLM models. Global batch size is measured in packed sequences of length 2,048. W/S/D gives the exact numbers of warmup, stable, and decay updates. Token counts are computed before the one-token shift used by the next-token objective.

Task	Brief Description
SciQ Welbl et al. (2017)	Science facts; locating evidence in a support passage
ARC-Easy Clark et al. (2018)	Elementary science knowledge and simple causal attribution
PIQA Bisk et al. (2020)	Object affordances, material properties, outcomes of actions

Appendix

Table 4: Three tasks in the Knowledge group.

Task	Brief Description
ARC-Challenge Clark et al. (2018)	Multi-step inference over harder science items
WinoGrande Sakaguchi et al. (2020)	Coreference resolution requiring commonsense
OpenBookQA Mihaylov et al. (2018)	Combining a retrieved science fact with additional knowledge
HellaSwag Zellers et al. (2019)	Plausible event continuation; temporal and causal structure
CommonsenseQA Talmor et al. (2019)	ConceptNet-style relational reasoning over use, location
ProofWriter Tafjord et al. (2021)	Entailment under natural-language facts and rules

Appendix

Table 5: Eight tasks in the Reasoning group. The last three tasks are the ones whose difficulty is controlled by an explicit compositional parameter.

Subtask	Brief Description
disambiguation_qa	Resolves pronoun antecedents and identifies cases that remain genuinely ambiguous.
salient_translation_error_detection	Classifies salient semantic errors in German-to-English translations, such as altered entities, numbers, negation, or omitted content.
reasoning_about_colored_objects	Answers attribute, spatial-relation, and counting questions about colored objects described in natural language.
causal_judgement	Determines commonsense causal attribution and whether an action or outcome was intentional.
date_understanding	Infers calendar dates from relative temporal expressions and performs date arithmetic.
hyperbaton	Selects the sentence exhibiting the grammatically natural ordering of English adjectives.

Appendix

Table 6: The selected 14 BBH subtasks and their evaluated capabilities.

Model	ARC-E	ARC-C	SciQ	MMLU	HELLA	OBQA	PIQA	RACE	WINO	CSQA	Avg.
AdamW
nonloop (12 $\times$ 1)	59.30	25.68	81.50	26.06	33.64	22.00	67.52	31.67	51.46	20.64	41.95
nonloop (3 $\times$ 1)	53.41	20.90	74.30	23.10	29.04	18.40	63.38	27.56	50.99	20.31	38.14
BaseLoop (3 $\times$ 4)	56.23	23.81	77.00	23.00	31.05	21.40	64.20	30.33	53.28	19.49	39.98
LoopFormer (3 $\times$ 4)	56.73	23.89	77.90	24.59	30.97	20.00	64.64	29.28	51.62	19.98	39.96
Ouro (3 $\times$ 4)	55.85	22.87	77.90	23.03	30.74	18.00	63.82	29.95	52.49	19.57	39.42

Appendix

Table 7: Pilot study comparing AdamW and Muon for pre-training different non-recurrent and recurrent architectures. Results are reported on ten downstream evaluation benchmarks together with their average. Failed indicates that the corresponding pre-training run did not successfully converge.

Figure 9: Average results of BaseLoop and CoreLoop models from Qwen3-0.6B with physical layers of 4 on Overall , Knowledge , and Reasoning benchmarks. The dashed vertical line marks the effective training depth of 28 layers. Results at or below this depth correspond to evaluation within the training depth budget, while the yellow-shaded region denotes extrapolation beyond it.

Figure 10: Representation dynamics of BaseLoop and CoreLoop models with two physical layers from Llama3.1-1B. The horizontal axes in (b) - (d) are normalized by each configuration’s training iteration count. (a): The angular distance between consecutive layer states. Crosses mark coda layers at the ends of the recorded trajectories. (b): The same angular measure between recurrent states before and after each complete loop iteration. (c): The per-loop update magnitude relative to the preceding state, plotted on a logarithmic scale. (d): Recurrent-state variance relative to its value at the training iteration count, plotted on a logarithmic scale.

Figure 11: Representation dynamics of BaseLoop and CoreLoop models with five physical layers from Llama3.1-1B. BaseLoop

5\times 4

is compared with three CoreLoop variants that allocate two fixed layers between the prelude and coda.

Figure 12: Representation dynamics of BaseLoop and CoreLoop models with ten physical layers from Llama3.1-1B. BaseLoop

10\times 2

is compared with six CoreLoop variants

p+5\times 3+(5-p)

p=0,\ldots,5

, covering all allocations of five fixed layers between the prelude and coda.

Figure 13: Initial-state input injection on Llama3.1-1B. Columns correspond to four shared-stack configurations; rows report Overall, Knowledge, and Reasoning scores. Curves include all available evaluation depths. Vertical dashed lines mark the training depth

D=20

, and shading denotes extrapolation. BaseLoop uses no injection; NonLoop

20\times 1

is the unshared reference.

		$\bm{D=20}$			$\bm{D=40}$
Configuration	Method	Overall	Knowledge	Reasoning	Overall	Knowledge	Reasoning
$2\times 10$	BaseLoop	36.83	61.10	27.73	34.58	50.37	28.66
	Scalar	37.26	62.33	27.85	34.90	53.64	27.87
	Channel-wise	37.54	60.98	28.75	35.01	53.41	28.10
	Residual Channel-wise	37.02	60.46	28.23	32.52	41.39	29.19
	Dense	36.66	59.23	28.19	29.96	32.78	28.90

Appendix

Table 8: Full initial-state injection results on Llama3.1-1B at the training depth

D=20

and twice that depth

D=40

. Configuration denotes the shared-stack depth and training loop count. Bold marks the best result within each recurrent configuration for each metric and evaluation depth. NonLoop is evaluated only at its native depth.

		$\bm{D=28}$			$\bm{D=56}$
Configuration	Method	Overall	Knowledge	Reasoning	Overall	Knowledge	Reasoning
$2\times 14$	BaseLoop	36.85	58.70	28.66	37.31	57.30	29.81
	Scalar	36.27	58.00	28.12	35.71	56.89	27.77
	Channel-wise	35.87	58.01	27.57	35.59	56.56	27.73
	Residual Channel-wise	36.77	57.88	28.85	32.73	42.49	29.07
	Dense	35.18	55.95	27.39	30.30	33.15	29.24

Appendix

Table 9: Full initial-state injection results on Qwen3-0.6B at the training depth

D=28

and twice that depth

D=56

		$\bm{D=28}$			$\bm{D=56}$			$\bm{D=84}$
Parameterization	Injection	Overall	Knowledge	Reasoning	Overall	Knowledge	Reasoning	Overall	Knowledge	Reasoning
–	BaseLoop	36.80	59.44	28.31	36.59	57.33	28.81	35.60	52.29	29.34
Scalar	Initial-state	36.16	58.35	27.84	36.34	56.05	28.95	35.69	52.10	29.53
	History ( $w=1$ )	36.45	59.41	27.84	35.74	57.67	27.51	34.69	52.23	28.12
	History ( $w=2$ )	36.46	58.83	28.07	36.78	59.18	28.38	35.75	55.83	28.22
	History ( $w=4$ )	36.61	58.90	28.25	36.80	59.41	28.32	37.24	59.18	29.01

Appendix

Table 10: Full history-state input injection results on Qwen3-0.6B with the

4\times 7

shared-stack configuration.

D=28

is the training depth;

D=56

and

D=84

correspond to

2\times

and

3\times

depth extrapolation. Initial-state and History denote initial-state-only and history-state-only injection, respectively;

w

denotes the history window size. BaseLoop uses no injection. All scores are percentages, with higher values indicating better performance. Bold marks the best result in each column across all listed configurations.

		$\bm{D=20}$			$\bm{D=40}$
Configuration	Method	Overall	Knowledge	Reasoning	Overall	Knowledge	Reasoning
$4\times 5$	BaseLoop	37.87	62.80	28.52	37.11	52.11	31.49
	Loop Gating	37.99	63.32	28.49	37.43	56.17	30.41
	Branch Gating	37.99	62.39	28.84	37.23	56.60	29.97
	AdaLN	38.58	62.65	29.56	37.91	59.68	29.75
$20\times 1$	NonLoop	39.83	65.38	30.25	–	–	–

Appendix

Table 11: Timestep conditioning results on Llama3.1-1B at the training depth

D=20

and twice that depth

D=40

. All timestep variants use rescaled time grids at inference. Configuration denotes the shared-stack depth and training loop count. Bold marks the best result within the recurrent configuration for each metric and evaluation depth. NonLoop is evaluated only at its native depth.

		$\bm{D=28}$			$\bm{D=56}$
Configuration	Method	Overall	Knowledge	Reasoning	Overall	Knowledge	Reasoning
$2\times 14$	BaseLoop	36.85	58.70	28.66	37.31	57.30	29.81
	Loop Gating	36.16	57.51	28.16	35.18	57.07	26.97
	Branch Gating	36.62	58.20	28.53	36.64	58.01	28.63
	AdaLN	35.97	58.44	27.54	35.35	56.54	27.41
$4\times 7$	BaseLoop	36.80	59.44	28.31	36.59	57.33	28.81

Appendix

Table 12: Timestep conditioning results on Qwen3-0.6B at the training depth

D=28

and twice that depth

D=56

. All timestep variants use rescaled time grids at inference. Configuration denotes the shared-stack depth and training loop count. Bold marks the best result within each recurrent configuration for each metric and evaluation depth. NonLoop is evaluated only at its native depth.

Figure 14: Results of Scalar initial plus history injection combination and their individual components across history windows

w=1,2,4

with the Qwen3-0.6B

4\times 7

configuration.

Figure 15: Results of Channel-wise initial plus history injection combination and their individual components across history windows

w=1,2,4

with the Qwen3-0.6B

4\times 7

configuration.

Figure 16: Results of Dense initial plus history injection combination and their individual components across history windows

w=1,2,4

with the Qwen3-0.6B

4\times 7

configuration.

Figure 17: Results of Dense initial plus timestep combination and their individual components with the Qwen3-0.6B

4\times 7

configuration.

Figure 18: Results of Dense history injection plus timestep combination and their individual components for history windows

w=1,2

with the Qwen3-0.6B

4\times 7

configuration.

Figure 19: Results of Channel-wise history injection combined with BG and their individual components across history windows

w=1,2,4

with the Qwen3-0.6B

4\times 7

configuration.

Figure 20: Results of Channel-wise history injection combined with LG and their individual components across history windows

w=1,2,4

with the Qwen3-0.6B

4\times 7

configuration.

Figure 21: Results of three-mechanism combination and their individual components across history windows

w=1,2

with the Qwen3-0.6B

4\times 7

configuration.

Figure 22: Average results for Qwen3-0.6B BaseLoop

4\times 7

with Scalar, Channel-wise, and Residual Channel-wise initial-state input injection under an extended effective layer of 84 (3x training budget). The vertical boundary marks the training depth of 28 layers, with shading indicating depth extrapolation. Initialization noise has a limited effect on performance, supporting our default choice of initialization without noise.

Figure 23: Additional computational interaction results for Qwen3-0.6B BaseLoop

4\times 7

, using the probes defined in App. C.4 . All state conditioning is channel-wise, with history window

w=2

. (a) Mean unnormalized response

N(d)

for pairwise conditioning, showing all measured lags. (b) Ratios of mean

C

for H+LG versus I+H and I+LG, separately for each of the 16 shared source/target block pairs over lags

1

6

. (c) Triple/double ratios of mean

C

, mean

N

, and mean

B

over lags

2

6

, with the gate type and history settings matched. (d) Triple/double ratios of mean

C

for each shared-block pair over lags

2

6

. All means weight occurrence pairs equally on the indicated lag support;

B_{j}

is repeated for each included source paired with target

j

. Configuration ratios divide these means, and mean

C

is the mean of

N_{i\to j}/B_{j}

, not the ratio of mean

N

to mean

B

. In ( b,d ),

s{:}q

denotes one-based source and target block indices; vertical separators distinguish source blocks. Lines between block pairs or metrics are visual guides. Dashed curves with open circles denote

D=56

; solid curves with filled triangles denote

D=84

. Horizontal references at one indicate equal values.

Explore similar work

Aug 18, 2026

Allocating Recurrent Compute in Looped Language Models

Looped language models improve reasoning and knowledge manipulation by applying shared computation repeatedly. Existing systems usually repeat an entire layer stack, although a mixer and a dense feed-forward network (FFN) perform different operations and have different costs. We ask a narrower question: what should loop? We view recurrence as repeated composition of a state update and argue that an application is valuable when it exposes a new cross-position influence direction that remains observable at the task readout. Iterative Transport Rank (ITR) describes the cumulative influence trajectory; marginal ITR describes the nonredundant influence contributed by successive applications. This view motivates MixerLoop, which repeats each Gated DeltaNet mixer while applying its dense FFN once. We compare MixerLoop with no recurrence and full-block recurrence at 15M and 110M parameters under the same data, initialization, and architecture. A finite context-off intervention tests whether later mixer applications produce distinct, non-negligible, and beneficial changes at the final language-model readout. MixerLoop surpasses FullLoop on aggregate CORE at 15M and retains 41.5% of its CORE improvement at 110M while reducing recurrent-backbone projection FLOPs by 45.9%. These results show that the benefits of recurrent depth can be retained without repeatedly executing the dense FFN.

Ruhai Lin, Yiyang Guo, Rui-Jie Zhu +2

University of California, Santa Cruz

Recurrent Transformers

Ask PDF

Sep 3, 2026

RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory

Repeating a small block of middle layers increases a language model's effective inference depth without adding parameters or generating extra tokens, and recent work shows that this latent recurrence improves reasoning. However, two design choices limit these gains. Each iteration sees only the previous output and cannot directly access earlier computations. Moreover, a fixed loop count wastes depth on easy inputs while leaving hard ones with too little computation. We introduce RecurTrace, which addresses both limitations using the loop's own trajectory. Specifically, Loop Memory Attention lets each looped layer attend to its own states from previous iterations along the loop-time axis, so the model can revisit earlier computations instead of relying on the latest state alone. A halting head then reads the loop state and predicts whether to continue, with supervision from an oracle that identifies when additional depth still reduces loss. In a controlled MathQA comparison on the same looped backbone, RecurTrace achieves 56.9% accuracy with an average of 2.0 loops, exceeding the best fixed loop depth by 2.2 points at matched compute. By comparison, ACT and PonderNet collapse to one loop, and CALM reaches only 54.1% with 5.6 loops, while the stronger LoopUS-Conf and TaH-Mismatch baselines reach 55.3% at 3.2 loops and 55.7% at 2.1 loops. Finally, RecurTrace improves generation accuracy over same-budget fine-tuned baselines at 0.6B, 1.7B, 4B, and 8B, with the gain growing with model size from 0.6 to 3.4 points.

Yuxiang Wang, Kunyu Feng, Yingda Shen +3

The Chinese University of Hong Kong, Shenzhen · The Chinese University of Hong Kong · Tianjin University +2

Memory-Augmented Language Models Recurrent Transformers LLM Inference Acceleration Adaptive Computation

Ask PDF

Sep 30, 2026

Closing the Loop: Practical Training Recipes for Looped Language Models

Looped language models increase effective depth by repeatedly applying a shared block of layers, but existing large-scale recipes require multi-stage training over trillions of tokens, while the benefits of recurrence remain difficult to separate from differences in data and training. In this work, we establish practical training recipes for looped language models, with three main results. (1) We develop a compute-efficient from-scratch pipeline that reduces the training budget from 7.7T tokens in Ouro to 310B tokens while retaining strong reasoning performance. Pretraining followed by high-quality mid-training, together with learning-rate warmup and stronger exit-gate regularization, enables stable recurrent training without prior multi-stage schedules. (2) Under controlled comparisons, our 1.4B LoopLM outperforms a parameter-matched dense model trained on the same data and token budget on all 12 evaluated benchmarks, including +14 points on GSM8K, +10 on MATH, and +22 on DROP. At matched inference compute, it approaches a 3.9B dense model on mathematical reasoning and reading comprehension while using only 36% as many parameters. (3) We introduce a minimal recipe for converting pretrained dense models into looped ones: a single learned input-mixing scalar and a smoothed exit loss, with no step-specific parameters. Applied to Qwen3-1.7B-Base, Looped Qwen improves over an identically continued dense baseline on every evaluated benchmark across two data regimes, with statistically clear gains on GSM8K, MATH, and MMLU-Pro on the curated mixture. Together, these results make looped language models substantially cheaper to train from scratch and practical to introduce into existing pretrained checkpoints, while isolating the gains due to recurrence itself.

Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin +5

RND NLP, DAIMLD, Russian Federation

Efficient Language Model Training LLM Training Recursive Language Models

Ask PDF