Towards Looped Models Done Right, Part I: Topology, Input Injection, Recurrent-State Design

Benhao Huang; Chufan Shi; Junlin Chen; Shicheng Wen; Zhengzhong Liu; Eric Xing; Xuezhe Ma

cs.LGOct 16, 2026

Towards Looped Models Done Right, Part I: Topology, Input Injection, Recurrent-State Design

Authors: Benhao Huang, Chufan Shi, Junlin Chen, Shicheng Wen, Zhengzhong Liu, Eric Xing, Xuezhe Ma

Organizations: Institute of Foundation Models · CMU · USC

Abstract

Looped language models reuse physical Transformer blocks across logical depth, but established architectures entangle recurrence placement, input injection, and recurrent state organization. We isolate these axes in models trained from scratch at matched parameter scale, logical depth, and token budgets, and then test whether the resulting design principles transfer to resident- and active-parameter-matched Mixture-of-Experts (MoE) models. The prelude-recurrent block-coda sandwich architecture consistently improves performance on context extraction and reasoning-intensive tasks. Input injection primarily enhances knowledge retrieval and context utilization, whereas random initialization and separate H/L recurrent states provide mixed or negative returns. Taken together, these results favor the Huginn-style design: at the 730M-parameter and 336B-token scale, the complete Huginn architecture outperforms Ouro across all ten benchmarks. The advantages of Huginn over Ouro transfer to MoE models. At the 8BA0.8B (32BA3B when unrolled) scale trained on 500B tokens, Huginn outperforms Ouro on nearly all benchmarks spanning knowledge, reasoning, and coding, while also exhibiting more balanced expert routing. Under matched training and inference FLOPs, the Huginn-style model uses 75% fewer resident parameters than a 112-layer feedforward MoE baseline. Despite its smaller physical parameters, it outperforms the feedforward baseline on DROP and GSM8K, matches it on MATH500, and remains competitive on the other benchmarks.

Figures & tables

Figure 2: Selected looped-model works organized by family and quarter. Huginn-family: [ 3 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 6 , 5 , 27 , 28 , 29 , 30 , 31 ] . Ouro-family: [ 2 , 32 , 33 , 34 , 35 , 12 , 36 , 37 , 38 , 9 , 39 ] . Cross-family mechanistic comparisons remain limited [ 40 ] . Counts include all cited family-assigned papers 2 2 2 Literature cutoff: July 24, 2026. ; the axis markers show a visual subset.

Panel of Figure 3 — Figure 3: Architecture comparisons. Feedforward models use distinct blocks. Ouro-style models reuse a full Transformer stack. Huginn-like models initialize the recurrent hidden stream from a prior, place a tied iterative body between untied prelude and coda stacks, and inject the prelude representation into the iterations.

Model specification	Ouro -style	Huginn -style
Prelude $P_{\theta}$	identity	8 untied layers
Initializer $\phi_{\theta}$	$\mathbf{z}_{0}=\mathbf{x}_{0}$	$\mathbf{z}_{0}=\bm{\xi}$
Input write $W_{\theta}$	$W_{\theta}(\mathbf{z}_{t},\mathbf{e})=\mathbf{z}_{t}$	$\bm{\alpha}\odot\mathbf{z}_{t}+\bm{\delta}\odot(\mathbf{W}_{\mathrm{in}}\mathbf{e})$
Shared body $R_{\theta}$	tied 28-block stack	tied 12-block middle body
Coda $C_{\theta}$	identity	8 untied layers
Iterations $T$	4	8

Table 1: Ouro - and Huginn -style instantiations of the common iterative-model notation, when given a layer budget of 112. Each row isolates one architectural choice.

Panel of Figure 4 — Figure 4: The controlled architecture ablations in Q1 & Q2. Q1 moves recurrence from the full stack into a sandwich loop between untied prelude and coda layers; Q2 then further injects the evolving $\mathbf{z}_{t}$ with the fixed prelude embedding $\mathbf{e}$ before every recurrent-core iteration.

Group	Benchmark	$D_{28}$	$D_{112}$	$R_{28}^{4}$	$P_{8}R_{12}^{8}C_{8}$	$\Delta$
		Feedforward reference		Loop topology
General	ARC-C	36.39	43.78	39.40	38.45	$-0.94$
	HSwag	41.73	50.28	44.71	45.01	$+0.30$
	MMLU	39.86	54.85	46.32	47.27	$+0.95$
	BBH-CoT	34.42	55.91	45.83	48.30	$+2.47$
	TQA	25.21	38.83	28.51	28.60	$+0.09$

Table 2: Q1 envelope comparison at 460 TPP (730M parameters; 336B training tokens). Feedforward columns are references; bold compares only the loop variants, and

\Delta

is sandwich minus full-stack. Green and red text marks positive and negative changes, respectively.

Figure 5: Evaluation trajectories for the full-stack and middle-loop sandwich variants.

		Middle loop			Full-stack Ouro
Group	Benchmark	no write	prelude write	$\Delta$	no write	raw-token write	$\Delta$
General	ARC-C	38.45	37.60	$-0.86$	39.40	39.06	$-0.34$
	HSwag	45.01	45.37	$+0.36$	44.71	45.65	$+0.94$
	MMLU	47.27	49.79	$+2.53$	46.32	48.11	$+1.79$
	BBH-CoT	48.30	54.94	$+6.63$	45.83	47.63	$+1.80$
	TQA	28.60	29.42	$+0.82$	28.51	28.53	$+0.02$

Table 3: Recurrent input-injection controls at 460 TPP (730M parameters; 336B training tokens). For the middle loop,

\Delta

denotes prelude-injection minus no injection; for full-stack Ouro , it denotes raw-token injection minus no injection. Boldface compares models within the same family, while green and red values indicate positive and negative changes, respectively.

Figure 6: Evaluation trajectories over matched token budgets for input-injection ablations. The upper two rows compare the middle loop without and with prelude injection; the lower two rows compare full-stack ( Ouro -style) recurrence without and with token embedding injection.

Panel of Figure 7 — Figure 7: Shared high/low latent-state update. The same recurrent body alternates low-state refinements with a high-state update.

Group	Benchmark	$\mathbf{z}_{0}=\mathbf{e}$	$\mathbf{z}_{0}\sim\mathcal{N}$	$\mathrm{H/L}$	$\Delta$
		Latent state			$\mathrm{H/L}-(\mathbf{z}_{0}=\mathbf{e})$
General	ARC-C	37.60	40.94	39.83	$+2.23$
	HSwag	45.37	45.47	45.90	$+0.53$
	MMLU	49.79	48.50	49.49	$-0.30$
	BBH-CoT	54.94	54.43	55.12	$+0.18$
	TQA	29.42	28.97	30.17	$+0.75$

Table 4: Matched recurrent-state comparison at 460 TPP (730M parameters; 336B training tokens). Bold marks the best available score.

\Delta

is shared high/low (

\mathrm{H/L}

) minus the input-initialized single state (

\mathbf{z}_{0}=\mathbf{e}

). Green and red text mark positive and negative changes.

Figure 8: Evaluation benchmark trajectories for the three latent-state organizations.

Figure 9: Raw benchmark scores along the cumulative dense construction path at 336B training tokens. Starting from full-stack Ouro (

\mathrm{O}

), the stages add the untied prelude/coda sandwich envelope (

+\mathrm{P/C}

), prelude-conditioned input injection (

+\mathrm{W}

), and a separate random-initialized recurrent state (

+z

), yielding complete Huginn -style looped models. Horizontal lines show the matched

D_{28}

and

D_{112}

feedforward references.

		MoE architecture
Group	Benchmark	Feedforward $D_{112}^{\mathrm{MoE}}$	Ouro $R_{28}^{4}$	Huginn $P_{8}R_{12}^{8}C_{8}$	$\Delta$
General	ARC-C	53.22	51.33	50.47	$-0.86$
	HSwag	57.30	54.82	55.10	$+0.28$
	MMLU	67.13	64.03	66.81	$+2.78$
	BBH-CoT	71.76	69.05	68.85	$-0.20$
	TQA	59.02	50.98	51.04	$+0.07$

Table 5: Benchmark performance of looped MoE models and the feedforward MoE reference after training on 500B tokens. The looped models contain 8.0B resident and 793.9M active parameters per physical pass, corresponding to approximately 630 training tokens per active parameter. Bold compares only the recurrent rows;

\Delta

is Huginn MoE minus Ouro MoE ; green and red mark positive and negative changes, respectively.

Figure 10: Normalized MoE load-balancing loss across training for the recurrent models and the 112-layer feedforward MoE reference (lower is better). Curves show 0.5B-token bin medians after robust smoothing. At 500B tokens, Huginn remains below both comparators, and the feedforward reference lies between Huginn and Ouro .

Figure 11: Iteration-dependent top-2 expert activation at the 500B-token checkpoint across all 12 recurrent layers, computed on a 48-input probe set with eight examples each from ARC-Challenge, MMLU, GSM8K, MATH500, HumanEval, and MBPP. Each panel shows the fraction of valid tokens activating each expert over the eight loop iterations, using one shared color scale.

Figure 12: Original accuracy and accuracy after iterations 2–8 reuse the loop-1 top-2 expert identities on MMLU, ARC-Easy, BoolQ, ARC-Challenge, PIQA, and COPA. The gap at right is original minus frozen-route accuracy.

Figure 13: Dense and MoE gaps to the 112-layer feedforward references. For each model family

f\in\{\mathrm{dense},\mathrm{MoE}\}

g_{f}=S(D_{112}^{f})-S(\textsc{Huginn}^{f})

; positive values favor the feedforward reference. Panel (a) compares the individual benchmark gaps at the shared endpoint. Each green or red

\Delta

g_{\mathrm{MoE}}-g_{\mathrm{dense}}

, so negative values indicate that MoE narrows the recurrent-to-feedforward gap. Panel (b) follows the same gap comparison over training on the fixed seven-task intersection (ARC-C, HSwag, MMLU, BBH-CoT, TQA, DROP, and HumanEval + ).

	Geometry			MoE			Parameters
Model	Phys.	Topology	MoE phys.	$E$	Top- $k$	$d_{\mathrm{expert}}$	Real	Active	Loop-equiv.	Unrolled active
Feedforward reference
Feedforward MoE	112	$D_{112}$	112	25	2	2432	32.0B	3.2B	n/a	n/a
Looped models
Huginn MoE	28	$P_{8}R_{12}^{8}C_{8}$	28	25	2	2432	8.0B	793.9M	32.0B	3.2B
Ouro MoE	28	$R_{28}^{4}$	28	25	2	2432	8.0B	793.9M	32.0B	3.2B

Table 6: Model geometry and non-embedding parameter accounting for the MoE models.

Task	Split (examples)	Shots	Metric	Input / output cap
ARC-C	Shared local test (1,165)	0	$\mathrm{Acc}_{\mathrm{tok}}$	512 / –
HellaSwag	Validation (10,042)	0	Acc	256 / –
MMLU	Test, 57 subjects (14,042)	5	Micro Acc	2,048 / –
BBH-CoT	Full test (6,511)	3	EM	4,096 / 1,024
TriviaQA	Local full test (11,313)	5	F1	512 / 24
DROP	Test (9,535)	3	F1	2,500 / 64

Table 7: Task-level evaluation settings for the reported benchmark results.

Recurrent Transformers

Recurrent Transformers reuse layers or state across steps to extend context, reasoning depth, and sequence modeling with less parameter growth.

See all 183 papers

What Makes Recurrence Effective in Looped Language Models?
Sep 29, 2026Xinlin Zhuang, Siyuan Wang, Imran Razzak +1Recursive Language Models Efficient Language Model Inference
The Chinese University of Hong Kong · MBZUAI
Closing the Loop: Practical Training Recipes for Looped Language Models
Sep 30, 2026Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin +5Efficient Language Model Training LLM Training
RND NLP, DAIMLD, Russian Federation
Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts
Oct 1, 2026Di He, Pengxiang Li, Da Chang +3Mixture-of-Experts Language Models Recurrent Transformers
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Peng Cheng Laboratory · University of Chinese Academy of Sciences +5

Sep 29, 2026J/K move · Enter open · S save

What Makes Recurrence Effective in Looped Language Models?

Xinlin Zhuang, Siyuan Wang, Imran Razzak, Weiyang Liu

The Chinese University of Hong Kong · MBZUAI

Recursive Language Models Efficient Language Model Inference Recurrent Transformers Inference-Time Scaling

Open paper AskPDF arXiv

Looped language models (LoopLMs) increase computational depth through parameter sharing, offering a path to scale inference computation without adding parameters. However, it remains unclear when additional recurrence is beneficial and how architectural choices affect its effectiveness. Through controlled experiments, we systematically examine (1) when recurrence helps, (2) where it should be applied, and (3) how its conditioning affects performance. Our evaluation covers inference budgets below, within, and beyond the training horizon under knowledge and reasoning tasks. (1) We find that recurrence can improve reasoning beyond the training horizon while degrading knowledge performance, but harder reasoning instances do not consistently benefit more. (2) Performance also depends on how distinct layers and recurrent iterations are allocated, showing that effective depth alone is insufficient to predict behavior. Non-recurrent output layers improve robustness to under-unrolling, while the preferred placement of input and output layers varies with inference budget. (3) Finally, we find that conventional initial-state injection offers limited robustness to varying recurrence depth. We therefore propose history-state injection as an alternative, and show that channel-wise history-state injection combined with timestep conditioning offers a low-cost and more effective design, better preserving knowledge under extended unrolling while improving robustness across inference budgets. Overall, our results clarify when recurrent computation helps, where it fails, and offer practical guidelines for designing LoopLMs across variable inference budgets.

Abstract

Recurrent Transformers

Explore similar work

What Makes Recurrence Effective in Looped Language Models?

Closing the Loop: Practical Training Recipes for Looped Language Models

Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts