Allocating Recurrent Compute in Looped Language Models

Ruhai Lin; Yiyang Guo; Rui-Jie Zhu; Hao Ye; Jason K. Eshraghian

cs.LGAug 18, 2026

Allocating Recurrent Compute in Looped Language Models

Authors: Ruhai Lin, Yiyang Guo, Rui-Jie Zhu, Hao Ye, Jason K. Eshraghian

Organizations: University of California, Santa Cruz

Abstract

Looped language models improve reasoning and knowledge manipulation by applying shared computation repeatedly. Existing systems usually repeat an entire layer stack, although a mixer and a dense feed-forward network (FFN) perform different operations and have different costs. We ask a narrower question: what should loop? We view recurrence as repeated composition of a state update and argue that an application is valuable when it exposes a new cross-position influence direction that remains observable at the task readout. Iterative Transport Rank (ITR) describes the cumulative influence trajectory; marginal ITR describes the nonredundant influence contributed by successive applications. This view motivates MixerLoop, which repeats each Gated DeltaNet mixer while applying its dense FFN once. We compare MixerLoop with no recurrence and full-block recurrence at 15M and 110M parameters under the same data, initialization, and architecture. A finite context-off intervention tests whether later mixer applications produce distinct, non-negligible, and beneficial changes at the final language-model readout. MixerLoop surpasses FullLoop on aggregate CORE at 15M and retains 41.5% of its CORE improvement at 110M while reducing recurrent-backbone projection FLOPs by 45.9%. These results show that the benefits of recurrent depth can be retained without repeatedly executing the dense FFN.

Figures & tables

Figure 1: Overview of the motivation and architecture. A loop repeatedly composes a state update, and useful applications expose cross-position effects that remain observable at the final readout. MixerLoop repeats the shared GDN mixer within each physical layer while applying the dense FFN once.

Model	Computation	Applications $(A,F)$
NoLoop	$\mathcal{F}_{L}\!\circ\!\mathcal{A}_{L}\!\circ\!\cdots$	$(1,1)$
MixerLoop	$(\mathcal{F}_{L}\!\circ\!\mathcal{A}_{L}^{T})\!\circ\!\cdots$	$(T,1)$
FullLoop	$(\mathcal{F}_{L}\!\circ\!\mathcal{A}_{L}\!\circ\!\cdots)^{T}$	$(T,T)$

Table 1: Recurrent boundaries studied in this work.

\mathcal{A}

denotes the GDN token mixer and

\mathcal{F}

denotes the dense FFN. All models use the same physical backbone and differ only in the placement of recurrent computation.

15M parameters / 52.43B processed tokens
Model	Hella-ZS	Jeop.	WikiQA	ARC-E	ARC-C	COPA	CSQA	PIQA	OBQA	LAMB.	BoolQ	CORE
No Loop	4.20	0.14	8.42	18.01	-3.07	-16.00	1.21	20.13	3.73	13.04	-26.43	5.01
Full Loop (LT2)	4.61	0.05	6.38	19.08	-2.62	-8.00	-0.12	20.78	4.80	14.67	-24.01	5.52
MixerLoop	4.07	0.19	1.56	16.67	-3.98	-16.00	2.13	19.48	5.87	15.02	-6.87	6.52
110M parameters / 52.43B processed tokens
No Loop	19.62	1.32	31.48	39.11	3.41	2.00	0.70	37.65	10.67	30.60	-19.75	14.16

Table 2: CORE after 52.43B ClimbMix tokens with

T=4

. Panels follow the fixed evaluation split. Scores are chance-adjusted and multiplied by 100; CORE averages 22 tasks, and Manip. averages the lower-panel 11. Bold marks the best result within each scale.

Figure 2: Finite iterative transport at 15M (top) and 110M (bottom). For each physical layer, contextual mixer passes are restored in execution order. Left: cumulative readout ITR. Middle: per-pass squared Hellinger effect. Right: ground-truth next-token

\Delta\log p

. Middle/right values are uniformly averaged over evaluated layers, held-out windows, and prediction positions.

Figure 3: Layer-wise marginal ITR for contextual passes two through four. Each cell reports the fraction of a pass’s finite readout update that lies outside the span of earlier updates in the same physical layer. MixerLoop retains broadly distributed novelty at 15M and a more selective pattern at 110M, whereas FullLoop maintains high geometric novelty across the stack.

Explore similar work

What Makes Recurrence Effective in Looped Language Models?
Sep 29, 2026Xinlin Zhuang, Siyuan Wang, Imran Razzak +1Recursive Language Models Efficient Language Model Inference
The Chinese University of Hong Kong · MBZUAI
Closing the Loop: Practical Training Recipes for Looped Language Models
Sep 30, 2026Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin +5Efficient Language Model Training LLM Training
RND NLP, DAIMLD, Russian Federation
One Model to Loop Them All: Elastic Latent Recurrence for Language Models
Oct 8, 2026Faisal Ladhak, Shantanu Acharya, Santiago Akle Serano +3Language Modeling Recurrent Transformers
NVIDIA

Sep 29, 2026J/K move · Enter open · S save

What Makes Recurrence Effective in Looped Language Models?

Xinlin Zhuang, Siyuan Wang, Imran Razzak, Weiyang Liu

The Chinese University of Hong Kong · MBZUAI

Recursive Language Models Efficient Language Model Inference Recurrent Transformers Inference-Time Scaling

Open paper AskPDF arXiv

Looped language models (LoopLMs) increase computational depth through parameter sharing, offering a path to scale inference computation without adding parameters. However, it remains unclear when additional recurrence is beneficial and how architectural choices affect its effectiveness. Through controlled experiments, we systematically examine (1) when recurrence helps, (2) where it should be applied, and (3) how its conditioning affects performance. Our evaluation covers inference budgets below, within, and beyond the training horizon under knowledge and reasoning tasks. (1) We find that recurrence can improve reasoning beyond the training horizon while degrading knowledge performance, but harder reasoning instances do not consistently benefit more. (2) Performance also depends on how distinct layers and recurrent iterations are allocated, showing that effective depth alone is insufficient to predict behavior. Non-recurrent output layers improve robustness to under-unrolling, while the preferred placement of input and output layers varies with inference budget. (3) Finally, we find that conventional initial-state injection offers limited robustness to varying recurrence depth. We therefore propose history-state injection as an alternative, and show that channel-wise history-state injection combined with timestep conditioning offers a low-cost and more effective design, better preserving knowledge under extended unrolling while improving robustness across inference budgets. Overall, our results clarify when recurrent computation helps, where it fails, and offer practical guidelines for designing LoopLMs across variable inference budgets.

Abstract

Explore similar work

What Makes Recurrence Effective in Looped Language Models?

Closing the Loop: Practical Training Recipes for Looped Language Models

One Model to Loop Them All: Elastic Latent Recurrence for Language Models