One Model to Loop Them All: Elastic Latent Recurrence for Language Models

Faisal Ladhak; Shantanu Acharya; Santiago Akle Serano; Simeng Sun; Boris Ginsburg; Krishna C Puvvada

cs.CLOct 8, 2026

One Model to Loop Them All: Elastic Latent Recurrence for Language Models

Authors: Faisal Ladhak, Shantanu Acharya, Santiago Akle Serano, Simeng Sun, Boris Ginsburg, Krishna C Puvvada

Organizations: NVIDIA

Abstract

A latent recurrent language model applies a shared block of layers repeatedly, so one checkpoint can trade inference compute for accuracy by changing the loop count. At one loop, however, such models typically fall well short of a non-recurrent model of the same size, a one-loop tax that makes a separate non-recurrent model the better choice whenever latency matters. We present Elastic Latent Recurrence (ELR), a recipe for training looped language models without this tax. At one loop, an ELR model costs the same to serve as a non-recurrent model of the same size and, trained on the same data, reaches similar accuracy. Each additional loop then buys more accuracy for more inference compute. Three choices make this work: a gated state update with a learned step size, loop counts mixed within each optimizer update, and conditioning each pass on its loop index and budget. The first two preserve one-loop accuracy, and the third lowers loss at every additional loop. Mixed depths can be introduced late in pretraining to limit the added training cost, and with zero-initialized conditioning added at the switch, this late schedule reaches lower loss than mixed-depth training from scratch. In MoE models, reusing first-loop expert routes on later loops cuts mixed-depth update time by 11.7% with no measurable change in accuracy. At inference, keeping only the last loop's attention KV cache holds attention KV memory at its one-loop level while retaining most of the gain from additional loops. On 4B dense and 30B-A3B MoE hybrid models, ELR at one loop is 0.52 points below and 1.04 points above its matched baseline on a ten-task average, and four loops add 7.25 and 4.75 points. Carried through post-training, the recipe yields Koi-4B, a single checkpoint that scores an estimated 18 on the Artificial Analysis Intelligence Index at one loop, matching Qwen3.5-4B, and 29 at four loops.

Figures & tables

Figure 1: Elastic Latent Recurrence. The shared recurrent block

\mathcal{R}_{\theta}

is applied

k

times between the input

\mathcal{I}

and output

\mathcal{O}

blocks. Each loop

i

re-injects

h^{\mathrm{in}}

and updates the state with the adaptive-delta rule (Eq. 2 ), using a gate computed from

\tilde{s}_{i}

. At

k=1

, every unique layer executes exactly once.

Panel of Figure 2 — Figure 2: Average downstream benchmark accuracy across ten tasks versus inference loop count. Each curve represents a single model evaluated at multiple loop counts. Dashed horizontal lines mark the non-recurrent baselines. Per-task scores are in Tables 8 and 9 .

Routing	$\Delta$ Val Loss at $k=4$	$\Delta$ Training Update Time		$\Delta$ Inference Forward Time
Routing	$\Delta$ Val Loss at $k=4$	Mixed-depth	Fixed $k{=}4$	$\Delta$ Inference Forward Time
(A) Dynamic routing	reference
(B) Inference-only replay	$+0.0055$	–	–	$-1.2\%$
(C) Replay training	$+0.0038$	$-11.7\%$	$-19.6\%$	$-1.2\%$

Table 1: Cross-loop route replay on GDP-Hybrid-30B-A3B, relative to dynamically routed ELR (A). (B) applies replay to A’s checkpoint at inference only while (C) is trained with replay. Validation loss and inference forward time are measured at

k=4

on 8K-token sequences. Training update time is averaged over the mixed-depth training phase and profiled at fixed

k=4

Panel of Figure 3 — Figure 3: Koi-4B evaluated at $k=1,\ldots,4$ . (a) AA-Index v4.1 estimates, rounded to the nearest integer; dashed lines mark non-recurrent reference models. (b) Output tokens per rollout relative to $k=1$ , as a geometric mean over eight of the AA-Index benchmarks (Appendix C.1 ). All results come from our evaluation harness, not official Artificial Analysis results.

Benchmark	Per-loop		Last-loop	$\Delta$ [1pt]Score	Output [1pt]Tokens
Benchmark	$k=1$	$k=4$	$k=4$	$\Delta$ [1pt]Score	Output [1pt]Tokens
Terminal-Bench 2.1	12.1	28.0	27.4	$-0.6$	$0.80\times$
$\tau^{3}$ -Bench Banking	11.3	21.8	21.8	$\phantom{-}0.0$	$0.86\times$
Humanity’s Last Exam	5.9	11.9	10.5	$-1.4$	$0.93\times$
AA-Omniscience accuracy	8.1	9.3	8.5	$-0.7$	$0.82\times$
SciCode	17.7	33.4	32.0	$-1.4$	$0.97\times$

Table 2: Koi-4B with per-loop and last-loop KV caches.

\Delta

Score is last-loop minus per-loop at

k=4

, in percentage points, computed before rounding. Output Tokens is the ratio of mean output tokens per rollout, last-loop to per-loop, at

k=4

. Results at

k=2,3

are in Appendix C.2 .

Panel of Figure 4 — Figure 4: Architectural and training ablations on GPT-0.5B. (a) State update rules: solid lines mark rules with learned step sizes, and dashed lines mark parameter-free rules. (b) Fixed-depth versus mixed-depth training: outlined cells mark matching training and evaluation depths.

$p_{1}$	Sampling	Depths/Update	Step Time	$k=1$	$k=2$	$k=3$	$k=4$	$k=5$
–	Non-recurrent baseline	–	–	1.6852	–	–	–	–
0.20	Synchronized, uniform	1	$1.00\times$	1.7077	1.6473	1.6344	1.6318	1.6313
0.20	Independent, uniform	$\leq 5$	$1.36\times$	1.6937	1.6330	1.6193	1.6157	1.6147
0.32	Synchronized, weighted	1	$1.00\times$	1.6871	1.6436	1.6343	1.6320	1.6316
	Synchronized, 15% override	$\leq 2$	$1.00\times$	1.6830	1.6391	1.6299	1.6278	1.6273
	Independent, weighted	$\leq 5$	$1.36\times$	1.6774	1.6334	1.6231	1.6200	1.6194

Table 3: Loop sampling schemes on GPT-0.5B (validation loss). Weighted sampling spreads the probability remaining after

p_{1}

uniformly over

k\in\{2,\ldots,5\}

, so rows with the same

p_{1}

share the same marginal distribution over

k

. Depths/Update is the maximum number of distinct loop counts contributing gradients to one update. Step time is relative to synchronized uniform sampling.

Panel of Figure 5 — Figure 5: Training choices on GPT-0.5B. Each x-axis point is a separately trained model, and each line shows validation loss at one inference loop count $k$ across these models. The dashed line marks the non-recurrent baseline (1.685). (a) Point at which training switches from one loop to mixed depth ( $f=0.15$ ); 0% is mixed-depth training from scratch, and added compute is estimated from Eq. 5 . (b) One-loop override fraction $f$ , with mixed-depth training from scratch.

Figure 6: Loop-conditioning on GPT-0.5B. Delayed models switch to mixed depth after 75% of tokens, and the from-scratch model trains at mixed depth throughout.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

	GPT-0.5B	GDP-Hybrid-4B	GDP-Hybrid-30B-A3B
Layers (GDP / attention / FF)	0 / 15 / 15	23 / 6 / 23	23 / 6 / 23
ELR partition ( $\mathcal{I}$ / $\mathcal{R}_{\theta}$ / $\mathcal{O}$ )	8 / 14 / 8	14 / 24 / 14	14 / 24 / 14
Feed-forward type	Dense	Dense	MoE
Activation	SwiGLU	Squared ReLU	Squared ReLU
Hidden size $d$	1,024	3,072	2,688
FF intermediate size	3,968	12,288	1,856 per expert

Appendix

Table 4: Backbone configurations, shared by each ELR model and its non-recurrent baseline. Koi-4B uses the GDP-Hybrid-4B configuration. Partial RoPE rotates 50% of each attention head’s dimensions.

	GPT-0.5B	GDP-Hybrid-4B	GDP-Hybrid-30B-A3B
Training tokens	300B	1T	300B
Optimizer	AdamW	AdamW	Muon + AdamW
AdamW $(\beta_{1},\beta_{2})$ / weight decay	(0.9, 0.95) / 0.1	(0.9, 0.95) / 0.1	(0.9, 0.95) / 0.1
Peak / minimum learning rate	$1.8{\times}10^{-3}$ / $1.8{\times}10^{-5}$	$1.2{\times}10^{-3}$ / $1.2{\times}10^{-5}$	$1.2{\times}10^{-3}$ / $1.2{\times}10^{-5}$
Warmup tokens	8.4B	33.6B	8.4B
Sequence length / global batch	8,192 / 768	8,192 / 768	8,192 / 768

Appendix

Table 5: Optimization and ELR training settings. Each non-recurrent baseline uses the same data, token budget, and optimization settings as its ELR counterpart.

p_{1}

and

\bar{k}

are nominal values from Eq. 3 .

Stage	Data and budget	Loop schedule
Pretraining	Nemotron 3 Ultra subset; 7T tokens at 8K context	One loop for the first 50%; then uniform over $k=1,\ldots,4$ with a 10% one-loop rank override ( $p_{1}=0.325$ , $\bar{k}=2.35$ )
Mid-training	Context extension; 50B tokens at 256K context	Same as the pretraining mixed-depth phase
SFT	Nemotron 3.5 Super text-only data; four epochs	Same as the pretraining mixed-depth phase
RLVR	Nemotron 3.5 Super text-only data; 200 steps	Sampling ratio $4:1:1:1$ for $k=1,2,3,4$ ( $p_{1}\approx 0.57$ , $\bar{k}\approx 1.86$ )

Appendix

Table 6: Koi-4B training stages.

p_{1}

and

\bar{k}

are nominal values from Eq. 3 or the sampling ratio.

Benchmark	Domain	Shots	Metric
MMLU ( Hendrycks et al., 2021 )	Knowledge	5	Accuracy
MMLU-Pro ( Wang et al., 2024 )	Knowledge	5	Accuracy
ARC-Challenge ( Clark et al., 2018 )	Science QA	25	Accuracy
WinoGrande ( Sakaguchi et al., 2021 )	Commonsense	5	Accuracy
CommonsenseQA ( Talmor et al., 2019 )	Commonsense	7	Accuracy
RACE-high ( Lai et al., 2017 )	Reading comprehension	0	Accuracy

Appendix

Table 7: Downstream evaluation benchmarks.

Benchmark	Baseline	ELR
Benchmark	Baseline	$k=1$	$k=2$	$k=3$	$k=4$
MMLU	62.1	62.5	66.7	67.2	67.6
MMLU-Pro	41.3	39.5	45.0	47.0	47.3
ARC-Challenge	79.5	79.8	83.8	84.6	85.0
WinoGrande	70.8	73.5	74.8	75.2	75.6
CommonsenseQA	71.4	68.8	75.5	77.2	77.8

Appendix

Table 8: Downstream benchmark scores (%) for GDP-Hybrid-4B trained for 1T tokens.

Benchmark	Baseline	ELR				ELR + Route Replay
Benchmark	Baseline	$k=1$	$k=2$	$k=3$	$k=4$	$k=1$	$k=2$	$k=3$	$k=4$
MMLU	62.04	65.55	68.48	69.32	69.28	65.23	68.51	69.15	69.17
MMLU-Pro	39.07	38.40	44.35	45.78	45.94	39.27	44.23	45.42	45.68
ARC-Challenge	77.39	79.78	83.70	84.73	84.73	79.35	84.90	85.32	85.32
WinoGrande	74.74	74.03	76.72	77.19	78.69	74.03	75.45	75.85	76.01
CommonsenseQA	69.53	72.56	77.31	78.13	77.72	71.66	75.59	77.31	77.64

Appendix

Table 9: Downstream benchmark scores (%) for GDP-Hybrid-30B-A3B trained for 300B tokens.

Benchmark (index weight)	$k=1$	$k=2$	$k=3$	$k=4$
AA-Index v4.1 estimate	17.52	25.57	27.93	29.23
GDPval-AA v2, Elo (20%)	611	899	960	997
Terminal-Bench 2.1 (16%)	12.1	22.9	26.5	28.0
$\tau^{3}$ -Bench Banking (14%)	11.3	16.8	19.8	21.8
Humanity’s Last Exam (12%)	5.9	9.1	10.1	11.9
AA-Omniscience accuracy (8%)	8.1	8.8	8.6	9.3

Appendix

Table 10: Koi-4B benchmark scores across inference loop counts. Parenthesized values are AA-Index v4.1 weights. GDPval-AA v2 is reported as Elo, and all other benchmarks as percentages.

Benchmark	$k=1$	$k=2$	$k=3$	$k=4$
GDPval-AA v2	154,205	96,979	83,979	78,506
Terminal-Bench 2.1	215,788	146,397	121,387	107,927
$\tau^{3}$ -Bench Banking	35,543	27,169	24,325	22,905
Humanity’s Last Exam	28,907	29,766	28,519	28,187
AA-Omniscience	7,254	6,356	5,826	5,639
SciCode	41,061	35,408	31,861	30,253

Appendix

Table 11: Mean output tokens per rollout for Koi-4B. The last row is

\bar{r}_{k}

(Eq. 6 ).

Benchmark	$k=2$			$k=3$
Benchmark	Per-loop	Last-loop	Output [1pt]Tokens	Per-loop	Last-loop	Output [1pt]Tokens
Terminal-Bench 2.1	22.89	23.60	$0.81\times$	26.54	29.35	$0.80\times$
$\tau^{3}$ -Bench Banking	16.75	19.85	$0.86\times$	19.85	20.36	$0.86\times$
Humanity’s Last Exam	9.08	8.94	$0.95\times$	10.10	9.87	$0.94\times$
AA-Omniscience accuracy	8.80	8.03	$0.80\times$	8.57	8.28	$0.82\times$
SciCode	27.95	29.12	$0.99\times$	30.82	29.99	$0.94\times$

Appendix

Table 12: Per-loop and last-loop KV-cache results at

k=2,3

. Scores are percentages. Output Tokens is the ratio of mean output tokens per rollout, last-loop to per-loop, at the same loop count. Both AA-Omniscience metrics use the same generations.

Panel of Figure 7 — Figure 7: GDP-Hybrid-4B ten-task average accuracy versus (a) TTFT and (b) per-token decode latency, normalized to the non-recurrent baseline ( $1\times$ ). Numbers indicate loop counts, and horizontal bars show 95% bootstrap intervals.

Metric	Value
Retained experts (%)	53.17 [52.37, 53.89]
Aggregate frequency cosine	0.978
Routed-output cosine	0.767 / 0.227
Full-MoE output cosine	0.918 / 0.718

Appendix

Table 13: Cross-loop routing and output comparisons, averaged over ten recurrent MoE layers. Routing statistics compare loops one and four in A, with a 95% document-bootstrap interval in brackets. Output cosines report first-loop replay / shifted-route control on the same fourth-loop inputs. Full-MoE outputs include shared experts and exclude the outer residual.

Recurrent-block input $\tilde{s}_{i}$	$k=1$	$k=2$	$k=3$	$k=4$	$k=5$
Non-recurrent baseline	1.6852	–	–	–	–
$s_{i-1}$	1.6830	1.6391	1.6299	1.6278	1.6273
$s_{i-1}+h^{\mathrm{in}}$	1.6858	1.6442	1.6352	1.6329	1.6325

Appendix

Table 14: Input re-injection on GPT-0.5B (validation loss).

Explore similar work

Aug 18, 2026

Allocating Recurrent Compute in Looped Language Models

Looped language models improve reasoning and knowledge manipulation by applying shared computation repeatedly. Existing systems usually repeat an entire layer stack, although a mixer and a dense feed-forward network (FFN) perform different operations and have different costs. We ask a narrower question: what should loop? We view recurrence as repeated composition of a state update and argue that an application is valuable when it exposes a new cross-position influence direction that remains observable at the task readout. Iterative Transport Rank (ITR) describes the cumulative influence trajectory; marginal ITR describes the nonredundant influence contributed by successive applications. This view motivates MixerLoop, which repeats each Gated DeltaNet mixer while applying its dense FFN once. We compare MixerLoop with no recurrence and full-block recurrence at 15M and 110M parameters under the same data, initialization, and architecture. A finite context-off intervention tests whether later mixer applications produce distinct, non-negligible, and beneficial changes at the final language-model readout. MixerLoop surpasses FullLoop on aggregate CORE at 15M and retains 41.5% of its CORE improvement at 110M while reducing recurrent-backbone projection FLOPs by 45.9%. These results show that the benefits of recurrent depth can be retained without repeatedly executing the dense FFN.

Ruhai Lin, Yiyang Guo, Rui-Jie Zhu +2

University of California, Santa Cruz

Recurrent Transformers

Ask PDF

Sep 29, 2026

What Makes Recurrence Effective in Looped Language Models?

Looped language models (LoopLMs) increase computational depth through parameter sharing, offering a path to scale inference computation without adding parameters. However, it remains unclear when additional recurrence is beneficial and how architectural choices affect its effectiveness. Through controlled experiments, we systematically examine (1) when recurrence helps, (2) where it should be applied, and (3) how its conditioning affects performance. Our evaluation covers inference budgets below, within, and beyond the training horizon under knowledge and reasoning tasks. (1) We find that recurrence can improve reasoning beyond the training horizon while degrading knowledge performance, but harder reasoning instances do not consistently benefit more. (2) Performance also depends on how distinct layers and recurrent iterations are allocated, showing that effective depth alone is insufficient to predict behavior. Non-recurrent output layers improve robustness to under-unrolling, while the preferred placement of input and output layers varies with inference budget. (3) Finally, we find that conventional initial-state injection offers limited robustness to varying recurrence depth. We therefore propose history-state injection as an alternative, and show that channel-wise history-state injection combined with timestep conditioning offers a low-cost and more effective design, better preserving knowledge under extended unrolling while improving robustness across inference budgets. Overall, our results clarify when recurrent computation helps, where it fails, and offer practical guidelines for designing LoopLMs across variable inference budgets.

Xinlin Zhuang, Siyuan Wang, Imran Razzak +1

The Chinese University of Hong Kong · MBZUAI

Recursive Language Models Efficient Language Model Inference Recurrent Transformers Inference-Time Scaling

Ask PDF

Sep 30, 2026

Closing the Loop: Practical Training Recipes for Looped Language Models

Looped language models increase effective depth by repeatedly applying a shared block of layers, but existing large-scale recipes require multi-stage training over trillions of tokens, while the benefits of recurrence remain difficult to separate from differences in data and training. In this work, we establish practical training recipes for looped language models, with three main results. (1) We develop a compute-efficient from-scratch pipeline that reduces the training budget from 7.7T tokens in Ouro to 310B tokens while retaining strong reasoning performance. Pretraining followed by high-quality mid-training, together with learning-rate warmup and stronger exit-gate regularization, enables stable recurrent training without prior multi-stage schedules. (2) Under controlled comparisons, our 1.4B LoopLM outperforms a parameter-matched dense model trained on the same data and token budget on all 12 evaluated benchmarks, including +14 points on GSM8K, +10 on MATH, and +22 on DROP. At matched inference compute, it approaches a 3.9B dense model on mathematical reasoning and reading comprehension while using only 36% as many parameters. (3) We introduce a minimal recipe for converting pretrained dense models into looped ones: a single learned input-mixing scalar and a smoothed exit loss, with no step-specific parameters. Applied to Qwen3-1.7B-Base, Looped Qwen improves over an identically continued dense baseline on every evaluated benchmark across two data regimes, with statistically clear gains on GSM8K, MATH, and MMLU-Pro on the curated mixture. Together, these results make looped language models substantially cheaper to train from scratch and practical to introduce into existing pretrained checkpoints, while isolating the gains due to recurrence itself.

Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin +5

RND NLP, DAIMLD, Russian Federation

Efficient Language Model Training LLM Training Recursive Language Models

Ask PDF