LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation

Zefeng He; Yunkai Chai; Xiaoyang Chen; Tong Zhu; Zongliang Wu; Yun Luo; Yafu Li; Yu Cheng

cs.LGOct 14, 2026

LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation

Authors: Zefeng He, Yunkai Chai, Xiaoyang Chen, Tong Zhu, Zongliang Wu, Yun Luo, Yafu Li, Yu Cheng

Organizations: SJTU · Shanghai AI Lab · USTC · CUHK ·

Abstract

Looped Mixture-of-Experts (MoE) models combine parameter reuse with sparse computation to improve performance under matched budgets. Prior work demonstrates that different loop visits can play different roles even when they share parameters. To probe the contributions of different loop visits, we learn separate residual coefficients in a model that applies a shared block twice, observing a smaller scale in the first visit. Fixed coefficient experiments show that reducing the first visit's scale preserves or improves performance, whereas the same reductions to the second hurt it. This asymmetry motivates us to explore asymmetric compute allocation. We propose LatentLoop, which compresses the first visit's sequence, retains full resolution in the second, and reallocates saved computation to increase the hidden size. We further learn how strongly the first visit updates the full sequence, and share MoE weights across visits while using separate attention parameters to accommodate their different sequence resolutions. Across 15 configurations spanning 1.3B--7B parameters, with the largest budget exceeding $10^{21}$ FLOPs, LatentLoop consistently lowers validation NLL. At a fitted NLL of 1.988, predicted compute savings are 19.0% and 7.9% relative to vanilla MoE and the looped baseline. It also improves average scores on nine benchmarks at all sizes in 0-shot and 5-shot settings. These results suggest asymmetric compute allocation is a promising direction for looped MoE models.

Figures & tables

Figure 1: The four steps of budget matching. We adjust attention projection dimensions, model width, the number of stored experts, and training steps in this order.

Budget	SMELT	This work
Backbone parameters	1.0 %	0.6 %
KV cache	3.6 %	0.8 %
FLOPs per token	3.9 %	0.9 %

Table 1: Maximum absolute budget deviations.

Panel of Figure 2 — Figure 2: Residual scale asymmetry at 1.3B and 3B. (a) Learned coefficients for the two visits. (b) Validation NLL with fixed coefficients; rows and columns correspond to the first and second visits. Colors show differences from the same size $(1,1)$ baseline using scores rounded to three decimals.

Figure 3: LatentLoop Architecture. LatentLoop compresses the first visit’s sequence and retains full resolution in the second, sharing MoE weights with independent attention blocks.

Shot	Params	Token	Model	LD-O	LD-S	HS	PIQA	WG	ARC-E	ARC-C	SciQ	MMLU	Avg
			MoE Base	31.44	29.30	48.73	70.46	53.91	69.40	35.41	90.40	32.16	51.24
	1.3B / 0.2B	40B	MoE Loop	31.15	28.72	49.61	69.91	56.04	67.42	36.26	89.60	31.86	51.18
			LatentLoop	32.99	30.20	50.36	69.80	56.35	67.38	37.12	91.10	31.93	51.91
			MoE Base	40.89	36.17	57.89	73.18	57.93	73.40	42.49	93.10	35.12	56.69
5-shot	3B / 0.5B	70B	MoE Loop	39.72	37.26	59.48	74.70	59.19	72.90	43.69	92.90	36.09	57.33
			LatentLoop	43.86	41.74	60.57	74.65	57.62	74.41	45.14	92.80	35.83	58.51

Table 2: Downstream benchmark scores (%). Params denote approximate total/active sizes; token budgets are FLOPs matching reference labels. LD-O/LD-S denote LAMBADA OpenAI/Standard, HS is HellaSwag, WG is WinoGrande, and MMLU uses continuation scoring.

Figure 4: Fitted compute-optimal NLL. Solid segments lie within the convex hull of observed

(\log F,\log D)

points; dashed segments involve extrapolation.

Baselines
Strategy	NLL $\downarrow$
MoE Base	2.0426
MoE Loop	2.0293
Initial first visit reductions
Residual width	2.0361
Expert intermediate width	2.0364

Table 3: Compute reduction strategies.

Table 8

Figure 5: Learned residual coefficients with three and four visits, for 1.3B and 3B.

Figure 6: Learned scaled decoder coefficients at 3B/40B. (a) Channel means during training, with 25th–75th percentile bands. (b) Final channel distributions.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Layers	$d$	$d_{e}$	$d_{h}$	$E$	$P_{b}$	$P_{\rm all}$
1.3B
MoE Base	12/12	1024	512	128	64	1.240232	1.305768
MoE Loop	12/18	864	432	86	91	1.241561	1.296857
LatentLoop	12/18	944	472	102	75	1.238587	1.299003
3B
MoE Base	16/16	1408	704	176	64	3.125857	3.215969

Appendix

Table 6: Actual architecture configurations. The Layers column gives stored/executed depth;

d

d_{e}

, and

d_{h}

are model width, expert intermediate width, and attention head dimension.

E

is the stored expert count per layer. Parameter counts are in billions.

1.3B
Model	KV (MiB)	$F_{\rm step}$	$\Delta P_{b}$	$\Delta P_{\rm all}$	$\Delta$ KV	$\Delta F_{\rm step}$
MoE Base	48.00	1.676	+0.00	+0.00	+0.00	+0.00
MoE Loop	48.38	1.685	+0.11	-0.68	+0.78	+0.54
LatentLoop	47.81	1.672	-0.13	-0.52	-0.39	-0.25
3B
MoE Base	88.00	3.767	+0.00	+0.00	+0.00	+0.00

Appendix

Table 7: Single-step resources and deviations from MoE Base at the same tier. KV is BF16 K/V tensor storage per 4096-token sequence;

F_{\rm step}

is in PFLOPs. All

\Delta

columns are signed percentage differences computed before rounding.

1.3B
Model	Updates	Token	$\Delta D$	$C$	$\Delta C$
MoE Base	38,146	39.999B	+0.00	0.6392	+0.00
MoE Loop	37,941	39.784B	-0.54	0.6392	+12.47
LatentLoop	38,240	40.098B	+0.25	0.6392	-0.98
3B
MoE Base	66,757	70.000B	+0.00	2.5146	+0.00

Appendix

Table 8: Actual training budgets at the largest endpoint of each tier. Token counts are in billions and

C

is in units of

10^{20}

FLOPs. Signed deviations from the same tier’s MoE Base are computed before rounding:

\Delta D

is in percent and

\Delta C

10^{-4}\%

Hyperparameter	Value
Training precision	BF16 mixed precision
Context length	4096
Global batch size	256 sequences
Prediction tokens per update	1,048,576
Optimizer	AdamW ( Loshchilov and Hutter, 2017 )
$(\beta_{1},\beta_{2})$	$(0.9,0.95)$

Appendix

Table 9: Common training hyperparameters. Warmup is applied once per trajectory started from scratch. The variable

p\in[0,1]

denotes progress through the annealing phase.

Size	Token budgets (B)	Points
1.3B	5, 10, 20, 40	4
3B	5, 10, 20, 40, 70	5
7B	5, 10, 20, 40, 70, 140	6

Appendix

Table 10: Reference token budgets.

Benchmark	Harness task	Metric
LAMBADA OpenAI	lambada_openai	acc
LAMBADA Standard	lambada_standard	acc
HellaSwag	hellaswag	acc_norm
PIQA	piqa	acc_norm
WinoGrande	winogrande	acc
ARC-Easy	arc_easy	acc_norm

Appendix

Table 11: Downstream task identifiers and scoring metrics. Both shot settings use the same metric. acc_norm is the harness length-normalized accuracy.

Figure 7: GPU time normalized to MoE Base at each model size. GPU time is elapsed training time multiplied by GPU count.

Table 18

Shot	Params	Token	Model	LD-O	LD-S	HS	PIQA	WG	ARC-E	ARC-C	SciQ	MMLU	Avg
5-shot	1.4B / 1.4B	250B	SpiralFormer-L	46.32	45.29	51.66	70.51	59.51	60.61	31.91	93.20	30.35	54.37
	7B / 1B	140B	LatentLoop	49.21	45.92	68.44	76.33	63.93	79.55	51.71	96.20	39.35	63.40
0-shot	1.4B / 1.4B	250B	SpiralFormer-L	52.86	46.69	50.88	69.64	56.43	50.59	28.75	80.70	29.24	51.75
	7B / 1B	140B	LatentLoop	54.92	45.22	66.91	75.68	62.59	74.20	46.50	90.40	37.69	61.57

Appendix

Table 16: System comparison on downstream benchmarks (%). Params denote approximate total/active parameter counts. Benchmark abbreviations follow Table 2 ; Avg is the unweighted mean of the nine scores.

Params	Tokens	MoE Base	MoE Loop	LatentLoop
1.3B	5B	2.4093	2.4105	2.3979
	10B	2.3026	2.3004	2.2863
	20B	2.2201	2.2161	2.2022
	40B	2.1572	2.1519	2.1393
3B	5B	2.3157	2.3077	2.2971
	10B	2.2010	2.1910	2.1805

Appendix

Table 17: Validation NLL (

\downarrow

) at all scaling endpoints. Parameter tiers denote approximate total parameter counts. The lowest NLL at each matched budget is bold.

Mixture-of-Experts Models

Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.

See all 456 papers

LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling
Jun 3, 2026Wenkai Chen, Tianshu Li, Wenyong Huang +3Mixture-of-Experts Language Models Language Modeling
Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Co.,Ltd.
Scaling Laws for Looped Mixture of Experts
Sep 30, 2026Yanbei Chen, Anirudh Goyal, Raghuraman KrishnamoorthiMixture-of-Experts Language Models Language Model Scaling Laws
Meta AI
Sparse Layers are Critical to Scaling Looped Language Models
May 9, 2026Ryan Lee, Jacob Biloki, Edward J. Hu +1Mixture-of-Experts Language Models Recurrent Transformers
1USC Information Sciences Institute · 2Netflix · 3Independent Researcher

Jun 3, 2026J/K move · Enter open · S save

LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling

Wenkai Chen, Tianshu Li, Wenyong Huang, Yichun Yin+2

Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Co.,Ltd.

Mixture-of-Experts Language Models Language Modeling Recursive Language Models

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) and looped architectures scale models along two orthogonal axes, namely parameter capacity and effective depth. However, mainstream looped architectures rely on dense backbones that couple parameter count with per-token FLOPs, which makes it impossible to isolate the effect of iterative computation under matched budgets. To this end, we present LoopMoE, a looped MoE language model that integrates sparse routing with iterative weight-shared computation through two designs. The first is IterAdaLN, which resolves weight-sharing symmetry via a modulation signal jointly conditioned on the iteration index and the per-token hidden state. The second is a capacity-balancing strategy that recovers the attention-to-FFN active parameter ratio of well-tuned non-looped references. Together, these designs enable the first strictly controlled, head-to-head evaluation of a looped MoE against a Vanilla MoE under identical total parameters, per-token FLOPs, and active sublayer ratios. At the 3B scale, LoopMoE outperforms the Vanilla MoE on 8 of 9 downstream benchmarks with an average improvement exceeding 1 point. At the 9B scale, LoopMoE continues to outperform the matched Vanilla MoE, indicating that the architectural gain persists at larger scale. Our work establishes a controlled synthesis of sparsity and recurrence, and suggests a promising direction for looped language models.

Abstract

Mixture-of-Experts Models

Explore similar work

LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling

Scaling Laws for Looped Mixture of Experts

Sparse Layers are Critical to Scaling Looped Language Models