Scaling Laws for Looped Mixture of Experts

Yanbei Chen; Anirudh Goyal; Raghuraman Krishnamoorthi

cs.LGSep 30, 2026

Scaling Laws for Looped Mixture of Experts

Authors: Yanbei Chen, Anirudh Goyal, Raghuraman Krishnamoorthi

Organizations: Meta AI

Abstract

Looped transformers and Mixture-of-Experts (MoE) offer complementary routes to efficient scaling: recurrence increases computational depth at fixed parameters, while MoE sparsity expands total capacity at fixed active compute. Yet existing scaling laws model recurrence or sparsity in isolation. In this work, we introduce Loop Scaling Laws, the first scaling law to jointly model recurrence and sparsity alongside model size and data. At its core is a bounded, sparsity-conditional recurrence mapping that characterizes the effective-parameter gain from looping and how sparsity raises this gain. The laws predict the held-out loss of looped models more accurately than prior alternatives, and recover the standard dense and MoE scaling laws as special cases. Beyond prediction, the fitted laws provide a principled foundation for designing looped MoE models under compute and memory constraints. Downstream evaluations further demonstrate the complementary benefits of the two axes: sparsity delivers ~3x active-parameter efficiency, recurrence yields ~2x total-parameter efficiency on reasoning, and joint scaling further advances the performance frontier. As a practical extension, we show these gains hold at trillion-token scale: at matched training compute, a looped MoE with law-derived recurrence matches a ~2x larger non-looped MoE on the reasoning benchmarks, while enabling test-time scaling through recurrence.

Figures & tables

Figure 1: (a) Dense looped transformers reuse the same FFNs, whereas looped MoE models can route tokens to different experts across recurrent passes. (b) At fixed active parameters and training tokens, increasing recurrence yields diminishing returns, with the dense model ( $E{=}1$ ) saturating earlier than MoE models ( $E{>}1$ ). (c) Our Loop Scaling Laws model a bounded effective-parameter gain conditioned on recurrence $R$ and sparsity (varied by expert count $E$ ), unlike prior linear and power-law mappings that assume unbounded gains.

		Scaling axes					Optimality
LLM Category	Scaling law	$N/F$	$D$	$R$	$S/E$	Recurrence mapping	Compute	Memory
Dense	Chinchilla ( 2022 )	✓	✓	$\times$	$\times$	–	✓	$\times$
MoE	Unified routed ( 2022 )	✓	$\times$	$\times$	✓	–	$\times$	$\times$
MoE	Joint MoE ( 2025 )	✓	✓	$\times$	✓	–	✓	✓
Looped	Parcae ( 2026 /04 )	✓	✓	✓	$\times$	linear; unbounded †	✓	$\times$
Looped	Iso-Depth ( 2026 /05 )	✓	✓	✓	$\times$	power law; unbounded	✓	$\times$

Table 1: Comparison of scaling laws for training LLMs.

\checkmark

indicates variables explicitly modeled by each law.

N/F

denotes model size

N

/ training FLOPs

F

S/E

denotes MoE sparsity

S

(varied by expert expansion

E

Figure 2: (a) Illustration of effective-parameter gain under linear, power-law, bounded recurrence mappings. (b) Predictive loss by different recurrence mappings: each mapping is fitted on

R\leq 8

and applied to predict the held-out-

R

up to

R=16

. (c) Formula on recurrence mappings and their effective-parameter gains (top) and their evaluation on held-out recurrence

R

, model

N

, and data

D

with RMSE scores (bottom).

Figure 3: (a) Illustration of effective-parameter gain with sparsity-conditional recurrence mapping. (b) Predictive loss under different MoE recurrence mappings fitted on runs with

R\leq 8

; markers denote observations used for fitting. (c) Formula on MoE recurrence mappings and their effective-parameter gains (top), and their evaluation on held-out recurrence

R

, expert count

E

, model size

N

, and data

D

with RMSE scores (bottom).

Figure 4: (a,b) Effective-parameter multiplier over recurrence at

N_{\text{act}}{=}1.0

B, and its asymptote over expert count. (c,d) Recurrence IsoFLOP profiles for dense (

E{=}1

) and MoE (

E{=}8

) models; markers denote minima.

Figure 5: Predicted compute-optimal recurrence $R^{\star}$ of dense ( $E{=}1$ ) and MoE ( $E{=}8$ ) models. (a) Recurrence with compute-optimal loss across training-compute budgets. (b) IsoFLOP profiles over recurrence at $\bar{F}_{\text{train}}{=}5{\times}10^{21}$ FLOPs; stars mark $R^{\star}$ . (c) $R^{\star}$ across compute and active model size for $E{=}8$ (top) and $E{=}1$ (bottom).

Figure 6: Predicted memory-optimal expert count

E^{\star}

and joint optimum

(N_{\text{act}}^{\star},E^{\star},R^{\star})

\bar{F}_{\text{train}}{=}5{\times}10^{21}

FLOPs. (a)

E^{\star}

across bf16 weight-memory budgets at

R{=}4

(top) and

R{=}1

(bottom). (b)

E^{\star}

across INT4 weight memory and recurrence. (c) Predicted joint optimum

(N_{\text{act}}^{\star},E^{\star},R^{\star})

across INT4 weight memory.

Figure 7: Downstream scaling of sparsity and recurrence. (a,b) Scaling

E

R{=}1

; (c,d) Scaling

R

E{=}8

, with the dense baseline

E{=}1,R{=}1

. (a,c) report Overall performance, and (b,d) report Reasoning performance; dashed lines mark compute-matched comparisons.

Model	$\mathbf{R}$	$\displaystyle\frac{F_{\text{inf}}^{\text{0.3B}}(R)}{F_{\text{inf}}^{\text{0.6B}}(1)}$	Reasoning		Science			Commonsense Reasoning				Reading		Knowledge
Model	$\mathbf{R}$		BBH 3	GSM8K 8	ARC-C 25	ARC-E	OBQA	HS	PIQA	SIQA	Wino	BoolQ	DROP 3	MMLU 5	NQ 5	TQA 5	Overall ( $\Delta$ )
A0.6B-2.9B MoE	1	$1.0\times$	29.8	42.9	50.0	66.9	40.8	68.1	77.2	51.6	64.4	70.1	39.7	46.9	13.8	39.8	50.2
A0.3B-1.3B LoopMoE	1	$0.4\times$	18.6	10.5	36.1	57.3	33.8	52.7	70.7	43.8	54.6	58.1	23.0	29.2	5.7	18.2	36.6
	2	$0.8\times$	26.7	30.0	42.2	60.3	38.2	59.9	72.5	46.4	58.5	62.8	33.7	37.6	8.3	24.1	43.0 $\bm{(+6.4)}$
	3	$1.1\times$	30.1	41.0	44.9	60.9	39.6	62.4	73.2	50.2	61.3	65.9	38.4	41.6	8.9	25.6	46.0 $\bm{(+9.4)}$
	4	$1.5\times$	31.5	41.5	46.5	63.9	39.0	62.9	73.9	51.1	60.6	64.4	39.3	42.2	9.8	27.1	46.7 $\bm{(+10.1)}$

Table 2: Downstream results of A0.6B-2.9B MoE and A0.3B-1.3B LoopMoE across test-time recurrence

R

, trained at matched compute FLOPs

{\approx}1.5{\times}10^{22}

. Relative inference compute is normalized to the non-looped baseline.

\Delta

reports gains over LoopMoE at

R{=}1

. Evaluation protocols are provided in Appendix D.1 .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

$E$	$R{=}1$	$R{=}2$	$R{=}4$	$R{=}8$
1	$1.0000\pm 0.0000$	$1.0000\pm 0.0000$	$1.0000\pm 0.0000$	$1.0000\pm 0.0000$
2	$1.0000\pm 0.0000$	$1.1212\pm 0.0692$	$1.2165\pm 0.0465$	$1.5088\pm 0.0529$
4	$1.0000\pm 0.0000$	$1.1917\pm 0.0974$	$1.2973\pm 0.0703$	$1.5300\pm 0.0785$
8	$1.0000\pm 0.0000$	$1.2099\pm 0.0868$	$1.4159\pm 0.0901$	$1.7035\pm 0.0861$

Appendix

Table B.1: Expert-path diversity

\Psi

across varying recurrence

R

and expert count

E

(mean

\pm

std).

	Architecture dimensions					$N_{\text{total}}(E)$ (B)
$N_{\text{act}}$ (B)	$d_{m}$	$n_{h}$	$n_{\ell}$	$n_{\ell,\text{loop}}$	$N_{\text{loop}}$ (B)	$E{=}1$	$E{=}2$	$E{=}4$	$E{=}8$	$E{=}16$
0.3	768	12	20	16	0.1	0.3	0.5	0.8	1.3	2.5
0.6	1024	16	26	22	0.3	0.6	0.9	1.6	2.9	5.5
1.0	1280	20	32	28	0.7	1.0	1.6	2.9	5.4	10.5
1.6	1536	24	38	34	1.2	1.6	2.7	4.8	9.1	17.7
2.4	1792	28	44	40	1.8	2.4	4.1	7.5	14.3	27.8

Appendix

Table C.1: Model scaling ladder over five model scales. Parameter counts are in billions (B).

N_{\text{act}}

and

N_{\text{total}}

include embedding parameters, whereas

N_{\text{loop}}

includes only parameters in the recurrent block.

Base scaling law coefficients					MoE coefficients						Recurrence coefficients			Fit
$A$	$\alpha$	$B$	$\beta$	$c$	$\delta$	$\gamma$	$\omega$	$\zeta$	$E_{\text{start}}$	$E_{\max}$	$\kappa_{1}$	$\kappa_{2}$	$\theta$	RMSE
0.8085	$-0.1951$	3.4291	$-0.7548$	1.3555	$-0.1935$	$-0.0137$	$-0.2056$	0.0881	1.3174	57.1201	0.3682	1.4037	0.3286	0.0037

Appendix

Table C.2: Fitted coefficients of MoE Loop Scaling Law in Eq. 9 . RMSE: root-mean-square error of the fit.

Coefficient	Estimate	90% CI
$\kappa_{1}$	0.3682	$[0.3658,\,0.3723]$
$\kappa_{2}$	1.4037	$[1.3896,\,1.4243]$
$\theta$	0.3286	$[0.3183,\,0.3350]$

Appendix

Table C.3: Bootstrap results on the MoE loop scaling law. (a) Coefficients fitted on full-data or the 90% sampled subsets with intervals spanning the 5th–95th percentiles over 100 refit runs. (b) Derived joint optimum under the compute and memory budgets (with INT4 weights) used in Figure 6 (c) on the 100 refit runs.

Category	Benchmark	Task name	$n$ -shot	Metric
Reasoning	BBH	leaderboard_bbh	3	acc_norm
	GSM8K	gsm8k_cot	8	exact_match,flexible-extract
Science	ARC-C	arc_challenge	25	acc_norm
	ARC-E	arc_easy	0	acc_norm
	OBQA	openbookqa	0	acc_norm
Commonsense Reasoning	HellaSwag	hellaswag	0	acc_norm

Appendix

Table D.1: Evaluation setups of 14 foundational benchmarks. Task names and metrics follow lm-eval .

Explore similar work

Sparse Layers are Critical to Scaling Looped Language Models
May 9, 2026Ryan Lee, Jacob Biloki, Edward J. Hu +1Transformer Architectures Layer-Wise
1USC Information Sciences Institute · 2Netflix · 3Independent Researcher
LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling
Jun 3, 2026Wenkai Chen, Tianshu Li, Wenyong Huang +3Mixture-Of-Experts Large Language Models Mixture-Of-Experts
Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Co.,Ltd.
Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts
Oct 1, 2026Di He, Pengxiang Li, Da Chang +3Mixture-Of-Experts Large Language Models Mixture-Of-Experts
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Peng Cheng Laboratory · University of Chinese Academy of Sciences +5

May 9, 2026, cs.LGJ/K move · Enter open · S save

Sparse Layers are Critical to Scaling Looped Language Models

Ryan Lee, Jacob Biloki, Edward J. Hu, Jonathan May

1USC Information Sciences Institute · 2Netflix · 3Independent Researcher

Transformer Architectures Layer-Wise Localized Lora-Moe Mixture-Of-Experts Peak Memory

Open paper AskPDF arXiv

Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries. However, looped models do not scale as favorably as standard transformers with unique layers. We compare standard and Mixture-of-Experts (MoE) transformers, with and without looping, and find two main results. First, we find Looped-MoE models scale better than the standard baseline while dense looped models do not. We trace this to routing divergence between loops: in Looped-MoE models, different experts are activated on each pass through the same shared layers, recovering expressivity without additional parameters. Our second finding is that looped models have better compute-quality trade-offs with early exits than standard models. Because each loop ends with the same layers that produce the final output, loop boundaries are superior exit points, as confirmed by earlier output convergence at these points. In sum, we provide a clear direction for scaling looped models: a Looped-MoE model with early exits can not only beat standard transformers at scale, but also enable significant memory and inference savings with minimal degradation in quality.

Abstract

Explore similar work

Sparse Layers are Critical to Scaling Looped Language Models

LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling

Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts