Generalization and Scaling Laws for Mixture-of-Experts Transformers

Mansour Zoubeirou a Mayaki

cs.LGApr 10, 2026

Generalization and Scaling Laws for Mixture-of-Experts Transformers

Authors: Mansour Zoubeirou a Mayaki

Organizations: Univ Lyon, Lyon 2, LIRIS UMR 5205 F-69676, Lyon, France

Abstract

We develop a theory of generalization and scaling for Mixture-of-Experts (MoE) Transformers that cleanly separates active per-input capacity from routing combinatorics. By conditioning on fixed routing patterns and union-bounding across them, we derive a sup-norm covering-number bound whose metric entropy scales with the active parameter budget and incurs a MoE-specific routing overhead. Combined with a standard ERM analysis for squared loss, this yields a generalization bound under a $d$ -dimensional manifold data model and $C^β$ targets, showing that approximation and estimation trade off as in dense networks once active parameters are accounted for appropriately. We further prove a constructive approximation theorem for MoE architectures, showing that, under the approximation construction, error can decrease either by scaling active capacity or by increasing the number of experts, depending on the dominant bottleneck. From these results we derive neural scaling laws for model size, data size, and compute-optimal tradeoffs. Overall, our results provide a transparent statistical reference point for reasoning about MoE scaling, clarifying which behaviors are certified by worst-case theory and which must arise from data-dependent routing structure or optimization dynamics.

Figures & tables

Figure 1: Routing ablation across expert pool size $M$ and active experts $k$ . (a) In the moderate regime ( $M/k\leq 8$ ), validation loss increases with the routing complexity term $k\log(eM/k)$ , consistent with its interpretation as a worst-case routing overhead. (b) Over the full range, performance improves again for sufficiently large $M/k$ , suggesting gains from expert specialization not captured by the worst-case analysis.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Values Adopted for Scaling Analysis
Model (number of parameters)	Tinystories		Wikitext-103		OpenWebText
Model (number of parameters)	Mean (CI)	Median	Mean (CI)	Median	Mean (CI)	Median
gpt2 (117 million)	$22.9\pm 2.9$	$23.1$	$31.6\pm 2.9$	$32.4$	$46.7\pm 6.2$	$49.8$
gpt2-medium (345 million)	$21.4\pm 1.1$	$21.9$	$31.6\pm 1.7$	$32.9$	$43.7\pm 2.6$	$45.0$
gpt2-large (774 million)	$22.1\pm 0.9$	$22.4$	$31.5\pm 1.6$	$32.1$	$44.7\pm 2.6$	$47.7$
gpt2-xl (1558 million)	$19.7\pm 0.6$	$19.6$	$30.3\pm 1.0$	$31.1$	$43.0\pm 1.7$	$43.9$

Appendix

Table 1: Intrinsic Dimension (ID) Summary across Models and Datasets. The ID estimates remain remarkably stable across different model sizes for each dataset. For the subsequent scaling analysis (Figure 5 and 4 ), we chose the median ID values for each dataset.

Dataset	Chosen $d$	Best-Fit $\beta$	Theoretical Exponents			Empirical Exponents
Dataset	Chosen $d$	Best-Fit $\beta$	$\boldsymbol{\alpha_{N}=\frac{2\beta}{d}}$	$\boldsymbol{\alpha_{D}=\frac{2\beta}{2\beta+d}}$	Consistency $\frac{\alpha_{N}}{1+\alpha_{N}}$	$\boldsymbol{\widehat{\alpha}_{N}}$	$\boldsymbol{\widehat{\alpha}_{D}}$
WikiText-103	$32$	$1.0$	$0.063$	$0.059$	$0.059$	$\mathbf{0.060}$	$\mathbf{0.058}$
OpenWebText	$45$	$1.0$	$0.044$	$0.043$	$0.042$	$\mathbf{0.045}$	$\mathbf{0.043}$
OpenWebText	$45$	$1.5$	$0.067$	$0.062$	$0.063$	$\mathbf{0.068}$	$\mathbf{0.062}$
TinyStories	$23$	$1.0$	$0.087$	$0.080$	$0.080$	$\mathbf{0.085}$	$\mathbf{0.081}$

Appendix

Table 2: Theoretical Exponents derived from the Chosen

d

and

\beta

Panel of Figure 2 — Figure 2: Intrinsic Dimension (ID) Evolution Across Model Depth. The plots show the Maximum Likelihood Estimation (MLE) of the Intrinsic Dimension, $d$ , calculated for the representations at each layer of the four GPT-2 models (varying size). The ID remains stable across different model sizes for a given dataset but varies significantly across data corpora. (a) OpenWebText (OWT): Shows the highest intrinsic dimension, reflecting its diverse and complex nature. (b) TinyStories: Exhibits the lowest intrinsic dimension, consistent with its synthetic, simple structure. (c) WikiText-103 (WT-103): Has an intermediate intrinsic dimension.

Panel of Figure 3 — Figure 3: Intrinsic Dimension Scaling Law: $\bar{d}$ versus Number of Parameters ( $N$ ). The plots show the relationship between the Average Intrinsic Dimension ( $\bar{d}$ ) (log-log scale) and the Number of Parameters ( $N$ ) across the four GPT-2 model sizes for each dataset. This relationship is modeled as a power law, $\bar{d}\sim N^{\beta}$ , where the slope of the linear fit determines the ID Scaling Exponent $\beta$ . The results suggest that $\bar{d}$ exhibits weak or negligible scaling with model size across all three corpora.

$L_{T}$	$d_{ff}$	$N_{\mathrm{act}}$	$d_{\mathrm{model}}$	$n_{\mathrm{heads}}$	$M$	$k$
2	256	3,148,800	512	4	8	2
2	384	3,673,600	512	4	8	2
2	512	4,198,400	512	4	8	2
2	768	5,248,000	512	4	8	2
2	1024	6,297,600	512	4	8	2
4	256	6,297,600	512	4	8	2

Appendix

Table 3: Model configurations used in the scaling experiments for each data set . Each row corresponds to a distinct Transformer-MoE model used in the model-scaling sweep. The effective active parameter count

N_{\mathrm{act}}

varies with the number of layers and FFN width, while

d_{\mathrm{model}}=512

n_{\mathrm{heads}}=4

M=8

k=2

, and sequence length

\ell=256

are held fixed.

$L_{T}$	$d_{\mathrm{ff}}$	$N_{\mathrm{act}}$	$d_{\mathrm{model}}$	$n_{\mathrm{heads}}$	$M$	$k$	$D$
4	512	8,396,800	512	4	8	2	$5\times 10^{4}$
4	512	8,396,800	512	4	8	2	$1\times 10^{5}$
4	512	8,396,800	512	4	8	2	$2\times 10^{5}$
4	512	8,396,800	512	4	8	2	$4\times 10^{5}$
4	512	8,396,800	512	4	8	2	$8\times 10^{5}$

Appendix

Table 4: Model configuration used in the data-scaling experiments. The architecture is held fixed while the training token budget

D

is varied. Unless stated otherwise, the configuration corresponds to a representative mid-scale Transformer–MoE model with fixed active parameter budget

N_{\mathrm{act}}

Panel of Figure 4 — Figure 4: Data-scaling results (fixed $d$ , varying $\beta$ ). Empirical estimates of the data-scaling law (loss versus data size $D$ ) across datasets, compared with theoretical lines obtained by fixing the dataset-specific intrinsic dimension $d$ and varying $\beta$ . The empirical fits are most consistent with $\beta=1.5$ for OpenWebText and $\beta=1.0$ for TinyStories and WikiText-103.

Panel of Figure 5 — Figure 5: Model-scaling results (fixed $d$ , varying $\beta$ ). Empirical estimates of the model-scaling law (loss versus active parameter count $N_{\mathrm{act}}$ ) across datasets. For each plot, the dataset-specific intrinsic dimension $d$ is fixed and theoretical lines are shown for varying $\beta$ , with theoretical exponent $\alpha_{N}=2\beta/d$ . The empirical fits are most consistent with $\beta=0.5$ across all three datasets.

$E$	$\mu(E)$	$\nu(E)$	$\alpha_{N}$	$\alpha_{D}$	$\alpha_{C}$	$\alpha_{D}^{\text{pred}}$	$\alpha_{C}^{\text{pred}}$	$r_{D}$	$r_{C}$	$\mathrm{rel\_err}_{D}$	$\mathrm{rel\_err}_{C}$
1	-0.1817	-0.1965	0.1817	0.1965	0.0944	0.1537	0.0832	0.0427	0.0111	0.2779	0.1335
2	-0.1780	-0.2065	0.1780	0.2065	0.0955	0.1511	0.0817	0.0553	0.0138	0.3666	0.1697
4	-0.1731	-0.2192	0.1731	0.2192	0.0967	0.1475	0.0796	0.0716	0.0170	0.4855	0.2142
8	-0.1676	-0.2338	0.1676	0.2338	0.0976	0.1435	0.0773	0.0902	0.0203	0.6287	0.2625
16	-0.1617	-0.2494	0.1617	0.2494	0.0980	0.1391	0.0748	0.1102	0.0232	0.7917	0.3114
32	-0.1557	-0.2652	0.1557	0.2652	0.0981	0.1347	0.0722	0.1304	0.0258	0.9684	0.3582

Appendix

Table 5: Per-

E

identity checks.We set

\alpha_{N}(E)=-\mu(E)

and

\alpha_{D}(E)=-\nu(E)

. The compute-optimal exponent is

\alpha_{C}(E)=\frac{\alpha_{N}(E)\alpha_{D}(E)}{\alpha_{N}(E)+\alpha_{D}(E)}

. Predicted exponents use

\alpha_{D}^{\text{pred}}=\alpha_{N}/(1+\alpha_{N})

and

\alpha_{C}^{\text{pred}}=\alpha_{N}/(2+\alpha_{N})

. Residuals are

r_{D}=\alpha_{D}-\alpha_{D}^{\text{pred}}

and

r_{C}=\alpha_{C}-\alpha_{C}^{\text{pred}}

; relative errors are

r/\text{pred}

Panel of Figure 6 — Figure 6: Empirical amplification patterns across experts.

Panel of Figure 7 — Figure 7: Cross-budget compute-optimal allocations. (a) Cross-budget slopes for each $E$ . (b) Tokens-to-parameters drift across experts (per budget).

Mixture-of-Experts Models

Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.

See all 456 papers

Scaling Laws for Looped Mixture of Experts
Sep 30, 2026Yanbei Chen, Anirudh Goyal, Raghuraman KrishnamoorthiMixture-of-Experts Language Models Language Model Scaling Laws
Meta AI
Distributionally Robust Mixture-of-Experts Training
Oct 5, 2026Xin Teng, Muxiao Li, Hongyi WenMixture-of-Experts Language Models Sparse Mixture-of-Experts
New York University Center for Data Science, NYU Shanghai
Towards a Statistical Understanding of Mixture-of-Experts
Sep 3, 2026Siyuan He, Bokai Yang, Jie Hu +2Mixture of Experts Sparse Mixture-of-Experts
Qiuzhen College, Tsinghua University, Beijing, China. · KLATASDS-MOE, School of Statistics, East China Normal University, Shanghai, China. · Yau Mathematical Sciences Center, Tsinghua University, Beijing, China. +1

Sep 30, 2026J/K move · Enter open · S save

Scaling Laws for Looped Mixture of Experts

Yanbei Chen, Anirudh Goyal, Raghuraman Krishnamoorthi

Meta AI

Mixture-of-Experts Language Models Language Model Scaling Laws Inference-Time Scaling Recurrent Transformers AI Scaling Laws

Open paper AskPDF arXiv

Looped transformers and Mixture-of-Experts (MoE) offer complementary routes to efficient scaling: recurrence increases computational depth at fixed parameters, while MoE sparsity expands total capacity at fixed active compute. Yet existing scaling laws model recurrence or sparsity in isolation. In this work, we introduce Loop Scaling Laws, the first scaling law to jointly model recurrence and sparsity alongside model size and data. At its core is a bounded, sparsity-conditional recurrence mapping that characterizes the effective-parameter gain from looping and how sparsity raises this gain. The laws predict the held-out loss of looped models more accurately than prior alternatives, and recover the standard dense and MoE scaling laws as special cases. Beyond prediction, the fitted laws provide a principled foundation for designing looped MoE models under compute and memory constraints. Downstream evaluations further demonstrate the complementary benefits of the two axes: sparsity delivers ~3x active-parameter efficiency, recurrence yields ~2x total-parameter efficiency on reasoning, and joint scaling further advances the performance frontier. As a practical extension, we show these gains hold at trillion-token scale: at matched training compute, a looped MoE with law-derived recurrence matches a ~2x larger non-looped MoE on the reasoning benchmarks, while enabling test-time scaling through recurrence.

Abstract

Mixture-of-Experts Models

Explore similar work

Scaling Laws for Looped Mixture of Experts

Distributionally Robust Mixture-of-Experts Training

Towards a Statistical Understanding of Mixture-of-Experts