Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design

Junzhuo Li; Peijie Jiang; Changxin Tian; Jia Liu; Zhiqiang Zhang; Xuming Hu

cs.LGMar 11, 2026

Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design

Authors: Junzhuo Li, Peijie Jiang, Changxin Tian, Jia Liu, Zhiqiang Zhang, Xuming Hu

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · Ant Group

Abstract

This paper presents a novel extension of neural scaling laws to Mixture-of-Experts (MoE) models, focusing on the optimal allocation of compute between expert and attention sub-layers. As MoE architectures have emerged as an efficient method for scaling model capacity without proportionally increasing computation, determining the optimal expert-attention compute ratio becomes critical. We define the ratio $r$ as the fraction of total FLOPs per token dedicated to the expert layers versus the attention layers, and explore how this ratio interacts with the overall compute budget and model sparsity. Through extensive experiments with GPT-style MoE Transformers, we empirically find that the optimal ratio $r^*$ follows a power-law relationship with total compute and varies with sparsity. Our analysis leads to an explicit formula for $r^*$ , enabling precise control over the expert-attention compute allocation. We generalize the Chinchilla scaling law by incorporating this architectural parameter, providing a new framework for tuning MoE models beyond size and data. Our findings offer practical guidelines for designing efficient MoE models, optimizing performance while respecting fixed compute budgets.

Figures & tables

Panel of Figure 1 — Figure 1: Loss as a function of FLOPs ratio and total compute. Black dashed lines trace optimal $r^{*}$ . Color indicates active parameters. Low-sparsity models (left) favor higher $r^{*}$ at scale.

Panel of Figure 2 — Figure 2: (a) Relationship between the optimal FLOPs ratio $r^{*}$ and total per-token compute $C$ , fitted with the power law $r^{*}=\alpha_{r}C^{\beta_{r}}$ . (b) Dependence of the fitted coefficient $\alpha_{r}$ on the fraction of activated experts $(1-S)$ , showing a clear power-law trend. (c) Dependence of the fitted exponent $\beta_{r}$ on $(1-S)$ , also following a power-law. All axes are plotted on log–log scales.

Panel of Figure 3 — Figure 3: Scaling law fit on data obtained from training compute-optimal models. (a) shows the fit on the data used to estimate the coefficients for Eq. 2 , (b) validates these coefficients on a held-out dataset. All data points with S = 97.67% were excluded from the fitting process for out-of-sample validation.

Coefficient	$\alpha$	$\beta$	$\lambda$	$\gamma$	$\tau$	a	b	c	d
Estimate	0.6288	0.0453	0.4228	0.0431	13.7354	15.12	18.62	39.55	0.0499

Table 1: Estimated values for coefficients in Equation 2 .

Figure 4: Comparison between the actual training curve of a model with 30M activation parameters and 550M total parameters (with 95.38% sparsity) and the fitted curve derived from Equation 2 .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Label	Layers ( $n_{\text{layer}}$ )	Heads ( $n_{\text{head}}$ )	Batch Size	Learning Rate
20M	8	8	96	0.0015
30M	8	8	160	0.0013
55M	10	10	224	0.0011
100M	14	12	320	0.0009
200M	16	16	512	0.0008

Appendix

Table 2: Key Hyperparameters

Coefficients	Initial Values
$\log(a),\log(b),\log(c),\log(d)$	[0, 10, 20]
$\alpha,\beta,\lambda,\gamma$	[0, 0.25, 0.5, 0.75, 1, 1.25]
$\log(\tau)$	1.5

Appendix

Table 3: Initial values used to estimate coefficients in Equation 2 .

Panel of Figure 5 — Figure 5: Comparison of observed versus predicted validation losses for two alternative scaling formulas. (a) Predictions using the Wang et al. (2024) formulation (Equation 5 ), which fails to generalize across modern high-sparsity settings. (b) Predictions using the Wang et al. (2024) formulation, which fits large models well but underperforms on smaller ones. The solid diagonal line indicates perfect agreement.

Panel of Figure 6 — Figure 6: Loss as a function of FLOPs ratio and total compute. Black dashed lines trace optimal $r^{*}$ . Color indicates active parameters. Low-sparsity models (left) favor higher $r^{*}$ at scale.

Mixture-of-Experts Models

Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.

See all 454 papers

Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
Mar 23, 2026Weilin Wan, Jingtao Han, Debing Zhang +2Mixture-of-Experts Models AI Scaling Laws
College of Computer Science and Artificial Intelligence, Fudan University · Hi Lab, Xiaohongshu Inc. · Project Numina +1
DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks
Mar 2, 2026Gökdeniz GülmezMixture-of-Experts Models Expert Routing
Machine Learning Research, Stuttgart, Germany
Slicing and Dicing: Configuring Optimal Mixtures of Experts
May 12, 2026Margaret Li, Sneha Kudugunta, Danielle Rothermel +1Mixture-of-Experts Language Models Mixture of Experts
Paul G Allen School of Computer Science University of Washington Seattle, USA · Courant School of Data Science New York University New York, USA

Mar 23, 2026J/K move · Enter open · S save

Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization

Weilin Wan, Jingtao Han, Debing Zhang, Weizhong Zhang+1

College of Computer Science and Artificial Intelligence, Fudan University · Hi Lab, Xiaohongshu Inc. · Project Numina+1

Mixture-of-Experts Models AI Scaling Laws Language Model Scaling Laws Neural Architecture Search

Open paper AskPDF arXiv

Scaling laws govern macroscopic resource allocation for LLMs, yet precise architectural configurations for Mixture-of-Experts (MoE) models remain guided by heuristics. Existing MoE scaling studies either incorporate MoE-specific variables into scaling formulas, causing fitted coefficients to grow rapidly without proportional experimental support, or fix all non-MoE factors, implicitly assuming global architecture does not influence local MoE scaling. We propose a reusable framework for holistic MoE architectural optimization. We first show that MoE architectures decouple FLOPs per token (

M

), active parameters (

N_a

), and total parameters (

N

), which serve as distinct proxies for computation, weight access, and parameter storage. We therefore establish a joint constraint triad of

M

, active parameters (

N_a

), and total parameters (

N

) for rigorous MoE characterization. To tame the resulting

O(n^{16})

search space, we employ mathematical decoupling: structural constraints and a rank-preserving property of the hidden dimension factorize the optimization into an

O(n^3) + O(n^2)

two-phase search. Through extensive validation across 670+ MoE models spanning

10^{18}

3 \times 10^{20}

FLOPs, we derive globally applicable scaling laws mapping any compute budget to its optimal architecture. A key finding is that the near-optimal configuration band widens with scale, providing a quantitative basis for trade-offs between scaling law recommendations and engineering constraints. Our work delivers actionable blueprints for optimal MoE design under arbitrary compute budgets.

Abstract

Mixture-of-Experts Models

Explore similar work

Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization

DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks

Slicing and Dicing: Configuring Optimal Mixtures of Experts