Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Training

Charafeddine Mouzouni

cs.LGApr 5, 2026

Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Training

Authors: Charafeddine Mouzouni

Organizations: OPIT – Open Institute of Technology, and Cohorte AI, Paris, France.

Abstract

We model Mixture-of-Experts (MoE) token routing as a congestion game with a single effective parameter, the congestion coefficient gamma_eff, that quantifies the balance-quality tradeoff. Tracking gamma_eff across training checkpoints of two open-source MoE models, OLMoE-1B-7B (20 checkpoints, with dense sampling in the surge region) and OpenMoE-8B (6 checkpoints), reveals a three-phase trajectory: a surge phase where the router learns to balance load (gamma_eff: 14 to 36-39, peaking in the step 30K-40K region), a stabilization phase where experts specialize under steady balance (B_0: 2.4 to 2.3, steps 100K-400K), and a relaxation phase where the router trades balance for quality as experts differentiate (gamma_eff: 27 to 9, steps 400K-1.2M). This non-monotone trajectory, invisible to post-hoc analysis of converged models, reveals that early MoE training prioritizes balance while late training prioritizes quality. The theoretical framework is honest about its limits: the single-type equilibrium reduces to temperature-scaled softmax (held-out L1: MFG = 0.199 vs. softmax = 0.200). The game is not a better predictor; it reveals what the temperature means and, critically, how that temperature evolves. We complement the dynamics with an effective congestion decomposition, a multi-type extension that improves load prediction via token clustering on all 16 layers (mean: 30%), scope diagnostics (K/M, epsilon_l), and robustness verification across four independent quality estimators (r >= 0.89). All confidence intervals are from bootstrap resampling over 50 independent text batches.

Figures & tables

Figure 1. Effective congestion

\gamma_{\mathrm{eff}}

across 20 training checkpoints of OLMoE-1B-7B. The three-phase trajectory—surge, stabilization, relaxation—is the paper’s central finding. Shaded band: 95% bootstrap CIs (where available). Open circles: dense-sample checkpoints (20 texts, no CI). The inverted-U shape, with a

{\geq}\,4.2\times

peak-to-final ratio, is invisible to analysis of the converged model alone.

Step	Phase	$\gamma_{\mathrm{eff}}$	$B_{0}$	$H$
5K	Surge	13.7 [13.3, 17.0]	4.10	0.923
10K		11.4 [10.1, 13.3]	3.65	0.943
15K		23.0	3.51	0.954
20K		31.4	3.34	0.962
25K		31.5 [28.4, 35.3]	3.09	0.969
30K		36.4	2.98	0.970

Table 1. Training dynamics of OLMoE-1B-7B across 20 checkpoints. The surge region (steps 5K–50K) is sampled at 5K resolution.

\gamma_{\mathrm{eff}}

: effective congestion (layer average; 95% bootstrap CIs from 50-text resampling where shown).

B_{0}

: expert quality spread.

H

: normalized routing entropy.

Tokens	$\gamma_{\mathrm{eff}}$	$B_{0}$	$H$	Phase
200B	0.0	2.86	0.952	Dormant
400B	0.0	2.99	0.925
600B	0.0	3.12	0.925
800B	3.3	2.72	0.961	Surge
1T	35.6	2.00	0.969
1.1T	27.3	1.71	0.969	Relaxation

Table 2. Training dynamics of OpenMoE-8B across 6 checkpoints. The three-phase pattern replicates: a dormant phase (200B–600B), a surge (600B–1T), and an early relaxation (1T–1.1T). 30 texts per checkpoint, 4 MoE layers.

Figure 2. Robustness of the three-phase trajectory to quality estimation method. All four estimators reproduce the surge–stabilization–relaxation pattern (

r\geq 0.89

vs. default mean). The three-phase finding is not an artifact of the quality proxy.

	Uniform	Temp-softmax	MFG	Multi-type MFG
Early layers (0–7)	0.252	0.143	0.146	0.094
Late layers (8–15)	0.349	0.258	0.252	0.187
All 16 layers	0.301	0.200	0.199	0.140

Table 3. Held-out

L^{1}

error on OLMoE-1B-7B at convergence (119 texts, 1160 held-out tokens, three-way split). The single-type MFG and temperature-scaled softmax are statistically indistinguishable.

Layer	Uniform	Temp-softmax	MFG	MT-MFG	Improv. (%)
0	0.243	0.156	0.163	0.123	24.5
1	0.165	0.101	0.102	0.078	23.5
2	0.253	0.127	0.135	0.092	31.9
3	0.240	0.160	0.158	0.117	25.9
4	0.265	0.134	0.135	0.082	39.3
5	0.278	0.148	0.151	0.083	45.0

Table 4. Per-layer held-out

L^{1}

error. Multi-type MFG (

K=4

types) wins on all 16 layers. Improvement is relative to the single-type MFG.

Layer group	Mean $\hat{\gamma}$	$\gamma_{\mathrm{explicit}}$	$\gamma_{\mathrm{implicit}}$	Status
0, 2, 4, 5, 6, 11	11.8	0.64	$+11.2$	Implicit $18\times$ explicit
1, 3, 9, 10	54.4	0.64	$+53.8$	Implicit $85\times$ explicit
7, 8, 12–15	$\to 0$	0.64	N/A	Out of scope

Table 5. Effective congestion decomposition for OLMoE-1B-7B at convergence. The 10 in-scope layers (where

\hat{\gamma}>0.05

) all have

\hat{\gamma}\gg\gamma_{\mathrm{explicit}}=0.64

: training internalizes far more balance than the auxiliary loss provides. The remaining 6 layers have

\hat{\gamma}\to 0

: the single-type model is out of scope.

Model	Arch.	$M$	$K$	$K/M$	$L^{1}_{\mathrm{MFG}}$	$L^{1}_{\mathrm{unif}}$	MFG wins?
JetMoE-8B	Dec.	8	2	0.250	$\mathbf{0.086}$	$0.127$	Yes
OLMoE-1B-7B	Dec.	64	8	0.125	$0.199$	$0.301$	Yes
Switch-Base-8	Enc-dec	8	1	0.125	$0.351$	$0.355$	Marginal
Switch-Base-16	Enc-dec	16	1	0.063	$0.487$	$0.421$	No
Switch-Base-32	Enc-dec	32	1	0.031	$0.759$	$0.512$	No
Switch-Base-64	Enc-dec	64	1	0.016	$0.546$	$0.489$	No

Table 6. MFG fit across six MoE models, sorted by

K/M

. The MFG outperforms uniform only when

K>1

, consistent with Theorem 5.6 .

Explore similar work

Apr 1, 2026

Routing-Free Mixture-of-Experts

Standard Mixture-of-Experts (MoE) models rely on centralized routing mechanisms that introduce rigid inductive biases. We propose Routing-Free MoE which eliminates any hard-coded centralized designs including external routers, Softmax, Top-K and load balancing, instead encapsulating all activation functionalities within individual experts and directly optimized through continuous gradient flow, enabling each expert to determine its activation entirely on its own. We introduce a unified adaptive load-balancing framework to simultaneously optimize both expert-balancing and token-balancing objectives through a configurable interpolation, allowing flexible and customizable resource allocation. Extensive experiments show that Routing-Free MoE can consistently outperform baselines with better scalability and robustness. We analyze its behavior in detail and offer insights that may facilitate future MoE design ad optimization.

Yilun Liu, Jinru Han, Sikuan Yan +2

Ludwig Maximilian University of Munich · Munich Center for Machine Learning · University of California, Los Angeles

Mixture-of-Experts Models Expert Load Balancing

Ask PDF

May 8, 2026

Hierarchical Mixture-of-Experts with Two-Stage Optimization

Sparse Mixture-of-Experts (MoE) models scale capacity by routing each token to a small subset of experts. However, their routers exhibit a fundamental trade-off: strong load balancing can suppress expert specialization, while aggressive diversity often causes routing collapse. We propose Hi-MoE, a grouped MoE framework that decomposes routing control into two coupled levels: (i) inter-group balancing that enforces fair traffic across expert groups, and (ii) intra-group specialization that promotes complementary expert behaviors while preventing within-group collapse. Our analysis provides a principled explanation of how our hierarchical objectives reshape the router, thereby promoting stable specialization and mitigating collapse. We observe consistent improvements over recent sparse-routing and grouped-MoE baselines across NLP and vision benchmarks, and confirm robustness via scaling studies (model size, expert count) and targeted ablations. In large-scale pre-training on 58B tokens, Hi-MoE-7B achieves a 5.6% perplexity reduction and a 40% improvement in expert balance over OLMoE-7B across diverse evaluation domains.

Gleb Molodtsov, Alexander Miasnikov, Aleksandr Beznosikov

BRAIn Lab, Moscow, Russia

Expert Load Balancing Mixture of Experts Sparse Mixture-of-Experts Expert Routing Mixture-of-Experts Models

Ask PDF

May 8, 2026

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models

Mixture-of-Experts (MoE) language models route each token to a small subset of experts, but whether the routes selected by a trained top-

k

router are good ones is rarely evaluated directly. Holding the model fixed, we compare each standard route against sampled equal-compute alternatives for the same token and score each by the next-token probability it assigns to the realized token in a verified reasoning trajectory. The result is sharply token-conditional: the standard router is well-aligned with route utility on confident tokens but uninformative on the fragile tokens that drive hard reasoning, where lower-loss equal-compute routes consistently exist inside the frozen model but are not selected. The same pattern holds across Qwen3-30B-A3B, GPT-OSS-20B, DeepSeek-V2-Lite, and OLMoE-1B-7B, and follows structurally from how standard top-

k

training evaluates routing decisions: the language modeling loss scores only the executed route, and load balancing depends only on aggregate routing statistics. A minimal router-only update to the final-layer router, leaving every expert and every other router frozen, is sufficient to shift pass@K on AIME 2024+2025 and HMMT 2025 for both Qwen3-30B-A3B and GPT-OSS-20B, suggesting that at least part of the failure reflects router-reachable misallocation rather than expert capacity alone.

Youngsik Yoon, Siwei Wang, Wei Chen +1

Department of Computer Science and Engineering, POSTECH, South Korea · Microsoft Research Asia, Beijing, China · Graduate School of Artificial Intelligence, POSTECH, South Korea

Mixture-of-Experts Language Models LLM Routing Mixture-of-Experts Models

Ask PDF