Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Training

Charafeddine Mouzouni

cs.LGApr 5, 2026

Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Training

Authors: Charafeddine Mouzouni

Organizations: OPIT – Open Institute of Technology, and Cohorte AI, Paris, France.

Abstract

We model Mixture-of-Experts (MoE) token routing as a congestion game with a single effective parameter, the congestion coefficient gamma_eff, that quantifies the balance-quality tradeoff. Tracking gamma_eff across training checkpoints of two open-source MoE models, OLMoE-1B-7B (20 checkpoints, with dense sampling in the surge region) and OpenMoE-8B (6 checkpoints), reveals a three-phase trajectory: a surge phase where the router learns to balance load (gamma_eff: 14 to 36-39, peaking in the step 30K-40K region), a stabilization phase where experts specialize under steady balance (B_0: 2.4 to 2.3, steps 100K-400K), and a relaxation phase where the router trades balance for quality as experts differentiate (gamma_eff: 27 to 9, steps 400K-1.2M). This non-monotone trajectory, invisible to post-hoc analysis of converged models, reveals that early MoE training prioritizes balance while late training prioritizes quality. The theoretical framework is honest about its limits: the single-type equilibrium reduces to temperature-scaled softmax (held-out L1: MFG = 0.199 vs. softmax = 0.200). The game is not a better predictor; it reveals what the temperature means and, critically, how that temperature evolves. We complement the dynamics with an effective congestion decomposition, a multi-type extension that improves load prediction via token clustering on all 16 layers (mean: 30%), scope diagnostics (K/M, epsilon_l), and robustness verification across four independent quality estimators (r >= 0.89). All confidence intervals are from bootstrap resampling over 50 independent text batches.

Figures & tables

Figure 1. Effective congestion

\gamma_{\mathrm{eff}}

across 20 training checkpoints of OLMoE-1B-7B. The three-phase trajectory—surge, stabilization, relaxation—is the paper’s central finding. Shaded band: 95% bootstrap CIs (where available). Open circles: dense-sample checkpoints (20 texts, no CI). The inverted-U shape, with a

{\geq}\,4.2\times

peak-to-final ratio, is invisible to analysis of the converged model alone.

Step	Phase	$\gamma_{\mathrm{eff}}$	$B_{0}$	$H$
5K	Surge	13.7 [13.3, 17.0]	4.10	0.923
10K		11.4 [10.1, 13.3]	3.65	0.943
15K		23.0	3.51	0.954
20K		31.4	3.34	0.962
25K		31.5 [28.4, 35.3]	3.09	0.969
30K		36.4	2.98	0.970

Table 1. Training dynamics of OLMoE-1B-7B across 20 checkpoints. The surge region (steps 5K–50K) is sampled at 5K resolution.

\gamma_{\mathrm{eff}}

: effective congestion (layer average; 95% bootstrap CIs from 50-text resampling where shown).

B_{0}

: expert quality spread.

H

: normalized routing entropy.

Tokens	$\gamma_{\mathrm{eff}}$	$B_{0}$	$H$	Phase
200B	0.0	2.86	0.952	Dormant
400B	0.0	2.99	0.925
600B	0.0	3.12	0.925
800B	3.3	2.72	0.961	Surge
1T	35.6	2.00	0.969
1.1T	27.3	1.71	0.969	Relaxation

Table 2. Training dynamics of OpenMoE-8B across 6 checkpoints. The three-phase pattern replicates: a dormant phase (200B–600B), a surge (600B–1T), and an early relaxation (1T–1.1T). 30 texts per checkpoint, 4 MoE layers.

Figure 2. Robustness of the three-phase trajectory to quality estimation method. All four estimators reproduce the surge–stabilization–relaxation pattern (

r\geq 0.89

vs. default mean). The three-phase finding is not an artifact of the quality proxy.

	Uniform	Temp-softmax	MFG	Multi-type MFG
Early layers (0–7)	0.252	0.143	0.146	0.094
Late layers (8–15)	0.349	0.258	0.252	0.187
All 16 layers	0.301	0.200	0.199	0.140

Table 3. Held-out

L^{1}

error on OLMoE-1B-7B at convergence (119 texts, 1160 held-out tokens, three-way split). The single-type MFG and temperature-scaled softmax are statistically indistinguishable.

Layer	Uniform	Temp-softmax	MFG	MT-MFG	Improv. (%)
0	0.243	0.156	0.163	0.123	24.5
1	0.165	0.101	0.102	0.078	23.5
2	0.253	0.127	0.135	0.092	31.9
3	0.240	0.160	0.158	0.117	25.9
4	0.265	0.134	0.135	0.082	39.3
5	0.278	0.148	0.151	0.083	45.0

Table 4. Per-layer held-out

L^{1}

error. Multi-type MFG (

K=4

types) wins on all 16 layers. Improvement is relative to the single-type MFG.

Layer group	Mean $\hat{\gamma}$	$\gamma_{\mathrm{explicit}}$	$\gamma_{\mathrm{implicit}}$	Status
0, 2, 4, 5, 6, 11	11.8	0.64	$+11.2$	Implicit $18\times$ explicit
1, 3, 9, 10	54.4	0.64	$+53.8$	Implicit $85\times$ explicit
7, 8, 12–15	$\to 0$	0.64	N/A	Out of scope

Table 5. Effective congestion decomposition for OLMoE-1B-7B at convergence. The 10 in-scope layers (where

\hat{\gamma}>0.05

) all have

\hat{\gamma}\gg\gamma_{\mathrm{explicit}}=0.64

: training internalizes far more balance than the auxiliary loss provides. The remaining 6 layers have

\hat{\gamma}\to 0

: the single-type model is out of scope.

Model	Arch.	$M$	$K$	$K/M$	$L^{1}_{\mathrm{MFG}}$	$L^{1}_{\mathrm{unif}}$	MFG wins?
JetMoE-8B	Dec.	8	2	0.250	$\mathbf{0.086}$	$0.127$	Yes
OLMoE-1B-7B	Dec.	64	8	0.125	$0.199$	$0.301$	Yes
Switch-Base-8	Enc-dec	8	1	0.125	$0.351$	$0.355$	Marginal
Switch-Base-16	Enc-dec	16	1	0.063	$0.487$	$0.421$	No
Switch-Base-32	Enc-dec	32	1	0.031	$0.759$	$0.512$	No
Switch-Base-64	Enc-dec	64	1	0.016	$0.546$	$0.489$	No

Table 6. MFG fit across six MoE models, sorted by

K/M

. The MFG outperforms uniform only when

K>1

, consistent with Theorem 5.6 .

Expert Load Balancing

Expert load balancing distributes tokens and compute across mixture-of-experts models to improve utilization without erasing expert specialization.

See all 38 papers

Routing-Free Mixture-of-Experts
Apr 1, 2026Yilun Liu, Jinru Han, Sikuan Yan +2Mixture-of-Experts Models Expert Load Balancing
Ludwig Maximilian University of Munich · Munich Center for Machine Learning · University of California, Los Angeles
Hierarchical Mixture-of-Experts with Two-Stage Optimization
May 8, 2026Gleb Molodtsov, Alexander Miasnikov, Aleksandr BeznosikovExpert Load Balancing Mixture of Experts
BRAIn Lab, Moscow, Russia
When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
May 8, 2026Youngsik Yoon, Siwei Wang, Wei Chen +1Mixture-of-Experts Language Models LLM Routing
Department of Computer Science and Engineering, POSTECH, South Korea · Microsoft Research Asia, Beijing, China · Graduate School of Artificial Intelligence, POSTECH, South Korea

Apr 1, 2026J/K move · Enter open · S save

Routing-Free Mixture-of-Experts

Yilun Liu, Jinru Han, Sikuan Yan, Volker Tresp+1

Ludwig Maximilian University of Munich · Munich Center for Machine Learning · University of California, Los Angeles

Mixture-of-Experts Models Expert Load Balancing

Open paper AskPDF arXiv

Standard Mixture-of-Experts (MoE) models rely on centralized routing mechanisms that introduce rigid inductive biases. We propose Routing-Free MoE which eliminates any hard-coded centralized designs including external routers, Softmax, Top-K and load balancing, instead encapsulating all activation functionalities within individual experts and directly optimized through continuous gradient flow, enabling each expert to determine its activation entirely on its own. We introduce a unified adaptive load-balancing framework to simultaneously optimize both expert-balancing and token-balancing objectives through a configurable interpolation, allowing flexible and customizable resource allocation. Extensive experiments show that Routing-Free MoE can consistently outperform baselines with better scalability and robustness. We analyze its behavior in detail and offer insights that may facilitate future MoE design ad optimization.

Abstract

Expert Load Balancing

Explore similar work

Routing-Free Mixture-of-Experts

Hierarchical Mixture-of-Experts with Two-Stage Optimization

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models