Rethinking Routing Exploration in Modern Mixture-of-Experts

Bingcong Li; Hao Ma; Michael Muehlebach; Niao He; Yi Ma

cs.LGOct 15, 2026

Rethinking Routing Exploration in Modern Mixture-of-Experts

Authors: Bingcong Li, Hao Ma, Michael Muehlebach, Niao He, Yi Ma

Organizations: ETH Zurich · The University of Hong Kong · Max Planck Institute for Intelligent Systems · WU Vienna

Abstract

Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio, and the latter introduces experts that are active for every token. We identify an exploration dilemma between these designs: shared experts can make routing less exploratory by reducing the variability of token representations. This conflicts with the greater need for exploration induced by fine-grained expert segmentation due to the combinatorially enlarged routing space. Consequently, the additional expressiveness offered by fine-grained segmentation may not be fully realized in practice. MirrorMoE is proposed to reconcile this exploration dilemma by shrinking the exploration space through token-adaptive suppression of unnecessary experts. MirrorMoE consistently improves performance across model scales.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison of MoEs with and without shared experts (SE) and the proposed MirrorMoE. MirrorMoE consistently i) achieves the best performance, ii) realizes the expressiveness gains from larger $K$ , and iii) enables shared-expert MoEs to outperform their non-shared counterparts.

Panel of Figure 2 — Figure 2: Perplexity increase after ablating individual experts. In the right subplot, the two rightmost columns, separated by a solid line, correspond to shared experts.

Panel of Figure 3 — Figure 3: Shared experts make MoE routing less exploratory.

Figure 4: Preview of how MirrorMoE in Alg. 1 actively suppresses experts on a 100M MoE model. The rebound in

K_{\mathsf{eff}}

results from annealing

\rho

Model	Experts	Depth	PPL ( $\downarrow$ )		iteration time (ms) ( $\downarrow$ )
Total-activated params.	$K$ / $R$ / $S$		Vanilla	MirrorMoE	Vanilla	MirrorMoE
MoE-100M-A38M	8/64/2	1D + 8S	54.78	51.77	7871.3 $\pm$ 65.9	7844.4 $\pm$ 85.6
MoE-498M-A121M	8/64/2	1D + 12S	28.31	27.36	13550.0 $\pm$ 63.0	13426.7 $\pm$ 83.3
MoE-625M-A100M	8/128/2	1D + 8S	28.46	27.84	9527.5 $\pm$ 45.4	9462.5 $\pm$ 61.2
MoE-2.4B-A270M	8/128/2	1D + 15S	21.17	20.50	25711.1 $\pm$ 176.2	25635.4 $\pm$ 109.4

Table 1: Pretraining performance of MirrorMoE across scales. D/S in the depth column denotes dense/sparse Transformer layers.

Model	Method	ARC-E	OBQA	HellaSwag	PIQA	BoolQ	SIQA	Avg ( $\uparrow$ )
MoE-100M-A38M	Vanilla	32.11	26.40	26.62	57.56	42.66	34.29	36.61
MoE-100M-A38M	MirrorMoE	34.22	26.00	25.76	58.00	42.48	34.49	36.82
MoE-498M-A121M	Vanilla	39.44	26.80	29.87	61.43	58.65	36.69	42.15
MoE-498M-A121M	MirrorMoE	41.04	27.80	30.33	64.31	59.20	36.95	43.27
MoE-625M-A100M	Vanilla	41.12	28.20	29.97	61.10	61.47	37.36	43.20
MoE-625M-A100M	MirrorMoE	41.37	29.80	30.23	62.08	61.16	36.59	43.53

Table 2: Downstream performance of MoE models trained with MirrorMoE.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	MoE-100M-A38M	MoE-498M-A121M	MoE-625M-A100M	MoE-2.4B-A270M
Dense / MoE layers	1 / 8	1 / 12	1 / 8	1 / 15
Layer layout	1D + 8S	1D + 12S	1D + 8S	1D + 15S
Hidden size ( $d_{\mathrm{model}}$ )	256	512	512	768
Dense FFN hidden	1368	2736	2736	3648
MoE FFN hidden (per expert)	176	352	352	512
Shared expert FFN (2 $\times$ )	352	704	704	1024

Appendix

Table 4: Model configurations used in our experiments.

Panel of Figure 6 — Figure 6: Router variance under different levels of Gumbel perturbation.

	Top- $(K-S)$	Sto- $(K-S)$	Top-2 with Sto- $(K-S-2)$
PPL	54.78	55.74	55.38

Appendix

Table 5: Intrinsic difficulty of increasing exploration

	vanilla balancing loss	our load balancing loss ( 6 )
PPL	52.94	51.77
runtime	7727.1 $\pm$ 91.8	7844.4 $\pm$ 85.6

Appendix

Table 6: Comparison of different load balancing loss for MirrorMoE

Model	Experts	Depth	PPL ( $\downarrow$ )
Total-activated params	$K$ / $R$ / $S$		Vanilla	MirrorMoE
MoE-NS-97M-A36M	8/64/0	8S	52.01	51.23
MoE-NS-480M-A115M	8/64/0	12S	28.24	28.20
MoE-NS-614M-A95M	8/128/0	8S	28.42	28.49
MoE-NS-2.5B-A270M	8/128/0	16S	20.85	20.83

Appendix

Table 7: MirrorMoE has less gains when pretraining MoEs without shared experts.

Method	PPL ( $\downarrow$ )	# activated experts at first iteration	# activated experts at last iteration
Vanilla	54.78	8	8
MirrorMoE	51.77	8	8
Top- $p$ , $p=0.4$	55.64	18.65	6.73
Top- $p$ , $p=0.6$	54.09	29.46	10.79

Appendix

Table 8: Comparison with top-

p

routing. Activated experts are averaged over layers and tokens for top-

p

routing.

Mixture-of-Experts Models

Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.

See all 454 papers

Routing-Free Mixture-of-Experts
Apr 1, 2026Yilun Liu, Jinru Han, Sikuan Yan +2Mixture-of-Experts Models Expert Load Balancing
Ludwig Maximilian University of Munich · Munich Center for Machine Learning · University of California, Los Angeles
Adaptive Inverted-Index Routing for Granular Mixtures-of-Experts
May 6, 2026Klaus-Rudolf Kladny, Maximilian Mordig, Bernhard Schölkopf +1Mixture-of-Experts Inference Expert Routing
Max Planck Institute for Intelligent Systems, Tübingen · Tübingen AI Center · ETH Zurich +1
Path-Constrained Mixture-of-Experts
Mar 18, 2026Zijin Gu, Tatiana Likhomanenko, Vimal Thilak +2Mixture-of-Experts Language Models Sparse Mixture-of-Experts
Apple · †Google

Apr 1, 2026J/K move · Enter open · S save

Routing-Free Mixture-of-Experts

Yilun Liu, Jinru Han, Sikuan Yan, Volker Tresp+1

Ludwig Maximilian University of Munich · Munich Center for Machine Learning · University of California, Los Angeles

Mixture-of-Experts Models Expert Load Balancing

Open paper AskPDF arXiv

Standard Mixture-of-Experts (MoE) models rely on centralized routing mechanisms that introduce rigid inductive biases. We propose Routing-Free MoE which eliminates any hard-coded centralized designs including external routers, Softmax, Top-K and load balancing, instead encapsulating all activation functionalities within individual experts and directly optimized through continuous gradient flow, enabling each expert to determine its activation entirely on its own. We introduce a unified adaptive load-balancing framework to simultaneously optimize both expert-balancing and token-balancing objectives through a configurable interpolation, allowing flexible and customizable resource allocation. Extensive experiments show that Routing-Free MoE can consistently outperform baselines with better scalability and robustness. We analyze its behavior in detail and offer insights that may facilitate future MoE design ad optimization.

Abstract

Mixture-of-Experts Models

Explore similar work

Routing-Free Mixture-of-Experts

Adaptive Inverted-Index Routing for Granular Mixtures-of-Experts

Path-Constrained Mixture-of-Experts