Softmax Routing Dynamics in Mixture-of-Experts Models: Collapse and Recovery

Shengze Xu; Maolin Deng; Tieyong Zeng

cs.LGOct 29, 2026

Softmax Routing Dynamics in Mixture-of-Experts Models: Collapse and Recovery

Authors: Shengze Xu, Maolin Deng, Tieyong Zeng

Organizations: The Chinese University of Hong Kong · Beijing Normal-Hong Kong Baptist University · Guangzhou Nanfang College

Abstract

Softmax routing in mixture-of-experts models can concentrate probability on a subset of experts, suppressing access to the others. We study how this concentration forms, why it persists, and how suppressed probabilities recover. For fixed experts, a stationary-loss barrier identifies the face supporting the optimal routed output. In a common-target square-loss model with freely trainable expert outputs, slow adaptation tracks fixed-expert dynamics on finite horizons, while every fixed positive adaptation rate yields an interior routing limit. An exact normal-velocity decomposition separates complementary-expert covariance from relative gradient advantage. Under positive two-sided power bounds along recovery paths, uniform over initial masses, the resulting coefficient's order of vanishing gives logarithmic or power-law recovery times. Integrations of constructed fields reproduce these orders. An exact discrete log-odds identity relates neural updates to threshold crossing. In paired Transformer-MoE training, removing a class-conditional router-gradient contrast delays confirmed recovery and reduces accumulated log-odds on fixed tokens, both with and without capacity constraints.

Figures & tables

Geometry	Barrier gap	Entry time	Outside mass $(\times 10^{-3})$	Projection error $(\times 10^{-3})$
Triangle	0.1225	5.38	1.62	1.64
Pentagon edge	0.3360	18.88	1.08	1.22
Redundant hexagon face	0.0400	91.78	1.53	1.59
Cube bottom facet	0.4880	26.60	2.51	2.51
Cube boundary control	0.4880	—	320	482

Table 1: Fixed-expert face selection. Barrier gap is

\ell_{\rm comp}-r_{\ast}^{2}

; entry time is the first sample with

\mathcal{L}<\ell_{\rm comp}

. Outside-face mass

D_{F^{\ast}}(p)

and projection error

\|E^{\top}p-z^{\ast}\|

are measured at

T=2000

. A dash denotes no entry for the boundary control.

Figure 1: Face selection and expert-adaptation time scales. (a) Triangle trajectories approach

\Delta_{F^{\ast}}

; circles mark initial states and diamonds mark entry below the stationary-loss barrier. (b) Outside-face mass under fixed and adapting experts; shading marks

D_{F^{\ast}}<0.05

. (c) Finite-horizon routing error follows the

O(\eta)

prediction of Theorem 4.1 .

Figure 2: Recovery-time scaling in constructed fields. (a) Fixed positive covariance and zero mean advantage give logarithmic recovery. (b) Zero covariance and uniform complementary weights give inverse-probability recovery under a fixed positive advantage. Lines show theoretical hitting times and symbols show numerical events for

n=4,8

, with

\theta=0.05

; the curves in (a) coincide. (c) Fitted exponents agree with prescribed orders in

\Gamma_{j}=x^{\alpha}

;

\alpha=0

uses a separate logarithmic fit.

Dispatch	Control	Projection	RMST difference (95% CI)	Sign-test $p$
No drop	796.25	840.50	$+44.25\;[31.25,57.50]$	$9.77\times 10^{-4}$
Capacity $1.25$	810.50	869.50	$+59.00\;[39.50,81.00]$	$7.81\times 10^{-3}$

Table 2: Effect of router-gradient projection on restricted mean time to confirmed recovery through 1,024 steps. Confirmation requires mean target probability

\geqslant 0.05

at two consecutive 64-step evaluations. Differences are projection minus control, with 95% paired-seed bootstrap CIs. The no-drop and capacity settings use 16 and eight paired seeds; sign-test

p

-values are exact and two-sided, excluding ties.

Figure 3: Router-gradient projection delays confirmed recovery. (a), (b) Fractions without confirmed recovery under no-drop and capacity-constrained dispatch (factor

1.25

), using 16 and eight paired seeds, respectively. Curves weight seeds and their layer–initial-mass combinations equally; their areas through 1,024 steps define RMST. (c) Circles show paired-seed RMST differences (projection minus control); diamonds and bars give means and 95% paired-seed bootstrap CIs.

Dispatch	Cumulative log-odds difference (95% CI)	Difference in fraction meeting threshold (pp; 95% CI)
No drop	$-0.194\;[-0.252,-0.133]$	$-1.06\;[-1.64,-0.54]$
Capacity $1.25$	$-0.228\;[-0.304,-0.140]$	$-2.28\;[-3.71,-0.82]$

Table 3: Log-odds response of initially subthreshold tokens at step 1,024. Differences are projection minus control, with 95% paired-seed bootstrap CIs from eight seeds per setting. Threshold attainment is evaluated for each token before averaging; pp denotes percentage points.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure B.1 : Face selection across triangle initializations. Colors distinguish initial loss below the stationary-loss barrier, later sublevel entry with face selection, and trajectories that have not selected the face by

T=2000

Figure B.2 : Residual decay and interior routing under joint training. (a) Squared residual norm and its exponential envelope. (b) Expert routing probabilities along the same trajectory.

Figure B.3 : Fitting-window effects on recovery exponents. Fitted exponent errors decrease as the upper end of the initial-mass window approaches zero. Results use

n=4

;

n=8

agrees at plotting precision.

Figure B.4 : Recovery with changing advantages and nonquadratic joint training. (a) Accumulated sign-changing advantage relative to the flow-specific recovery threshold; zero marks threshold attainment. (b) Nonquadratic PL loss and its exponential envelope.

Figure C.1 : Initial routing separation under a shared affine router. Observed slopes agree with

\chi_{\rm feat}

in random feature–residual configurations; nullspace controls have zero first-order separation.

Synthetic data
Router	Mean $\ell^{1}$ deviation	Dominant experts
Shared global	$0$	$1$
Input-conditioned, softmax	$1.012\pm 0.120$	$3.1\pm 0.5$
Input-conditioned, entropy	$0.305\pm 0.047$	$3.2\pm 0.4$
Input-conditioned, capacity	$1.072\pm 0.084$	$4$
AG News

Appendix

Table C.1 : Routing structure across synthetic and external datasets. Values are means

\pm

standard deviations over ten paired seeds, except twelve for UCI HAR. Single values denote invariant statistics. The synthetic shared-global row summarizes the identical routing-structure statistics of softmax, load-balanced, and top-2 variants.

Study	Mean difference	95% CI
Four-domain small model	0.444	[0.404,0.478]
Four-domain large model	0.236	[0.217,0.255]
Four-domain large, capacity	0.174	—
AG News, no drop	0.102	[0.081,0.128]
AG News, capacity	0.102	[0.087,0.117]

Appendix

Table C.2 : Mean between-group

\ell^{1}

deviation differences across settings (input-conditioned minus shared global), averaged over evaluated middle layers. Intervals are paired-seed bootstrap 95% CIs. Each study uses eight seeds except the four-domain capacity comparison, whose four-seed mean is reported descriptively. Capacity factor is

1.25

Figure D.1 : Paired training with and without class-conditional gradient projection. Branches share the calibrated state and minibatches; projection acts before clipping and AdamW.

Dispatch	$\theta$	RMST difference (95% CI)	Control recovery (%)	Projection recovery (%)
No drop	0.04	$56.50\;[46.00,67.25]$	71.48	59.38
	0.05	$44.25\;[31.25,57.50]$	44.53	35.94
	0.06	$25.00\;[15.25,37.50]$	22.27	18.36
	0.08	$10.50\;[4.50,17.50]$	6.64	5.08
Capacity $1.25$	0.04	$75.50\;[55.50,96.50]$	65.63	53.13
	0.05	$59.00\;[39.50,81.00]$	40.63	32.03

Appendix

Table E.1 : Confirmed-recovery sensitivity to the probability threshold. RMST differences are projection minus control in optimizer steps, with pointwise 95% paired-seed bootstrap CIs. Percentages give confirmation by step 1,024. The no-drop and capacity settings use 16 and eight seeds, with equal weight on seeds and layer–initial-mass combinations within seeds.

Figure E.1 : Accumulated log-odds and fractions of tokens meeting the threshold. (a), (b) Mean accumulated log-odds

A_{m}

. (c), (d) Fractions satisfying

A_{m}(x)\geqslant B(x)

at each evaluation, computed before averaging. Each setting uses eight paired seeds, with equal mass and seed weights; capacity factor is

1.25

$\varepsilon$	Dispatch	RMST difference (steps)	Control recovery (%)	Projection recovery (%)
0.005	No drop	$+14$	10.9	6.3
	Capacity	$+22$	12.5	3.1
0.010	No drop	$+33$	23.4	17.2
	Capacity	$+48$	21.9	12.5
0.020	No drop	$+62$	57.8	39.1
	Capacity	$+94$	46.9	37.5

Appendix

Table E.2 : Confirmed recovery by initial probability

\varepsilon

. Descriptive RMST differences are projection minus control through 1,024 steps at threshold

0.05

, averaging four layers within each seed and then seeds equally. Percentages give confirmation by step 1,024. No-drop and capacity settings use 16 and eight seeds, respectively; capacity factor is

1.25

Mixture of Experts

Mixture of Experts (MoE) models route inputs to specialized subnetworks, aiming to scale capacity and adapt computation without activating every parameter.

See all 226 papers

How Sparse Probability Maps Shape Mixture-of-Experts Routing
Oct 5, 2026Tomás Brogueira, Marcos Treviso, Miguel CouceiroMixture-of-Experts Language Models Mixture-of-Experts Inference
Técnico, Universidade de Lisboa · INESC-ID · Instituto de Telecomunicações +2
ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts
Jun 1, 2026Heng Zhao, Zilei Shao, Guy Van den Broeck +1Sparse Mixture-of-Experts Expert Routing
Department of Computer Science, University of Virginia, Charlottesville, USA · Department of Computer Science, University of California, Los Angeles, Los Angeles, USA
Soft-to-Hard Routing in Sparse Mixture-of-Experts Models
May 4, 2026Reza RastegarSparse Mixture-of-Experts Expert Routing
Meta Platforms, Inc Woodinville, WA

Oct 5, 2026J/K move · Enter open · S save

How Sparse Probability Maps Shape Mixture-of-Experts Routing

Tomás Brogueira, Marcos Treviso, Miguel Couceiro

Técnico, Universidade de Lisboa · INESC-ID · Instituto de Telecomunicações+2

Mixture-of-Experts Language Models Mixture-of-Experts Inference Expert Routing

Open paper AskPDF arXiv

Mixture-of-experts (MoE) routers typically apply softmax to the router scores and keep the top-K experts, making every token use exactly K experts. Sparsity-inducing probability maps such as sparsemax, alpha-entmax and normmax can adaptively assign exact zeros to selected experts, and therefore appear to offer token-dependent expert participation, even when using the same top-K machinery. In this work, we study whether and how this sparsity survives training. We train matched 300M and 1B top-2 MoE language models with softmax, 1.5-entmax, sparsemax and 2-normmax, and find that the maps behave very differently once trained: at 1B, entmax discards 30% less probability mass than softmax while almost never dropping a selected expert, sparsemax retains the most mass, and normmax routes 21% of tokens to a single expert. These outcomes are not properties of the maps alone. Each map drops a selected expert only when the gap between the two largest scores reaches a fixed threshold, and the trained routers differ in the score distribution they learn: the entmax router learns scores with roughly half the spread of softmax's, which keeps its top-2 gaps below its threshold, while sparsemax and normmax, which share the same threshold, learn different gap distributions and hence different participation. Routers thus co-adapt their scores to the map, and a map's capacity to produce zeros does not by itself determine expert participation. While none of the sparse maps improves validation loss over softmax, they make the trained models far less sensitive to selecting more experts at inference: sparsemax trained with K=2 loses 0.02 nats when run with K=8, where softmax loses 0.58. Our results indicate that adaptive MoE routing has to be designed around the joint behavior of the probability map and the learned scores, rather than around the map alone.

Abstract

Mixture of Experts

Explore similar work

How Sparse Probability Maps Shape Mixture-of-Experts Routing

ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts

Soft-to-Hard Routing in Sparse Mixture-of-Experts Models