More Experts, Worse Dynamics: Inverse Scaling and Spectral Bias in Mixture-of-Experts State-Space Models

Chandresh Pandey

cs.LGAug 22, 2026

More Experts, Worse Dynamics: Inverse Scaling and Spectral Bias in Mixture-of-Experts State-Space Models

Authors: Chandresh Pandey

Organizations: Computer Science and Engineering, Delhi Technological University, India

Abstract

Mixture-of-Experts (MoE) architectures are commonly motivated as a way to increase expressivity by decomposing complex systems into simpler local dynamics. This intuition has recently been extended to spectral state-space models, where mixing stable operators is assumed to enable adaptation to heterogeneous or regime-switching time series. We critically evaluate this assumption in a controlled synthetic setting designed to isolate dynamical rather than representational challenges. We study a next-step prediction task on sequences composed of three regimes: chaotic dynamics generated by the Mackey-Glass system, a stable oscillatory regime, and a noise-dominated autoregressive regime. Across extensive ablations including capacity scaling, oracle routing, frozen-expert variants, and comparisons to output-level MoE baselines, operator-level mixture models consistently fail to outperform a single-expert baseline. Increasing the number of experts leads to inverse scaling, routing collapses or fails to induce meaningful specialization, and even perfect regime supervision does not prevent degradation in global performance. Furthermore, we show that apparent improvements in mean squared error on chaotic trajectories can be misleading. Phase-space analysis reveals that lower error often arises from temporal smoothing that destroys the geometry of the underlying attractor rather than from faithful modeling of the dynamics. These results identify a likely limitation of operator interpolation under the studied parameterization and training protocol, and underscore the need for geometry-aware evaluation when assessing regime-switching dynamical systems.

Figures & tables

Figure 1: Test NMSE (mean $\pm$ standard deviation over five seeds) as a function of the number of experts $K$ . A small improvement is occasionally observed for $K=2$ , but performance degrades and variance increases as capacity grows beyond a single expert.

Figure 2: Prediction and routing behavior across regimes. Top: One-step prediction for the single-expert baseline ( $K=1$ ) and the GMS model ( $K=4$ ) across chaotic, oscillatory, and noise-dominated segments. Bottom: Router assignment probabilities over time for the $K=4$ GMS model. Despite explicit regime boundaries, routing collapses to a dominant expert ( 2 ; 4 ) for most timesteps, with only transient deviations during regime transitions.

Figure 3: Phase-space reconstruction of the chaotic regime using delay-coordinate embedding $(x(t),x(t-\tau))$ with $\tau=17$ . Left : ground truth Mackey–Glass attractor. Middle : single-expert baseline ( $K=1$ ). Right : GMS with operator mixing ( $K=4$ ). Despite a $61\%$ reduction in MSE, operator mixing collapses the attractor toward a smooth, low-variance manifold, destroying fine-scale chaotic structure.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Value
Input dimension	$d_{in}=1$
State dimension	$d_{state}=64$
Output dimension	$d_{out}=1$
Number of experts	$K\in\{1,2,4,8\}$
State transition	Diagonal spectral operator
Eigenvalue type	Complex-valued

Appendix

Table 1: Architectural hyperparameters shared across all models.

Experts $K$	$d_{state}$ per expert	Total state dim	Relative capacity
1	64	64	1.0×
2	64	128	2.0×
4	64	256	4.0×
8	64	512	8.0×

Appendix

Table 2: Effective latent dimensionality and relative capacity as a function of number of experts.

Router Component	Value
Expert input projections	$K$ linear layers: $1\rightarrow 16$
Router input dimension	$16K+64$
Hidden layer	Linear( $16K+64\rightarrow 4K$ )
Activation	ReLU
Output layer	Linear( $4K\rightarrow K$ )
Normalization	Softmax

Appendix

Table 3: Router architecture and hyperparameters.

Hyperparameter	Value
Optimizer	AdamW
Learning rate	$1\times 10^{-3}$
Weight decay	$1\times 10^{-4}$
Loss function	Mean squared error
Gradient clipping	$\ell_{2}$ norm clipped to $1.0$
Batch size	Full batch

Appendix

Table 4: Optimization hyperparameters.

Seed	Baseline ( $K=1$ )	GMS ( $K=4$ )
42	0.02479	0.03442
100	0.02513	0.03480
1337	0.02617	0.03259
2024	0.02650	0.02091
999	0.02630	0.01893

Appendix

Table 5: Test NMSE across five random seeds for the baseline and GMS models.

Model	Mean NMSE	Std
Baseline ( $K=1$ )	0.02578	0.00069
GMS ( $K=4$ )	0.02833	0.00694

Appendix

Table 6: Mean and standard deviation of test NMSE across seeds.

Figure 4: (A) Histogram of eigenvalue magnitudes $|\lambda|$ for each expert in the $K=4$ GMS model. (B) Per-regime MSE comparison between GMS ( $K=4$ ) and the single-expert baseline ( $K=1$ ). Experts differ primarily in degree of contraction, while regime-wise errors reveal a trade-off between improved chaotic prediction and degraded performance on simpler regimes.

Model	NMSE	Distinct Experts Used
Baseline ( $K=1$ )	0.02001	1/3
GMS Oracle ( $K=4$ )	0.02294	2/3

Appendix

Table 7: Global NMSE under oracle routing supervision.

Regime	Baseline ( $K=1$ )	GMS Oracle ( $K=4$ )	$\Delta$
Chaos	0.00765	0.00265	+65.3%
Oscillator	0.01068	0.01760	-64.8%
Noise	0.05630	0.06581	-16.9%

Appendix

Table 8: Per-regime MSE under oracle routing.

Model	NMSE
Frozen Experts ( $K=4$ )	0.98584

Appendix

Table 9: Global NMSE when expert operators are frozen and only the router is trained.

Model	NMSE
Operator Mixing (GMS, $K=4$ )	0.08385
Output Mixing (MoE, $K=4$ )	0.07054

Appendix

Table 10: Comparison of operator-level and output-level mixture models.

Mixture-of-Experts Models

Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.

See all 456 papers

Towards a Statistical Understanding of Mixture-of-Experts
Sep 3, 2026Siyuan He, Bokai Yang, Jie Hu +2Mixture of Experts Sparse Mixture-of-Experts
Qiuzhen College, Tsinghua University, Beijing, China. · KLATASDS-MOE, School of Statistics, East China Normal University, Shanghai, China. · Yau Mathematical Sciences Center, Tsinghua University, Beijing, China. +1
How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization
May 13, 2026Leena Chennuru Vankadara, Moritz Haas, Luke Hayward +2Mixture-of-Experts Models Dynamical Mean-Field Theory
1Gatsby Computational Neuroscience Unit, University College London · 2Amazon · University of Tübingen, Tübingen AI Center
Generalization and Scaling Laws for Mixture-of-Experts Transformers
Apr 10, 2026Mansour Zoubeirou a MayakiMixture-of-Experts Models AI Scaling Laws
Univ Lyon, Lyon 2, LIRIS UMR 5205 F-69676, Lyon, France

Sep 3, 2026J/K move · Enter open · S save

Towards a Statistical Understanding of Mixture-of-Experts

Siyuan He, Bokai Yang, Jie Hu, Ziwen Gao+1

Qiuzhen College, Tsinghua University, Beijing, China. · KLATASDS-MOE, School of Statistics, East China Normal University, Shanghai, China. · Yau Mathematical Sciences Center, Tsinghua University, Beijing, China.+1

Mixture of Experts Sparse Mixture-of-Experts Mixture-of-Experts Models Statistical Learning Theory

Open paper AskPDF arXiv

Mixture-of-experts (MoE) architectures increase model capacity by combining a collection of expert predictors through input-dependent routing, while often activating only a small subset of experts for each input. Despite their growing importance in modern large-scale models, the statistical roles of their design choices, especially routing, sparse activation, and shared experts, remain only partially understood, as existing theory has largely focused on parametric or correctly specified MoE models. In this paper, we view MoE as a form of localized aggregation and show how this localization reshapes the approximation-estimation-computation tradeoff. We derive oracle risk bounds for learning dense and sparse routing with evolving experts, separating approximation, expert-learning, and router-estimation errors, and characterize how sparse Top-K routing can retain the benefits of localized aggregation while controlling per-input computation. We also interpret gating through the geometry of input space, relating routing performance to regions of local expert advantage, and show how shared experts, as adopted in architectures such as DeepSeekMoE, can extract common predictive structure so that routed experts focus on residual local variation. Together, these results provide a unified statistical framework for understanding MoE through input-dependent expert aggregation, in which expert specialization and computational tradeoffs are governed by local predictive structure.

Abstract

Mixture-of-Experts Models

Explore similar work

Towards a Statistical Understanding of Mixture-of-Experts

How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization

Generalization and Scaling Laws for Mixture-of-Experts Transformers