When Does Sparse Routing Help? The Geometry of Expert Separation in Mixture of Experts

Huy Nguyen; Dung Le; Alessandro Rinaldo; Nhat Ho

stat.MLNov 1, 2026

When Does Sparse Routing Help? The Geometry of Expert Separation in Mixture of Experts

Authors: Huy Nguyen, Dung Le, Alessandro Rinaldo, Nhat Ho

Organizations: Department of Statistics and Data Science, The University of Texas at Austin

Abstract

Sparse mixtures of experts combine multiple prediction models while evaluating only a small subset for each input. Although this architecture reduces computation, how expert separation and routing sparsity jointly govern expert recovery remains an open problem. In this paper, we address this question for location Gaussian mixtures of experts with a fixed number of experts by establishing how expert separation and spatial arrangement determine recovery rates. For dense models, our results reveal that when experts form well-separated clusters, the separation geometry is governed by the size of the largest cluster; without a prescribed cluster structure, this geometry is instead driven by the total number of experts. For sparse models, the separation geometry on retained routing regions is determined by the number of active experts, while the contribution of omitted regions is controlled by their total probability. Consequently, activating fewer experts reduces the separation complexity in these bounds. Furthermore, Top-1 routing yields a particularly striking result: experts can be recovered at a rate independent of the separation quantity. Our technical approach combines polynomial interpolation and Gaussian reproducing-kernel Hilbert space (RKHS) test functions with quantitative restriction inequalities that extend pointwise comparisons to expert functions.

Figures & tables

Setting	Rate	Statistical result
Dense, one cluster	$\Delta_{\mathrm{sep}}^{-(2k-2)}\varepsilon_{n,\rm den}$	Corollary 5.3 (i), $q=1$
Dense, several clusters	$\Delta_{\mathrm{sep}}^{-(2s_{\max}-2)}\varepsilon_{n,\rm den}$	Corollary 5.3 (i)
Dense, unstructured	$\Delta_{\mathrm{sep}}^{-(2k-2)}\varepsilon_{n,\rm den}$	Corollary 5.3 (ii)
Top- $K$ , retained regions	$C_{K,\lambda}^{-1}q_{n,K}$	Eq. ( 4.11 ); Theorem 5.5
Top- $K$ , full partition	$C_{K,\lambda}^{-1}q_{n,K}+\sqrt{\lambda}$	Corollary 5.6
Top- $K$ , balanced threshold	$q_{n,K}^{1/(2\alpha+1)}$	Proposition 5.7

Table 1: Likelihood recovery rates under the corresponding uniform hypotheses. All dense rows control globally matched experts under their respective inverse hypotheses. The unstructured row uses no prescribed cluster partition. Top-

K

rows use regional matchings; Top-1 concerns the selected function. Fixed constants are suppressed; the balanced row uses the oracle threshold in Proposition 5.7 .

Sparse Mixture-of-Experts

Sparse mixture-of-experts models route each input to a small subset of specialized networks to increase capacity while limiting computation.

See all 128 papers

Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing
Jul 30, 2026Huiyuan Tian, Bonan Xu, Shijian LiSparse Mixture-of-Experts Expert Routing
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University
Geometric Routing Enables Causal Expert Control in Mixture of Experts
Apr 15, 2026Ivan Ternovtsii, Yurii BilakMixture-of-Experts Interpretability Mixture-of-Experts Expert Specialization
Department of Software Systems, Uzhhorod National University, Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes
Grassmannian Mixture-of-Experts: Concentration-Controlled Routing on Subspace Manifolds
Feb 19, 2026Ibne Farabi Shihab, Sanjeda Akter, Anuj SharmaExpert Routing Mixture-of-Experts Language Models
Department of Computer Science, Iowa State University · Department of Civil, Construction and Environmental Engineering, Iowa State University

Jul 30, 2026J/K move · Enter open · S save

Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing

Huiyuan Tian, Bonan Xu, Shijian Li

College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University

Sparse Mixture-of-Experts Expert Routing Representation Geometry in Language Models Mixture-of-Experts Interpretability

Open paper AskPDF arXiv

Sparse mixture-of-experts (MoE) language models route each token to multiple experts, suggesting a geometric account of their benefit: co-selected experts should contribute distinct representation directions. Existing evidence often conflates route coherence, candidate quality, and candidate-by-context interaction. We distinguish these quantities using an Expert Subspace Separation Index (ESSI), matched-route residuals, and a prefix-controlled

2\times2

factorial; frozen-route interventions and a controlled Top-

k

study assess functional value. Three paired contrasts organize the findings. First, across six MoE architectures, expert subspaces overlap substantially, yet actual routes explain token representations better than matched alternatives. Second, across the 39 factorial cells in OLMoE, Mixtral, and DeepSeek, the selected candidate explains more of the residual representation than the strongest unselected rival in every cell, yet the actual prefix narrows this advantage throughout: all interactions are negative, and every 95% confidence interval lies below zero. Third, this geometric narrowing does not imply functional redundancy: adding later experts improves next-token prediction in 24 of 39 frozen-route comparisons, while the other 15 estimates are inconclusive; a controlled training study also favors Top-2 over Top-1 in all three seeds. We call this joint pattern coherent overlap: routing selects token-relevant experts from a shared geometric neighborhood, while useful multi-expert computation persists without disjoint linear coverage. Separating these quantities clarifies why geometric similarity alone cannot determine redundancy or pruning value.

Abstract

Sparse Mixture-of-Experts

Explore similar work

Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing

Geometric Routing Enables Causal Expert Control in Mixture of Experts

Grassmannian Mixture-of-Experts: Concentration-Controlled Routing on Subspace Manifolds