Grassmannian Mixture-of-Experts: Concentration-Controlled Routing on Subspace Manifolds

Ibne Farabi Shihab; Sanjeda Akter; Anuj Sharma

cs.LGFeb 19, 2026

Grassmannian Mixture-of-Experts: Concentration-Controlled Routing on Subspace Manifolds

Authors: Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma

Organizations: Department of Computer Science, Iowa State University · Department of Civil, Construction and Environmental Engineering, Iowa State University

Abstract

Mixture-of-Experts models rely on learned routers to assign tokens to experts, yet standard softmax gating provides no principled mechanism to control the tradeoff between sparsity and utilization. We propose Grassmannian MoE (GrMoE), a routing framework that operates on the Grassmannian manifold of subspaces, where gating weights arise from the concentration parameters of Matrix Bingham distributions. This construction yields a single, interpretable knob -- the concentration matrix $Λ$ -- that continuously controls routing entropy, replacing discrete top- $k$ selection with a smooth, geometrically principled sparsity mechanism. We further develop an amortized variational inference procedure for posterior routing distributions, enabling uncertainty-aware expert assignment that naturally resists expert collapse. We formally prove tight bounds relating the Bingham concentration spectrum to routing entropy, expected top- $k$ mass, and an exponential bound on expert collapse, establishing the first formal theory of concentration-controlled sparsity. On synthetic routing tasks, a 350M-parameter MoE language model with 8 experts, a 1.3B-parameter model with 16 experts, and a 2.7B-parameter model with 32 experts, GrMoE achieves 0% routing collapse across all seeds, comparable or better perplexity with 15--30% improved load balance, and a smooth monotonic relationship between concentration and effective sparsity that enables post-hoc sparsity tuning without retraining. Token-level analysis reveals that experts learn heterogeneous concentration values that correlate with linguistic specialization, providing interpretable routing behavior.

Figures & tables

Method	Assign. Acc $\uparrow$	CV $\downarrow$	Collapse $\downarrow$	Entropy
Softmax Top-1	82.4	.312	36%	1.02
Switch	85.1	.187	22%	1.41
Expert Choice	84.7	.042	0%	1.89
Hash	71.2	.031	0%	2.08
Soft MoE	86.8	.095	8%	1.55
vMF-Gate	86.3	.178	18%	1.38

Table 1: Synthetic routing task: recovery of ground truth expert assignments across 50 random seeds.

Figure 1: Post-hoc sparsity control via concentration scaling. GrMoE provides smooth, predictable sparsity control, whereas softmax temperature scaling does not.

Scale	Method	PPL $\downarrow$	CV $\downarrow$	Collapse $\downarrow$	$H$
350M / N=8	Softmax Top-2	$18.7_{\pm 0.4}$	$.241_{\pm.08}$	40%	$1.12$
	Switch	$19.1_{\pm 0.3}$	$.152_{\pm.04}$	20%	$0.89$
	Expert Choice	$18.9_{\pm 0.2}$	$.068_{\pm.01}$	0%	$1.71$
	Hash	$20.3_{\pm 0.1}$	$.038_{\pm.01}$	0%	$2.05$
	Soft MoE	$18.4_{\pm 0.2}$	$.112_{\pm.03}$	10%	$1.42$
	vMF-Gate	$18.5_{\pm 0.3}$	$.198_{\pm.06}$	20%	$1.18$

Table 2: MoE language models on OpenWebText. PPL on held-out set. Results averaged over 5 seeds (

\pm

std).

Figure 2: Training dynamics comparison. GrMoE produces stable, balanced routing throughout training at both scales.

Figure 3: Token-level routing analysis on the 1.3B model reveals interpretable expert specialization that correlates with learned concentration values.

Softmax Top- $k$ (must retrain for each $k$ ):
$\alpha$	Eff. Experts	PPL $\downarrow$	Throughput $\uparrow$
0.5	5.2	18.4	0.82 $\times$
1.0 (default)	3.1	18.1	1.0 $\times$
2.0	1.8	18.3	1.24 $\times$
5.0	1.1	19.0	1.41 $\times$
Top-2	2.0	18.7	1.0 $\times$

Table 3: Post-hoc sparsity tuning at inference on the 350M model (no retraining). Throughput is relative to

\alpha=1.0

Figure 4: Pareto frontier of PPL vs. throughput. A single GrMoE model (varying

\alpha

) traces a smooth frontier that dominates separately trained softmax top-

k

models at every operating point.

$\mathcal{L}_{\text{subspace}}$	PPL $\downarrow$	CV $\downarrow$	Collapse $\downarrow$	$\rho_{\max}$
None	$18.3_{\pm 0.3}$	$.168_{\pm.05}$	20%	0.72
$\rho_{0}=0.1$	$18.4_{\pm 0.2}$	$.052_{\pm.01}$	0%	0.10
$\rho_{0}=0.3$	$18.1_{\pm 0.2}$	$.074_{\pm.02}$	0%	0.28
$\rho_{0}=0.5$	$18.2_{\pm 0.2}$	$.098_{\pm.03}$	0%	0.46
$\rho_{0}=0.8$	$18.3_{\pm 0.3}$	$.142_{\pm.04}$	10%	0.71

Table 4: Ablation: subspace overlap regularization (350M model,

N=8

). Mean over 5 seeds.

$k_{r}$	PPL $\downarrow$	CV $\downarrow$	Collapse $\downarrow$
16	$18.9_{\pm 0.3}$	$.092_{\pm.02}$	0%
32	$18.4_{\pm 0.2}$	$.081_{\pm.02}$	0%
48	$\mathbf{18.1_{\pm 0.2}}$	$.074_{\pm.02}$	0%
64	$18.2_{\pm 0.2}$	$.078_{\pm.02}$	0%
96	$18.5_{\pm 0.3}$	$.105_{\pm.03}$	0%

Table 5: Routing rank ablation on the 350M model (

N=8

d=768

). PPL and CV averaged over 5 seeds.

Component	Softmax Top-2	GrMoE + Amort.
Attention	4.2 ms	4.2 ms
Routing	0.3 ms	0.4 ms
Expert FFN	7.1 ms	7.1 ms
Other	0.8 ms	0.8 ms
Total	12.4 ms	12.5 ms

Table 6: Wall-clock overhead breakdown on the 350M model (single A100, batch size 16, sequence length 1024).

Expert Routing

Expert routing in mixture-of-experts models selects which subnetworks process each input, balancing accuracy, computation, and expert workload.

See all 112 papers

Geometric Routing Enables Causal Expert Control in Mixture of Experts
Apr 15, 2026Ivan Ternovtsii, Yurii BilakMixture-of-Experts Interpretability Mixture-of-Experts Expert Specialization
Department of Software Systems, Uzhhorod National University, Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes
How Sparse Probability Maps Shape Mixture-of-Experts Routing
Oct 5, 2026Tomás Brogueira, Marcos Treviso, Miguel CouceiroMixture-of-Experts Language Models Mixture-of-Experts Inference
Técnico, Universidade de Lisboa · INESC-ID · Instituto de Telecomunicações +2
Routing-Free Mixture-of-Experts
Apr 1, 2026Yilun Liu, Jinru Han, Sikuan Yan +2Mixture-of-Experts Models Expert Load Balancing
Ludwig Maximilian University of Munich · Munich Center for Machine Learning · University of California, Los Angeles

Apr 15, 2026J/K move · Enter open · S save

Geometric Routing Enables Causal Expert Control in Mixture of Experts

Ivan Ternovtsii, Yurii Bilak

Department of Software Systems, Uzhhorod National University, Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes

Mixture-of-Experts Interpretability Mixture-of-Experts Expert Specialization Mixture of Experts Mixture-of-Experts Models Neural Network Interpretability Representation Geometry in Language Models

Open paper AskPDF arXiv

Sparse Mixture-of-Experts (MoE) models scale parameters while fixing active computation per token, but the specialization of individual experts remains opaque. In a companion paper we showed that routing topology is quality-neutral: five structurally different configurations converge to statistically equivalent language modeling quality. Here we show that expert identity is nonetheless causally meaningful: individual rank-1 experts are monosemantic by construction, and cosine-similarity routing in a low-dimensional metric space makes their specialization directly inspectable. We present four lines of evidence. First, projecting expert output vectors through the unembedding matrix yields a Semantic Dictionary: 15% of experts are monosemantic specialists spanning 10 categories (temporal, geographic, cardinal, discourse, emotional, financial, military, scientific). Second, routing exhibits a frequency-to-syntax gradient: early layers separate tokens by word frequency, deeper layers by syntactic class (Zipf-confound controls, all

p < 0.001

). Third, causal interventions confirm these labels: steering toward a temporal expert's centroid increases P(temporal) by +321% (median across 44 prompts); suppressing a geographic expert drops P(geographic) by -23%; rewriting an expert's output vector halves target-category probability, and effects compose additively across layers. Fourth, the interventions are not unique to cosine routing: linear routers support comparable steering, but only cosine routing provides geometric transparency -- expert specialization is readable directly from the centroid matrix. MoE expert-level specialization is a first-class interpretability primitive: architecturally monosemantic, causally validated, and controllable at inference with zero overhead.

Abstract

Expert Routing

Explore similar work

Geometric Routing Enables Causal Expert Control in Mixture of Experts

How Sparse Probability Maps Shape Mixture-of-Experts Routing

Routing-Free Mixture-of-Experts