Geometric Routing Enables Causal Expert Control in Mixture of Experts

Ivan Ternovtsii; Yurii Bilak

cs.AIApr 15, 2026

Geometric Routing Enables Causal Expert Control in Mixture of Experts

Authors: Ivan Ternovtsii, Yurii Bilak

Organizations: Department of Software Systems, Uzhhorod National University, Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes

Abstract

Sparse Mixture-of-Experts (MoE) models scale parameters while fixing active computation per token, but the specialization of individual experts remains opaque. In a companion paper we showed that routing topology is quality-neutral: five structurally different configurations converge to statistically equivalent language modeling quality. Here we show that expert identity is nonetheless causally meaningful: individual rank-1 experts are monosemantic by construction, and cosine-similarity routing in a low-dimensional metric space makes their specialization directly inspectable. We present four lines of evidence. First, projecting expert output vectors through the unembedding matrix yields a Semantic Dictionary: 15% of experts are monosemantic specialists spanning 10 categories (temporal, geographic, cardinal, discourse, emotional, financial, military, scientific). Second, routing exhibits a frequency-to-syntax gradient: early layers separate tokens by word frequency, deeper layers by syntactic class (Zipf-confound controls, all $p < 0.001$ ). Third, causal interventions confirm these labels: steering toward a temporal expert's centroid increases P(temporal) by +321% (median across 44 prompts); suppressing a geographic expert drops P(geographic) by -23%; rewriting an expert's output vector halves target-category probability, and effects compose additively across layers. Fourth, the interventions are not unique to cosine routing: linear routers support comparable steering, but only cosine routing provides geometric transparency -- expert specialization is readable directly from the centroid matrix. MoE expert-level specialization is a first-class interpretability primitive: architecturally monosemantic, causally validated, and controllable at inference with zero overhead.

Figures & tables

Figure 1: The Logit Lens. (a)

P(\text{target})

rises monotonically with each hop for both static and MLP experts, with MLP showing

4.8\times

larger refinement. (b) Per-layer MoE refinement: MLP experts contribute

+4.3\%

gain at Layer 3 vs

+0.9\%

for static. (c) Mean rank of target token decreases with hops, confirming directional progress.

Layer	Expert	Top Decoded Words	Archetype
L7	E760	before, after, prior, earlier	Temporal sequence
L3	E71	six, three, four, five, eight	Cardinal numbers
L7	E193	Moreover, Indeed, Furthermore	Discourse connectives
L6	E224	trouble, pain, disease, surgery	Medical/adversity
L6	E88	-ing, -ed, -ly, -tion, -ment	Morphological suffixes
L5	E733	London, Paris, Berlin, Rome	European capitals

Table 1: Semantic Dictionary: representative expert archetypes (converged 50K checkpoint).

Figure 2: Polysemy Branching. The token “bank” routes through 86% different experts depending on river vs. finance context. Shared experts (yellow) handle syntactic obligations; unique experts (teal/red) encode context-specific semantics.

Figure 3: Routing Statistics by Layer. Both Wide and Deep architectures exhibit a clear gradient: early layers (L0–L2) route selectively (high Gini, low entropy, high max probability), while late layers (L5–L7) distribute near-uniformly across experts. This pattern holds regardless of routing topology.

Category	$\Delta P$ (targeted)	$\Delta P$ (random)	KL (targeted)
Temporal (L4-E80)	$\mathbf{-33.7\%}$	$+0.0\%$	$\mathbf{0.139}$
Geographic (L4-E685)	$\mathbf{-23.2\%}$	$+0.0\%$	$\mathbf{0.512}$
Cardinal (L7-E1004)	$-0.0\%$	$+0.0\%$	0.000
Discourse (L7-E525)	$-0.4\%$	$-0.0\%$	0.000

Table 2: Expert suppression (

N{=}1

) on category-relevant prompts: blocking one correctly-identified expert produces large, selective output shifts. Random suppression of the same size has zero effect.

Explore similar work

Apr 10, 2026

The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise

Mixture of Experts (MoEs) are now ubiquitous in large language models, yet the mechanisms behind their "expert specialization" remain poorly understood. We show that, since MoE routers are linear maps, hidden state similarity is both necessary and sufficient to explain expert usage similarity, and specialization is therefore an emergent property of the representation space, not of the routing architecture itself. We confirm this at both token and sequence level across five pre-trained models. We additionally prove that load-balancing loss suppresses shared hidden state directions to maintain routing diversity, which might provide a theoretical explanation for specialization collapse under less diverse data, e.g. small batch. Despite this clean mechanistic account, we find that specialization patterns in pre-trained MoEs resist human interpretation: expert overlap between different models answering the same question is no higher than between entirely different questions (

\sim

60%); prompt-level routing does not predict rollout-level routing; and deeper layers exhibit near-identical expert activation across semantically unrelated inputs, especially in reasoning models. We conclude that, while the efficiency perspective of MoEs is well understood, understanding expert specialization is at least as hard as understanding LLM hidden state geometry, a long-standing open problem in the literature.

Xi Wang, Soufiane Hayou, Eric Nalisnick

Johns Hopkins University

Mixture of Experts Mixture-of-Experts Models Expert Routing LLM Interpretability Representation Geometry in Language Models

Ask PDF

Apr 15, 2026

Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality

Sparse Mixture-of-Experts (MoE) architectures employ increasingly sophisticated routing mechanisms -- learned routers, multi-hop trajectories, token-dependent gating. We ask: does routing topology actually determine language modeling quality? We build a geometric MoE (ST-MoE) using cosine-similarity routing against learned centroids in a low-dimensional space (

d_{space} = 64

), requiring 80% fewer routing parameters than standard linear routers. Through 62 controlled experiments on WikiText-103 at 76--84M parameters trained to convergence (50K steps, 1.64B tokens), we find that routing topology does not determine asymptotic perplexity (PPL): five cosine-routing variants are statistically equivalent within a 1-PPL margin (Two One-Sided Tests [TOST],

p < 0.05

for all 10 pairwise comparisons; 15 runs across 3 seeds, observed range 33.93--34.72). The finding extends to hash, random-fixed, and top-1 routing (single-seed; graceful 1.1--2.2 PPL degradation) and replicates on OpenWebText (0.03 PPL gap, 6 runs, 3 seeds each). A standard linear router with 5.3

\times

more routing parameters reaches PPL 32.76, but iso-parameter cosine routing closes 67% of this gap -- the true mechanism advantage is

\sim

1.2%. The mechanistic explanation is convergent redundancy: multi-hop updates are collinear (

\cos(Δh_0, Δh_1) = 0.805

), implementing magnitude amplification rather than compositional reasoning; a single learnable scalar replicates multi-hop performance. As a practical payoff, zero-shot relative-norm halting saves 25% of MoE FLOPs at +0.12% PPL. Expert-level specialization and causal controllability -- which coexist with topology-level equifinality -- are explored in a companion paper.

Ivan Ternovtsii, Yurii Bilak

Department of Software Systems, Uzhhorod National University Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes

Mixture-of-Experts Language Models Expert Routing Mixture-of-Experts Inference

Ask PDF

Jul 30, 2026

Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing

Sparse mixture-of-experts (MoE) language models route each token to multiple experts, suggesting a geometric account of their benefit: co-selected experts should contribute distinct representation directions. Existing evidence often conflates route coherence, candidate quality, and candidate-by-context interaction. We distinguish these quantities using an Expert Subspace Separation Index (ESSI), matched-route residuals, and a prefix-controlled

2\times2

factorial; frozen-route interventions and a controlled Top-

k

study assess functional value. Three paired contrasts organize the findings. First, across six MoE architectures, expert subspaces overlap substantially, yet actual routes explain token representations better than matched alternatives. Second, across the 39 factorial cells in OLMoE, Mixtral, and DeepSeek, the selected candidate explains more of the residual representation than the strongest unselected rival in every cell, yet the actual prefix narrows this advantage throughout: all interactions are negative, and every 95% confidence interval lies below zero. Third, this geometric narrowing does not imply functional redundancy: adding later experts improves next-token prediction in 24 of 39 frozen-route comparisons, while the other 15 estimates are inconclusive; a controlled training study also favors Top-2 over Top-1 in all three seeds. We call this joint pattern coherent overlap: routing selects token-relevant experts from a shared geometric neighborhood, while useful multi-expert computation persists without disjoint linear coverage. Separating these quantities clarifies why geometric similarity alone cannot determine redundancy or pruning value.

Huiyuan Tian, Bonan Xu, Shijian Li

College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University

Sparse Mixture-of-Experts Expert Routing Representation Geometry in Language Models

Ask PDF