Geometric Routing Enables Causal Expert Control in Mixture of Experts
Organizations: Department of Software Systems, Uzhhorod National University, Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes
Abstract
Sparse Mixture-of-Experts (MoE) models scale parameters while fixing active computation per token, but the specialization of individual experts remains opaque. In a companion paper we showed that routing topology is quality-neutral: five structurally different configurations converge to statistically equivalent language modeling quality. Here we show that expert identity is nonetheless causally meaningful: individual rank-1 experts are monosemantic by construction, and cosine-similarity routing in a low-dimensional metric space makes their specialization directly inspectable. We present four lines of evidence. First, projecting expert output vectors through the unembedding matrix yields a Semantic Dictionary: 15% of experts are monosemantic specialists spanning 10 categories (temporal, geographic, cardinal, discourse, emotional, financial, military, scientific). Second, routing exhibits a frequency-to-syntax gradient: early layers separate tokens by word frequency, deeper layers by syntactic class (Zipf-confound controls, all ). Third, causal interventions confirm these labels: steering toward a temporal expert's centroid increases P(temporal) by +321% (median across 44 prompts); suppressing a geographic expert drops P(geographic) by -23%; rewriting an expert's output vector halves target-category probability, and effects compose additively across layers. Fourth, the interventions are not unique to cosine routing: linear routers support comparable steering, but only cosine routing provides geometric transparency -- expert specialization is readable directly from the centroid matrix. MoE expert-level specialization is a first-class interpretability primitive: architecturally monosemantic, causally validated, and controllable at inference with zero overhead.
Figures & tables
| Layer | Expert | Top Decoded Words | Archetype |
| L7 | E760 | before, after, prior, earlier | Temporal sequence |
| L3 | E71 | six, three, four, five, eight | Cardinal numbers |
| L7 | E193 | Moreover, Indeed, Furthermore | Discourse connectives |
| L6 | E224 | trouble, pain, disease, surgery | Medical/adversity |
| L6 | E88 | -ing, -ed, -ly, -tion, -ment | Morphological suffixes |
| L5 | E733 | London, Paris, Berlin, Rome | European capitals |
| Category | (targeted) | (random) | KL (targeted) | KL (random) |
|---|---|---|---|---|
| Temporal (L4-E80) | 0.000 | |||
| Geographic (L4-E685) | 0.000 | |||
| Cardinal (L7-E1004) | 0.000 | 0.000 | ||
| Discourse (L7-E525) | 0.000 | 0.000 |