The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise

Xi Wang; Soufiane Hayou; Eric Nalisnick

cs.AIApr 10, 2026

The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise

Authors: Xi Wang, Soufiane Hayou, Eric Nalisnick

Organizations: Johns Hopkins University

Abstract

Mixture of Experts (MoEs) are now ubiquitous in large language models, yet the mechanisms behind their "expert specialization" remain poorly understood. We show that, since MoE routers are linear maps, hidden state similarity is both necessary and sufficient to explain expert usage similarity, and specialization is therefore an emergent property of the representation space, not of the routing architecture itself. We confirm this at both token and sequence level across five pre-trained models. We additionally prove that load-balancing loss suppresses shared hidden state directions to maintain routing diversity, which might provide a theoretical explanation for specialization collapse under less diverse data, e.g. small batch. Despite this clean mechanistic account, we find that specialization patterns in pre-trained MoEs resist human interpretation: expert overlap between different models answering the same question is no higher than between entirely different questions ( $\sim$ 60%); prompt-level routing does not predict rollout-level routing; and deeper layers exhibit near-identical expert activation across semantically unrelated inputs, especially in reasoning models. We conclude that, while the efficiency perspective of MoEs is well understood, understanding expert specialization is at least as hard as understanding LLM hidden state geometry, a long-standing open problem in the literature.

Figures & tables

Figure 1: Empirical verification of Prop. 1 . On the middle layer of 5 models, using 16 sequences from OpenWebText, we compare the RMS distance (Eq. ( 5 )) of all token pairs’ hidden state v.s. their router logits. Bottom left corners: Across all models, similar hidden states induce similar router logits, i.e. similar expert usage. Middle right: When the hidden states are different, the router logits can be similar or different. Outlier dots are from distances to the massive activation of the first token, i.e. attention sink ( Sun et al., 2024 ) .

Panel of Figure 2 — Figure 2: On gpt-oss-20b and Trinity-Mini-Base, we studied tokens in 16 random sequences from OpenWebText. Lines / bars denote the mean / 1-std. over 16 sequences. Two models use different training configurations, resulting in distinct patterns across depth. Top Left : Token hidden states become more correlated with depth, indicated by stronger energy (Eq. ( 7 )) from the shared direction (first right singular vector). Top Middle : After layer 11, the router output does not show a significant increase in cosine similarity, while hidden states do (Eq. ( 10 )). Top Right : Looking at the router, it learns to suppress the shared direction to maintain diversity of router outputs: The trace of router’s projection (averaged over rows) onto the shared direction (red line, Eq. ( 8 )) aligns with the shape of router output similarity (blue lines, Eq. ( 11 )) Bottom left : Under Muon trained model, token hidden states does not show increased correlation with depth. Bottom middle and right : When no load balancing loss is used, the router does not show evidence of suppressing shared direction.

Panel of Figure 3 — Figure 3: We shuffled and reversed 16 sequences from OpenWebText to simulate OOD inputs. Router outputs show less confidence (measured by maximum softmax probability, Eq. ( 12 )) on OOD inputs compared with the original ones (green line), due to the reduced alignment data router alignment (Eq. ( 4 ), verified in Fig. 23 ).

Panel of Figure 4 — Figure 4: Same token (top) / sentence (bottom), different context, the expert usage is context dependent in deep layers.

Figure 5: Pairwise comparisons of inputs drawn from Nemotron-CC-Math-v1, OpenWebText, and across these two datasets, plotting pooled hidden-state similarity (Eq. ( 17 )) against expert-usage frequency similarity (Eq. ( 14 )). Consistent with the token-level findings in Fig. 1 , more similar sequences tend to activate more similar experts.

Figure 6: Expert overlap (Jaccard similarity at Top-

p{=}0.8

, Eq. ( 19 )) between rollout pairs on 30 HMMT February 2025 questions, averaged over 4 seeds and three MoEs. Different models solving the same question (orange) share only

\sim

60% of their most-used experts, on par with the same model solving different questions (blue), and well below the same-question, same-model baseline (green). The gap is consistent across all layers.

Figure 7: Expert-usage similarity between paired queries on Ling-mini , tracked from prompt through rollout. The dashed line marks the prompt boundary. Across all three query pairs, prompt-phase similarity is near-identical; once generation begins, similarity either holds or drops sharply depending on the query and is not predictable from the prompt alone.

Figure 8: On gpt-oss-20b, we keep the

12

most-used experts (out of 32) in the last

k

layers, using usage frequency from a single hand-written paragraph. First four columns: Pruning up to 2 layers has a negligible effect (

<

10% NLL increase) on prompt-only inputs, indicating router collapse during prefilling. Last two columns: Adding completions makes the pruned experts consequential, with NLL increasing substantially for on- or off-policy rollout.

Figure 9: Per-expert activation frequencies for an HLE question (orange, 411 tokens) and a Wimbledon Wikipedia passage (purple, 2500 tokens) across four layers and five generation lengths for answering the HLE question. During prefilling, the two unrelated sequences activate nearly identical experts in deeper layers. As the generation proceeds, their activation profiles diverge, with the split emerging earlier in later layers.

Panel of Figure 10 — Figure 10: Cross-domain expert-usage similarity under increasing input duplication (1 $\times$ to 12 $\times$ ) across models; dashed grey shows intra-domain similarity. For models except gpt-oss with chat template ( 10(c) ), hidden state similarity alone does not produce router collapse under no duplication. Duplication amplifies the dominant hidden-state direction, progressively collapsing cross-domain expert usage toward the intra-domain baseline.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11: Same model architecture, different optimizers induce different feature space properties. At the same layer, given the same sequence, both models show a similar distribution of pairwise cosine similarity (over all tokens). However, embeddings in the model trained with AdamW show, on average, a higher RMS distance, indicating that different tokens exhibit high variation in norm (25 - 35). In contrast, Muon trained model shows more concentrated distribution of feature norm (12 - 14) and smaller pairwise RMS distances.

Figure 12: On dense model, except for Gemma3, which uses sandwich norm, token similarities also increase with depth.

Panel of Figure 13 — Figure 13: On Deepseek v2 lite, empirical comparison of the naive bound $\|P\|_{2}\|\delta_{i,j}\|$ versus the tighter projection-based bound $\|P\Pi_{r}\|_{2}\|\delta_{i,j}\|_{\Pi_{r}}$ from Proposition 1 , where $\delta_{i,j}=\boldsymbol{h}_{i}-\boldsymbol{h}_{j}$ . Left: the residual ratio $|(I-\Pi_{r})\delta_{i,j}|\,/\,|\delta_{i,j}|_{\Pi_{r}}$ is sharply concentrated near zero under $r=1200$ , confirming that the residual term $\|P(I-\Pi_{r})\delta_{i,j}\|_{2}$ is negligible and the bound in Eq. ( 4 ) is approximately tight with respect to the SVD truncation. Right: the actual router logit differences $|P\delta|$ (orange dots) fall far below both upper bounds. While the projection-based bound $\|P\Pi_{r}\|_{2}\,|\delta|_{\Pi_{r}}$ (blue line) is theoretically tighter than the naive bound $\|P\|_{2}\,|\delta|$ (orange line), $P$ is already well-aligned with the principal subspace of the data, the gap between the two bounds is less significant in pre-trained models.

Panel of Figure 14 — Figure 14: Hidden states similarity (measured by Eq. ( 17 )) v.s. depth for the prompt / question part v.s. question + rollout on two different reasoning models tested on 10 questions from HLE. Left On gpt-oss, the similarity of the rollout part falls below the question-only line and does not show collapsing, aligning with the observations in Fig. 9 . Right On a dense reasoning model from different lab, while we did see an increment of hidden state similarity across depth, we did not observe a clear separation in deeper layers.

Panel of Figure 15 — Figure 15: Similar to Fig. 2 in the main text, but for more models

Figure 16: gpt-oss-20b (w/ auxiliary loss)

Figure 17: Trinity Mini Base (Muon, no auxiliary loss)

Figure 18: DeepSeek V2 Lite (AdamW, w/ auxiliary loss)

Figure 19: Moonlight 16B A3B, same architecture as DeepSeek V2 Lite, trianed with Muon and no auxiliary loss

Figure 23: OOD inputs reduce router data alignment (

\left\lVert PQ\right\rVert_{2}

), where

Q=V_{r}V_{r}^{\top}

denotes the orthogonal projector onto the subspace spanned by the data’s top

r

right singular vectors.

Figure 24: Keeping only top-K directions (

{V}_{K}

) of the data (i.e. the post attention hidden states, denoted by

h_{t}

), we study the ratio of expert usage agreement compared with the original inputs, averaged over 50,000 tokens from nvidia/Nemotron-CC-Math-v1 at the

10

th layer of each model. The expert receiving the largest logits (brown line,

m=1

) shows high agreement over all tokens. Lower rank experts (e.g. the 6th expert,

m=6

) demonstrate a higher degree of perturbations when the tail directions are removed; more than 40% of the tokens end up using a different expert compared with the original inputs at

K=1200

(out of

2048

directions in total).

Figure 25: Similiar to Fig. 6 in the main text, but for more values of

p

Figure 26: On Ernie-4.5, duplicating sequences increases token hidden states similarity (first column), i.e. stronger shared direction, resulting in higher sequence hidden state similarity (second column). The increment in hidden state space results in increased expert usage similarity (last two columns).

Figure 27: Comparison of expert usage under duplication for gpt-oss-20b with (bottom) and without (top) chat template on two pieces of input.

Figure 28: Comparison of expert usage under duplication for ERNIE-4.5-21B-A3B with (bottom) and without (top) chat template on two pieces of input.

Figure 29: Duplication increases logit magnitude, increasing model confidence and causes random decoding to collapse.

Explore similar work

Apr 15, 2026

Geometric Routing Enables Causal Expert Control in Mixture of Experts

Sparse Mixture-of-Experts (MoE) models scale parameters while fixing active computation per token, but the specialization of individual experts remains opaque. In a companion paper we showed that routing topology is quality-neutral: five structurally different configurations converge to statistically equivalent language modeling quality. Here we show that expert identity is nonetheless causally meaningful: individual rank-1 experts are monosemantic by construction, and cosine-similarity routing in a low-dimensional metric space makes their specialization directly inspectable. We present four lines of evidence. First, projecting expert output vectors through the unembedding matrix yields a Semantic Dictionary: 15% of experts are monosemantic specialists spanning 10 categories (temporal, geographic, cardinal, discourse, emotional, financial, military, scientific). Second, routing exhibits a frequency-to-syntax gradient: early layers separate tokens by word frequency, deeper layers by syntactic class (Zipf-confound controls, all

p < 0.001

). Third, causal interventions confirm these labels: steering toward a temporal expert's centroid increases P(temporal) by +321% (median across 44 prompts); suppressing a geographic expert drops P(geographic) by -23%; rewriting an expert's output vector halves target-category probability, and effects compose additively across layers. Fourth, the interventions are not unique to cosine routing: linear routers support comparable steering, but only cosine routing provides geometric transparency -- expert specialization is readable directly from the centroid matrix. MoE expert-level specialization is a first-class interpretability primitive: architecturally monosemantic, causally validated, and controllable at inference with zero overhead.

Ivan Ternovtsii, Yurii Bilak

Department of Software Systems, Uzhhorod National University, Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes

Mixture of Experts Mixture-of-Experts Models Neural Network Interpretability Representation Geometry in Language Models

Ask PDF

Apr 8, 2026

MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale

Sparse Mixture-of-Experts (MoE) architectures are increasingly popular for frontier large language models (LLM) but they introduce training challenges due to routing complexity. Fully leveraging parameters of an MoE model requires all experts to be well-trained and to specialize in non-redundant ways. Assessing this, however, is complicated due to lack of established metrics and, importantly, many routing techniques exhibit similar performance at smaller sizes, which is often not reflective of their behavior at large scale. To address this challenge, we propose the MoE Routing Testbed, a setup that gives clearer visibility into routing dynamics at small scale while using realistic data. The testbed pairs a data mix with clearly distinguishable domains with a reference router that prescribes ideal routing based on these domains, providing a well-defined upper bound for comparison. This enables quantifiable measurement of expert specialization. To demonstrate the value of the testbed, we compare various MoE routing approaches and show that balancing scope is the crucial factor that allows specialization while maintaining high expert utilization. We confirm that this observation generalizes to models 35x larger.

Tobias Falke, Nicolas Anastassacos, Samson Tan +6

Amazon AGI

Mixture of Experts Sparse Mixture-of-Experts Expert Routing Expert Load Balancing

Ask PDF

Mar 6, 2026

MoE Lens -- An Expert Is All You Need

Mixture of Experts (MoE) models enable parameter-efficient scaling through sparse expert activations, yet optimizing their inference and memory costs remains challenging due to limited understanding of their specialization behavior. We present a systematic analysis of expert specialization in MoEs through two complementary approaches: domain-specific routing patterns and an early decoding framework that tracks expert contributions to output representations. Our analysis of the DeepSeekMoE model reveals that despite having 64 routed experts with 6 active for each layer's computation, the model predominantly relies on a few specialized experts, with the top-weighted expert's output closely approximating the full ensemble prediction. We quantitatively validate these findings through a systematic analysis of the token routing distribution, demonstrating that very few experts handle over 50% of routing decisions across different specialized domains. Hidden state similarity between single and ensemble experts for every layer is extremely high, with some layers having cosine similarity as high as 0.95 and perplexity increasing by only 5% when using a single expert across all three domains. Our results indicate that Mixture of Experts models exhibit concentrated expertise highlighting potential opportunities for inference optimization through targeted expert pruning while maintaining model performance and opening avenues towards studying localization of learned knowledge in these models.

Marmik Chaudhari, Idhant Gulati, Nishkal Hundia +2

The Pennsylvania State University, University Park, PA 16802 USA · University of Maryland, College Park, MD 20742 USA · Harvard University, Cambridge, MA 02138 USA

Mixture-of-Experts Models Mixture-of-Experts Language Models Mixture of Experts LLM Routing

Ask PDF