RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing

Ilya Lasy; Nora Yinuo Cai; Kola Ayonrinde

cs.AIOct 8, 2026

RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing

Authors: Ilya Lasy, Nora Yinuo Cai, Kola Ayonrinde

Organizations: Faculty of Informatics, TU Wien · Independent · UK AI Security Institute

Abstract

Sparse Mixture of Experts (MoE) models scale more efficiently than dense models by routing tokens to modular expert networks that are only active for processing a fraction of tokens. A leading hypothesis for the performance of MoE models is that each expert specialises in a single, coherent domain. However, interpretability efforts that assume this hypothesis have generally been unsuccessful. We propose and present evidence for an alternative account that we call the Superposed Specialisation Hypothesis (SSH): experts specialise in a disjoint union of fine-grained features rather than one broad domain. Leveraging the SSH, we introduce RouterInterp, a method for interpreting expert routing that identifies Sparse Autoencoder features most predictive of routing decisions and produces unified natural language explanations. On gpt-oss-20b, RouterInterp explains expert routing with ${\sim}65\%$ higher detection accuracy than prior token statistics based methods. This work provides a scalable method for generating more accurate explanations of expert routing and increases our understanding of a previously uninterpretable component of foundation models.

Figures & tables

Figure 1: Illustration of two alternative hypotheses for the expert specialisation in MoE models. Left: Under the Domain Specialisation Hypothesis, examples (points) from the same semantic domain (coloured region) are routed to the same expert, so each expert specialises in a single coherent domain. Right: Under the Superposed Specialisation Hypothesis, examples routed to the same expert are scattered across different domains. Here the clusters of adjacent points routed to the same expert represent micro-domains (fine-grained features of a particular domain) and each expert specialises in a disjoint union of them. We suggest that the SSH is a more accurate description of expert specialisation in practice.

Figure 2: Each expert covers a semantically diverse range of micro-domains: the latents that best predict its activation are spread across many distinct semantic clusters. For each expert we take its top $n{=}20$ predictive SAE latents ${\mathbb{F}}_{i}$ and cluster them semantically by their explanations. The DSH predicts one shared cluster; the SSH predicts a separate cluster per latent. We observe that experts’ latents are grouped into far more than one cluster: 10.8 clusters on average for OLMoE-1B-7B (layer 15) and 11.3 for gpt-oss-20b (layer 20).

Figure 3: RouterInterp explains routing decisions as a combination of interpretable features. Sparse autoencoder (SAE) latents, sparse feature representations extracted from model activations, pick out directions in activation space that correspond to interpretable features. RouterInterp identifies SAE latents that are most predictive of routing, collects inputs where these features and the expert are jointly active, and explains the expert in natural language. While no single monosemantic concept captures an expert’s behaviour, aggregating multiple features makes routing understandable.

Figure 4: Leveraging the SSH, RouterInterp generates expert explanations from examples where SAE features and experts are jointly active. It outperforms both a method based on token-statistics ( Jiang et al., 2024 ; Zoph et al., 2022 ) and Expert Impact AutoInterp ( Herbst et al., 2026 ) , which generates explanations from activating windows ranked by expert impact on the residual stream, at every model depth ( ${\sim}65\%$ / ${\sim}76\%$ mean F1 over Unigram Lookup on gpt-oss-20b / OLMoE-1B-7B; ${\sim}28\%$ over Expert Impact AutoInterp on gpt-oss-20b). Shown for layers 4, 16, 20 of gpt-oss-20b and layers 3, 11, 15 of OLMoE-1B-7B.

	Layer 4	Layer 8	Layer 12	Layer 16	Layer 20
Unigram Lookup	0.295	0.300	0.278	0.305	0.315
Bigram Lookup	0.309	0.320	0.283	0.320	0.331
Unigram AutoInterp	0.332	0.406	0.320	0.356	0.348
Bigram AutoInterp	0.429	0.453	0.385	0.407	0.411
Expert Activations AutoInterp	0.415	0.475	0.345	0.425	0.425
RouterInterp ( $s{=}64$ )	0.490	0.542	0.465	0.509	0.470

Table 1: Expert explanations improve when an LLM summarises evidence rather than using raw lookup tables, but n-grams and expert activations still cannot cover an expert’s disjoint micro-domains (both

0.417

mean F1). RouterInterp uses SAE features to enumerate those micro-domains and gains

+18\%

over the best of these LLM-based explainers. RouterInterp’s success is also robust to SAE sparsity (

0.495

for

s{=}64

vs.

0.492

for

s{=}128

). Results are reported on gpt-oss-20b.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

	OLMoE-1B-7B	gpt-oss-20b
Unigram Baseline	0.564	0.296
Bigram Baseline	0.633	0.356
Neuron Basis Probe	0.666	0.586
PCA Basis Probe	0.680	0.536
SAE Predictor (Ours)	0.740	0.730

Appendix

Table 2: SAE latents predict MoE routing better than token-level statistics and sparse probes. Neuron and PCA basis probes operate on sparse residual stream representations matched in sparsity to the SAE. N-gram baselines predict routing from token co-occurrence frequencies. Results reported over

\sim

1M tokens as macro-F1 for layer 11 of OLMoE-1B-7B and layer 12 of gpt-oss-20b.

Figure 5: Gradient-based attribution is the strongest feature-selection criterion across five layers of gpt-oss-20b and every feature set size $n$ . For each feature selection method, we select the top- $n$ features per expert and train a linear probe to predict routing to an expert using only these features’ activations. The mean F1 across layers improves only gradually beyond ${\sim}50$ features.

Figure 6: Up to $n{=}45$ features, every additional feature still improves RouterInterp explanations, but the gains shrink as descriptions grow. Explanation length grows roughly linearly with the feature set size $n$ , while the explanation score improves steeply at small $n$ and then slows: from $n{=}25$ to $n{=}45$ , descriptions grow by ${\sim}90$ tokens while the score improves by only ${\sim}0.03$ . Each point is a feature set size $n\in\{1,5,15,25,45\}$ , with score and length averaged across experts over 5 layers of gpt-oss-20b.

Figure 7: Expert routing distributions across Pile domains provide evidence against macro-domain DSH and are consistent with SSH. Adapting the Mixtral analysis of Jiang et al. (2024) on gpt-oss-20b, we plot how often an expert’s routed tokens come from a subset, relative to this subset’s fraction of the corpus (the expert routing preference ). We show routing preferences for randomly selected experts across 16 subsets (left) and their density over all expert–subset pairs (right), at an early (layer 4), a middle (layer 12), and a late (layer 20) layer. The densities are tightly concentrated at the corpus rate (dashed line at $1$ ). If expert domains coincided with Pile subsets, preferences would instead spread far from $1$ : near zero for most pairs, and at many times the corpus rate for a few dedicated specialists. Values above 4.0 are capped for better readability.

	Layer 4	Layer 8	Layer 12	Layer 16	Layer 20
SAE Predictor ( $s{=}128$ )	0.569	0.702	0.730	0.439	0.789
SAE Predictor ( $s{=}64$ )	0.552	0.725	0.685	0.297	0.773
Neuron Probe ( $s{=}128$ )	0.519	0.638	0.586	0.559	0.630
Neuron Probe ( $s{=}64$ )	0.476	0.603	0.509	0.524	0.570
PCA Probe ( $s{=}128$ )	0.544	0.623	0.536	0.494	0.366
PCA Probe ( $s{=}64$ )	0.517	0.603	0.523	0.482	0.384

Appendix

Table 3: The learned SAE basis captures more routing-relevant structure than standard bases at matched sparsity across most layers of gpt-oss-20b. All methods use sparse linear probes with matched sparsity (

s

active dimensions). At

s{=}128

, the SAE predictor achieves strictly higher macro-F1 than the best neuron or PCA probe at four of five layers. All values reported as macro-F1.

	Layer 4	Layer 8	Layer 12	Layer 16	Layer 20
Unigram Baseline	0.472	0.442	0.296	0.343	0.369
Bigram Baseline	0.535	0.511	0.356	0.406	0.411
SAE Predictor ( $m=1$ )	0.183	0.260	0.185	0.089	0.280
SAE Predictor ( $m=8$ )	0.235	0.380	0.267	0.249	0.617
SAE Predictor ( $m=16$ )	0.349	0.408	0.434	0.387	0.694
SAE Predictor ( $m=32$ )	0.490	0.640	0.632	0.414	0.749

Appendix

Table 4: Routing prediction improves significantly as more SAE features are used, consistent with polysemantic expert behaviour predicted by the SSH. Results are shown for gpt-oss-20b with SAE sparsity

s=128

. Performance improves with

m

across all reported layers, peaking at

m{=}128

in four of five layers, indicating that routing depends on multiple distinct concepts rather than a single domain.

Layer	FVU	Dead features (%)
3	0.007	4.11
7	0.024	0.51
11	0.064	0.59
15	0.218	0.51

Appendix

Table 5: SAE reconstruction quality for OLMoE-1B-7B (Top-K; 32,768 features, 16

\times

expansion,

s{=}32

). FVU increases with depth while dead feature rates remain low.

	FVU		Dead features (%)
Layer	$s{=}64$	$s{=}128$	$s{=}64$	$s{=}128$
4	0.050	0.037	0.04	0.90
8	0.101	0.077	0.02	0.29
12	0.172	0.137	0.00	0.01
16	0.232	0.191	0.00	0.00
20	0.146	0.117	0.02	0.00

Appendix

Table 6: SAE reconstruction quality for gpt-oss-20b (BatchTopK; 131,072 features,

{\sim}

\times

expansion). FVU peaks at layer 16 for both sparsity levels;

s{=}128

consistently achieves lower FVU than

s{=}64

Metric	Human–Human	Human–LLM
Pooled agreement	0.807 $\pm$ 0.070	0.857 $\pm$ 0.050
Cohen’s $\kappa$	0.487 $\pm$ 0.165	0.602 $\pm$ 0.150

Appendix

Table 7: The LLM scorer is at least as consistent a judge as a human rater. Pooled agreement is the fraction of examples where both raters gave the same label; Cohen’s

\kappa

is a stricter metric that accounts for the fact that two raters could coincidentally agree just by both labelling most examples the same way. Values are mean

\pm

std over repeated resampling; measured on 150 examples across 25 experts from layer 12 of gpt-oss-20b.

Layer	Macro-F1	Spearman $\rho$	MSE
4	0.569	0.258	0.0278
8	0.702	0.809	0.0294
12	0.730	0.741	0.0295
16	0.439	0.413	0.0285
20	0.789	0.817	0.0296

Appendix

Table 8: The predicted probabilities from SAE-based probes align well with the actual MoE router probabilities, even though the probes are trained only with binary cross-entropy on top-

k

membership. Spearman

\rho

is the rank correlation between the predicted and true probability assigned to each expert, computed per token and averaged; high values mean the probe recovers which experts the router assigns more probability to. MSE is the mean squared error between predicted and true per-expert probabilities (

\approx 0.03

; RMSE

\approx 0.17

). Spearman correlates with macro-F1 across layers (both strong at 8, 12, 20; weaker at 4 and 16). gpt-oss-20b; SAE predictor with

s{=}128

Level	Explanation
Expert 2	This expert activates across following distinct contexts: (1) Physical trauma and medical conditions—high activation on “bruis” and variants (“bruised”, “bruises”), plus fragments like “blister”, “abras”, “haemat”, “infarct”, and “stunned”. (2) Technical and structured writing—code, mathematical expressions, scientific notation, legal citations, and medical terminology from programming, mathematics, science, law, and academia, including punctuation and formatting elements. (3) Brand and term suffixes—final segments of brand names, company names, or technical terms before spaces or punctuation. (4) Electronic components—activation on “diode”, “resistor”, and “rectifier” in patents, diagrams, and technical specifications.
Feature 41223	Physical trauma terms—“bruis” and variants (“bruised”, “bruises”), plus fragments of injury-related terms (“blister”, “abras”, “haemat”, “infarct”, “stunned”) in clinical or descriptive contexts.
	Activating Examples:
	[1] suffered scr apes , bruis es , bleeding
	[2] . Her face was bruis ed and there
Feature 71261	Programming and scientific notation—code snippets, structured data, and formal keywords with high activation on punctuation, symbols, and domain-specific terms.

Appendix

Table 9: RouterInterp produces an expert-level explanation from activating examples grouped by SAE feature. Feature-level descriptions below let the reader see how the expert’s stated specialisations correspond to individual SAE features. For example, Expert 2 from gpt-oss-20b layer 12 specialises in physical trauma ( Feature 41223 ), programming and scientific notation ( Feature 71261 ), structured punctuation ( Feature 17613 ), brand-name suffixes ( Feature 1738 ), and electronic components ( Feature 84536 ). We display five features selected by gradient-based attribution.

Explore similar work

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
May 8, 2026Youngsik Yoon, Siwei Wang, Wei Chen +1Mixture-of-Experts Language Models LLM Routing
Department of Computer Science and Engineering, POSTECH, South Korea · Microsoft Research Asia, Beijing, China · Graduate School of Artificial Intelligence, POSTECH, South Korea
Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing
Jul 30, 2026Huiyuan Tian, Bonan Xu, Shijian LiSparse Mixture-of-Experts Expert Routing
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University
Self-Routing: Parameter-Free Expert Routing from Hidden States
Apr 1, 2026Jama Hussein Mohamud, Drew Wagner, Mirco RavanelliParameter-Free Mixture-of-Experts Routing LLM Routing
Mila - Quebec AI Institute · Universite de Montreal · Concordia University

May 8, 2026, cs.LGJ/K move · Enter open · S save

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models

Youngsik Yoon, Siwei Wang, Wei Chen, Jungseul Ok

Department of Computer Science and Engineering, POSTECH, South Korea · Microsoft Research Asia, Beijing, China · Graduate School of Artificial Intelligence, POSTECH, South Korea

Mixture-of-Experts Language Models LLM Routing Mixture-of-Experts Models

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) language models route each token to a small subset of experts, but whether the routes selected by a trained top-

k

router are good ones is rarely evaluated directly. Holding the model fixed, we compare each standard route against sampled equal-compute alternatives for the same token and score each by the next-token probability it assigns to the realized token in a verified reasoning trajectory. The result is sharply token-conditional: the standard router is well-aligned with route utility on confident tokens but uninformative on the fragile tokens that drive hard reasoning, where lower-loss equal-compute routes consistently exist inside the frozen model but are not selected. The same pattern holds across Qwen3-30B-A3B, GPT-OSS-20B, DeepSeek-V2-Lite, and OLMoE-1B-7B, and follows structurally from how standard top-

k

training evaluates routing decisions: the language modeling loss scores only the executed route, and load balancing depends only on aggregate routing statistics. A minimal router-only update to the final-layer router, leaving every expert and every other router frozen, is sufficient to shift pass@K on AIME 2024+2025 and HMMT 2025 for both Qwen3-30B-A3B and GPT-OSS-20B, suggesting that at least part of the failure reflects router-reachable misallocation rather than expert capacity alone.

Abstract

Explore similar work

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models

Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing

Self-Routing: Parameter-Free Expert Routing from Hidden States