Organizations: The Pennsylvania State University, University Park, PA 16802 USA · University of Maryland, College Park, MD 20742 USA · Harvard University, Cambridge, MA 02138 USA
Mixture of Experts (MoE) models enable parameter-efficient scaling through sparse expert activations, yet optimizing their inference and memory costs remains challenging due to limited understanding of their specialization behavior. We present a systematic analysis of expert specialization in MoEs through two complementary approaches: domain-specific routing patterns and an early decoding framework that tracks expert contributions to output representations. Our analysis of the DeepSeekMoE model reveals that despite having 64 routed experts with 6 active for each layer's computation, the model predominantly relies on a few specialized experts, with the top-weighted expert's output closely approximating the full ensemble prediction. We quantitatively validate these findings through a systematic analysis of the token routing distribution, demonstrating that very few experts handle over 50% of routing decisions across different specialized domains. Hidden state similarity between single and ensemble experts for every layer is extremely high, with some layers having cosine similarity as high as 0.95 and perplexity increasing by only 5% when using a single expert across all three domains. Our results indicate that Mixture of Experts models exhibit concentrated expertise highlighting potential opportunities for inference optimization through targeted expert pruning while maintaining model performance and opening avenues towards studying localization of learned knowledge in these models.
Figures & tables
Figure 1: Expert Specialization in DeepSeekMoE. We visualize the distribution of tokens that are routed to an expert for our English, French-QA, and GSM8K datasets. The y-axis shows the routing percentage per expert, with the red dashed line indicating a uniform routing baseline ( ≈ 9.4%). See Appendix A.1 for extended plots for other models and layers.
Figure 2: An example of early decoding using LogitLens for DeepSeekMoE on an example input: “When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction”. Each cell shows the top-1 token prediction after the final token “these” across layers (rows) for layer output, routed experts + residual stream for various top- k values. Color intensity indicates prediction confidence. The expert index is denoted by the lower-left subscript number and the top-right superscript indicates expert weight. See Appendix A.2 for other domains.
Figure 3: (Left) Normalized log perplexity across different values of top-k experts for various domains for next-token prediction task. (Right) Cosine similarity between the hidden states of Htℓ1 and Htℓ6 across all 27 layers shows consistently high alignment.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Expert Specialization (Part 1) of DeepSeekMoE for various layers. We visualize how frequently tokens from different domains are routed to the 64 experts using top- k=6 routing. The y-axis shows routing percentage per expert, with the red dashed line indicating uniform routing baseline ( ≈ 9.4%).
Figure 5: Expert Specialization (Part 2) of DeepSeekMoE for various layers. We visualize how frequently tokens from different domains are routed to the 64 experts using top- k=6 routing. The y-axis shows routing percentage per expert, with the red dashed line indicating uniform routing baseline ( ≈ 9.4%).
Figure 6: Expert Specialization (Part 1) of Qwen 1.5 MoE for various layers. We visualize how frequently tokens from different domains are routed to the 60 experts using top- k=4 routing. The y-axis shows routing percentage per expert, with the red dashed line indicating uniform routing baseline ( ≈ 6.67%).
Figure 7: Expert Specialization (Part 2) of Qwen 1.5 MoE for various layers. We visualize how frequently tokens from different domains are routed to the 60 experts using top- k=4 routing. The y-axis shows routing percentage per expert, with the red dashed line indicating uniform routing baseline ( ≈ 6.67%).
Figure 8: Expert Specialization (Part 2) of OLMoE for various layers. We visualize how frequently tokens from different domains are routed to the 64 experts using top- k=8 routing. The y-axis shows routing percentage per expert, with the red dashed line indicating uniform routing baseline ( ≈ 12.5%).
Figure 9: Expert Specialization (Part 2) of OLMoE for various layers. We visualize how frequently tokens from different domains are routed to the 66 experts using top- k=8 routing. The y-axis shows routing percentage per expert, with the red dashed line indicating uniform routing baseline ( ≈ 12.5%).
Figure 10: LogitLens visualization for DeepSeekMoE on the french input sequence: “Dans le silence feutre de la nuit, les etoiles semblent murmurer d’anciens secrets a ceux qui prennent le”. Each cell shows the top-1 token prediction after “le” across layers (rows) for layer output, routed experts with residual stream for various top- k values. Color intensity indicates prediction confidence. The lower-left subscript indicates expert indices and the top-right superscript indicates expert weight.
Figure 11: LogitLens visualization for DeepSeekMoE on the english input sequence: “One might expect language modeling performance to depend on model architecture, the size of neural models, the computing power used to train them, and the data available for this”. Each cell shows the top-1 token prediction after “this” across layers (rows) for layer output, routed experts with residual stream for various top- k values. Color intensity indicates prediction confidence. The lower-left subscript indicates expert indices and the top-right superscript indicates expert weight.
Mixture of Experts (MoEs) are now ubiquitous in large language models, yet the mechanisms behind their "expert specialization" remain poorly understood. We show that, since MoE routers are linear maps, hidden state similarity is both necessary and sufficient to explain expert usage similarity, and specialization is therefore an emergent property of the representation space, not of the routing architecture itself. We confirm this at both token and sequence level across five pre-trained models. We additionally prove that load-balancing loss suppresses shared hidden state directions to maintain routing diversity, which might provide a theoretical explanation for specialization collapse under less diverse data, e.g. small batch. Despite this clean mechanistic account, we find that specialization patterns in pre-trained MoEs resist human interpretation: expert overlap between different models answering the same question is no higher than between entirely different questions (∼60%); prompt-level routing does not predict rollout-level routing; and deeper layers exhibit near-identical expert activation across semantically unrelated inputs, especially in reasoning models. We conclude that, while the efficiency perspective of MoEs is well understood, understanding expert specialization is at least as hard as understanding LLM hidden state geometry, a long-standing open problem in the literature.
Expert specialization in Mixture-of-Experts (MoE) models remains poorly understood, with traditional evaluations conflating architectural load-balancing with functional specialization. We introduce DBES, a comprehensive diagnostic framework combining a multi-domain benchmark with five theoretically grounded metrics: Routing Specialization, Normalized Effective Rank, Domain Isolation, Routing Stiffness Score, and N-gram Expertise measures. Critical findings demonstrate distinct specialization paradigms across models: Qwen-series exhibit modular specialization with high domain isolation, while DeepSeek and GLM employ distributed collaboration. However, we emphasize that specialization is a diagnostic dimension, necessary but not sufficient for downstream performance. Most crucially, interventional evidence validates the actionability of these metrics: by using DBES to identify high-specialization expert paths during domain-specific post-training, we achieved 66% to 94.48% improvement in specialized domains with only 15% of original training resources, demonstrating that these diagnostic tools can be converted into concrete optimization operators. This work provides the first systematic methodology for evaluating expert specialization independently of accuracy metrics, offering crucial insights for the design and post-training optimization of next-generation MoE systems.
Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs) while maintaining sparse activations and reduced per-token compute. However, in memory-constrained inference settings, expert weights must be offloaded to CPU, creating a performance bottleneck from CPU-GPU transfers during decoding. We propose an expert prefetching scheme that leverages currently computed internal model representations to speculate future experts, enabling memory transfers to overlap with computation. Across multiple MoE architectures, we demonstrate that future experts can be reliably predicted by these internal representations. We also demonstrate that executing speculated experts generally maintains downstream task accuracy, thus preserving more effective compute-memory overlap by eliminating the need to re-fetch true router-selected experts. Integrated into an optimized inference engine, our approach achieves up to 14% reduction in time per output token (TPOT) over on-demand loading of experts from CPU memory. For MoEs where speculative execution alone yields suboptimal accuracy, we further examine lightweight estimators that improve expert prediction hit rates, thereby reducing performance degradation. Our code is released in open-source at https://github.com/axonn-ai/yalis/tree/offload_prefetch.