Mixture-of-Experts Interpretability

Follow this topic Ask about this topic

Latest papers

23 papers5 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

SAE Interprets MoE: Sparse AutoEncoders Reveal That MoE Experts Exhibit Human-Interpretable Fine-Grained Specialization
Nov 5, 2026Anyuan Zhuo, Shaohua Zhang, Yuan Lin +1Mixture-of-Experts Expert Specialization Mixture-of-Experts Interpretability
ByteDance Seed · Shanghai University of Finance and Economics
Routes of Reasoning: Reading and Reshaping Test-Time Compute in MoE Models
Nov 4, 2026Shengjie Wang, Mohsen Fayyaz, Nanyun PengEfficient Language Model Reasoning Adaptive Reasoning in Language Models
University of California, Los Angeles
PRISM-Net: Dynamic Routing Network with VLM-Guided Disentangled Attribute Distillation for Explainable Fine-Grained Product Retrieval
Oct 30, 2026Chung-Ming Lo, Che-Wei ChangFine-Grained Image Retrieval Visual Representation Learning
Institute of Artificial Intelligence Innovation, Industry Academia Innovation School, National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Graduate Institute of Library, Information and Archival Studies, National Chengchi University, Taipei, Taiwan
Decomposing Irregular Time Series into Interpretable Temporal Patterns with Mixture-of-Experts
Oct 30, 2026Jenna L. Ballard, Zijie Pan, Shivani Shekhar +6Irregular Time-Series Modeling Multiscale Time-Series Modeling
University of Pennsylvania · PayPal AI
CalibCause: A Quality-Gated, Interpretable and Calibrated Deep Learning Framework for Diabetic Retinopathy Grading
Oct 28, 2026Sohaib Sulman, Muhammad Atif Saeed, Akhtar Jamil +2Diabetic Retinopathy Grading Domain Generalization
Department of Data Science and Artificial Intelligence National University of Computer and Emerging Sciences (FAST-NUCES) Islamabad 44000, Pakistan · Department of Computer Science National University of Computer and Emerging Sciences (FAST-NUCES) Islamabad 44000, Pakistan · Department of Computer Engineering Istinye University Istanbul 34408, Turkey +1
TempoMoE: Identifying and Steering Fast-Slow Thinking Experts for Efficient Multimodal Reasoning
Oct 19, 2026Xiyue Zhang, Keru Zhou, Ziyi Wang +2Mixture-of-Experts Inference Efficient Multimodal Inference
Zhejiang University · Tsinghua University · Wenzhou Business College
RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing
Oct 8, 2026Ilya Lasy, Nora Yinuo Cai, Kola AyonrindeMixture-of-Experts Interpretability Mixture of Experts
Faculty of Informatics, TU Wien · Independent · UK AI Security Institute
ExpertLens: Visualizing Embedding Spaces for Post-Hoc Explainability in MoE Enhanced Retrievers
Sep 5, 2026Effrosyni Sokli, Isaac Roberts, Alexander Schulz +2Concept Activation Vectors Interpretable ML
Department of Informatics, Systems and Communication, DISCo University of Milano-Bicocca Milan, Italy · CITEC - Center for Cognitive Interaction Technology Bielefeld University Bielefeld, Germany
A Heterogeneous Mixture of Experts Framework for Interpretable Machine Learning
Aug 25, 2026Soham Chatterjee, Rwitobroto Dey, Smarajit BoseMixture-of-Experts Interpretability Mixture-of-Experts Models
Indian Statistical Institute, 203 B.T. Road, Kolkata, 700108, India · Interdisciplinary Statistical Research Unit, Indian Statistical Institute, 203 B.T. Road, Kolkata, 700108, India
Beyond the Trace: Coupling an Interpretable Reasoning-State Readout to Native MoE Routing
Aug 18, 2026Kang Chen, Sihan Zhao, Yixin Cao +1Mixture-of-Experts Interpretability LLM Interpretability
Fudan University · Shanghai Innovation Institute
Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrast
Jul 31, 2026Yifan Wang, Jinyi Mu, Mayank Jobanputra +4Reward Modeling Mixture-of-Experts Interpretability
Saarland University · University of California San Diego · Bielefeld University +2
Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing
Jul 30, 2026Huiyuan Tian, Bonan Xu, Shijian LiSparse Mixture-of-Experts Expert Routing
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University
Understanding Structured Health Data through Interaction-Aware Mixture-of-Experts
Jul 14, 2026Ji Hwan Park, Ying Ding, Tianjin GuoMulti-View Learning Clinical Outcome Prediction
Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Information, The University of Texas at Austin, Austin, Texas, USA · Kelley School of Business, Indiana University, Bloomington, Indiana, USA
Does Role Specialization Matter for Explanation Faithfulness in Mixture-of-Experts?
Jun 28, 2026Yeji Kim, Housam Babiker, Mi-Young Kim +1Multimodal Model Interpretability Mixture-of-Experts Models
University of Alberta, Edmonton, AB, Canada · University of Waikato, Hamilton, New Zealand · University of Alberta, Camrose, AB, Canada
How Modular Is a Frontier Mixture-of-Experts? A Pre-registered Causal Test in Which Apparent Expert Modularity Mostly Dissolves
Jun 23, 2026Tony Salomone, Deep Gandhi, Ali AsariaSparse Mixture-of-Experts Mixture-of-Experts Models
Transformer Lab
From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Models
Jun 9, 2026Leonard Engmann, Christian Medeiros Adriano, Holger GieseMixture-of-Experts Pruning LLM Auditing
Hasso Plattner Institute, University of Potsdam, Germany.
Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs
Apr 20, 2026Charles Ye, Bo Yuan, Lee SharkeyMixture-of-Experts Interpretability LLM Interpretability
Independent · Georgia Institute of Technology · Goodfire
Geometric Routing Enables Causal Expert Control in Mixture of Experts
Apr 15, 2026Ivan Ternovtsii, Yurii BilakMixture-of-Experts Interpretability Mixture-of-Experts Expert Specialization
Department of Software Systems, Uzhhorod National University, Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes
The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level
Apr 2, 2026Jeremy Herbst, Stefan Wermter, Jae Hee LeeMixture-of-Experts Interpretability Mixture-of-Experts Models
Department of Informatics, University of Hamburg, Hamburg, Germany.
Task-Conditioned Routing Signatures in Sparse Mixture-of-Experts Transformers
Mar 11, 2026Mynampati Sri Ranganadha AvinashMixture-of-Experts Interpretability Mixture-of-Experts Language Models
Independent Researcher | Asthra Labs
MoE Lens -- An Expert Is All You Need
Mar 6, 2026Marmik Chaudhari, Idhant Gulati, Nishkal Hundia +2Mixture-of-Experts Expert Specialization Mixture-of-Experts Models
The Pennsylvania State University, University Park, PA 16802 USA · University of Maryland, College Park, MD 20742 USA · Harvard University, Cambridge, MA 02138 USA
Sparse Crosscoders for diffing MoEs and Dense models
Mar 6, 2026Marmik Chaudhari, Nishkal Hundia, Idhant GulatiMixture-of-Experts Interpretability Mixture-of-Experts Models
University of California, Berkeley · University of Maryland
Routing-Aware Explanations for Mixture of Experts Graph Models in Malware Detection
Feb 22, 2026Hossein Shokouhinejad, Roozbeh Razavi-Far, Griffin Higgins +1Mixture-of-Experts Interpretability Mixture-of-Experts Models
1*University of New Brunswick, Faculty of Computer Science, 3 Bailey Dr., Fredericton, E3B5A3, New Brunswick, Canada.

Nov 5, 2026J/K move · Enter open · S save

SAE Interprets MoE: Sparse AutoEncoders Reveal That MoE Experts Exhibit Human-Interpretable Fine-Grained Specialization

Anyuan Zhuo, Shaohua Zhang, Yuan Lin, Pinyan Lu

ByteDance Seed · Shanghai University of Finance and Economics

Mixture-of-Experts Expert Specialization Mixture-of-Experts Interpretability

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models route each token to a few of many experts, yet what an individual expert is responsible for remains unsettled. Existing analyses count how often an expert is selected on domain-labeled corpora, an instrument that fixes the taxonomy in advance, stays coarse, and records selection rather than contribution. We instead study specialization in feature space by training a sparse autoencoder (SAE) on the output of an MoE layer. Since this output is a router-weighted sum of expert outputs and the SAE encoder is affine, the pre-activation of every latent decomposes exactly over the experts. Aggregated over a corpus, this gives each expert a dominant set of latents whose activation it principally drives. On layers 4, 30 and 46 of Qwen3-30B-A3B, contribution is far more concentrated than gating, most latents have only one or two dominators, and the overlap between dominant sets concentrates on a few expert pairs. Auto-interpreting these sets reveals a division of labor below coarse domains, with distinct experts for Java and front-end web code, or for geometry and for arithmetic and polynomials, while the coarse domains themselves take very unequal numbers of experts. Our SAEs and code are available at https://github.com/Bytedance-Seed/SAE-Interprets-MoE.