Mixture-of-Experts Interpretability

Latest papers

18 papers1 paper in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

TempoMoE: Identifying and Steering Fast-Slow Thinking Experts for Efficient Multimodal Reasoning
Oct 19, 2026Xiyue Zhang, Keru Zhou, Ziyi Wang +2Mixture-of-Experts Inference Efficient Multimodal Inference
Zhejiang University · Tsinghua University · Wenzhou Business College
RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing
Oct 8, 2026Ilya Lasy, Nora Yinuo Cai, Kola AyonrindeMixture-of-Experts Interpretability Mixture of Experts
Faculty of Informatics, TU Wien · Independent · UK AI Security Institute
ExpertLens: Visualizing Embedding Spaces for Post-Hoc Explainability in MoE Enhanced Retrievers
Sep 5, 2026Effrosyni Sokli, Isaac Roberts, Alexander Schulz +2Interpretable ML Neural Network Interpretability
Department of Informatics, Systems and Communication, DISCo University of Milano-Bicocca Milan, Italy · CITEC - Center for Cognitive Interaction Technology Bielefeld University Bielefeld, Germany
A Heterogeneous Mixture of Experts Framework for Interpretable Machine Learning
Aug 25, 2026Soham Chatterjee, Rwitobroto Dey, Smarajit BoseMixture-of-Experts Interpretability Mixture-of-Experts Models
Indian Statistical Institute, 203 B.T. Road, Kolkata, 700108, India · Interdisciplinary Statistical Research Unit, Indian Statistical Institute, 203 B.T. Road, Kolkata, 700108, India
Beyond the Trace: Coupling an Interpretable Reasoning-State Readout to Native MoE Routing
Aug 18, 2026Kang Chen, Sihan Zhao, Yixin Cao +1Mixture-of-Experts Interpretability LLM Interpretability
Fudan University · Shanghai Innovation Institute
Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrast
Jul 31, 2026Yifan Wang, Jinyi Mu, Mayank Jobanputra +4Reward Modeling Mixture-of-Experts Interpretability
Saarland University · University of California San Diego · Bielefeld University +2
Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing
Jul 30, 2026Huiyuan Tian, Bonan Xu, Shijian LiSparse Mixture-of-Experts Expert Routing
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University
Understanding Structured Health Data through Interaction-Aware Mixture-of-Experts
Jul 14, 2026Ji Hwan Park, Ying Ding, Tianjin GuoMulti-View Learning Clinical Outcome Prediction
Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Information, The University of Texas at Austin, Austin, Texas, USA · Kelley School of Business, Indiana University, Bloomington, Indiana, USA
Does Role Specialization Matter for Explanation Faithfulness in Mixture-of-Experts?
Jun 28, 2026Yeji Kim, Housam Babiker, Mi-Young Kim +1Multimodal Model Interpretability Mixture-of-Experts Models
University of Alberta, Edmonton, AB, Canada · University of Waikato, Hamilton, New Zealand · University of Alberta, Camrose, AB, Canada
How Modular Is a Frontier Mixture-of-Experts? A Pre-registered Causal Test in Which Apparent Expert Modularity Mostly Dissolves
Jun 23, 2026Tony Salomone, Deep Gandhi, Ali AsariaSparse Mixture-of-Experts Mixture-of-Experts Models
Transformer Lab
From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Models
Jun 9, 2026Leonard Engmann, Christian Medeiros Adriano, Holger GieseMixture-of-Experts Pruning LLM Auditing
Hasso Plattner Institute, University of Potsdam, Germany.
Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs
Apr 20, 2026Charles Ye, Bo Yuan, Lee SharkeyMixture-of-Experts Interpretability LLM Interpretability
Independent · Georgia Institute of Technology · Goodfire
Geometric Routing Enables Causal Expert Control in Mixture of Experts
Apr 15, 2026Ivan Ternovtsii, Yurii BilakMixture-of-Experts Interpretability Mixture-of-Experts Expert Specialization
Department of Software Systems, Uzhhorod National University, Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes
The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level
Apr 2, 2026Jeremy Herbst, Stefan Wermter, Jae Hee LeeMixture-of-Experts Interpretability Mixture-of-Experts Models
Department of Informatics, University of Hamburg, Hamburg, Germany.
Task-Conditioned Routing Signatures in Sparse Mixture-of-Experts Transformers
Mar 11, 2026Mynampati Sri Ranganadha AvinashMixture-of-Experts Interpretability Mixture-of-Experts Language Models
Independent Researcher | Asthra Labs
MoE Lens -- An Expert Is All You Need
Mar 6, 2026Marmik Chaudhari, Idhant Gulati, Nishkal Hundia +2Mixture-of-Experts Expert Specialization Mixture-of-Experts Models
The Pennsylvania State University, University Park, PA 16802 USA · University of Maryland, College Park, MD 20742 USA · Harvard University, Cambridge, MA 02138 USA
Sparse Crosscoders for diffing MoEs and Dense models
Mar 6, 2026Marmik Chaudhari, Nishkal Hundia, Idhant GulatiMixture-of-Experts Interpretability Mixture-of-Experts Models
University of California, Berkeley · University of Maryland
Routing-Aware Explanations for Mixture of Experts Graph Models in Malware Detection
Feb 22, 2026Hossein Shokouhinejad, Roozbeh Razavi-Far, Griffin Higgins +1Mixture-of-Experts Interpretability Mixture-of-Experts Models
1*University of New Brunswick, Faculty of Computer Science, 3 Bailey Dr., Fredericton, E3B5A3, New Brunswick, Canada.

Oct 19, 2026J/K move · Enter open · S save

TempoMoE: Identifying and Steering Fast-Slow Thinking Experts for Efficient Multimodal Reasoning

Xiyue Zhang, Keru Zhou, Ziyi Wang, Bing Hu+1

Zhejiang University · Tsinghua University · Wenzhou Business College

Mixture-of-Experts Inference Efficient Multimodal Inference VLM Reasoning Multimodal CoT Reasoning Mixture-of-Experts Interpretability

Open paper AskPDF arXiv

Multimodal Mixture-of-Experts (MoE) reasoning models rely on long chain-of-thought (CoT) for complex tasks, yet often overthink easy inputs, increasing token cost and latency, and sometimes degrading accuracy. Existing methods optimize reasoning length during training or impose external control at inference, but leave the underlying reasoning mechanism a black box. We find that fast and slow thinking exhibit distinct expert-routing patterns: holding the input fixed and varying only the prompted reasoning tempo produces stable routing shifts. Suppressing slow-thinking experts shortens the reasoning chain, whereas suppressing fast-thinking experts lengthens it, providing causal evidence that these experts influence reasoning depth. Based on this finding, we propose TempoMoE, a training-free, two-stage framework. It identifies fast- and slow-thinking experts via tempo-contrastive routing analysis and redistributes routing weights within the original top-k set according to sample difficulty. It amplifies fast-thinking experts and suppresses slow-thinking experts on easy inputs and reverses this adjustment on hard ones, enabling difficulty-adaptive control over reasoning depth. We evaluate TempoMoE on Kimi-VL, Qwen3-VL, and ERNIE-4.5-VL across six benchmarks. Compared with Vanilla, TempoMoE consistently improves accuracy while reducing output tokens, with accuracy gains of up to 2.6 points and token reductions of up to 35.4%.