Mixture-of-Experts Expert Specialization

Follow this topic Ask about this topic

Latest papers

26 papers6 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

SAE Interprets MoE: Sparse AutoEncoders Reveal That MoE Experts Exhibit Human-Interpretable Fine-Grained Specialization
Nov 5, 2026Anyuan Zhuo, Shaohua Zhang, Yuan Lin +1Mixture-of-Experts Expert Specialization Mixture-of-Experts Interpretability
ByteDance Seed · Shanghai University of Finance and Economics
When Does Sparse Routing Help? The Geometry of Expert Separation in Mixture of Experts
Nov 1, 2026Huy Nguyen, Dung Le, Alessandro Rinaldo +1Sparse Mixture-of-Experts Mixture-of-Experts Models
Department of Statistics and Data Science, The University of Texas at Austin
MoEDDI: Semantic Mixtures of Experts for Long Tailed Drug Drug Interaction Prediction
Oct 31, 2026Dinh Tan Nguyen, Hien Quang Kha, Minh-Toan Dinh +5Drug-Drug Interaction Prediction Mixture-of-Experts Models
University of Technology Sydney, Australia · Taipei Medical University, Taiwan · National Central University, Taiwan +4
Learning What to Share: Expert Specialization for Time-Series Forecasting
Oct 24, 2026Bashar Khoury, Shadi Omari, Assaf SchusterMultivariate Time Series Forecasting Mixture-of-Experts Expert Specialization
Department of Computer Science, Technion – Israel Institute of Technology
GaP-MoE : Explicit Expert Specialization in Mixture-of-Experts via Orthogonal Gradient Accumulation and Projection
Oct 21, 2026Donghyun Jeong, Yoon-Yeong KimSparse Mixture-of-Experts Mixture-of-Experts Models
Department of Statistics and Data Science University of Seoul Republic of Korea · Department of Statistics University of Seoul Republic of Korea
Explaining the Roles of Experts in Mixture-of-Experts LLMs for Personalization
Oct 15, 2026Sekh Mainul Islam, Pepa Atanasova, Isabelle AugensteinMixture-of-Experts Expert Specialization Personalized Language Models
University of Copenhagen
RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing
Oct 8, 2026Ilya Lasy, Nora Yinuo Cai, Kola AyonrindeMixture-of-Experts Interpretability Mixture of Experts
Faculty of Informatics, TU Wien · Independent · UK AI Security Institute
Dynamic Expert Pruning for Multi-Agent Systems
Oct 2, 2026Jabin Koo, Soheil Abbasloo, Sungjae Lee +1Mixture-of-Experts Pruning Mixture-of-Experts Language Models
Pohang University of Science and Technology (POSTECH) · Microsoft Research Asia
Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
Sep 10, 2026Atindra Jha, Margaret Li, Jure Leskovec +2Mixture of Experts Sparse Mixture-of-Experts
Stanford University · Paul G. Allen School of Computer Science, University of Washington
Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts
Sep 1, 2026Nikolaos Xiros, Dimitrios Damianos, Maria-Eleni Zoumpoulidi +3Mixture-of-Experts Language Models Expert Routing
Institute for Language and Speech Processing, Athena Research Center, Greece
UMoE:Unlocking Every Expert in Domain-Specific Training
Jul 13, 2026Xuefeng Li, Pengfei LiuMixture-of-Experts Pruning Mixture-of-Experts Language Models
1SII · 2SJTU · 3GAIR
Does the Same Token Mean the Same State? MoE Routing as Signal for Reasoning Control
Jun 22, 2026Kang Chen, Minshen Yu, Junjie Nian +3Mixture-of-Experts Language Models LLM Routing
Fudan University · Shanghai Innovation Institute
A theoretical model for task routing in mixture-of-expert transformers
Jun 12, 2026Vinoth Nandakumar, Yongli Xiang, Yunzhi Yao +2Transformer Expressivity Mixture of Experts
University of Sydney · Zhejiang University · Google Research
Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts
May 20, 2026Gene Tangtartharakul, Katherine R. StorrsVisual Representation Learning Mixture of Experts
School of Psychology · University of Auckland · Auckland, New Zealand
DBES: A Systematic Benchmark and Metric Suite for Evaluating Expert Specialization in Large-Scale MoEs
May 18, 2026Jing Wang, Hongxuan Lu, Jazze Young +2Mixture-of-Experts Language Models Mixture-of-Experts Models
Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap
May 8, 2026Feilong LiuTransformer Interpretability Neural Representation Geometry
UniPool: Learning Expert-to-Layer Ownership from Brief Global Access
May 7, 2026Minbin Huang, Han Shi, Chuanyang Zheng +5Mixture-of-Experts Language Models Mixture of Experts
The Chinese University of Hong Kong · Huawei Technologies · The University of Hong Kong
EMO: Pretraining Mixture of Experts for Emergent Modularity
May 7, 2026Ryan Wang, Akshita Bhagia, Sewon MinLanguage Model Pretraining Mixture-of-Experts Language Models
University of California, Berkeley · Allen Institute for AI
E = T*H/(O+B): A Dimensionless Control Parameter for Mixture-of-Experts Ecology
May 7, 2026Qingjun ZhangExpert Load Balancing Mixture of Experts
School of Integrated Circuits, Wuxi Taihu University Department of Communication Engineering · Department of Communication Engineering
Affinity Is Not Enough: Recovering the Free Energy Principle in Mixture-of-Experts
May 1, 2026Man Yung WongSparse Mixture-of-Experts Mixture-of-Experts Inference
Russell · Independent Researcher
Geometric Metrics for MoE Specialization: From Fisher Information to Early Failure Detection
Apr 16, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuInformation Geometry Mixture of Experts
The University of Hong Kong · Hong Kong, China · Brain Investing Limited
Geometric Routing Enables Causal Expert Control in Mixture of Experts
Apr 15, 2026Ivan Ternovtsii, Yurii BilakMixture-of-Experts Interpretability Mixture-of-Experts Expert Specialization
Department of Software Systems, Uzhhorod National University, Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes
Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling
Apr 15, 2026Sanghyeok Chu, Pyunghwan Ahn, Gwangmo Song +3Mixture-of-Experts Expert Specialization Mixture-of-Experts Models
ECE, Seoul National University · LG AI Research · University of Michigan +1
The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise
Apr 10, 2026Xi Wang, Soufiane Hayou, Eric NalisnickMixture-of-Experts Expert Specialization Mixture of Experts
Johns Hopkins University
KALAVAI: Predicting When Independent Specialist Fusion Works -- A Quantitative Model for Post-Hoc Cooperative LLM Training
Mar 24, 2026Ramchand KumaresanLanguage Model Merging Mixture-of-Experts Expert Specialization
Murai Labs
MoE Lens -- An Expert Is All You Need
Mar 6, 2026Marmik Chaudhari, Idhant Gulati, Nishkal Hundia +2Mixture-of-Experts Expert Specialization Mixture-of-Experts Models
The Pennsylvania State University, University Park, PA 16802 USA · University of Maryland, College Park, MD 20742 USA · Harvard University, Cambridge, MA 02138 USA

Nov 5, 2026J/K move · Enter open · S save

SAE Interprets MoE: Sparse AutoEncoders Reveal That MoE Experts Exhibit Human-Interpretable Fine-Grained Specialization

Anyuan Zhuo, Shaohua Zhang, Yuan Lin, Pinyan Lu

ByteDance Seed · Shanghai University of Finance and Economics

Mixture-of-Experts Expert Specialization Mixture-of-Experts Interpretability

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models route each token to a few of many experts, yet what an individual expert is responsible for remains unsettled. Existing analyses count how often an expert is selected on domain-labeled corpora, an instrument that fixes the taxonomy in advance, stays coarse, and records selection rather than contribution. We instead study specialization in feature space by training a sparse autoencoder (SAE) on the output of an MoE layer. Since this output is a router-weighted sum of expert outputs and the SAE encoder is affine, the pre-activation of every latent decomposes exactly over the experts. Aggregated over a corpus, this gives each expert a dominant set of latents whose activation it principally drives. On layers 4, 30 and 46 of Qwen3-30B-A3B, contribution is far more concentrated than gating, most latents have only one or two dominators, and the overlap between dominant sets concentrates on a few expert pairs. Auto-interpreting these sets reveals a division of labor below coarse domains, with distinct experts for Java and front-end web code, or for geometry and for arithmetic and polynomials, while the coarse domains themselves take very unequal numbers of experts. Our SAEs and code are available at https://github.com/Bytedance-Seed/SAE-Interprets-MoE.