Mixture-of-Experts Expert Specialization

Latest papers

21 papers3 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Explaining the Roles of Experts in Mixture-of-Experts LLMs for Personalization
Oct 15, 2026Sekh Mainul Islam, Pepa Atanasova, Isabelle AugensteinMixture-of-Experts Expert Specialization Personalized Language Models
University of Copenhagen
RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing
Oct 8, 2026Ilya Lasy, Nora Yinuo Cai, Kola AyonrindeMixture-of-Experts Interpretability Mixture of Experts
Faculty of Informatics, TU Wien · Independent · UK AI Security Institute
Dynamic Expert Pruning for Multi-Agent Systems
Oct 2, 2026Jabin Koo, Soheil Abbasloo, Sungjae Lee +1Mixture-of-Experts Pruning Mixture-of-Experts Language Models
Pohang University of Science and Technology (POSTECH) · Microsoft Research Asia
Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
Sep 10, 2026Atindra Jha, Margaret Li, Jure Leskovec +2Mixture of Experts Sparse Mixture-of-Experts
Stanford University · Paul G. Allen School of Computer Science, University of Washington
Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts
Sep 1, 2026Nikolaos Xiros, Dimitrios Damianos, Maria-Eleni Zoumpoulidi +3Mixture-of-Experts Language Models Expert Routing
Institute for Language and Speech Processing, Athena Research Center, Greece
UMoE:Unlocking Every Expert in Domain-Specific Training
Jul 13, 2026Xuefeng Li, Pengfei LiuMixture-of-Experts Pruning Mixture-of-Experts Language Models
1SII · 2SJTU · 3GAIR
Does the Same Token Mean the Same State? MoE Routing as Signal for Reasoning Control
Jun 22, 2026Kang Chen, Minshen Yu, Junjie Nian +3Mixture-of-Experts Language Models LLM Routing
Fudan University · Shanghai Innovation Institute
A theoretical model for task routing in mixture-of-expert transformers
Jun 12, 2026Vinoth Nandakumar, Yongli Xiang, Yunzhi Yao +2Transformer Expressivity Mixture of Experts
University of Sydney · Zhejiang University · Google Research
Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts
May 20, 2026Gene Tangtartharakul, Katherine R. StorrsVisual Representation Learning Mixture of Experts
School of Psychology · University of Auckland · Auckland, New Zealand
DBES: A Systematic Benchmark and Metric Suite for Evaluating Expert Specialization in Large-Scale MoEs
May 18, 2026Jing Wang, Hongxuan Lu, Jazze Young +2Mixture-of-Experts Language Models Mixture-of-Experts Models
Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap
May 8, 2026Feilong LiuTransformer Interpretability Neural Representation Geometry
UniPool: Learning Expert-to-Layer Ownership from Brief Global Access
May 7, 2026Minbin Huang, Han Shi, Chuanyang Zheng +5Mixture-of-Experts Language Models Mixture of Experts
The Chinese University of Hong Kong · Huawei Technologies · The University of Hong Kong
EMO: Pretraining Mixture of Experts for Emergent Modularity
May 7, 2026Ryan Wang, Akshita Bhagia, Sewon MinLanguage Model Pretraining Mixture-of-Experts Language Models
University of California, Berkeley · Allen Institute for AI
E = T*H/(O+B): A Dimensionless Control Parameter for Mixture-of-Experts Ecology
May 7, 2026Qingjun ZhangExpert Load Balancing Mixture of Experts
School of Integrated Circuits, Wuxi Taihu University Department of Communication Engineering · Department of Communication Engineering
Affinity Is Not Enough: Recovering the Free Energy Principle in Mixture-of-Experts
May 1, 2026Man Yung WongSparse Mixture-of-Experts Mixture-of-Experts Inference
Russell · Independent Researcher
Geometric Metrics for MoE Specialization: From Fisher Information to Early Failure Detection
Apr 16, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuInformation Geometry Mixture of Experts
The University of Hong Kong · Hong Kong, China · Brain Investing Limited
Geometric Routing Enables Causal Expert Control in Mixture of Experts
Apr 15, 2026Ivan Ternovtsii, Yurii BilakMixture-of-Experts Interpretability Mixture-of-Experts Expert Specialization
Department of Software Systems, Uzhhorod National University, Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes
Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling
Apr 15, 2026Sanghyeok Chu, Pyunghwan Ahn, Gwangmo Song +3Mixture-of-Experts Expert Specialization Mixture-of-Experts Models
ECE, Seoul National University · LG AI Research · University of Michigan +1
The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise
Apr 10, 2026Xi Wang, Soufiane Hayou, Eric NalisnickMixture-of-Experts Expert Specialization Mixture of Experts
Johns Hopkins University
KALAVAI: Predicting When Independent Specialist Fusion Works -- A Quantitative Model for Post-Hoc Cooperative LLM Training
Mar 24, 2026Ramchand KumaresanLanguage Model Merging Mixture-of-Experts Expert Specialization
Murai Labs
MoE Lens -- An Expert Is All You Need
Mar 6, 2026Marmik Chaudhari, Idhant Gulati, Nishkal Hundia +2Mixture-of-Experts Expert Specialization Mixture-of-Experts Models
The Pennsylvania State University, University Park, PA 16802 USA · University of Maryland, College Park, MD 20742 USA · Harvard University, Cambridge, MA 02138 USA

Oct 15, 2026J/K move · Enter open · S save

Explaining the Roles of Experts in Mixture-of-Experts LLMs for Personalization

Sekh Mainul Islam, Pepa Atanasova, Isabelle Augenstein

University of Copenhagen

Mixture-of-Experts Expert Specialization Personalized Language Models Mixture-of-Experts Language Models Mixture of Experts LLM Interpretability

Open paper AskPDF arXiv

Personalization of Large Language Models (LLMs) is increasingly important for user-centric applications, yet prior methods mainly adapt dense LLMs, whose polysemantic neurons, each encoding multiple unrelated concepts, hinder localizing the components behind personalization, and thus understanding and controlling it. We propose to personalize Mixture-of-Experts (MoE) LLMs, which sparsely route each token to a few specialized experts, reducing polysemanticity and enabling such localization. We adapt three open-weight MoE LLMs on a large-scale personalization dataset and train crosscoders on paired activations of the personalized and non-personalized models to learn interpretable personalized latents, from which we identify the experts encoding distinct personalization features. The personalized MoE LLMs perform competitively with dense LLMs while activating only a fraction of their parameters, exhibit lower polysemanticity than dense LLMs with similar active parameters, and, within the same model family, yield personalized latents that are less shared with the non-personalized model. Causal interventions show that these latents suffice to recover the personalized model's behavior and that boosting their associated experts moves the non-personalized model toward it, revealing distinct routing paths for different personalization features.