Mixture-of-Experts Models

Follow this topic Ask about this topic

Mixture-of-experts models divide computation among expert networks and use a router to select which experts process each input, often activating only a small subset. Common approaches include top-k sparse routing, shared or hierarchical experts, and parameter-efficient expert adapters; researchers also study load balancing to prevent some experts from being overused or ignored. These models are used in language, vision, multimodal learning, forecasting, and robotics, while systems work targets the memory, communication, and latency costs of serving and training them. A central tension is that specialization can improve capacity and efficiency, but balanced routing does not guarantee useful expertise, and routing overhead or weaker performance under changed conditions can offset the gains.

Latest papers

461 papers75 papers in the last four weeks, up 97% on the four weeks before. 0.4% of all new papers.

MoE Lens -- An Expert Is All You Need
Mar 6, 2026Marmik Chaudhari, Idhant Gulati, Nishkal Hundia +2Mixture-of-Experts Expert Specialization Mixture-of-Experts Models
The Pennsylvania State University, University Park, PA 16802 USA · University of Maryland, College Park, MD 20742 USA · Harvard University, Cambridge, MA 02138 USA
Sparse Crosscoders for diffing MoEs and Dense models
Mar 6, 2026Marmik Chaudhari, Nishkal Hundia, Idhant GulatiMixture-of-Experts Interpretability Mixture-of-Experts Models
University of California, Berkeley · University of Maryland
Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation
Mar 5, 2026Yilong Chen, Naibin Gu, Junyuan Shang +8Mixture of Experts Mixture-of-Experts Models
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Baidu Inc.
Timer-S1: A Billion-Scale Time Series Foundation Model with Serial Scaling
Mar 5, 2026Yong Liu, Xingjian Su, Shiyu Wang +7Time Series Forecasting Time-Series Foundation Models
Tsinghua University · ByteDance
ECG-MoE: Mixture-of-Expert Electrocardiogram Foundation Model
Mar 4, 2026Yuhao Xu, Xiaoda Wang, Yi Wu +3Electrocardiogram Foundation Models Electrocardiography
Department of Computer Science, Emory University, Atlanta, GA 30307 · School of Computer Science, University of Oklahoma, Norman, OK 73019 · Center of Data Science, Emory University, Atlanta, GA 30307
Feature-level Interaction Explanations in Multimodal Transformers
Mar 4, 2026Yeji Kim, Housam Khalifa Bashier Babiker, Mi-Young Kim +1Multimodal Understanding Feature Attribution
University of Alberta, Edmonton, AB, Canada · University of Waikato, Hamilton, New Zealand · University of Alberta, Camrose, AB, Canada
RANGER: Sparsely-Gated Mixture-of-Experts with Adaptive Retrieval Re-ranking for Pathology Report Generation
Mar 4, 2026Yixin Chen, Ziyu Su, Hikmat Khan +1Medical Report Generation Multimodal RAG
Department of Biomedical Engineering, The Ohio State University, Columbus, OH, USA · Department of Pathology, The Ohio State University, Columbus, OH, USA
CMoE: Contrastive Mixture of Experts for Motion Control and Terrain Adaptation of Humanoid Robots
Mar 3, 2026Shihao Ma, Hongjin Chen, Zijun Xu +6Humanoid Robot Locomotion Terrain-Aware Robot Locomotion
College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China, 200433
Practical FP4 Training for Large-Scale MoE Models on Hopper GPUs
Mar 3, 2026Wuyue Zhang, Chongdong Huang, Chunbo You +3Low-Precision Training LLM Training
Zhejiang Lab, Hangzhou, China
DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks
Mar 2, 2026Gökdeniz GülmezMixture-of-Experts Models Expert Routing
Machine Learning Research, Stuttgart, Germany
PathMoE: Interpretable Multimodal Interaction Experts for Pediatric Brain Tumor Classification
Mar 2, 2026Jian Yu, Joakim Nguyen, Jinrui Fang +10Brain Tumor Classification Multimodal Classification
University of Texas, Austin, TX, USA · Dell Children’s Medical Center, Austin, TX, USA · University of North Carolina at Chapel Hill, NC, USA +1
Multimodal Mixture-of-Experts with Retrieval Augmentation for Protein Active Site Identification
Mar 2, 2026Jiayang Wu, Jiale Zhou, Rubo Wang +5Mixture-of-Experts Models Uncertainty-Aware Multimodal Fusion
Westlake University · Chinese Academy of Sciences · Mohamed bin Zayed University of Artificial Intelligence +2
Mixture of Sequence: Theme-Aware Mixture-of-Experts for Long-Sequence Recommendation
Mar 1, 2026Xiao Lin, Zhicheng Tang, Weilin Cong +14Sequential Recommendation User Preference Modeling
UIUC United States · Meta United States
MME: Mixture of Mesh Experts with Random Walk Transformer Gating
Feb 28, 2026Amir Belder, Ayellet TalMixture-of-Experts Models Expert Routing
Technion - Israel Institute of Technology, Israel
Specializing Foundation Models via Mixture of Low-Rank Experts for Comprehensive Head CT Analysis
Feb 28, 2026Youngjin Yoo, Han Liu, Bogdan Georgescu +14Medical Imaging Foundation Models Low-Rank Adaptation
Digital Technology and Innovation, Siemens Healthineers, Princeton, NJ USA · Department of Computed Tomography, Siemens Healthineers, Forchheim, Germany · Department of Radiology, University of Maryland Medical Center, Baltimore, MD USA +4
CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging
Feb 28, 2026Jie Cao, Zhenxuan Fan, Zhuonan Wang +8Mixture-of-Experts Models Parameter-Efficient Fine-Tuning
Zhejiang University · Wechat, Tencent
CoME: Empowering Channel-of-Mobile-Experts with Informative Hybrid-Capabilities Reasoning
Feb 27, 2026Yuxuan Liu, Weikai Xu, Kun Huang +9Mobile GUI Agents Agentic Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China · MiLM Plus, Xiaomi Inc. · Nanyang Technological University +1
A Mixture-of-Experts Model for Multimodal Emotion Recognition in Conversations
Feb 26, 2026Soumya Dutta, Smruthi Balaji, Sriram GanapathyEmotion Recognition in Conversations Multimodal Emotion Recognition
LEAP Lab, Department of Electrical Engineering, IISc Bangalore, Bangalore, 560012, Karnataka, India · Microsoft, Bangalore, Karnataka, India
pMoE: Prompting Diverse Experts Together Wins More in Visual Adaptation
Feb 26, 2026Shentong Mo, Xufang Luo, Dongsheng LiVisual Prompt Learning Vision Foundation Model Adaptation
Carnegie Mellon University · Microsoft Research
NESTOR: A Nested MOE-based Neural Operator for Large-Scale PDE Pre-Training
Feb 25, 2026Dengdi Sun, Xiaoya Zhou, Xiao Wang +4Neural Operators PDE Surrogate Modeling
School of Artificial Intelligence, Anhui University, Hefei, China · School of Computer Science and Technology, Anhui University, Hefei, China
Excitation: Momentum For Experts
Feb 25, 2026Sagi ShaierSparse Mixture-of-Experts Mixture of Experts
Aleph Alpha Research, Heidelberg, Germany.
Modality-Guided Mixture of Structured Experts with Entropy-Triggered Routing for Multimodal Recommendation
Feb 24, 2026Ji Dai, Quan Fang, DeSheng CaiSparse Mixture-of-Experts Recommender Systems
Beijing University of Posts and Telecommunications, China · Beijing University of Posts and Telecommunications, Beijing, China · Tianjin University of Technology, China
Grouter: Decoupling Routing from Representation for Accelerated MoE Training
Feb 22, 2026Yuqi Xu, Rizhen Hu, Zihan Liu +2Mixture-of-Experts Models Expert Routing
School of Mathematical Sciences, Peking University, Beijing, China · Zhejiang Lab, Hangzhou, China · Yuanpei College, Peking University, Beijing, China +1
Routing-Aware Explanations for Mixture of Experts Graph Models in Malware Detection
Feb 22, 2026Hossein Shokouhinejad, Roozbeh Razavi-Far, Griffin Higgins +1Mixture-of-Experts Interpretability Mixture-of-Experts Models
1*University of New Brunswick, Faculty of Computer Science, 3 Bailey Dr., Fredericton, E3B5A3, New Brunswick, Canada.
Phase-Aware Mixture of Experts for Agentic Reinforcement Learning
Feb 19, 2026Shengtian Yang, Yu Li, Shuo He +4Reinforcement Learning RL for LLM Agents
School of Computer Science and Engineering, Southeast University, Nanjing, China · Kuaishou Technology, Beijing, China · College of Computing and Data Science, Nanyang Technological University, Singapore
Mixture-of-Experts under Finite-Rate Gating: Communication--Generalization Trade-offs
Feb 16, 2026Ali Khalesi, Mohammad Reza Deylam SalehiMixture-of-Experts Models Communication-Efficient Mixture-of-Experts
Institut Polytechnique des Sciences Avancées (IPSA) and LINCS Lab, Paris, France · IEEE member, Nice, France
DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Driving
Feb 16, 2026Chenxu Dang, Sining Ang, Yongkang Li +7VLMs for Autonomous Driving Diffusion-Based Planning
Huazhong University of Science and Technology · Xiaomi EV · Institute for AI Industry Research (AIR), Tsinghua University
OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale
Feb 5, 2026Jingze Shi, Zhangyang Peng, Yizhang Zhu +3Mixture-of-Experts Inference LLM Inference Acceleration
The Hong Kong University of Science and Technology (Guangzhou) · Beijing Academy of Artificial Intelligence
Multi-Modal Time Series Prediction via Mixture of Modulated Experts
Jan 29, 2026Lige Zhang, Ali Maatouk, Jialin Chen +3Multivariate Time Series Forecasting Time Series Forecasting
Emory University · Yale University · Amazon
L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts
Jan 29, 2026Minghao Yang, Ren Togo, Guang Li +2Expert Routing Mixture-of-Experts Models
Hokkaido University

Mar 6, 2026J/K move · Enter open · S save

MoE Lens -- An Expert Is All You Need

Marmik Chaudhari, Idhant Gulati, Nishkal Hundia, Pranav Karra+1

The Pennsylvania State University, University Park, PA 16802 USA · University of Maryland, College Park, MD 20742 USA · Harvard University, Cambridge, MA 02138 USA

Mixture-of-Experts Expert Specialization Mixture-of-Experts Models Mixture-of-Experts Language Models Mixture of Experts Mixture-of-Experts Interpretability LLM Routing

Open paper AskPDF arXiv

Mixture of Experts (MoE) models enable parameter-efficient scaling through sparse expert activations, yet optimizing their inference and memory costs remains challenging due to limited understanding of their specialization behavior. We present a systematic analysis of expert specialization in MoEs through two complementary approaches: domain-specific routing patterns and an early decoding framework that tracks expert contributions to output representations. Our analysis of the DeepSeekMoE model reveals that despite having 64 routed experts with 6 active for each layer's computation, the model predominantly relies on a few specialized experts, with the top-weighted expert's output closely approximating the full ensemble prediction. We quantitatively validate these findings through a systematic analysis of the token routing distribution, demonstrating that very few experts handle over 50% of routing decisions across different specialized domains. Hidden state similarity between single and ensemble experts for every layer is extremely high, with some layers having cosine similarity as high as 0.95 and perplexity increasing by only 5% when using a single expert across all three domains. Our results indicate that Mixture of Experts models exhibit concentrated expertise highlighting potential opportunities for inference optimization through targeted expert pruning while maintaining model performance and opening avenues towards studying localization of learned knowledge in these models.