Mixture-of-Experts Models

Follow this topic Ask about this topic

Mixture-of-experts models divide computation among expert networks and use a router to select which experts process each input, often activating only a small subset. Common approaches include top-k sparse routing, shared or hierarchical experts, and parameter-efficient expert adapters; researchers also study load balancing to prevent some experts from being overused or ignored. These models are used in language, vision, multimodal learning, forecasting, and robotics, while systems work targets the memory, communication, and latency costs of serving and training them. A central tension is that specialization can improve capacity and efficiency, but balanced routing does not guarantee useful expertise, and routing overhead or weaker performance under changed conditions can offset the gains.

Latest papers

461 papers75 papers in the last four weeks, up 97% on the four weeks before. 0.4% of all new papers.

ExFusion: Efficient Transformer Training via Multi-Experts Fusion
Mar 30, 2026Jiacheng Ruan, Daize Dong, Xiaoye Qu +5Transformer Mixture of Experts
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, No 800, Dongchuan Road, Minhang District, Shanghai, China, 200240 · Shanghai Artificial Intelligence Laboratory, Shanghai, China, 200240 · Huazhong University of Science and Technology, Hubei, China, 430074 +3
On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models
Mar 29, 2026Chongyang Zhao, Mingsong Li, Haodong Lu +1Continual Learning for VLMs Mixture-of-Experts Models
University of New South Wales (UNSW Sydney)
MP-MoE: Matrix Profile-Guided Mixture of Experts for Precipitation Forecasting
Mar 26, 2026Huyen Ngoc Tran, Dung Trung Tran, Hong Nguyen +2Precipitation Forecasting Weather Forecasting
School of Electronics and Electrical Engineering, Hanoi University of Science and Technology · Department of Electrical and Computer Engineering, University of Southern California, United States
MoE-GRPO: Optimizing Mixture-of-Experts via Reinforcement Learning in Vision-Language Models
Mar 26, 2026Dohwan Ko, Jinyoung Park, Seoung Choi +3Expert Routing Mixture-of-Experts Models
Korea University · KAIST
B-MoE: A Body-Part-Aware Mixture-of-Experts "All Parts Matter" Approach to Micro-Action Recognition
Mar 25, 2026Nishit Poddar, Aglind Reka, Diana-Laura Borza +4Video Action Recognition Mixture-of-Experts Models
INRIA · Birla Institute of Technology & Science, Hyderabad · Université Côte d’Azur
LGEST: Dynamic Spatial-Spectral Expert Routing for Hyperspectral Image Classification
Mar 25, 2026Jiawen Wen, Suixuan Qiu, Zihang Luo +2Hyperspectral Image Classification Hybrid CNN-Transformer Architectures
Information Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, 511453, China · Faculty of Geographical Science, Beijing Normal University, Beijing, 100875, China · School of Electronic and Communication Engineering, Guangzhou University, Guangzhou University, Guangzhou, Guangzhou, 510006, China
Similarity-Aware Mixture-of-Experts for Data-Efficient Continual Learning
Mar 24, 2026Connor Mclaughlin, Nigel Lee, Lili SuContinual Learning Data-Efficient Learning
Northeastern University, Boston, Massachusetts, USA · The Charles Stark Draper Laboratory, Inc., Cambridge, Massachusetts, USA
Mixture of Mini Experts: Overcoming the Linear Layer Bottleneck in Multiple Instance Learning
Mar 23, 2026Daniel Shao, Joel Runevic, Richard J. Chen +4Multiple Instance Learning Histopathology Image Classification
Massachusetts Institute of Technology, Cambridge MA · Harvard University, Cambridge MA · Emory University, Atlanta GA +1
Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
Mar 23, 2026Weilin Wan, Jingtao Han, Debing Zhang +2Mixture-of-Experts Models AI Scaling Laws
College of Computer Science and Artificial Intelligence, Fudan University · Hi Lab, Xiaohongshu Inc. · Project Numina +1
Mixture of Experts with Soft Nearest Neighbor Loss: Resolving Expert Collapse via Representation Disentanglement
Mar 20, 2026Abien Fred Agarap, Arnulfo P. AzcarragaMixture of Experts Mixture-of-Experts Models
De La Salle University, Manila, Philippines
Geometric Mixture-of-Experts with Curvature-Guided Adaptive Routing for Graph Representation Learning
Mar 20, 2026Haifang Cao, Yu Wang, Timing Li +2Ollivier-Ricci Curvature Graph Representation Learning
College of Intelligence and Computing, Tianjin University, Tianjin, China
SpecMoE: Spectral Mixture-of-Experts Foundation Model for Cross-Species EEG Decoding
Mar 17, 2026Davy Darankoum, Chloé Habermacher, Julien Volle +1EEG Decoding EEG Foundation Models
Univ. Grenoble Alpes, CNRS, Grenoble INP, LJK 38000 Grenoble, France · SynapCell SAS, ZAC ISIPARC, 38330 Saint-Ismier, France
Bridging Local and Global Knowledge: Cascaded Mixture-of-Experts Learning for Near-Shortest Path Routing
Mar 16, 2026Yung-Fu Chen, Anish AroraMixture-of-Experts Models Mixture-of-Experts Inference
The Ohio State University, Columbus, OH, USA
A Closer Look into LLMs for Table Understanding
Mar 16, 2026Jia Wang, Chuanyu Qin, Mingyu Zheng +3LLM Interpretability Transformer Interpretability
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
Mastering the Minority: An Uncertainty-guided Multi-Expert Framework for Challenging-tailed Sequence Learning
Mar 16, 2026Ye Wang, Zixuan Wu, Lifeng Shen +4Hierarchical Classification Long-Tail Learning
Key Laboratory of Cyberspace Big Data Intelligent Security, Ministry of Education; School of Artificial Intelligence, Chongqing University of Post and Telecommunications, Chongqing 400065, China. · School of Computer Science and Technology, East China Normal University, Shanghai, 200062, China. · Chongqing Key Laboratory of Brain-Inspired Cognitive Computing and Educational Rehabilitation for Children with Special Needs, Chongqing Normal University, Chongqing, 401331, China.
MoE-ACT: Scaling Multi-Task Bimanual Manipulation with Sparse Task-Conditioned Mixture-of-Experts Transformers
Mar 16, 2026Kangjun Guo, Haichao Liu, Yanji Sun +3Bimanual Robotic Manipulation Visuomotor Policy Learning
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China
Vision-Language Model Based Multi-Expert Fusion for CT Image Classification
Mar 16, 2026Jianfa Bai, Kejin Lu, Runtian Yuan +5Medical Image Classification Mixture-of-Experts Models
College of Computer Science and Artificial Intelligence, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University · University of Oxford · The Hong Kong University of Science and Technology
PA-Net: Precipitation-Adaptive Mixture-of-Experts for Long-Tail Rainfall Nowcasting
Mar 14, 2026Xinyu Xiao, Sen Lei, Eryun Liu +5Precipitation Nowcasting Mixture-of-Experts Models
Fudan University · the School of Information Science and Technology, Southwest Jiaotong University · Zhejiang University +2
Sparse-Dense Mixture of Experts Adapter for Multi-Modal Tracking
Mar 14, 2026Yabin Zhu, Jianqi Li, Chenglong Li +3Multimodal Visual Object Tracking Parameter-Efficient Fine-Tuning
School of Public Security and Emergency Management, Anhui University of Science and Technology, Hefei 231131, China. · Information Materials and Intelligent Sensing Laboratory of Anhui Province, Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, School of Artificial Intelligence, Anhui University, Hefei 230601, China. · School of Artificial Intelligence, Anhui University of Science and Technology, Hefei 231131, China. +1
Expert Pyramid Tuning: Efficient Parameter Fine-Tuning for Expertise-Driven Task Allocation
Mar 13, 2026Jia-Chen Zhang, Zhen-Wei Yan, Yu-Jie Xiong +1Parameter-Efficient Fine-Tuning Low-Rank Adaptation
School of Electronic and Electrical Engineering, Shanghai University of Engineering Science 333 Longteng Road, Shanghai, China
CrossEarth-SAR: A SAR-Centric and Billion-Scale Geospatial Foundation Model for Domain Generalizable Semantic Segmentation
Mar 12, 2026Ziqi Ye, Ziyang Gong, Ning Liao +10Domain Generalization for Semantic Segmentation Remote Sensing Image Segmentation
Fudan University · Shanghai Innovation Institute · Shanghai Jiao Tong University +7
Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancing
Mar 12, 2026Hanchi Sun, Yixin Liu, Yonghui Wu +1Expert Routing Mixture-of-Experts Models
Computer Science and Engineering, Lehigh University, Bethlehem, PA, USA · MD-HOBI-BIOMED INFORMATICS, University of Florida, Gainesville, FL, USA
Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design
Mar 11, 2026Junzhuo Li, Peijie Jiang, Changxin Tian +3Mixture-of-Experts Models AI Scaling Laws
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · Ant Group
Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformers
Mar 10, 2026Albus Yizhuo Li, Matthew WickerUncertainty Quantification Mixture-of-Experts Models
Department of Computing, Imperial College London.
CARE-Edit: Condition-Aware Routing of Experts for Contextual Image Editing
Mar 9, 2026Yucheng Wang, Zedong Wang, Yuetong Wu +2Diffusion-Based Image Editing Conditional Diffusion Models
The Hong Kong University of Science and Technology
FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach
Mar 9, 2026Ning Liao, Xiaoxing Wang, Xiaohan Qin +1Sparse Mixture-of-Experts Mixture-of-Experts Models
Shanghai Jiao Tong University
Mixture of experts architectures for machine learning interatomic potentials
Mar 9, 2026Yuzhi Liu, Duo Zhang, Anyang Peng +3Machine Learning Interatomic Potentials Mixture-of-Experts Models
AI for Science Institute, Beijing 100080, P. R. China. · DP Technology, Beijing 100080, P. R. China. · Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, P. R. China. +3
Scalable Training of Mixture-of-Experts Models with Megatron Core
Mar 8, 2026Zijie Yan, Hongxiao Bai, Xin Yao +42Low-Precision Training Mixture-of-Experts Models
NVIDIA
Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition
Mar 7, 2026Zhexian Huang, Bo Zhao, Hui Ma +5Multimodal Emotion Recognition Multimodal Learning
Great Bay University · The Chinese University of Hong Kong, Shenzhen · City University of Hong Kong (Dongguan) +1
Swimba: Switch Mamba Model Scales State Space Models
Mar 6, 2026Zhixu Du, Krishna Teja Chitty-Venkata, Murali Emani +3State Space Models Selective SSMs
Duke University · Red Hat, Inc. · Argonne National Laboratory

Mar 30, 2026J/K move · Enter open · S save

ExFusion: Efficient Transformer Training via Multi-Experts Fusion

Jiacheng Ruan, Daize Dong, Xiaoye Qu, Tong Zhu+4

School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, No 800, Dongchuan Road, Minhang District, Shanghai, China, 200240 · Shanghai Artificial Intelligence Laboratory, Shanghai, China, 200240 · Huazhong University of Science and Technology, Hubei, China, 430074+3

Transformer Mixture of Experts Mixture-of-Experts Models

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models substantially improve performance by increasing the capacity of dense architectures. However, directly training MoE models requires considerable computational resources and introduces extra overhead in parameter storage and deployment. Therefore, it is critical to develop an approach that leverages the multi-expert capability of MoE to enhance performance while incurring minimal additional cost. To this end, we propose a novel pre-training approach, termed ExFusion, which improves the efficiency of Transformer training through multi-expert fusion. Specifically, during the initialization phase, ExFusion upcycles the feed-forward network (FFN) of the Transformer into a multi-expert configuration, where each expert is assigned a weight for later parameter fusion. During training, these weights allow multiple experts to be fused into a single unified expert equivalent to the original FFN, which is subsequently used for forward computation. As a result, ExFusion introduces multi-expert characteristics into the training process while incurring only marginal computational cost compared to standard dense training. After training, the learned weights are used to integrate multi-experts into a single unified expert, thereby eliminating additional overhead in storage and deployment. Extensive experiments on a variety of computer vision and natural language processing tasks demonstrate the effectiveness of the proposed method.