Mixture-of-Experts Models

Follow this topic Ask about this topic

Mixture-of-experts models divide computation among expert networks and use a router to select which experts process each input, often activating only a small subset. Common approaches include top-k sparse routing, shared or hierarchical experts, and parameter-efficient expert adapters; researchers also study load balancing to prevent some experts from being overused or ignored. These models are used in language, vision, multimodal learning, forecasting, and robotics, while systems work targets the memory, communication, and latency costs of serving and training them. A central tension is that specialization can improve capacity and efficiency, but balanced routing does not guarantee useful expertise, and routing overhead or weaker performance under changed conditions can offset the gains.

Latest papers

462 papers75 papers in the last four weeks, up 97% on the four weeks before. 0.4% of all new papers.

FedCoE: Bridging Generalization and Personalization via Federated Coordinated Dual-level MoEs
May 20, 2026Penglin Dai, Fulian Li, Xincao Xu +3Federated Learning Aggregation Non-IID Federated Learning
School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu 611756, China · Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China, Shenzhen 518110, China · School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China
Task-Routed Mixture-of-Experts with Cognitive Appraisal for Implicit Sentiment Analysis
May 20, 2026Yaping Chai, Haoran Xie, Joe S. QinSentiment Analysis Multi-Task Learning
Division of Artificial Intelligence, Lingnan University, Hong Kong
HDMoE: A Hierarchical Decoupling-Fusion Mixture-of-Experts Framework for Multimodal Cancer Survival Prediction
May 20, 2026Huayi Wang, Haochao Ying, Yuyang Xu +5Cancer Survival Prediction Survival Analysis
Zhejiang University Hangzhou, China · Xinjiang University Urumqi, China · Hangzhou City University Hangzhou, China +1
Dynamic TMoE: A Drift-Aware Dynamic Mixture of Experts Framework for Non-Stationary Time Series Forecasting
May 20, 2026Jiawen Zhu, Shuhan Liu, Di Weng +1Concept Drift Mixture of Experts
School of Software Technology, Zhejiang University, Ningbo, China · State Key Lab of CAD&CG, Zhejiang University, Hangzhou, China
Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts
May 20, 2026Gene Tangtartharakul, Katherine R. StorrsVisual Representation Learning Mixture of Experts
School of Psychology · University of Auckland · Auckland, New Zealand
GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems
May 19, 2026Sourish Wawdhane, Avinash Kumar, Poulami DasExpert Load Balancing Mixture-of-Experts Inference
The University of Texas at Austin
DBES: A Systematic Benchmark and Metric Suite for Evaluating Expert Specialization in Large-Scale MoEs
May 18, 2026Jing Wang, Hongxuan Lu, Jazze Young +2Mixture-of-Experts Language Models Mixture-of-Experts Models
Patch-MoE Mamba: A Patch-Ordered Mixture-of-Experts State Space Architecture for Medical Image Segmentation
May 18, 2026Diego Adame, Fabian Vazquez, Jose A. Nunez +7Mamba Image Segmentation
University of Texas Rio Grande Valley, Edinburg, TX 78539, USA
Stable Routing for Mixture-of-Experts in Class-Incremental Learning
May 17, 2026Zirui Guo, Quan Cheng, Da-Wei Zhou +1Class-Incremental Learning Expert Routing
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China
CHoE: Cross-Domain Heterogeneous Graph Prompt Learning via Structure-Conditioned Experts
May 15, 2026Peiyuan Li, Yongqi Huang, Jitao Zhao +3Heterogeneous GNNs Few-Shot Learning
School of Computer Science and Technology, Tianjin University, Tianjin, China · Department of Health Technology and Informatics, and Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Kowloon, Hong Kong
When Does Sparse MoE Help in Vision? The Role of Backbone Compute Leverage in Sparse Routing
May 15, 2026Libo Sun, Po-wei Harn, Peixiong He +1Sparse Mixture-of-Experts Expert Routing
Department of Computer Science and Software Engineering Auburn University, Auburn, AL 36849, USA · Department of Information Management National Central University, Taoyuan 320317, Taiwan · Department of Computer Science and Software Engineering Auburn University
$φ$ -Balancing for Mixture-of-Experts Training
May 14, 2026Lizhang Chen, Jonathan Li, Qi Wang +5Expert Load Balancing Mirror Descent
University of Texas at Austin · 2Northwestern University.
RQ-MoE: Residual Quantization via Mixture of Experts for Efficient Input-Dependent Vector Compression
May 14, 2026Zhengjia Zhong, Shuyan Ke, Zaizhou Lin +3Residual VQ Vector Quantization
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, Xiamen, China.
MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unification
May 14, 2026Weisen Jiang, Shuhao Chen, Sinno Jialin PanPrivacy-Preserving ML Mixture-of-Experts Models
Department of Computer Science and Engineering, The Chinese University of Hong Kong · Department of Computer Science and Engineering, Hong Kong University of Science and Technology · Department of Computer Science and Engineering, Southern University of Science and Technology
How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization
May 13, 2026Leena Chennuru Vankadara, Moritz Haas, Luke Hayward +2Mixture-of-Experts Models Dynamical Mean-Field Theory
1Gatsby Computational Neuroscience Unit, University College London · 2Amazon · University of Tübingen, Tübingen AI Center
Dense vs Sparse Pretraining at Tiny Scale: Active-Parameter vs Total-Parameter Matching
May 13, 2026Abdalrahman WaelLanguage Model Pretraining Mixture-of-Experts Language Models
Independent Researcher
EMO: Frustratingly Easy Progressive Training of Extendable MoE
May 13, 2026Linghao Jin, Chufan Shi, Huijuan Wang +4Sparse Mixture-of-Experts Efficient Language Model Training
USC-ISI · MBZUAI-IFM
Sparse Mixture-of-Experts Routing in Visual Diffusion Transformers:Diagnosis, Boundary Calibration and Evolutionary Roadmap from Routing Collapse to Selective Deadlock
May 12, 2026Haiying ShaVideo Diffusion Models Diffusion Transformer
Independent Researcher
Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts
May 12, 2026Sagi Ahrac, Noya Hochwald, Mor GevaExpert Load Balancing Parameter-Free Mixture-of-Experts Routing
Blavatnik School of Computer Science and AI, Tel Aviv University
Fast Training of Mixture-of-Experts for Time Series Forecasting via Expert Loss Integration
May 11, 2026Btissame El Mahtout, Florian ZielOnline Time Series Forecasting Time Series Forecasting
Chair of Data Science in Energy and Environment, University of Duisburg-Essen, Germany · Chair of Econometrics and Statistics, Technical University of Dortmund, Germany
Mixture of Layers with Hybrid Attention
May 10, 2026Ivan Ternovtsii, Yurii BilakMixture-of-Experts Models Hybrid Attention
Department of Software Systems, Uzhhorod National University Narodna sq. 3, Uzhhorod, Ukraine, 88000
DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelism
May 10, 2026Zhichen Zeng, Chi-Chih Chang, Jiayi Wang +10Expert Parallelism Communication-Efficient Distributed Training
ByteDance Seed · University of Washington · Cornell University
TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tiling
May 10, 2026Hongyaoxing Gu, Xinzhe Chen, Lijuan Hu +1Efficient Inference Mixture-of-Experts Inference
Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences
VECTOR-Drive: Tightly Coupled Vision-Language and Trajectory Expert Routing for End-to-End Autonomous Driving
May 9, 2026Rui Zhao, Jianlin Yu, Zhenhai Gao +2VLMs for Autonomous Driving End-to-End Autonomous Driving
College of Automotive Engineering, Jilin University · 2The National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University · 3ReeFocus AI Technology
ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learning
May 9, 2026Chao Jin, Xinming Wei, Yinmin Zhong +6RL for Language Models Expert Load Balancing
School of Computer Science, Peking University · Independent Researcher
Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution
May 9, 2026Jongseok Park, Sunga Kim, Zhenyu Gu +2Activation Sparsity Mixture-of-Experts Inference
University of California, Berkeley · Work done during an internship at Advanced Micro Devices, Inc. · Advanced Micro Devices, Inc.
SDG-MoE: Signed Debate Graph Mixture-of-Experts
May 8, 2026Stepan Kulibaba, Kirill Labzin, Artem Dzhalilov +4Mixture-of-Experts Language Models Mixture of Experts
Innopolis University, Innopolis · Sirius University, Sirius · Innopolis University, Innopolis; Sirius University, Sirius +1
Hierarchical Mixture-of-Experts with Two-Stage Optimization
May 8, 2026Gleb Molodtsov, Alexander Miasnikov, Aleksandr BeznosikovExpert Load Balancing Mixture of Experts
BRAIn Lab, Moscow, Russia
When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
May 8, 2026Youngsik Yoon, Siwei Wang, Wei Chen +1Mixture-of-Experts Language Models LLM Routing
Department of Computer Science and Engineering, POSTECH, South Korea · Microsoft Research Asia, Beijing, China · Graduate School of Artificial Intelligence, POSTECH, South Korea
Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap
May 8, 2026Feilong LiuTransformer Interpretability Neural Representation Geometry

May 20, 2026J/K move · Enter open · S save

FedCoE: Bridging Generalization and Personalization via Federated Coordinated Dual-level MoEs

Penglin Dai, Fulian Li, Xincao Xu, Junhua Wang+2

School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu 611756, China · Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China, Shenzhen 518110, China · School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China

Federated Learning Aggregation Non-IID Federated Learning Mixture-of-Experts Models Federated Learning

Open paper AskPDF arXiv

Federated Learning (FL) has emerged as a promising paradigm for privacy-preserving distributed learning. However, existing FL methods face a fundamental challenge. Traditional averaging-based approaches suffer from parameter divergence under non-IID conditions, while personalized FL methods overfit to local data and fail to generalize to new clients (cold-start problem). Mixture-of-Experts naturally addresses this by routing heterogeneous data to specialized experts rather than forcing uniform aggregation. In this paper, we propose FedCoE, a Federated Coordinated dual-level mixture-of-Experts framework that effectively balances global generalization with local personalization. FedCoE maintains multiple independent global expert models on the server and employs a shared gating network to dynamically model client-expert correlations during aggregation, effectively mitigating expert drift and gating inconsistency. To address the cold-start challenge, we introduce an adaptive mechanism that enables new clients to immediately leverage the global expert pool without extensive local training. Extensive experiments demonstrate that FedCoE achieves 78.00% global accuracy and 89.32% personalized accuracy on average, outperforming the baseline by 8.82% and 29.19%, respectively. In cold-start scenarios, FedCoE delivers 77.27% accuracy without any local fine-tuning, outperforming baselines by over 12.54%.