Mixture-of-Experts Models

Follow this topic Ask about this topic

Mixture-of-experts models divide computation among expert networks and use a router to select which experts process each input, often activating only a small subset. Common approaches include top-k sparse routing, shared or hierarchical experts, and parameter-efficient expert adapters; researchers also study load balancing to prevent some experts from being overused or ignored. These models are used in language, vision, multimodal learning, forecasting, and robotics, while systems work targets the memory, communication, and latency costs of serving and training them. A central tension is that specialization can improve capacity and efficiency, but balanced routing does not guarantee useful expertise, and routing overhead or weaker performance under changed conditions can offset the gains.

Latest papers

461 papers75 papers in the last four weeks, up 97% on the four weeks before. 0.4% of all new papers.

HiMoE-VLA: Hierarchical Mixture-of-Experts for Generalist Vision-Language-Action Policies
Dec 5, 2025Zhiying Du, Bei Liu, Yaobo Liang +7Visuomotor Policy Learning Vision-Language-Action Models
1Fudan University · 2Microsoft Research Asia · 3Xi’an Jiaotong University +1
PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
Nov 6, 2025Yushu Zhao, Zheng Wang, Minjia ZhangLLM Compression Mixture-of-Experts Models
1Tsinghua University · †Work done while intern at UIUC. · University of Illinois Urbana-Champaign.
Variational Mixture of Graph Neural Experts for Alzheimer's Disease Recognition across Frequency Bands in EEG Brain Networks
Oct 13, 2025Jun-En Ding, Anna Zilverstand, Shihao Yang +2Alzheimer's Disease Electroencephalography
Department of Industrial and Systems Engineering, Rutgers University, Piscataway, NJ, USA · Department of Psychiatry and Behavioral Sciences, University of Minnesota, Minneapolis, MN, USA · Institute of Brain Science, College of Medicine, National Yang-Ming Chiao Tung University, Taipei City, Taiwan
Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspective
Oct 3, 2025Yehuda DarMixture-of-Experts Models Statistical Learning Theory
Faculty of Computer and Information Science, Ben-Gurion University
Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization
Sep 30, 2025Yaoxiang Wang, Qingguo Hu, Yucheng Ding +6Mixture-of-Experts Language Models Mixture-of-Experts Models
Xiamen University · Shanghai Jiao Tong University · University of Science and Technology of China +1
SHMoAReg: Spark Deformable Image Registration via Spatial Heterogeneous Mixture of Experts and Attention Heads
Sep 24, 2025Yuxi Zheng, Jianhui Feng, Tianran Li +2Medical Image Registration Mixture-of-Experts Models
Institute of Science and Technology for Brain-inspired Intelligence Fudan University Shanghai, China · Department of Radiology Leiden University Medical Center Leiden, The Netherlands
GateTS: Versatile and Efficient Forecasting via Attention-Inspired routed Mixture-of-Experts
Aug 24, 2025Kyrylo Yemets, Mykola Lukashchuk, Ivan IzoninTime Series Forecasting Mixture-of-Experts Models
Department of Artificial Intelligence, Lviv Polytechnic National University, Lviv, 79005, Ukraine · Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, 5612 AE, Netherlands · The Bartlett School of Sustainable Construction, University College London, London, WC1E 7HB, United Kingdom
Shape Distribution Matters: Shape-Specific Mixture-of-Experts for Amodal Segmentation Under Diverse Occlusions
Aug 3, 2025Zhixuan Li, Yujia Liu, Chen Hui +3Amodal Segmentation Occlusion Reasoning
College of Computing and Data Science, Nanyang Technological University, Singapore · School of Computer Science, Peking University, Beijing, China · National Key Laboratory for Multimedia Information Processing, Peking University, Beijing, China +3
Macro Graph of Experts for Billion-Scale Multi-Task Recommendation
Jun 12, 2025Hongyu Yao, Zijin Hong, Hao Chen +6Graph Representation Learning Multi-Task Learning
The Hong Kong Polytechnic University Hong Kong SAR, China · National University of Defense Technology Changsha, China · Beihang University Beijing, China
Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing
Apr 25, 2024Peizhuang Cong, Aomufei Yuan, Shimao Chen +4Mixture-of-Experts Language Models Mixture-of-Experts Models
Peking University
Optimal Transport Aggregation for Distributed Mixture-of-Experts
Dec 15, 2023Faïcel Chamroukhi, Nhat Thien PhamMixture-of-Experts Models Optimal Transport
IRT SystemX, 91120 Palaiseau, France · Normandie Univ, UNICAEN, CNRS, LMNO, 14000 Caen, France

Dec 5, 2025J/K move · Enter open · S save

HiMoE-VLA: Hierarchical Mixture-of-Experts for Generalist Vision-Language-Action Policies

Zhiying Du, Bei Liu, Yaobo Liang, Yichao Shen+6

1Fudan University · 2Microsoft Research Asia · 3Xi’an Jiaotong University+1

Visuomotor Policy Learning Vision-Language-Action Models Mixture-of-Experts Models Negative Transfer

Open paper AskPDF arXiv

Generalist vision--language--action (VLA) policies are typically trained on heterogeneous mixtures of robot demonstrations spanning diverse embodiments, action spaces, and observation configurations. Modeling such heterogeneity with a shared dense action module can induce negative transfer, particularly when action spaces or visual observations differ across data sources. We address this issue with HiMoE-VLA, a VLA framework built around a Hierarchical Mixture-of-Experts (HiMoE) action module. HiMoE uses Action-Space MoE layers at the input/output boundaries to specialize computation for distinct action spaces, Heterogeneity-Balancing MoE layers in neighboring layers to provide balanced capacity for residual variation in observations, scenes, and embodiments, and dense Transformer blocks in the middle to integrate shared representations. Two auxiliary objectives further guide this hierarchy: a contrastive Action-Space Regularization objective for boundary specialization and a load-balancing objective for stable expert utilization. HiMoE-VLA reaches 3.98 on CALVIN, 98.0% on LIBERO, and 75.0% and 63.7% average success on real xArm7 and ALOHA tasks; under controlled heterogeneous co-training, it turns the negative transfer observed in strong baselines into positive transfer. The code and models are publicly available at https://github.com/ZhiyingDu/HiMoE-VLA.