Expert Load Balancing

Follow this topic Ask about this topic

Sparse mixture-of-experts (MoE) models activate only a few experts per token, but uneven routing can overload some experts while others sit idle, slowing training and inference. Training methods use auxiliary balance losses, dynamic router biases or thresholds, hierarchical routing, and quantile- or control-based corrections to spread assignments while preserving specialization. Systems methods predict demand, replicate or migrate experts, and place or schedule them with awareness of GPU speed and network topology; evaluation spans large-scale pretraining, distributed serving, and language and multimodal workloads. The central tension is balancing utilization without degrading task quality or suppressing specialization, while avoiding communication and planning overhead, since token counts alone may not reflect actual compute time.

Latest papers

41 papers5 papers in the last four weeks, up 67% on the four weeks before. 0.0% of all new papers.

E = T*H/(O+B): A Dimensionless Control Parameter for Mixture-of-Experts Ecology
May 7, 2026Qingjun ZhangExpert Load Balancing Mixture of Experts
School of Integrated Circuits, Wuxi Taihu University Department of Communication Engineering · Department of Communication Engineering
Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
Apr 25, 2026Abhimanyu Bambhaniya, Geonhwa Jeong, Jason Park +6Expert Load Balancing Mixture-of-Experts Inference
1Georgia Institute of Technology, USA · 2Meta Platforms, Inc.
Mixture of Heterogeneous Grouped Experts for Language Modeling
Apr 25, 2026Zhicheng Ma, Xiang Liu, Zhaoxiang Liu +5Expert Load Balancing Mixture-of-Experts Language Models
1Data Science & Artificial Intelligence Research Institute, China Unicom · 2Unicom Data Intelligence, China Unicom
MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
Apr 19, 2026Bo Li, Chuan Wu, Shaolin ZhuEfficient Multimodal Inference Expert Load Balancing
School of Software, Tsinghua University, Beijing, China · TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China · School of New Media and Communication, Tianjin University, China
MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
Apr 8, 2026Tobias Falke, Nicolas Anastassacos, Samson Tan +6Mixture of Experts Sparse Mixture-of-Experts
Amazon AGI
Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Training
Apr 5, 2026Charafeddine MouzouniCongestion Games Expert Load Balancing
OPIT – Open Institute of Technology, and Cohorte AI, Paris, France.
Routing-Free Mixture-of-Experts
Apr 1, 2026Yilun Liu, Jinru Han, Sikuan Yan +2Mixture-of-Experts Models Expert Load Balancing
Ludwig Maximilian University of Munich · Munich Center for Machine Learning · University of California, Los Angeles
Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancing
Mar 12, 2026Hanchi Sun, Yixin Liu, Yonghui Wu +1Expert Routing Mixture-of-Experts Models
Computer Science and Engineering, Lehigh University, Bethlehem, PA, USA · MD-HOBI-BIOMED INFORMATICS, University of Florida, Gainesville, FL, USA
MoEless: Efficient MoE LLM Serving with Serverless Experts
Mar 6, 2026Hanfei Yu, Bei Ouyang, Shwai He +2LLM Inference Efficiency Expert Load Balancing
University of Maryland College Park
Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation
Mar 5, 2026Yilong Chen, Naibin Gu, Junyuan Shang +8Mixture of Experts Mixture-of-Experts Models
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Baidu Inc.
A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs
Feb 23, 2026Zijie Liu, Jie Peng, Jinhao Duan +7Expert Load Balancing Mixture of Experts
University of North Carolina at Chapel Hill · University of Minnesota Twin Cities · North Carolina State University +2

May 7, 2026J/K move · Enter open · S save

E = T*H/(O+B): A Dimensionless Control Parameter for Mixture-of-Experts Ecology

Qingjun Zhang

School of Integrated Circuits, Wuxi Taihu University Department of Communication Engineering · Department of Communication Engineering

Expert Load Balancing Mixture of Experts Sparse Mixture-of-Experts Mixture-of-Experts Models Mixture-of-Experts Expert Specialization

Open paper AskPDF arXiv

We introduce E = T*H/(O+B), a dimensionless control parameter that predicts whether Mixture-of-Experts (MoE) models will develop a healthy expert ecology or collapse into dead experts. E combines four hyperparameters -- routing temperature T, routing entropy weight H, oracle weight O, and balance weight B -- into a single quantity. Through 12 controlled experiments (8 vision, 4 language) totaling over 11,000 training epochs, we establish that E >= 0.5 alone is sufficient to guarantee zero dead experts, removing the necessity for handcrafted load-balancing auxiliary losses. We validate this cross-modally on CIFAR-10, CIFAR-100, TinyImageNet-200, WikiText-2, and WikiText-103. Six additional findings emerge: (1) dead experts can resuscitate -- triggered by balance loss driving router re-exploration; (2) ortho toxicity is dataset-dependent, not universal; (3) task complexity shifts the critical E threshold; (4) model overfitting is decoupled from expert ecological health; (5) three-tier MoE spontaneously collapses into a two-tier functional structure; (6) ecological structure is temperature-invariant across a 50x range. We propose that E serves as a unified diagnostic for MoE training, analogous to the Reynolds number in fluid dynamics.