Expert Load Balancing

Follow this topic Ask about this topic

Sparse mixture-of-experts (MoE) models activate only a few experts per token, but uneven routing can overload some experts while others sit idle, slowing training and inference. Training methods use auxiliary balance losses, dynamic router biases or thresholds, hierarchical routing, and quantile- or control-based corrections to spread assignments while preserving specialization. Systems methods predict demand, replicate or migrate experts, and place or schedule them with awareness of GPU speed and network topology; evaluation spans large-scale pretraining, distributed serving, and language and multimodal workloads. The central tension is balancing utilization without degrading task quality or suppressing specialization, while avoiding communication and planning overhead, since token counts alone may not reflect actual compute time.

Latest papers

41 papers5 papers in the last four weeks, up 67% on the four weeks before. 0.0% of all new papers.

Muon Makes the First Mixture-of-Experts Layer Harder to Balance
Oct 27, 2026Marcos HernanzMixture-of-Experts Language Models Expert Load Balancing
Escuela Politécnica Superior, Universidad Autónoma de Madrid
LRMoE: Demand-Side Load Re-Calibration for Efficient MoE Training on Memory-Constrained Nodes
Oct 27, 2026Jiaming Hu, Zhida Jiang, Zhaolong Xing +3Mixture-of-Experts Models Memory-Efficient Training
JD.com, Beijing, China
3D-MoE: Complementarity-Guided Hardware-Software Co-Design for MoE Decode on 3D-Stacked NMP
Oct 19, 2026Yudong Pan, Lian Liu, Shixin Zhao +8Neural Network Hardware Acceleration Mixture-of-Experts Inference
SKLP, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · University of Edinburgh, Edinburgh, United Kingdom
BRANCH-MoE: Balance-Aware Tree Routing for Large Embedding Models
Oct 5, 2026Gang Fu, Adel Javanmard, MohammadHossein Bateni +1Expert Load Balancing Mixture-of-Experts Models
Google Research · University of Southern California
CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training
Oct 5, 2026Jing Li, Jian Meng, Yingmeng Gao +8Expert Load Balancing Expert Routing
Tongji University · Cornell University · Harbin Institute of Technology, Shenzhen +1
ID Balancing: Stable Training of Extremely Sparse MoE via PID-Based Load Control
Sep 30, 2026Peng Jin, Zihan Qiu, Zekun Wang +8Expert Load Balancing Mixture-of-Experts Language Models
Qwen Team, Alibaba Token Hub, Alibaba Group
TopoEP: Topology-Aware Load Balancing for Expert-Parallel MoE Training
Sep 28, 2026Jiacheng Zhu, Xie Zhao, Gongming Zhao +3Expert Parallelism Expert Load Balancing
University of Science and Technology of China
Function Over Form: Distributional Orthogonalization in Mixture-of-Experts with Replica Expert Mechanism
Sep 26, 2026Jinfan He, Yunzhuo Liu, Kai Zhang +3Mixture-of-Experts Language Models Mixture of Experts
School of EECS, Peking University, China · Tencent Hunyuan, China
Exact Quantile Balancing and Load-Error Injection for Mixture-of-Experts
Sep 23, 2026Pit Neitemeier, Jiaze Li, Alessio Serra +2Expert Load Balancing Expert Routing
Aleph Alpha
When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models
Sep 3, 2026Berkcan Kapusuzoglu, Connor Pryor, Sangwoo Cho +4Mixture-of-Experts Pruning Expert Routing
Capital One, AI Foundations, McLean, VA 22102, USA
FreeBalance: Pre-Routing Online Moe Load Balancing via Residual Workload Prediction
Aug 14, 2026Pengfei Chen, Yize Wu, Shouxu Kuang +2Expert Load Balancing Mixture-of-Experts Inference
Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes
Aug 13, 2026Jie Li, Chenxin Jia, Jinliang Shen +5Expert Parallelism Expert Load Balancing
KlingAI Research
RoutePack: Replay-Guided Expert Placement and Rollout Trajectory Scheduling for MoE Reinforcement Learning
Aug 12, 2026Yibo Shen, Xudong Han, Xiaowei Zhu +2RL for Language Models Expert Load Balancing
Ant Group
The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism
Aug 9, 2026Jiguo LiExpert Parallelism Expert Load Balancing
This report was completed with the assistance of Codex.
EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference
Aug 8, 2026Yize Wu, Ke Gao, Ling Li +1Expert Load Balancing Mixture-of-Experts Inference
Intelligent Software Research Center, Institute of Software, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
REFLEX: Rethinking MoE Inference as Refinement-Aware Compute Allocation in Diffusion Language Models
Aug 3, 2026Xiang Xia, Cheng Yan, Yiming Zhang +3Expert Load Balancing Mixture-of-Experts Inference
University of Science and Technology of China, Hefei, China
Relax Within, Balance Across: Geometry-Guided Load Balancing for Vision-Language Mixture-of-Experts
Aug 1, 2026Ziang Wu, Peng Jin, Qishen Yin +4Expert Load Balancing Vision-Language Models
2Peng Cheng Laboratory · School of Software and Microelectronics, Peking University · 1Shenzhen Graduate School, Peking University +2
Hierarchical Copula-Gumbel-Top-\texorpdfstring{ $K$ }{K} Routing: Two-Sided Dependence Control for Frozen Mixture-of-Experts at Fixed Per-Token Routing Laws
Jul 25, 2026Richard Yi Da XuCopula Models Expert Load Balancing
Hong Kong Baptist University and TadReamk Limited
Optimal Scheduling in a Question-Answering Forum of Knowledge Workers
Jun 18, 2026Rohit Negi, Mustafa YilmazExpert Load Balancing Question Answering
Carnegie Mellon University
UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing
Jun 2, 2026Xinming Wei, Chao Jin, Tuo Dai +10Expert Parallelism Expert Load Balancing
School of Computer Science, Peking University · Xiaohongshu Inc. · Shanghai AI Laboratory +1
ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving
May 30, 2026Seokjin Go, Marko Scrbak, Ephrem Wu +2Expert Load Balancing Mixture-of-Experts Inference
Georgia Institute of Technology Atlanta, GA, USA · Advanced Micro Devices, Inc. Austin, TX, USA · Advanced Micro Devices, Inc. Santa Clara, CA, USA +1
A Minimal Bifurcation Model of Load Imbalance in a Softmax Mixture-of-Experts Router
May 27, 2026O. M. KiselevDynamical Systems Expert Load Balancing
Innopolis University, Innopolis, Russia
Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory
May 20, 2026Bole Ma, Jan Eitzinger, Harald Koestler +1Expert Parallelism Expert Load Balancing
GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems
May 19, 2026Sourish Wawdhane, Avinash Kumar, Poulami DasExpert Load Balancing Mixture-of-Experts Inference
The University of Texas at Austin
UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Models
May 15, 2026Van-Tuan Tran, Hong-Hanh Nguyen-Le, Marco Ruffini +1Expert Load Balancing Sparse Mixture-of-Experts
School of Computer Science and Statistics, Trinity College Dublin, Ireland · School of Computer Science, University College Dublin, Ireland
$φ$ -Balancing for Mixture-of-Experts Training
May 14, 2026Lizhang Chen, Jonathan Li, Qi Wang +5Expert Load Balancing Mirror Descent
University of Texas at Austin · 2Northwestern University.
Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts
May 12, 2026Sagi Ahrac, Noya Hochwald, Mor GevaExpert Load Balancing Parameter-Free Mixture-of-Experts Routing
Blavatnik School of Computer Science and AI, Tel Aviv University
Fast MoE Inference via Predictive Prefetching and Expert Replication
May 12, 2026Ankit Jyothish, Ali Jannesari, Aishwarya Sarkar +1Expert Load Balancing GPU Acceleration
Iowa State University Ames, USA
ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learning
May 9, 2026Chao Jin, Xinming Wei, Yinmin Zhong +6RL for Language Models Expert Load Balancing
School of Computer Science, Peking University · Independent Researcher
Hierarchical Mixture-of-Experts with Two-Stage Optimization
May 8, 2026Gleb Molodtsov, Alexander Miasnikov, Aleksandr BeznosikovExpert Load Balancing Mixture of Experts
BRAIn Lab, Moscow, Russia

Oct 27, 2026J/K move · Enter open · S save

Muon Makes the First Mixture-of-Experts Layer Harder to Balance

Marcos Hernanz

Escuela Politécnica Superior, Universidad Autónoma de Madrid

Mixture-of-Experts Language Models Expert Load Balancing Sparse Mixture-of-Experts Muon Optimizer

Open paper AskPDF arXiv

Many open mixture-of-experts (MoE) language models train with Muon instead of AdamW. Most balance the load with a bias on each expert's routing score, adjusted after every step. Their recipes disagree on whether Muon should also train the router. In models with 55M and 108M active parameters, we measure how the optimizer changes the span of these biases, the largest minus the smallest in a layer. The span grows when all tokens favor the same experts, so it shows how hard a layer is to balance. At learning rates tuned on shorter runs, Muon raised the span of the first MoE layer at every seed, roughly doubling it on average, while the other layers changed much less. The rise came from Muon on the rest of the network, whichever optimizer trained the router. In those runs, the inputs to the first router shared a direction, and soon after the warmup its busiest expert averaged four to seven times its fair share. Over the first 3,000 steps, subtracting the batch mean from these inputs removed this overload while barely changing the loss. On the smaller model, halving Muon's learning rate removed most of the rise and slightly improved the loss. Moving only the router to AdamW made the loads shift more during the warmup and slightly raised the loss, so at this scale the results favor keeping the router on Muon.