Load imbalance poses a major bottleneck to the efficiency of expert parallelism in distributed inference of Mixture-of-Experts (MoE) models. The most heavily loaded rank stalls global execution due to skewed routing distributions, directly increasing latency. While offline expert placement can alleviate persistent imbalance, practical multi-task serving workloads exhibit layer- and batch-dependent routing dynamics, making online load balancing indispensable. Existing approaches rely on routing statistics collected after each MoE router, requiring expert weight load or migration to begin only after routing decisions are available, consequently placing migration overhead on the inference critical path. In this work, we observe that online balancing can instead be largely overlapped with computation before target routing (e.g., attention), if routing distributions can be predicted accurately in advance. Therefore, we propose FreeBalance, a lossless online load-balancing framework that overlaps expert migration with preceding computation stages via residual workload prediction. FreeBalance leverages cross-layer similarities in hidden representations within the residual network to build a lightweight workload predictor. This enables proactive expert migration planning before routing decisions are available, creating substantial overlap between weight transfer and computation-heavy pre-routing stages. Furthermore, a cost model constrains the number of swaps to fully hide the synchronization overhead within the available window. Experiments across models and datasets show that FreeBalance reduces the max-to-mean rank load ratio by 32.8% and end-to-end prefill latency by 13.1%. Specifically, our method hides balancing overhead of an average of 5.1 experts per layer, which would otherwise account for about 8.5% of the critical-path latency.
Figures & tables
Figure 1: Overview of FreeBalance and its four stages: ① residual workload prediction, ② budgeted expert-swap planning, ③ expert-weight migration overlapped with pre-routing attention, and ④ lossless MoE computation. The residual representation from layer ℓ−1 predicts the expert workload of layer ℓ before its router executes. The prediction-guided expert placement is available before the target expert FFN executes. The blue line denotes the FreeBalance execution flow (ours), whereas the black line denotes the original inference flow.
Figure 2: Per-layer relative expert loads for NarrativeQA and LCC. Experts are sorted independently at each layer by their NarrativeQA load, and the same ordering is used for LCC to expose the workload-dependent shift in expert popularity.
Figure 3: Geometric intuition for cross-layer residual similarity. The attention and MoE updates cause only a small directional change from hℓ−1 to hℓ .
Qwen3-30B-A3B-Instruct-2507
Moonlight-16B-A3B-Instruct
LongBench Subset
Vanilla
Vanilla +Ours
EPLB
EPLB +Ours
Vanilla
Vanilla +Ours
EPLB
EPLB +Ours
Single-Document QA
NarrativeQA
147.6
126.6
135.9
126.6
77.4
66.7
63.6
63.2
Qasper
102.6
94.6
105.6
92.3
57.1
55.4
57.5
55.3
MultiFieldQA (EN)
85.3
74.8
76.9
74.7
43.1
42.5
43.4
42.7
Multi-Document QA
Table 1: Prefill latency across 19 LongBench subsets and one mixed dataset. Columns compare Vanilla, Vanilla with FreeBalance, EPLB, and EPLB with FreeBalance for Qwen3-30B-A3B and Moonlight-16B-A3B. Each entry reports latency in seconds; lower is better.
Model
Dataset
Hidden cos.
Logit cos.
Top- k hit
Qwen
PassageRetrieval
0.7334
0.9952
0.7520
LSHT
0.7242
0.9951
0.7578
TriviaQA
0.7116
0.9947
0.7419
Moonlight
PassageRetrieval
0.9316
0.9900
0.8461
LSHT
0.9291
0.9896
0.8494
TriviaQA
0.9234
0.9910
0.8037
Table 2: Pre-routing quality across models and datasets. Router logit and hidden cosine similarities compare the early and normal routing passes; top- k hit measures assignment overlap.
Figure 4: Speedup over Vanilla as a function of expert swaps per layer. Lines show zero to four forced swaps per layer; stars show adaptive FreeBalance, whose average swap count ranges from 1.82 to 2.68 across sequence lengths.
Length
Vanilla
Ours
Overlap (ms)
TTFT
Max/Mean
TTFT
Max/Mean
1K
3.08
2.00
2.74
1.33
12.55
2K
4.99
2.02
4.43
1.34
17.35
4K
8.98
2.03
7.78
1.35
31.30
8K
20.45
2.04
15.68
1.37
36.80
Table 3: Performance under batch size 16. TTFT is reported in seconds, and overlap time is reported in milliseconds.
Load Balancing has emerged as a critical problem in expert-parallel distributed inference of Mixture-of-Experts (MoE) models. As routing distributions are typically skewed across experts, devices hosting lighter-loaded experts must idle to wait for the heaviest during expert computing, leading to inefficiency. Existing load-balancing approaches primarily rely on expert replication or migration within each layer, which introduce additional overhead and limit their flexibility and scalability. To address this problem, we propose EasyBalance, a cross-layer load balancing strategy that requires no modifications to the expert-device mapping, enabling instant adaptability and incurring essentially no additional overhead. Our key insights are that (1) experts of other layers can be viewed as naturally redundant for the current layer, and (2) cross-layer MoE workloads can be jointly executed to mitigate their individual imbalance. Based on these observations, EasyBalance greedily schedules a subset of cross-layer workloads to run at each MoE step and defers the remaining workloads for future balancing opportunities, effectively leveraging cross-layer imbalance mitigation. Extensive experiments across models, tasks, and configurations demonstrate that EasyBalance consistently accelerates distributed MoE inference, reducing GPU idling by mostly over 40%. Code is available at https://github.com/yize-wu/EasyInfra.
Yize Wu, Ke Gao, Ling Li +1
Intelligent Software Research Center, Institute of Software, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Mixture-of-Experts (MoE) models rely on balanced expert utilization to fully realize their scalability. However, existing load-balancing methods are largely heuristic and operate on noisy mini-batch assignment statistics, introducing bias relative to population-level objectives. We propose φ-balancing, a principled framework that directly targets population-level expert balance by minimizing a strictly convex, symmetric, and differentiable potential of the expected routing distribution. Using convex duality, we derive an equivalent min-max formulation and obtain a simple online algorithm via mirror descent, yielding an efficient EMA-based routing adjustment with negligible overhead. Across large-scale pretraining and downstream fine-tuning, φ-balancing consistently outperforms prior Switch-style and loss-free baselines, demonstrating more stable and effective expert utilization.
Lizhang Chen, Jonathan Li, Qi Wang +5
University of Texas at Austin · 2Northwestern University.
Standard Mixture-of-Experts (MoE) models rely on centralized routing mechanisms that introduce rigid inductive biases. We propose Routing-Free MoE which eliminates any hard-coded centralized designs including external routers, Softmax, Top-K and load balancing, instead encapsulating all activation functionalities within individual experts and directly optimized through continuous gradient flow, enabling each expert to determine its activation entirely on its own. We introduce a unified adaptive load-balancing framework to simultaneously optimize both expert-balancing and token-balancing objectives through a configurable interpolation, allowing flexible and customizable resource allocation. Extensive experiments show that Routing-Free MoE can consistently outperform baselines with better scalability and robustness. We analyze its behavior in detail and offer insights that may facilitate future MoE design ad optimization.
Yilun Liu, Jinru Han, Sikuan Yan +2
Ludwig Maximilian University of Munich · Munich Center for Machine Learning · University of California, Los Angeles