FreeBalance: Pre-Routing Online Moe Load Balancing via Residual Workload Prediction

Pengfei Chen; Yize Wu; Shouxu Kuang; Ke Gao; Ling Li

cs.AIAug 14, 2026

FreeBalance: Pre-Routing Online Moe Load Balancing via Residual Workload Prediction

Authors: Pengfei Chen, Yize Wu, Shouxu Kuang, Ke Gao, Ling Li

Organizations: Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Abstract

Load imbalance poses a major bottleneck to the efficiency of expert parallelism in distributed inference of Mixture-of-Experts (MoE) models. The most heavily loaded rank stalls global execution due to skewed routing distributions, directly increasing latency. While offline expert placement can alleviate persistent imbalance, practical multi-task serving workloads exhibit layer- and batch-dependent routing dynamics, making online load balancing indispensable. Existing approaches rely on routing statistics collected after each MoE router, requiring expert weight load or migration to begin only after routing decisions are available, consequently placing migration overhead on the inference critical path. In this work, we observe that online balancing can instead be largely overlapped with computation before target routing (e.g., attention), if routing distributions can be predicted accurately in advance. Therefore, we propose FreeBalance, a lossless online load-balancing framework that overlaps expert migration with preceding computation stages via residual workload prediction. FreeBalance leverages cross-layer similarities in hidden representations within the residual network to build a lightweight workload predictor. This enables proactive expert migration planning before routing decisions are available, creating substantial overlap between weight transfer and computation-heavy pre-routing stages. Furthermore, a cost model constrains the number of swaps to fully hide the synchronization overhead within the available window. Experiments across models and datasets show that FreeBalance reduces the max-to-mean rank load ratio by 32.8% and end-to-end prefill latency by 13.1%. Specifically, our method hides balancing overhead of an average of 5.1 experts per layer, which would otherwise account for about 8.5% of the critical-path latency.

Figures & tables

Figure 1: Overview of FreeBalance and its four stages: ① residual workload prediction, ② budgeted expert-swap planning, ③ expert-weight migration overlapped with pre-routing attention, and ④ lossless MoE computation. The residual representation from layer $\ell{-}1$ predicts the expert workload of layer $\ell$ before its router executes. The prediction-guided expert placement is available before the target expert FFN executes. The blue line denotes the FreeBalance execution flow (ours), whereas the black line denotes the original inference flow.

Figure 2: Per-layer relative expert loads for NarrativeQA and LCC. Experts are sorted independently at each layer by their NarrativeQA load, and the same ordering is used for LCC to expose the workload-dependent shift in expert popularity.

Figure 3: Geometric intuition for cross-layer residual similarity. The attention and MoE updates cause only a small directional change from $h_{\ell-1}$ to $h_{\ell}$ .

Single-Document QA
	Qwen3-30B-A3B-Instruct-2507				Moonlight-16B-A3B-Instruct
LongBench Subset	Vanilla	Vanilla +Ours	EPLB	EPLB +Ours	Vanilla	Vanilla +Ours	EPLB	EPLB +Ours
NarrativeQA	147.6	126.6	135.9	126.6	77.4	66.7	63.6	63.2
Qasper	102.6	94.6	105.6	92.3	57.1	55.4	57.5	55.3
MultiFieldQA (EN)	85.3	74.8	76.9	74.7	43.1	42.5	43.4	42.7
Multi-Document QA

Table 1: Prefill latency across 19 LongBench subsets and one mixed dataset. Columns compare Vanilla, Vanilla with FreeBalance, EPLB, and EPLB with FreeBalance for Qwen3-30B-A3B and Moonlight-16B-A3B. Each entry reports latency in seconds; lower is better.

Model	Dataset	Hidden cos.	Logit cos.	Top- $k$ hit
Qwen	PassageRetrieval	0.7334	0.9952	0.7520
	LSHT	0.7242	0.9951	0.7578
	TriviaQA	0.7116	0.9947	0.7419
Moonlight	PassageRetrieval	0.9316	0.9900	0.8461
	LSHT	0.9291	0.9896	0.8494
	TriviaQA	0.9234	0.9910	0.8037

Table 2: Pre-routing quality across models and datasets. Router logit and hidden cosine similarities compare the early and normal routing passes; top-

k

hit measures assignment overlap.

Figure 4: Speedup over Vanilla as a function of expert swaps per layer. Lines show zero to four forced swaps per layer; stars show adaptive FreeBalance, whose average swap count ranges from 1.82 to 2.68 across sequence lengths.

Length	Vanilla		Ours		Overlap (ms)
Length	TTFT	Max/Mean	TTFT	Max/Mean	Overlap (ms)
1K	3.08	2.00	2.74	1.33	12.55
2K	4.99	2.02	4.43	1.34	17.35
4K	8.98	2.03	7.78	1.35	31.30
8K	20.45	2.04	15.68	1.37	36.80

Table 3: Performance under batch size 16. TTFT is reported in seconds, and overlap time is reported in milliseconds.

Expert Load Balancing

Expert load balancing distributes tokens and compute across mixture-of-experts models to improve utilization without erasing expert specialization.

See all 39 papers

EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference
Aug 8, 2026Yize Wu, Ke Gao, Ling Li +1Expert Load Balancing Mixture-of-Experts Inference
Intelligent Software Research Center, Institute of Software, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
$φ$ -Balancing for Mixture-of-Experts Training
May 14, 2026Lizhang Chen, Jonathan Li, Qi Wang +5Expert Load Balancing Mirror Descent
University of Texas at Austin · 2Northwestern University.
Routing-Free Mixture-of-Experts
Apr 1, 2026Yilun Liu, Jinru Han, Sikuan Yan +2Mixture-of-Experts Models Expert Load Balancing
Ludwig Maximilian University of Munich · Munich Center for Machine Learning · University of California, Los Angeles

Aug 8, 2026J/K move · Enter open · S save

EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference

Yize Wu, Ke Gao, Ling Li, Yanjun Wu

Intelligent Software Research Center, Institute of Software, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Expert Load Balancing Mixture-of-Experts Inference

Open paper AskPDF arXiv

Load Balancing has emerged as a critical problem in expert-parallel distributed inference of Mixture-of-Experts (MoE) models. As routing distributions are typically skewed across experts, devices hosting lighter-loaded experts must idle to wait for the heaviest during expert computing, leading to inefficiency. Existing load-balancing approaches primarily rely on expert replication or migration within each layer, which introduce additional overhead and limit their flexibility and scalability. To address this problem, we propose EasyBalance, a cross-layer load balancing strategy that requires no modifications to the expert-device mapping, enabling instant adaptability and incurring essentially no additional overhead. Our key insights are that (1) experts of other layers can be viewed as naturally redundant for the current layer, and (2) cross-layer MoE workloads can be jointly executed to mitigate their individual imbalance. Based on these observations, EasyBalance greedily schedules a subset of cross-layer workloads to run at each MoE step and defers the remaining workloads for future balancing opportunities, effectively leveraging cross-layer imbalance mitigation. Extensive experiments across models, tasks, and configurations demonstrate that EasyBalance consistently accelerates distributed MoE inference, reducing GPU idling by mostly over 40%. Code is available at https://github.com/yize-wu/EasyInfra.

Abstract

Expert Load Balancing

Explore similar work

EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference

φφφ-Balancing for Mixture-of-Experts Training

Routing-Free Mixture-of-Experts

$φ$ -Balancing for Mixture-of-Experts Training