BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding

Ali Abbasi; Justin Shi; Soheil Kolouri

cs.LGSep 28, 2026

BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding

Authors: Ali Abbasi, Justin Shi, Soheil Kolouri

Organizations: College of Connected Computing, Vanderbilt University

Abstract

Large language models are increasingly expensive to serve. In large-scale serving systems, autoregressive decoding is often bottlenecked by transferring model weights from accelerator high-bandwidth memory into on-chip SRAM. Mixture-of-experts (MoE) models reduce computation by activating only a small subset of experts per token, but this sparsity does not translate directly to batched decoding. Different requests select different experts; therefore, the combined active set across many concurrent requests can span a substantial fraction of the expert pool and require significantly more expert weights to be transferred. Most expert-reduction techniques make retention decisions independently for each token and therefore do not address this batch-level expansion. More recently, batch-aware methods have attempted to coordinate expert use across concurrent requests and reuse experts already fetched for the batch. Yet their selection criteria are based primarily on router rankings or expert statistics collected during calibration. Consequently, these criteria are not directly tied to the output error caused by dropping an expert, nor do they capture how an expert's contribution changes across tokens at inference time. We instead rank experts according to how much their removal would change the MoE-layer output. To apply this criterion during serving, we train a lightweight linear predictor during calibration that estimates the expert removal cost for each incoming token, and develop custom GPU kernels for cost prediction and expert selection. Across three MoE architectures, BASE improves the quality-efficiency tradeoff without retraining. On Qwen3-30B-A3B, it improves average accuracy by 29.5 points over the strongest baseline at comparable throughput under a tight expert budget. At a higher expert budget, it is 60% faster than dense inference while remaining within 0.4 accuracy points.

Figures & tables

Figure 1: Cross terms affect error magnitude but have limited impact on expert selection. (a)

w_{t,u}w_{t,v}

-weighted distributions of cosine similarity between expert contributions routed to the same token. (b) Exact-to-additive error ratio as the literal number of omitted experts increases. The active set is the batch-wise union of each token’s top-1 expert; dashed curves show one-parameter fits. (c) Exact-error ratio between additive and cross-term-aware selection under the same fixed budget

M=12

. Panels (a) and (b) exclude the first and last MoE layers.

Figure 2: Binary union and budgeted selection produce different active sets. Constructed example with five tokens, eight experts, and two routed experts per token. (a) SERE and OEA fetch the union of each token’s top-1 expert, shown by dashed columns, producing four active experts. (b) BASE computes each routed pair’s predicted removal cost,

w_{t,u}^{2}\widehat{e}_{u}(z_{t})

, and sums these values across tokens to obtain the batch-level expert score

\widehat{D}_{u}

. (c) BASE fetches the

M=3

highest-scoring experts. Experts

e_{5}

and

e_{8}

enter the binary union despite low predicted removal costs, while

e_{3}

is selected by BASE despite never being a token’s top-1 expert.

		Methods \ Tasks	Exam			Math			Code		Avg. (Acc. $\uparrow$ )	tok/s ( $\uparrow$ )
		Methods \ Tasks	cmmlu	boolq	bbh	math	gsm8k	math 401	heval	mbpp	Avg. (Acc. $\uparrow$ )	tok/s ( $\uparrow$ )
Qwen3-30B-A3B		Dense	$84.79$	$89.82$	$76.93$	$72.34$	$88.40$	$80.55$	$88.41$	$78.80$	$82.50$	$845.3$
	Restricted $\approx S\!=\!1$	SERE S=1	$51.04$	$74.07$	$40.20$	29.26	$38.97$	$40.90$	$16.46$	$9.40$	$37.54$	1452.9
		OEA ${}_{k_{0}=1}$	$64.64$	86.51	$43.69$	$27.78$	$33.28$	$34.66$	43.90	$9.60$	$43.01$	$1372.5$
		Lynx ${}_{\alpha=0.47,\ \beta=1}$	68.46	$84.01$	$51.76$	$19.80$	$38.29$	$65.34$	$29.88$	$17.40$	46.87	$1304.9$
		ExFold ${}_{D=8,\ P=8}$	$40.92$	$67.52$	51.93	$15.42$	68.16	70.82	$15.24$	25.20	$44.40$	$1395.4$

Table 1: Main quality and throughput comparison across three MoE architectures under restricted and relaxed operating regimes. Avg. is the average across eight benchmarks, and tok/s reports end-to-end throughput at batch size 16 on a single H100 GPU. Within each model and regime, bold marks the two highest values and shading marks the best. Dense denotes the original model with no experts skipped.

Table 4

Figure 3: Effect of decode batch size on quality and throughput across three MoE architectures. The top row reports the average across HumanEval, MBPP, MATH-401, and BoolQ; the bottom row reports token throughput. The dotted gray line denotes the dense model. We vary

B

from 8 to 128, and BASE’s expert budget is speed-matched to the faster baseline at each batch size. Appendix A.7 gives the budgets and further details.

Mixture-of-Experts Inference

Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.

See all 177 papers

Aug 14, 2026

DeaMoE: Efficient MoE Structure for Fast Small-Batch Decoding

Mixture-of-Experts (MoE) models have been widely adopted in real-time interactive applications such as coding assistants, real-time audio-video interaction systems. To meet the extremely low response latency requirements of these scenarios, practitioners commonly employ small-batch decoding, under which MoE inference becomes memory-bound and is severely bottlenecked by expert weight loading. However, this bottleneck has received limited attention, and existing solutions such as post-training weight compression or fine-grained expert design during pre-training either degrade model accuracy or introduce additional computation and communication overhead. To tackle this issue, we propose DeaMoE, a decoding-efficient MoE architecture, in which the experts are grouped into several departments, and the experts belonging to the same department share most parameters since they come from the same professional field, and additionally each expert contains a few private parameters to reflect its uniqueness. Moreover, we design customized two-stage routing strategy for DeaMoE to avoid redundant loading, under which DeaMoE greatly improves the efficiency during LLM decoding. Compared with vanilla MoE, DeaMoE reduces per-step loaded weights by up to 50.9% and achieves up to 1.33 end-to-end TPOT speedup for the pre-trained 7B model on A40, and up to 2.00x and 1.97x peak speedup for DeepSeek-V3 on A40 and H100 in microbenchmarks.

Zewen Jin, Shen Fu, Zeping Duan +8

University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · Shanxi University

Mixture-of-Experts Inference Mixture-of-Experts Models LLM Inference Acceleration Expert Routing

Ask PDF

Mar 9, 2026

Speculating Experts Accelerates Inference for Mixture-of-Experts

Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs) while maintaining sparse activations and reduced per-token compute. However, in memory-constrained inference settings, expert weights must be offloaded to CPU, creating a performance bottleneck from CPU-GPU transfers during decoding. We propose an expert prefetching scheme that leverages currently computed internal model representations to speculate future experts, enabling memory transfers to overlap with computation. Across multiple MoE architectures, we demonstrate that future experts can be reliably predicted by these internal representations. We also demonstrate that executing speculated experts generally maintains downstream task accuracy, thus preserving more effective compute-memory overlap by eliminating the need to re-fetch true router-selected experts. Integrated into an optimized inference engine, our approach achieves up to 14% reduction in time per output token (TPOT) over on-demand loading of experts from CPU memory. For MoEs where speculative execution alone yields suboptimal accuracy, we further examine lightweight estimators that improve expert prediction hit rates, thereby reducing performance degradation. Our code is released in open-source at https://github.com/axonn-ai/yalis/tree/offload_prefetch.

Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2

University of Maryland · TogetherAI

Mixture-of-Experts Inference LLM Inference Acceleration Mixture-of-Experts Language Models

Ask PDF

Apr 25, 2026

Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns

Most recent state-of-the-art (SOTA) large language models (LLMs) use Mixture-of-Experts (MoE) architectures to scale model capacity without proportional per-token compute, enabling higher-quality outputs at manageable serving costs. However, MoE inference at scale is fundamentally bottlenecked by expert load imbalance and inefficient token routing, especially in multi-node deployments where tokens are not guaranteed to be routed to local experts, resulting in significant inter-node all-to-all communication overhead. To systematically characterize these challenges, we profile SOTA open-source MoE models, including Llama 4 Maverick, DeepSeek V3-671B, and Qwen3-230B-A22B, on various datasets and collected over 100k real expert activation traces. Upon studying the expert activation patterns, we uncover various persistent properties across all the frontier MoE models: variable expert load imbalance, domain-specific expert activation where expert popularity shifts across task families (code, math, chat, general), and a strong correlation between prefill and decode expert activations. Motivated by these findings, we propose workload-aware micro-batch grouping and an expert placement strategy to maximize token locality to the destination expert, thereby reducing inter-node communication. Across models and datasets, these optimizations help reduce all2all communication data up to 20, resulting in lower MoE decode latency and better accelerator utilization.

Abhimanyu Bambhaniya, Geonhwa Jeong, Jason Park +6

1Georgia Institute of Technology, USA · 2Meta Platforms, Inc.

Expert Load Balancing Mixture-of-Experts Inference LLM Routing Mixture-of-Experts Models Distributed Inference Communication-Efficient Mixture-of-Experts

Ask PDF

Abstract

Mixture-of-Experts Inference

Explore similar work

DeaMoE: Efficient MoE Structure for Fast Small-Batch Decoding

Speculating Experts Accelerates Inference for Mixture-of-Experts

Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns