MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios

Shuhuai Li; Jianghao Lin; Dongdong Ge; Yinyu Ye

cs.LGFeb 12, 2026

MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios

Authors: Shuhuai Li, Jianghao Lin, Dongdong Ge, Yinyu Ye

Organizations: Shanghai University, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China · Standford University, California, United States

Abstract

Mixture-of-Experts (MoE) models enable scalable performance but face severe memory constraints on edge devices. Existing offloading strategies struggle with I/O bottlenecks due to the dynamic, low-information nature of autoregressive expert activation. In this paper, we propose to repurpose Speculative Decoding (SD) not merely as a compute accelerator, but as an informative lookahead sensor for memory management, supported by our theoretical and empirical analyses. Hence, we introduce MoE-SpAc, an MoE inference framework that integrates a Speculative Utility Estimator to track expert demand, a Heterogeneous Workload Balancer to dynamically partition computation via online integer optimization, and an Asynchronous Execution Engine to unify the prefetching and eviction in the same utility space. Extensive experiments on seven benchmarks demonstrate that MoE-SpAc achieves a 42% improvement in TPS over the SOTA SD-based baseline, and an average 4.04x speedup over all standard baselines. Code is available at https://github.com/lshAlgorithm/MoE-SpAc .

Figures & tables

Figure 1: Two key observations how speculative decoding (SD, Bottom) better benefits MoE inference than autoregressive decoding (AR, Top). Theoretically (Left), SD enables expert reuse and transforms activation utility signals from AR into informative frequency-valued ones. Practically (Right), experts can be scheduled in parallel by proactively leaving more memory for experts to come and asynchronously prefetching experts during the drafting phase.

Figure 2: Overall framework of MoE-SpAc.

Panel of Figure 3 — Figure 3: Notation and MoE-SpAc pipeline illustration.

Method	MMLU-Pro		MT-bench		GSM8K		HumanEval		Alpaca		CNN/DM		QA		Average
Method	TPS $\uparrow$	Lat. $\downarrow$	TPS $\uparrow$	Lat. $\downarrow$	TPS $\uparrow$	Lat. $\downarrow$	TPS $\uparrow$	Lat. $\downarrow$	TPS $\uparrow$	Lat. $\downarrow$	TPS $\uparrow$	Lat. $\downarrow$	TPS $\uparrow$	Lat. $\downarrow$	TPS $\uparrow$	Lat. $\downarrow$
Accelerate	1.13	495.37	1.15	435.55	1.21	237.45	1.15	393.60	1.09	350.83	1.27	222.47	1.16	138.42	1.17	324.81
vLLM	2.65	203.74	2.67	187.88	2.67	94.63	2.63	158.46	2.67	137.72	2.57	104.28	2.68	57.43	2.65	134.88
llama.cpp	15.17	37.92	14.34	36.08	15.01	19.21	14.74	30.57	16.86	22.40	15.42	19.46	15.09	11.07	15.23	25.24
llama.cpp-w/ SD	17.29	32.88	17.58	29.03	19.03	15.09	18.78	24.01	18.02	21.51	16.46	18.36	16.49	10.43	17.66	21.62
Mixtral Offload	2.43	230.87	2.36	212.70	2.35	120.25	2.48	177.87	2.33	162.32	2.23	131.72	2.33	70.86	2.36	158.08

Table 1: The tokens per second (TPS) and latency (Lat.) performance of different inference methods. The best results are given in bold while the second-best values are underlined . Rel.Imprv. indicates the relative improvement of MoE-SpAc against the best baseline. Dashes denote unreported measurements; averages cover all seven benchmarks.

Figure 4: The hot-or-cold online prediction accuracy of MoE-SpAc (SD, step level) with different acceptance rate and HybriMoE (AR, layer level).

Model Variants	MMLU-Pro		HumanEval		CNN/DM
Model Variants	TPS $\uparrow$	Lat. $\downarrow$	TPS $\uparrow$	Lat. $\downarrow$	TPS $\uparrow$	Lat. $\downarrow$
MoE-SpAc (Full)	25.80	23.11	28.24	16.93	22.78	14.31
w/o SpecUE	19.53	29.48	20.56	25.27	18.43	19.53
w/o AdaBC	25.33	23.48	28.03	19.03	22.31	15.98
w/o HetWB	24.32	24.32	26.95	19.97	20.16	17.69
w/o SG-Prefetcher	17.27	32.91	18.79	27.44	16.46	21.23

Table 2: Ablation study of MoE-SpAc.

Figure 5: Stability and sensitivity analysis. Left: Impact of expert cache ratios; despite an OOM boundary at 21% due to draft model allocation, MoE-SpAc yields superior throughput compared to existing works. Middle: Scalability across generation lengths, showing consistent gains over baselines in long-context tasks. Right: Effect of the threshold cap K, where small values reduce performance by limiting the precision of the activation utility.

Hardware	NVIDIA RTX 4090			NVIDIA A100-80GB
Method	DeepSeek-V2-Lite (16B)			GLM-4.5-Air (106B)			Qwen3.5 (122B)			MiniMax-M2.5 (228B)
Method	MT-bench	HumanEval	CNN/DM	MT-bench	HumanEval	CNN/DM	MT-bench	HumanEval	CNN/DM	MT-bench	HumanEval	CNN/DM
HybriMoE	18.43	17.95	18.43	15.34	15.28	16.83	9.25	9.25	9.27	4.47	4.48	4.44
Llama.cpp (w/ SD)	30.78	30.49	33.23	17.11	18.73	17.03	12.46	12.78	12.44	5.35	5.87	5.37
MoE-SpAc (Ours)	47.10	47.26	50.17	27.47	27.79	26.98	20.76	21.42	20.27	10.13	10.89	10.03
Rel. Improv.	+53.0%	+55.0%	+51.0%	+60.5%	+48.4%	+58.4%	+66.6%	+67.6%	+62.9%	+89.3%	+85.5%	+86.8%

Table 3: TPS performance comparison across different models and datasets. We compare our MoE-SpAc against best baselines. Rel. Improv. is calculated against the strongest baseline.

Figure 6: TPS under different batch sizes, with experiment configuration in Section 4.1 .

Batch size	$\gamma=2$	$\gamma=4$	$\gamma=6$	$\gamma=8$	$\gamma=10$
1	20.79	22.23	23.92	24.10	23.88
2	20.67	23.42	24.13	26.01	24.93
4	23.83	23.94	25.78	27.52	26.63
8	24.62	24.89	28.01	29.23	28.44
16	25.37	26.41	29.39	30.48	29.31
32	27.50	27.22	29.64	31.15	31.03

Table 4: MoE-SpAc throughput (TPS) across batch size and draft length

\gamma

. It shows the best combination of batch sizes and draft lengths in practice.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Illustration of TPS metric to the expert cache ratio(%) on NVIDIA GeForce RTX 4090 (24GB). Target model is Qwen3-30B-A3B , and draft one is Qwen3-4B-FP8 . In edge scenario, less cache ratio precipitates only negligible decelerate, shown as red block with “Cons” label. Placing a small draft model can lead to huge speedup, shown as the green block with “Pros” label. Pros is about $6\times$ overweight Cons.

Panel of Figure 8 — Figure 8: The curve comparison between $\Omega(\gamma,\alpha)$ and $\frac{a}{b}$ , experimented on MT-bench with Qwen3-235B-A22B as verification model and Qwen3-4B-FP8 , taking $\alpha$ as $0.8$ . (a) shows the numerical result, while (b) illustrates the benefit area.

Figure 9: Activation frequency across iterations in SD is shown on the left, while the AR is shown on the right. It is much easier to estimate the light points where the experts are activated multiple times(hot experts).

Panel of Figure 10 — Figure 10: Relationship between utility and activated frequency, ratio of activated experts, etc..

Figure 11: Scheduling and cache management of the MoE-SpAc in implementation when $K=5$ .

Figure 12: MoE-SpAc implementation atop llama.cpp , unchanged operators is omitted.

Figure 13: Comparison of MoE inference pipelines under Speculative Decoding (SD). ‘GPU’ and ‘CPU’ denote the compute device for expert layers, while prime notation (e.g., $1^{\prime},1^{\prime\prime}$ ) indicates the layer depth. MoE-SpAc (bottom) minimizes pipeline bubbles via optimized heterogeneous scheduling.

	Adaptive (Ours)	Fix $\tau=1$	Fix $\tau=2$	Fix $\tau=4$
TPS	24.10	21.73	23.48	10.16

Appendix

Table 5: Performance with adaptive and fixed thresholds.

Test	Configuration	Worst NMSE
AGGRE (CUDA vs. CPU)	$E=256$ , $U=8$ , $T\in\{1,32\}$	0 (bit-exact)
Small split MoE	$E=8$ , $U\in\{2,8\}$ , $T\in\{1,7,32\}$	$4.1\times 10^{-6}$
Qwen3-like split MoE	$E=128$ , $U\in\{2,8\}$ , $T\in\{1,7,32\}$	$1.2\times 10^{-6}$
Realistic layer	embedding 2048, FFN 768, $E=128$ , $U=8$ , $T=32$	$8.3\times 10^{-7}$

Appendix

Table 6: Numerical validation.

E

U

, and

T

denote expert count, experts per token, and token count. Split-pipeline tests cover cache ratios of 0/50/100%. NMSE is normalized mean squared error.

Mixture-of-Experts Inference

Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.

See all 173 papers

APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference
Aug 12, 2026Alish Kanani, Layan Badawi, Umit Y. OgrasEdge Inference Mixture-of-Experts Inference
University of Wisconsin–Madison
Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading
Nov 18, 2025Wenfeng Wang, Xiaofeng Hou, Xinfeng Xia +5Mixture-of-Experts Inference LLM Inference Acceleration
Shanghai Jiao Tong University, Shanghai, China · Shanghai Jiao Tong University, China · Hong Kong University of Science and Technology, Hongkong, China
S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices
Aug 15, 2026Haochen Huang, Shengxuan Qiu, Meng LiLLM Inference Acceleration On-Device Language Model Inference
Institute for Artificial Intelligence, Peking University, Beijing, China · School of Integrated Circuits, Peking University, Beijing, China · School of Electronics Engineering and Computer Science, Peking University, Beijing, China

Aug 12, 2026J/K move · Enter open · S save

APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference

Alish Kanani, Layan Badawi, Umit Y. Ogras

University of Wisconsin–Madison

Edge Inference Mixture-of-Experts Inference Memory-Efficient Inference

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models are attractive for edge deployment because they provide high model capacity while activating only a small subset of parameters per token, improving compute efficiency. However, MoE inference at the edge is fundamentally limited by memory. Expert parameters are large and often reside in off-chip memory due to capacity, cost, and power constraints, putting expert loading to the critical path. We present APEX: Adaptive Expert Prefetching, a predictive resource management framework that overlaps expert loading with useful computation. APEX introduces a lightweight prefetch router that predicts candidate experts before the attention block to dynamically fetch additional experts using a learned confidence model. This adaptive strategy achieves over 99% overlap accuracy, significantly outperforming fixed top-k prefetching techniques. APEX supports two execution modes: a correctness-preserving mode that guarantees exact routing semantics, and a stall-free mode that eliminates residual stalls by operating on available experts with negligible impact on application accuracy. Across multiple MoE models, the correctness-preserving mode reduces per-token latency by up to 26% and improves energy-delay product (EDP) by up to 41% over state-of-the-art baselines, while the stall-free mode provides additional efficiency gains with negligible impact on application accuracy. These results establish adaptive, confidence-driven expert prefetching as an effective approach for efficient MoE inference on edge systems.

Abstract

Mixture-of-Experts Inference

Explore similar work

APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference

Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading

S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices