S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices

Haochen Huang; Shengxuan Qiu; Meng Li

cs.AIAug 15, 2026

S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices

Authors: Haochen Huang, Shengxuan Qiu, Meng Li

Organizations: Institute for Artificial Intelligence, Peking University, Beijing, China · School of Integrated Circuits, Peking University, Beijing, China · School of Electronics Engineering and Computer Science, Peking University, Beijing, China

Abstract

Deploying large language models (LLMs) for inference on edge devices is challenging due to severe memory and bandwidth constraints. While speculative decoding and Mixture-of-Experts (MoE) have been proposed to improve inference efficiency, naively combining them often incurs excessive verification overhead and poor expert reuse, limiting their effectiveness in memory-bound edge settings. In this work, we propose S2-MoE, an efficient self-speculative decoding framework for MoE inference on edge devices. S2-MoE reduces redundant verification through routing-aware adaptive speculative expansion, improves verification efficiency with reuse-aware expert gating, and aligns draft and target execution via shared context. Implemented in llama $.$ cpp, S2-MoE achieves up to $5.3\times$ speedup (about $2.0\times$ on average) over standard autoregressive decoding across diverse MoE models and datasets on edge devices. Code is available at https://github.com/angerybob/S2-MoE.

Figures & tables

Figure 1. Comparison of autoregressive and speculative decoding on dense and MoE models. (a) Autoregressive decoding on a dense model. (b) SD on a dense model, where draft tokens can be efficiently verified due to parameter reuse across tokens. (c) Autoregressive decoding on a MoE model. (d) Naïve SD on a MoE model, where draft tokens activate different experts, leading to low parameter reuse and high verification cost. (e) S 2 -MoE improves SD on MoE models by reducing redundant verification and improving parameter reuse.

Figure 2. (a) Layer-wise breakdown of verification cost for DeepSeekMoE evaluated in Natural Questions, highlighting redundant expert activation. (b) Distribution of Draft Confidence. (c) Verification Cost Distribution Across Confidence Levels. (d) Accuracy of draft expert routing in predicting target routing under different draft depths. (e) Expert reuse ratio across layers under different decoding intervals. (f) Distribution of top routing scores, revealing expert redundancy. (g) KL divergence between draft and target models over decoding steps w/ and w/o shared context.

Figure 3. S 2 -MoE overview.

Figure 4. Routing-aware adaptive expansion reduces verification cost under different acceptance outcomes. (a) Expansion without pruning. (b) Confidence-based pruning. (c) Our routing-aware adaptive expansion.

Figure 5. An illustrative example of reuse-aware expert gating with top-2 routing. Four draft tokens select two experts each from eight experts. Token-level confidence is used to aggregate expert importance and identify a small set of reuse-reward experts (

E_{1}

–

E_{3}

), whose logits are softly biased to improve cross-token expert reuse during verification.

Figure 6. Cost-model calibration against measured verification latency across models and memory budgets.

	Model configuration				SD hyperparameter
Model	Total Params	Activated Params	Total Experts	Experts/Token	Cap	Draft Experts
OLMoE	6.7	1.1	64	8	18	2
DeepSeek	15.3	1.35	64	6	16	1
Qwen3	30.5	3.3	128	8	14	3
GPT-OSS	116.8	5.7	128	4	6	1

Table 1. Model and SD configurations.

Figure 7. End-to-end speedup of different SD methods across MoE models, tasks, memory budgets on Jetson (16G/32G/64G), and RTX 4090.

Model	Method	GSM8K	HumanEval	ARC-E	ARC-C	Avg.	Gov.	NQA	Qasper	Mean KL $\downarrow$	RMS $\Delta p$ $\downarrow$	Top-1 $\uparrow$	PPL $\downarrow$
		Task accuracy					Long-context			Distribution consistency
OLMoE	Original	35.29	22.56	55.81	43.86	39.38	24.69	8.10	21.52	0	0	100	1.000
	S 2 -MoE	34.49	24.39	55.71	43.86	39.61	24.14	7.90	23.30	0.048	5.20	89.89	1.012
DeepSeek	Original	66.64	48.98	63.06	56.03	58.68	30.30	11.86	27.40	0	0	100	1.000
	S 2 -MoE	67.27	50.20	66.91	58.56	60.74	29.20	13.13	32.72	0.065	6.95	89.13	1.013

Table 2. Quality comparison across MoE models: task accuracy, long-context quality, and distribution consistency.

Figure 8. Ablation study on different models, showing the progressive contribution of each component in S 2 -MoE. CA: context-aligned self-speculative decoding; AE: adaptive expansion; RG: reuse-aware gating.

Model	Dataset	w/o Context-Aligned	w/ Context-Aligned	Acceptance rate $\uparrow$
DeepSeek	HumanEval	43.01	52.91	+23.02%
	GSM8K	36.08	48.71	+35.01%
OLMoE	HumanEval	32.46	40.31	+24.18%
	GSM8K	33.01	59.09	+79.01%

Table 3. Effect of context-aligned (CA) self-speculative decoding on acceptance rate (Acc.). Higher values indicate better draft fidelity.

Figure 9. Comparison of different speculative expansion strategies across datasets. We report the acceptance length and the effective verification cost, defined as the fraction of expert activations contributing to accepted tokens.

Dataset	w/o RG	w/ RG	Reuse Improvement (%)
HumanEval	0.6017	0.4453	25.99
GSM8K	0.5893	0.3943	33.09

Table 4. Effect of reuse-aware expert gating on expert parameter reuse. Expert reuse is measured using the reuse ratio defined in Challenge 2 (Sec. 3 ).

Figure 10. Sensitivity to key hyperparameters in S 2 -MoE.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Table 15

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,084 papers

SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding
Apr 11, 2026Jehyeon Bang, Eunyeong Cho, Ranggi Hwang +2Mixture-of-Experts Inference LLM Inference Acceleration
KAIST · UNIST
MoE-Spec: Expert Budgeting for Efficient Speculative Decoding
Feb 17, 2026Bradley McDanel, Steven Li, Sruthikesh Surineni +1Speculative Decoding LLM Inference Acceleration
Franklin and Marshall College · Meta Reality Labs
EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding
Aug 5, 2026Sangwoo Ha, Hyunwoo Seo, Yurim Jo +2AI Accelerator Inference On-Device Language Model Inference
KAIST Daejeon, South Korea

Apr 11, 2026J/K move · Enter open · S save

SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding

Jehyeon Bang, Eunyeong Cho, Ranggi Hwang, Jinha Chung+1

KAIST · UNIST

Mixture-of-Experts Inference LLM Inference Acceleration Memory-Efficient Inference Speculative Decoding

Open paper AskPDF arXiv

The Mixture-of-Experts (MoE) architecture has emerged as a promising approach to mitigate the rising computational costs of large language models (LLMs) by selectively activating parameters. However, its high memory requirements and sub-optimal parameter efficiency pose significant challenges for efficient deployment. Although CPU-offloaded MoE inference systems have been proposed in the literature, they offer limited efficiency, particularly for large batch sizes. In this work, we propose SpecMoE, a memory-efficient MoE inference system based on our self-assisted speculative decoding algorithm. SpecMoE demonstrates the effectiveness of applying speculative decoding to MoE inference without requiring additional model training or fine-tuning. Our system improves inference throughput by up to

4.30\times

, while significantly reducing bandwidth requirements of both memory and interconnect on memory-constrained systems.

Abstract

LLM Inference Acceleration

Explore similar work

SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding

MoE-Spec: Expert Budgeting for Efficient Speculative Decoding

EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding