S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices
Authors: Haochen Huang, Shengxuan Qiu, Meng Li
Organizations: Institute for Artificial Intelligence, Peking University, Beijing, China · School of Integrated Circuits, Peking University, Beijing, China · School of Electronics Engineering and Computer Science, Peking University, Beijing, China
Deploying large language models (LLMs) for inference on edge devices is challenging due to severe memory and bandwidth constraints. While speculative decoding and Mixture-of-Experts (MoE) have been proposed to improve inference efficiency, naively combining them often incurs excessive verification overhead and poor expert reuse, limiting their effectiveness in memory-bound edge settings. In this work, we propose S2-MoE, an efficient self-speculative decoding framework for MoE inference on edge devices. S2-MoE reduces redundant verification through routing-aware adaptive speculative expansion, improves verification efficiency with reuse-aware expert gating, and aligns draft and target execution via shared context. Implemented in llama.cpp, S2-MoE achieves up to 5.3× speedup (about 2.0× on average) over standard autoregressive decoding across diverse MoE models and datasets on edge devices. Code is available at https://github.com/angerybob/S2-MoE.
Figures & tables
Figure 1. Comparison of autoregressive and speculative decoding on dense and MoE models. (a) Autoregressive decoding on a dense model. (b) SD on a dense model, where draft tokens can be efficiently verified due to parameter reuse across tokens. (c) Autoregressive decoding on a MoE model. (d) Naïve SD on a MoE model, where draft tokens activate different experts, leading to low parameter reuse and high verification cost. (e) S 2 -MoE improves SD on MoE models by reducing redundant verification and improving parameter reuse.
Figure 2. (a) Layer-wise breakdown of verification cost for DeepSeekMoE evaluated in Natural Questions, highlighting redundant expert activation. (b) Distribution of Draft Confidence. (c) Verification Cost Distribution Across Confidence Levels. (d) Accuracy of draft expert routing in predicting target routing under different draft depths. (e) Expert reuse ratio across layers under different decoding intervals. (f) Distribution of top routing scores, revealing expert redundancy. (g) KL divergence between draft and target models over decoding steps w/ and w/o shared context.
Figure 3. S 2 -MoE overview.
Figure 4. Routing-aware adaptive expansion reduces verification cost under different acceptance outcomes. (a) Expansion without pruning. (b) Confidence-based pruning. (c) Our routing-aware adaptive expansion.
Figure 5. An illustrative example of reuse-aware expert gating with top-2 routing. Four draft tokens select two experts each from eight experts. Token-level confidence is used to aggregate expert importance and identify a small set of reuse-reward experts ( E1 – E3 ), whose logits are softly biased to improve cross-token expert reuse during verification.
Figure 6. Cost-model calibration against measured verification latency across models and memory budgets.
Model configuration
SD hyperparameter
Model
Total Params
Activated Params
Total Experts
Experts/Token
Cap
Draft Experts
OLMoE
6.7
1.1
64
8
18
2
DeepSeek
15.3
1.35
64
6
16
1
Qwen3
30.5
3.3
128
8
14
3
GPT-OSS
116.8
5.7
128
4
6
1
Table 1. Model and SD configurations.
Figure 7. End-to-end speedup of different SD methods across MoE models, tasks, memory budgets on Jetson (16G/32G/64G), and RTX 4090.
Task accuracy
Long-context
Distribution consistency
Model
Method
GSM8K
HumanEval
ARC-E
ARC-C
Avg.
Gov.
NQA
Qasper
Mean KL ↓
RMS Δp↓
Top-1 ↑
PPL ↓
OLMoE
Original
35.29
22.56
55.81
43.86
39.38
24.69
8.10
21.52
0
0
100
1.000
S 2 -MoE
34.49
24.39
55.71
43.86
39.61
24.14
7.90
23.30
0.048
5.20
89.89
1.012
DeepSeek
Original
66.64
48.98
63.06
56.03
58.68
30.30
11.86
27.40
0
0
100
1.000
S 2 -MoE
67.27
50.20
66.91
58.56
60.74
29.20
13.13
32.72
0.065
6.95
89.13
1.013
Table 2. Quality comparison across MoE models: task accuracy, long-context quality, and distribution consistency.
Figure 8. Ablation study on different models, showing the progressive contribution of each component in S 2 -MoE. CA: context-aligned self-speculative decoding; AE: adaptive expansion; RG: reuse-aware gating.
Model
Dataset
w/o Context-Aligned
w/ Context-Aligned
Acceptance rate ↑
DeepSeek
HumanEval
43.01
52.91
+23.02%
GSM8K
36.08
48.71
+35.01%
OLMoE
HumanEval
32.46
40.31
+24.18%
GSM8K
33.01
59.09
+79.01%
Table 3. Effect of context-aligned (CA) self-speculative decoding on acceptance rate (Acc.). Higher values indicate better draft fidelity.
Figure 9. Comparison of different speculative expansion strategies across datasets. We report the acceptance length and the effective verification cost, defined as the fraction of expert activations contributing to accepted tokens.
Dataset
w/o RG
w/ RG
Reuse Improvement (%)
HumanEval
0.6017
0.4453
25.99
GSM8K
0.5893
0.3943
33.09
Table 4. Effect of reuse-aware expert gating on expert parameter reuse. Expert reuse is measured using the reuse ratio defined in Challenge 2 (Sec. 3 ).
Figure 10. Sensitivity to key hyperparameters in S 2 -MoE.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
The Mixture-of-Experts (MoE) architecture has emerged as a promising approach to mitigate the rising computational costs of large language models (LLMs) by selectively activating parameters. However, its high memory requirements and sub-optimal parameter efficiency pose significant challenges for efficient deployment. Although CPU-offloaded MoE inference systems have been proposed in the literature, they offer limited efficiency, particularly for large batch sizes. In this work, we propose SpecMoE, a memory-efficient MoE inference system based on our self-assisted speculative decoding algorithm. SpecMoE demonstrates the effectiveness of applying speculative decoding to MoE inference without requiring additional model training or fine-tuning. Our system improves inference throughput by up to 4.30×, while significantly reducing bandwidth requirements of both memory and interconnect on memory-constrained systems.
Speculative decoding accelerates Large Language Model (LLM) inference by verifying multiple drafted tokens in parallel. However, for Mixture-of-Experts (MoE) models, this parallelism introduces a severe bottleneck: large draft trees activate many unique experts, significantly increasing memory pressure and diminishing speedups from speculative decoding relative to autoregressive decoding. Prior methods reduce speculation depth when MoE verification becomes expensive. We propose MoE-Spec, a training-free verification-time expert budgeting method that decouples speculation depth from memory cost by enforcing a fixed expert capacity limit at each layer, loading only the experts that contribute most to verification and dropping the long tail of rarely used experts that drive bandwidth overhead. Experiments across multiple model scales and datasets show that this method yields 10--30% higher throughput than state-of-the-art speculative decoding baselines (EAGLE-3) at comparable quality, with flexibility to trade accuracy for further latency reductions through tighter budgets.
Bradley McDanel, Steven Li, Sruthikesh Surineni +1
On-device deployment of Large Language Models (LLMs) has become essential for personalized edge applications. A primary bottleneck is external memory access (EMA) in feed-forward network (FFN) layers. Speculative decoding and mixture-of-experts (MoE) are promising solutions. Speculative decoding reduces the number of decoding stages by generating multiple tokens per stage, and MoE minimizes per-stage cost through sparse expert activation. However, there is an incompatibility when combining these two techniques. We propose EdgeXpert, a software-hardware co-designed LLM accelerator that resolves this incompatibility. In the prefill stage, the prompt-wise expert reuse reformulates routing as prompt-level expert reuse rather than independent per-token expert selection. It identifies important tokens using a lightweight encoder, constructs a shared expert set from them, and routes less important tokens with a reduced expert budget to lower expert EMA. In the decode stage, depth-aware expert coalescing exploits the contextual similarity and mutual exclusivity of same-depth candidate tokens. Rather than loading the union of all required channels, EdgeXpert loads only salient channels and applies computational calibration to recover accuracy without additional memory access. Synthesized in Samsung 28nm technology at 800 MHz, EdgeXpert achieves up to 56.3% latency reduction and 44.1% energy reduction compared to prior works, while maintaining near-baseline accuracy.