Speculative Decoding (SD) accelerates Large Language Model (LLM) inference by employing a lightweight draft model to propose candidate tokens, which are verified in parallel by the target model, without compromising generation quality. While Retrieval-based Speculative Decoding (RSD) is favored for its plug-and-play versatility, its potential is impeded by rigid lexical dependencies, rendering both retrieval and verification brittle to surface-level variations. To address this, we propose SENSE (Semantic Embedding Navigation with Soft-gated Evaluation). By anchoring retrieval on the hidden states of the target model, SENSE establishes robust semantic alignment, which empowers the Soft-gated Evaluation module to validate semantic equivalence rather than surface forms. To ensure rigorous benchmarking, we deconstruct existing methods into atomic primitives within a unified framework, facilitating granular, component-level comparison. Extensive experiments across diverse domains demonstrate that SENSE outperforms multiple baselines on the LLaMA and Qwen families, attaining up to 4.09 mean acceptance length and 3.26x speedup, while preserving generation quality. Our code will be released upon publication.
Figures & tables
Figure 1: (a) Vanilla RSD retrieves tokens using an exact match rule based on token spans. This approach suffers from lexical limitations, as it strictly requires the retrieved drafts to exactly match the next token ”employs”. (b) SENSE utilizes hidden states to perform semantic alignment retrieval. This allows it to retrieve semantically similar drafts and verify them through a soft-gated match mechanism, effectively overcoming rigid lexical constraints.
Figure 2: Overview of the SENSE framework. (Left) The Semantic Embedding Navigation module retrieves semantically aligned drafts based on the input’s hidden states. These drafts are flattened into a losse trie structure for efficient verification by the target model. (Right) The Soft-gated Evaluation module filters the verification results using a cascade of masks to robustly determine the final accepted tokens.
Figure 3: The atomic framework for SD. We decompose the inference pipeline into modular drafting and verification primitives, orchestrated by a unified connector to enable seamless integration and standardized comparison of diverse methodologies.
Model
Method
GSM8K
CodeAlapca
UltraChat
TriviaQA
Mean
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Llama2-7B
SpS
1.94
2.73
1.50
2.15
1.47
2.15
1.23
1.65
1.53
2.17
PLD
4.18
3.64
2.73
2.13
2.91
2.46
3.18
6.12
3.25
3.59
REST
2.00
1.43
2.58
1.88
2.27
1.88
2.00
1.83
2.21
1.76
DReSD
1.37
1.38
1.87
2.47
1.07
1.58
1.25
1.64
1.39
1.77
SENSE(OOD)
6.44
6.69
7.88
8.36
4.77
5.50
8.23
8.61
6.83
7.29
Table 1: Training-free method comparison reporting Speedup Ratio and Mean Accepted Tokens ( τ ). Vanilla LLM serves as the 1.0 baseline. Bold and underlined text denote the best and second-best results, respectively.
Model
Method
GSM8K
CodeAlpaca
Mean
Speedup
τ
Speedup
τ
Speedup
τ
Qwen2.5-14B
EAGLE-2
2.19
2.43
1.49
2.94
1.84
2.69
SENSE(ID)
3.00
3.48
2.48
2.90
2.74
3.19
Qwen3-8B
EAGLE-3
1.71
2.00
1.77
2.02
1.74
2.01
SENSE(ID)
2.48
2.63
2.75
3.02
2.61
2.83
Qwen3-14B
EAGLE-3
1.65
1.87
1.51
1.78
1.58
1.83
Table 2: Performance comparison with training-based methods. Bold indicates the best performance in each column.
Figure 4: Verification method comparison. Upper: speedup; Lower: τ . Our method outperforms baselines on most models.
Figure 5: Generation quality preservation. SENSE achieves comparable performance to the Vanilla across diverse domains.
Figure 6: Hyperparameter sensitivity analysis on Qwen2.5-7B (GSM8K). Left: Accuracy (higher is better); Right: Speedup (higher is faster). Each cell shows the mean performance across window sizes w∈{5,6,7,8,9} . The highlighted cell indicates our default configuration ( θe=0.05 , k=3 ), which achieves 98% accuracy at 2.8 × speedup.
Figure 7: Latency Decomposition. Average latency per decoding step, decomposed into draft generation (upper) and verification (lower). Top annotations indicate total latency. A symmetric log scale is used to accommodate the significant magnitude disparity.
Datastore
Method
GSM8K
CodeAlpaca
UltraChat
TriviaQA
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
OOD
SENSE
1.58
1.70
2.44
3.13
1.87
2.19
1.39
1.36
w/o SEN
1.14
1.35
2.16
2.34
1.36
1.41
1.24
1.24
w/o SE
1.05
1.31
2.28
2.41
1.56
1.64
1.12
1.35
ID
SENSE
2.48
2.64
2.44
3.13
2.07
1.18
2.22
2.12
w/o SEN
2.07
2.20
2.16
2.34
1.86
1.70
1.97
1.84
Table 3: Component-wise ablation study of SENSE using Qwen3-8B across four datasets. We evaluate the impact of the SEN and SE modules on Speedup and Mean Accepted Tokens ( τ ).
Figure 8: Compression efficiency analysis of the Losse Trie structure across diverse model scales and benchmarks.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Method
GSM8K
CodeAlpaca
UltraChat
TriviaQA
Mean
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Qwen2.5-7B
ARC
2.51
3.09
2.87
3.71
2.14
2.71
1.97
2.42
2.37
2.98
FLY
2.43
3.02
2.83
3.71
2.02
2.57
2.32
2.89
2.40
3.05
Ours
2.94
3.44
3.30
3.95
2.59
3.04
2.72
3.63
2.89
3.52
Qwen2.5-14B
ARC
2.66
3.26
2.49
3.12
1.89
1.33
1.85
2.28
2.22
2.50
FLY
2.51
3.10
2.49
3.14
1.77
2.20
1.87
2.32
2.16
2.69
Appendix
Table 4: Complete performance comparison of verification strategies. We fix the SEN retrieval module and compare different verification methods: ARC, FLY, and our Soft-gated Evaluation (SE). Bold indicates the best performance in each row.
k=2
k=3
k=4
k=10
θe
w
Acc
SpeedUp
τ
Acc
SpeedUp
τ
Acc
SpeedUp
τ
Acc
SpeedUp
τ
0.01
5
0.89
3.28
3.93
0.76
5.02
5.70
0.67
5.95
6.50
0.15
8.12
8.28
6
0.90
3.25
3.85
0.82
4.30
4.99
0.77
5.59
6.14
0.12
8.33
8.46
7
0.90
3.07
3.61
0.89
3.89
4.53
0.73
5.09
5.64
0.16
8.16
8.35
8
0.92
3.00
3.55
0.88
3.93
4.57
0.71
5.23
5.79
0.18
8.12
8.31
9
0.93
2.95
3.51
0.88
3.96
4.59
0.72
5.60
6.15
0.18
8.10
8.27
Appendix
Table 5: Comprehensive hyperparameter study on Qwen2.5-7B (GSM8K, ID datastore). We report Accuracy (%), Speedup ratio, and mean accepted tokens ( τ ) across 80 configurations of entropy threshold θe , Top- k , and window size w . Highlighted cells indicate our default configuration ( θe=0.05 , k=3 , w=6 ).
Model
Method
GSM8K
CodeAlpaca
UltraChat
TriviaQA
ACC
ROUGE-L
ROUGE-L
ACC
Qwen3-8B
Vanilla
0.93
0.427
0.206
0.45
SENSE-OOD
0.91
0.443
0.189
0.43
SENSE-ID
0.90
0.435
0.188
0.42
Qwen3-14B
Vanilla
0.92
0.294
0.204
0.56
SENSE-OOD
0.92
0.288
0.205
0.56
Appendix
Table 6: Accuracy preservation analysis across models and datasets. We compare vanilla decoding (baseline) against SENSE-OOD and SENSE-ID configurations. GSM8K and TriviaQA report accuracy (ACC); CodeAlpaca and UltraChat report ROUGE-L scores. Bold indicates highest value per model-dataset pair.
Figure 9: Case Study Prompts
Figure 10: Case Study Outputs
Figure 11: Failure Case Prompts
Figure 12: Failure Case Outputs
Metric
Mean
Min
Max
Std
Total RAM Memory
1770.06
105.63
3690.47
1221.90
Index Size on Disk
459.73
0.00
1087.39
399.87
Metadata Memory
1715.97
73.55
3610.37
1214.92
Cache Storage
54.03
28.02
80.03
22.10
Peak RSS Memory
4577.93
2131.49
7412.49
1630.39
RSS Delta Mean
7.25
1.61
24.15
6.77
Appendix
Table 7: Overall datastore resource statistics across all 24 configurations (6 models × 4 test sets). All memory values are in megabytes (MB).
Model
GSM8K
CodeAlpaca
TriviaQA
UltraChat
Mean
Llama2-7B
105.63
807.74
1796.68
2647.28
1339.33
Llama2-13B
320.54
749.01
593.77
778.22
610.38
Qwen2.5-7B
268.19
2839.18
1436.67
3277.00
1955.26
Qwen2.5-14B
325.25
3690.47
3114.02
3048.90
2544.66
Qwen3-8B
395.49
2575.56
3308.63
2584.98
2216.16
Qwen3-14B
406.60
1945.01
2890.36
2576.31
1954.57
Appendix
Table 8: Total RAM memory usage (MB) by model and dataset. Bold indicates lowest (best) value in each column.
Model
RSS Mean
Baseline RSS
Peak RSS
Peak RSS Max
Delta RSS Mean
Delta RSS Max
Avg RSS
Llama2-7B
3746.68
3726.80
3746.68
3748.57
19.89
718.29
3736.74
Llama2-13B
2757.59
2750.71
2757.59
2760.38
6.88
679.19
2754.15
Qwen2.5-7B
4609.42
4602.70
4609.42
4612.28
6.72
668.12
4606.06
Qwen2.5-14B
6122.53
6119.77
6122.53
6125.51
2.76
271.24
6121.15
Qwen3-8B
4997.82
4993.63
4997.82
4998.78
4.20
417.65
4995.73
Qwen3-14B
5233.50
5230.46
5233.50
5241.10
3.05
293.41
5231.98
Appendix
Table 9: Detailed RSS (Resident Set Size) memory metrics by model, averaged across all four datasets. Baseline RSS is measured before datastore loading. Delta RSS quantifies the memory overhead introduced by SENSE. All values are in megabytes (MB). Bold indicates lowest (best) value in each column.
Dataset
Template
Prompt
GSM8K
Math
Solve the following problem step by step. Problem: {instruction} Think through the problem carefully, then provide your final answer. Your response MUST end with “Final Answer: answer” on a new line, where answer is the final result.
TriviaQA
QA
Solve the following problem step by step. Problem: {instruction} Think through the problem carefully, then provide your final answer. Your response MUST end with “Final Answer: answer” on a new line, where answer is the final result.
CodeAlpaca
Alpaca
You are a skilled programmer. Complete the following coding task. ### Task: {instruction} ### Solution: Provide only the code implementation. Do not include explanations unless specifically requested.
UltraChat
Chat
{instruction}
Appendix
Table 10: Unified prompt templates used for both datastore construction and inference.
Implementation Details
Method
Dataset
Source Data
Test Data
REST
GSM8K
Shuffle and sample data from Train spilt of GSM8K.
Smaple test spilt of GSM8K.
CodeAlpaca
Shuffle and sample data from Train spilt of CodeAlpaca_20k.
Smaple test spilt of CodeAlpaca_20k.
UltraChat
Shuffle and sample data from train_sft spilt of ultrachat_200k.
Smaple test_sft spilt of ultrachat_200k.
TriviaQA
Shuffle and sample data from Train spilt of TriviaQA.
Smaple validation split of TriviaQA.
DReSD
GSM8K
Shuffle and sample data from Train spilt of GSM8K.
Smaple test spilt of GSM8K.
Appendix
Table 11: Implementation Details for Different Benchmarks.
Speculative decoding (SD) accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens and a stronger target model to verify them. However, standard SD is mainly designed for acceleration, and its output quality is typically constrained by the target model. In this work, we propose Entropy-Aware Speculative Decoding (EASD), a lightweight and training-free extension of SD that improves reasoning quality through token-level entropy-guided rejection. EASD detects cases where both draft and target models exhibit high uncertainty while strongly overlapping in their top predictions. In such uncertain-agreement cases, EASD rejects the aligned token and resamples from the target distribution, preventing low-confidence errors from propagating. Experiments on challenging reasoning benchmarks show that EASD consistently improves accuracy over standard SD and reward-guided variants while maintaining comparable inference efficiency. Notably, EASD can surpass the standalone performance of the target model, suggesting that speculative decoding can serve not only as an acceleration method but also as an effective mechanism for improving reasoning quality. The code is available at https://github.com/ECNU-Text-Computing/EASD.
Tiancheng Su, Meicong Zhang, Guoxiu He
School of Economics and Management East China Normal University
Large Language Models (LLMs) achieve strong performance across many tasks but suffer from high inference latency due to autoregressive decoding. The issue is exacerbated in Large Reasoning Models (LRMs), which generate lengthy chains of thought. While speculative decoding accelerates inference by drafting and verifying multiple tokens in parallel, existing methods operate at the token level and ignore semantic equivalence (i.e., different token sequences expressing the same meaning), leading to inefficient rejections. We propose SemanticSpec, a semantic-aware speculative decoding framework that verifies entire semantic sequences instead of tokens. SemanticSpec introduces a semantic probability estimation mechanism that probes the model's internal hidden states to assess the likelihood of generating sequences with specific meanings. Experiments on four benchmarks show that SemanticSpec achieves up to 2.7x speedup on DeepSeekR1-32B and 2.1x on QwQ-32B, consistently outperforming token-level and sequence-level baselines in both efficiency and effectiveness.
Ximing Dong, Shaowei Wang, Dayi Lin +2
Centre for Software Excellence, Huawei, Canada · Department of Computer Science, University of Manitoba, Canada · School of Computing, Queen’s University, Canada
Speculative decoding accelerates large language model inference by employing a compact draft model to propose candidate tokens that are verified in parallel by a target model. Conventional training relies on soft-label distillation, which forces the draft model to mimic the target's full output distribution, including high-entropy regions that degrade acceptance rates and increase early rejection risk. We propose RCSD (Reinforcement and Continual Learning for Speculative Decoding), a training framework that aligns draft model optimization with the system-level objective of maximizing acceptance length. RCSD leverages naturally occurring accepted and rejected sequences during speculative inference to construct implicit preference pairs, with masked loss to avoid penalizing target-aligned tokens. Evaluated across multiple benchmarks, RCSD improves average acceptance length by 35% over HASS and achieves 1.35x faster end-to-end inference than HASS on practical workloads, without compromising generation quality.