SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decoding

Shaowen Chen; Zhicheng Liao; Hongwei Wang

cs.CLApr 14, 2026

SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decoding

Authors: Shaowen Chen, Zhicheng Liao, Hongwei Wang

Organizations: Zhejiang University, Hangzhou, China.

Abstract

Speculative Decoding (SD) accelerates Large Language Model (LLM) inference by employing a lightweight draft model to propose candidate tokens, which are verified in parallel by the target model, without compromising generation quality. While Retrieval-based Speculative Decoding (RSD) is favored for its plug-and-play versatility, its potential is impeded by rigid lexical dependencies, rendering both retrieval and verification brittle to surface-level variations. To address this, we propose SENSE (Semantic Embedding Navigation with Soft-gated Evaluation). By anchoring retrieval on the hidden states of the target model, SENSE establishes robust semantic alignment, which empowers the Soft-gated Evaluation module to validate semantic equivalence rather than surface forms. To ensure rigorous benchmarking, we deconstruct existing methods into atomic primitives within a unified framework, facilitating granular, component-level comparison. Extensive experiments across diverse domains demonstrate that SENSE outperforms multiple baselines on the LLaMA and Qwen families, attaining up to 4.09 mean acceptance length and 3.26x speedup, while preserving generation quality. Our code will be released upon publication.

Figures & tables

Figure 1: (a) Vanilla RSD retrieves tokens using an exact match rule based on token spans. This approach suffers from lexical limitations, as it strictly requires the retrieved drafts to exactly match the next token ”employs”. (b) SENSE utilizes hidden states to perform semantic alignment retrieval. This allows it to retrieve semantically similar drafts and verify them through a soft-gated match mechanism, effectively overcoming rigid lexical constraints.

Figure 2: Overview of the SENSE framework. (Left) The Semantic Embedding Navigation module retrieves semantically aligned drafts based on the input’s hidden states. These drafts are flattened into a losse trie structure for efficient verification by the target model. (Right) The Soft-gated Evaluation module filters the verification results using a cascade of masks to robustly determine the final accepted tokens.

Figure 3: The atomic framework for SD. We decompose the inference pipeline into modular drafting and verification primitives, orchestrated by a unified connector to enable seamless integration and standardized comparison of diverse methodologies.

Model	Method	GSM8K		CodeAlapca		UltraChat		TriviaQA		Mean
Model	Method	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$
Llama2-7B	SpS	1.94	2.73	1.50	2.15	1.47	2.15	1.23	1.65	1.53	2.17
	PLD	4.18	3.64	2.73	2.13	2.91	2.46	3.18	6.12	3.25	3.59
	REST	2.00	1.43	2.58	1.88	2.27	1.88	2.00	1.83	2.21	1.76
	DReSD	1.37	1.38	1.87	2.47	1.07	1.58	1.25	1.64	1.39	1.77
	SENSE(OOD)	6.44	6.69	7.88	8.36	4.77	5.50	8.23	8.61	6.83	7.29

Table 1: Training-free method comparison reporting Speedup Ratio and Mean Accepted Tokens (

\tau

). Vanilla LLM serves as the 1.0 baseline. Bold and underlined text denote the best and second-best results, respectively.

Model	Method	GSM8K		CodeAlpaca		Mean
Model	Method	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$
Qwen2.5-14B	EAGLE-2	2.19	2.43	1.49	2.94	1.84	2.69
Qwen2.5-14B	SENSE(ID)	3.00	3.48	2.48	2.90	2.74	3.19
Qwen3-8B	EAGLE-3	1.71	2.00	1.77	2.02	1.74	2.01
Qwen3-8B	SENSE(ID)	2.48	2.63	2.75	3.02	2.61	2.83
Qwen3-14B	EAGLE-3	1.65	1.87	1.51	1.78	1.58	1.83

Table 2: Performance comparison with training-based methods. Bold indicates the best performance in each column.

Figure 4: Verification method comparison. Upper: speedup; Lower:

\tau

. Our method outperforms baselines on most models.

Figure 5: Generation quality preservation. SENSE achieves comparable performance to the Vanilla across diverse domains.

Figure 6: Hyperparameter sensitivity analysis on Qwen2.5-7B (GSM8K). Left: Accuracy (higher is better); Right: Speedup (higher is faster). Each cell shows the mean performance across window sizes $w\in\{5,6,7,8,9\}$ . The highlighted cell indicates our default configuration ( $\theta_{e}{=}0.05$ , $k{=}3$ ), which achieves 98% accuracy at 2.8 $\times$ speedup.

Figure 7: Latency Decomposition. Average latency per decoding step, decomposed into draft generation (upper) and verification (lower). Top annotations indicate total latency. A symmetric log scale is used to accommodate the significant magnitude disparity.

Datastore	Method	GSM8K		CodeAlpaca		UltraChat		TriviaQA
Datastore	Method	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$
OOD	SENSE	1.58	1.70	2.44	3.13	1.87	2.19	1.39	1.36
	w/o SEN	1.14	1.35	2.16	2.34	1.36	1.41	1.24	1.24
	w/o SE	1.05	1.31	2.28	2.41	1.56	1.64	1.12	1.35
ID	SENSE	2.48	2.64	2.44	3.13	2.07	1.18	2.22	2.12
	w/o SEN	2.07	2.20	2.16	2.34	1.86	1.70	1.97	1.84

Table 3: Component-wise ablation study of SENSE using Qwen3-8B across four datasets. We evaluate the impact of the SEN and SE modules on Speedup and Mean Accepted Tokens (

\tau

Figure 8: Compression efficiency analysis of the Losse Trie structure across diverse model scales and benchmarks.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	GSM8K		CodeAlpaca		UltraChat		TriviaQA		Mean
Model	Method	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$
Qwen2.5-7B	ARC	2.51	3.09	2.87	3.71	2.14	2.71	1.97	2.42	2.37	2.98
	FLY	2.43	3.02	2.83	3.71	2.02	2.57	2.32	2.89	2.40	3.05
	Ours	2.94	3.44	3.30	3.95	2.59	3.04	2.72	3.63	2.89	3.52
Qwen2.5-14B	ARC	2.66	3.26	2.49	3.12	1.89	1.33	1.85	2.28	2.22	2.50
	FLY	2.51	3.10	2.49	3.14	1.77	2.20	1.87	2.32	2.16	2.69

Appendix

Table 4: Complete performance comparison of verification strategies. We fix the SEN retrieval module and compare different verification methods: ARC, FLY, and our Soft-gated Evaluation (SE). Bold indicates the best performance in each row.

		$k=2$			$k=3$			$k=4$			$k=10$
$\theta_{e}$	$w$	Acc	SpeedUp	$\tau$	Acc	SpeedUp	$\tau$	Acc	SpeedUp	$\tau$	Acc	SpeedUp	$\tau$
0.01	5	0.89	3.28	3.93	0.76	5.02	5.70	0.67	5.95	6.50	0.15	8.12	8.28
	6	0.90	3.25	3.85	0.82	4.30	4.99	0.77	5.59	6.14	0.12	8.33	8.46
	7	0.90	3.07	3.61	0.89	3.89	4.53	0.73	5.09	5.64	0.16	8.16	8.35
	8	0.92	3.00	3.55	0.88	3.93	4.57	0.71	5.23	5.79	0.18	8.12	8.31
	9	0.93	2.95	3.51	0.88	3.96	4.59	0.72	5.60	6.15	0.18	8.10	8.27

Appendix

Table 5: Comprehensive hyperparameter study on Qwen2.5-7B (GSM8K, ID datastore). We report Accuracy (%), Speedup ratio, and mean accepted tokens (

\tau

) across 80 configurations of entropy threshold

\theta_{e}

, Top-

k

, and window size

w

. Highlighted cells indicate our default configuration (

\theta_{e}{=}0.05

k{=}3

w{=}6

Model	Method	GSM8K	CodeAlpaca	UltraChat	TriviaQA
Model	Method	ACC	ROUGE-L	ROUGE-L	ACC
Qwen3-8B	Vanilla	0.93	0.427	0.206	0.45
	SENSE-OOD	0.91	0.443	0.189	0.43
	SENSE-ID	0.90	0.435	0.188	0.42
Qwen3-14B	Vanilla	0.92	0.294	0.204	0.56
	SENSE-OOD	0.92	0.288	0.205	0.56

Appendix

Table 6: Accuracy preservation analysis across models and datasets. We compare vanilla decoding (baseline) against SENSE-OOD and SENSE-ID configurations. GSM8K and TriviaQA report accuracy (ACC); CodeAlpaca and UltraChat report ROUGE-L scores. Bold indicates highest value per model-dataset pair.

Metric	Mean	Min	Max	Std
Total RAM Memory	1770.06	105.63	3690.47	1221.90
Index Size on Disk	459.73	0.00	1087.39	399.87
Metadata Memory	1715.97	73.55	3610.37	1214.92
Cache Storage	54.03	28.02	80.03	22.10
Peak RSS Memory	4577.93	2131.49	7412.49	1630.39
RSS Delta Mean	7.25	1.61	24.15	6.77

Appendix

Table 7: Overall datastore resource statistics across all 24 configurations (6 models

\times

4 test sets). All memory values are in megabytes (MB).

Model	GSM8K	CodeAlpaca	TriviaQA	UltraChat	Mean
Llama2-7B	105.63	807.74	1796.68	2647.28	1339.33
Llama2-13B	320.54	749.01	593.77	778.22	610.38
Qwen2.5-7B	268.19	2839.18	1436.67	3277.00	1955.26
Qwen2.5-14B	325.25	3690.47	3114.02	3048.90	2544.66
Qwen3-8B	395.49	2575.56	3308.63	2584.98	2216.16
Qwen3-14B	406.60	1945.01	2890.36	2576.31	1954.57

Appendix

Table 8: Total RAM memory usage (MB) by model and dataset. Bold indicates lowest (best) value in each column.

Model	RSS Mean	Baseline RSS	Peak RSS	Peak RSS Max	Delta RSS Mean	Delta RSS Max	Avg RSS
Llama2-7B	3746.68	3726.80	3746.68	3748.57	19.89	718.29	3736.74
Llama2-13B	2757.59	2750.71	2757.59	2760.38	6.88	679.19	2754.15
Qwen2.5-7B	4609.42	4602.70	4609.42	4612.28	6.72	668.12	4606.06
Qwen2.5-14B	6122.53	6119.77	6122.53	6125.51	2.76	271.24	6121.15
Qwen3-8B	4997.82	4993.63	4997.82	4998.78	4.20	417.65	4995.73
Qwen3-14B	5233.50	5230.46	5233.50	5241.10	3.05	293.41	5231.98

Appendix

Table 9: Detailed RSS (Resident Set Size) memory metrics by model, averaged across all four datasets. Baseline RSS is measured before datastore loading. Delta RSS quantifies the memory overhead introduced by SENSE. All values are in megabytes (MB). Bold indicates lowest (best) value in each column.

Dataset	Template	Prompt
GSM8K	Math	Solve the following problem step by step. Problem: {instruction} Think through the problem carefully, then provide your final answer. Your response MUST end with “Final Answer: answer” on a new line, where answer is the final result.
TriviaQA	QA	Solve the following problem step by step. Problem: {instruction} Think through the problem carefully, then provide your final answer. Your response MUST end with “Final Answer: answer” on a new line, where answer is the final result.
CodeAlpaca	Alpaca	You are a skilled programmer. Complete the following coding task. ### Task: {instruction} ### Solution: Provide only the code implementation. Do not include explanations unless specifically requested.
UltraChat	Chat	{instruction}

Appendix

Table 10: Unified prompt templates used for both datastore construction and inference.

		Implementation Details
Method	Dataset	Source Data	Test Data
REST	GSM8K	Shuffle and sample data from Train spilt of GSM8K.	Smaple test spilt of GSM8K.
	CodeAlpaca	Shuffle and sample data from Train spilt of CodeAlpaca_20k.	Smaple test spilt of CodeAlpaca_20k.
	UltraChat	Shuffle and sample data from train_sft spilt of ultrachat_200k.	Smaple test_sft spilt of ultrachat_200k.
	TriviaQA	Shuffle and sample data from Train spilt of TriviaQA.	Smaple validation split of TriviaQA.
DReSD	GSM8K	Shuffle and sample data from Train spilt of GSM8K.	Smaple test spilt of GSM8K.

Appendix

Table 11: Implementation Details for Different Benchmarks.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 309 papers

Entropy-Aware Token Rejection for Improving Speculative Decoding
Dec 29, 2025Tiancheng Su, Meicong Zhang, Guoxiu HeSpeculative Decoding LLM Inference Acceleration
School of Economics and Management East China Normal University
Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States
Feb 3, 2026Ximing Dong, Shaowei Wang, Dayi Lin +2Speculative Decoding LLM Inference Acceleration
Centre for Software Excellence, Huawei, Canada · Department of Computer Science, University of Manitoba, Canada · School of Computing, Queen’s University, Canada
RCSD: Reinforcement and Continual Learning for Domain-Specific Speculative Decoding
Oct 21, 2026Mingkang Liu, Weiming Meng, Peipei Jiao +4Speculative Decoding LLM Inference Acceleration

Dec 29, 2025J/K move · Enter open · S save

Entropy-Aware Token Rejection for Improving Speculative Decoding

Tiancheng Su, Meicong Zhang, Guoxiu He

School of Economics and Management East China Normal University

Speculative Decoding LLM Inference Acceleration LLM Reasoning

Open paper AskPDF arXiv

Speculative decoding (SD) accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens and a stronger target model to verify them. However, standard SD is mainly designed for acceleration, and its output quality is typically constrained by the target model. In this work, we propose Entropy-Aware Speculative Decoding (EASD), a lightweight and training-free extension of SD that improves reasoning quality through token-level entropy-guided rejection. EASD detects cases where both draft and target models exhibit high uncertainty while strongly overlapping in their top predictions. In such uncertain-agreement cases, EASD rejects the aligned token and resamples from the target distribution, preventing low-confidence errors from propagating. Experiments on challenging reasoning benchmarks show that EASD consistently improves accuracy over standard SD and reward-guided variants while maintaining comparable inference efficiency. Notably, EASD can surpass the standalone performance of the target model, suggesting that speculative decoding can serve not only as an acceleration method but also as an effective mechanism for improving reasoning quality. The code is available at https://github.com/ECNU-Text-Computing/EASD.

Abstract

Speculative Decoding

Explore similar work

Entropy-Aware Token Rejection for Improving Speculative Decoding

Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States

RCSD: Reinforcement and Continual Learning for Domain-Specific Speculative Decoding