DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification

Ziyi Wang; Siva Rajesh Kasa; Ankith M S; Santhosh Kumar Kasa; Jiaru Zou; Sumit Negi; Ruqi Zhang; Nan Jiang

cs.CLApr 8, 2026

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification

Authors: Ziyi Wang, Siva Rajesh Kasa, Ankith M S, Santhosh Kumar Kasa, Jiaru Zou, Sumit Negi, Ruqi Zhang, Nan Jiang,

Organizations: Purdue University · Amazon Inc. · University of Illinois Urbana-Champaign · University of Texas at El Paso

Abstract

Speculative decoding is an effective technique for accelerating large language model inference by drafting multiple tokens in parallel. In practice, its speedup is often bottlenecked by a rigid verification step that strictly enforces the accepted token distribution to exactly match the target model. This constraint leads to the rejection of many plausible tokens, lowering the acceptance rate and limiting overall time speedup. To overcome this limitation, we propose Dynamic Verification Relaxed Speculative Decoding (DIVERSED), a relaxed verification framework that improves time efficiency while preserving generation quality. DIVERSED learns an ensemble-based verifier that blends the draft and target model distributions with a task-dependent and context-dependent weight. We provide theoretical justification for our approach and demonstrate empirically that DIVERSED achieves substantially higher inference efficiency compared to standard speculative decoding methods. Code is available at: https://github.com/comeusr/diversed.

Figures & tables

Figure 1: Our method improves upon the Pareto frontier achieved by static ensemble, attaining better trade-offs between inference time and generation quality. Results are reported on the CNNDM dataset using the target/draft model pair: Llama-3.1-8B-Instruct and Llama-3.2-1B-Instruct, with temperature being

0

. More results are in Appendix Figure 10 .

Figure 2: Compared with classic speculative decoding (shown in (a) rigid verification), our Diversed (c) achieves a higher acceptance rate, comparable accuracy, and lower wall-clock time. Compared with static ensemble (b), Diversed (c) attains higher accuracy via a tunable ensemble weight that adapts to the task and the context.

Figure 4: Normalized total time (lower is better) versus acceptance rate (higher is better) under speculative decoding with draft length

N{=}5

. Model pairs with higher acceptance achieve lower decoding time. Results on the CNNDM, XSUM, MBPP, and GSM8K datasets for three target/draft pairs. Additional results for draft lengths

N{=}3

and

N{=}7

are provided in Appendix Figures 8 and 9 .

	GSM8K		CNNDM		XSum		MBPP
Method	Accept Rate ( $\uparrow$ )	Quality Accuracy ( $\uparrow$ )	Accept Rate( $\uparrow$ )	Quality ROUGE-2 ( $\uparrow$ )	Accept Rate( $\uparrow$ )	Quality ROUGE-2 ( $\uparrow$ )	Accept Rate( $\uparrow$ )	Quality pass@1 ( $\uparrow$ )
Autoregressive	NA	67 $\%$	NA	9.86	NA	7.03	NA	53 $\%$
SD	44.60 $\%$	67 $\%$	21.60 $\%$	9.46	20.44 $\%$	7.09	26.30 $\%$	53 $\%$
SD (Lossy)	59.81 $\%$	66 $\%$	38.86 $\%$	10.51	40.91 $\%$	7.96	66.75 $\%$	49 $\%$
SpecCascade	61.53 $\%$	67 $\%$	47.29 $\%$	11.74	43.95 $\%$	7.61	73.92 $\%$	52 $\%$
Static Ensemble	69.49 $\%$	66 $\%$	61.06 $\%$	11.46	51.58 $\%$	7.22	68.70 $\%$	52 $\%$

Table 1: Benchmark acceptance rate and generation quality. Columns group results by task while panels (a)–(c) correspond to different Target / Draft model pairs. All experiments use temperature

1

and draft length

N=5

. Our Diversed achieves a higher acceptance rate and also maintains comparable quality. Additional results for temperatures

0

and

0.1

are reported in Appendix Tables 6 and 7 .

Figure 5: Diversed consistently accepts a larger average number of draft tokens per verification round than all baselines (draft length

N{=}5

, temperature

1

). Appendix Figure 7 shows result for temperature

0.1

Train/Test	Method	CNNDM
Dataset		Accept	ROUGE-2
		Rate ( $\uparrow$ )	( $\uparrow$ )
CNNDM	SD	21.06 $\%$	9.46
CNNDM/CNNDM	Diversed	69.96 $\%$	12.11
GSM8K/CNNDM	Diversed	51.07 $\%$	8.63
(a) Draft/Target: Llama-3.2-1B / Llama-3.1-8B

Table 2: Comparison of finetuned draft vs. Diversed at temperature

1

. Fine-tuned draft model yields higher acceptance rates but lower task performance.

	Temperature	GSM8K		XSum		MBPP
	Temperature	Accept Rate ( $\uparrow$ )	Accuracy ( $\uparrow$ )	Accept Rate ( $\uparrow$ )	ROUGE-2 ( $\uparrow$ )	Accept Rate ( $\uparrow$ )	Pass@1 ( $\uparrow$ )
SD	$1.0$	44.60 $\%$	67 $\%$	20.44 $\%$	7.09	26.30 $\%$	53 $\%$
SD (Finetuned draft)		66.44 $\%$	67 $\%$	35.41 $\%$	6.95	47.07 $\%$	53 $\%$
Diversed (Ours)		72.61 $\%$	67 $\%$	70.53 $\%$	7.23	85.03 $\%$	53 $\%$
(a) Draft/Target model pair is Llama-3.2-1B / Llama-3.1-8B.
SD	$1.0$	59.58 $\%$	91 $\%$	15.60 $\%$	4.95	58.72 $\%$	55 $\%$

Table 3: Comparison of finetuned draft vs. Diversed at two temperatures

T=1.0,0.1

. Columns are grouped by task. Each block compares SD, SD (Finetuned draft), and DIVERSED for a given Draft/Target pair. It shows fine-tuning does not reliably improve acceptance.

Figure 6: Acceptance rate decreases approximately linearly with the ensemble weight for static ensemble on CNNDM for the Target / Draft pair Llama-3.1-8B-Instruct/Llama-3.2-1B-Instruct at temperature

0

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Definition
$x_{0}$	Input prompt
$N$	Draft sequence length
$x_{1:N}$	Sampled accepted token sequence
$T_{N}$	Number of rejections when sampling a sequence of length $N$
$M_{q}$	Draft (small) model
$M_{p}$	Target (large) model

Appendix

Table 4: List of notations and their definitions.

	CNNDM	XSUM	MBPP	GSM8K
Generation budget	128	128	384	512
Hyper Parameter $\lambda$	0.5	0.5	0.3	0.1
Draft length $N$	$\{3,5,7\}$
Rollout Number	4
Learning Rate	5e-4
Training Epochs	2

Appendix

Table 5: Hyper-parameter configuration for each benchmark dataset.

	GSM8K		CNNDM		Xsum		MBPP
	Accept Rate ( $\uparrow$ )	Quality Accuracy ( $\uparrow$ )	Accept Rate( $\uparrow$ )	Quality ROUGE-2 ( $\uparrow$ )	Accept Rate( $\uparrow$ )	Quality ROUGE-2 ( $\uparrow$ )	Accept Rate( $\uparrow$ )	Quality pass@1 ( $\uparrow$ )
Autoregressive	NA	80 $\%$	NA	11.29	NA	8.42	NA	62 $\%$
SD	61.53 $\%$	80 $\%$	71.70 $\%$	11.26	50.24 $\%$	8.36	72.18 $\%$	62 $\%$
Static Ensemble	82.58 $\%$	79 $\%$	85.04 $\%$	11.45	61.44 $\%$	7.53	84.67 $\%$	61 $\%$
Diversed (ours)	84.82 $\%$	80 $\%$	81.92 $\%$	12.37	94.81 $\%$	7.66	85.03 $\%$	62 $\%$
(a) Target/Draft model pair is Llama-3.1-8B/Llama-3.2-1B.

Appendix

Table 6: Reported acceptance rate and generation quality. Columns are grouped by task. Each block compares three baselines and our method for a given Target / Draft pair. All results use temperature 0 and draft length

N=5

. Our Diversed enjoys a higher acceptance rate and also maintains comparable quality to baselines.

	GSM8K		CNNDM		XSum		MBPP
Method	Accept Rate ( $\uparrow$ )	Quality Accuracy ( $\uparrow$ )	Accept Rate( $\uparrow$ )	Quality ROUGE-2 ( $\uparrow$ )	Accept Rate( $\uparrow$ )	Quality ROUGE-2 ( $\uparrow$ )	Accept Rate( $\uparrow$ )	Quality pass@1 ( $\uparrow$ )
Autoregressive	NA	76 $\%$	NA	9.94	NA	7.56	NA	62 $\%$
SD	66.41 $\%$	76 $\%$	17.69 $\%$	9.92	37.67 $\%$	7.82	72.94 $\%$	61 $\%$
SD (Lossy)	67.42 $\%$	76 $\%$	17.85 $\%$	10.17	38.83 $\%$	7.97	73.37 $\%$	61 $\%$
SpecCascade	68.01 $\%$	76 $\%$	52.20 $\%$	10.19	40.22 $\%$	7.53	84.15 $\%$	61 $\%$
Static Ensemble	72.36 $\%$	75 $\%$	87.58 $\%$	10.46	51.20 $\%$	7.31	80.66 $\%$	61 $\%$

Appendix

Table 7: Reported acceptance rate and generation quality, under different temperature. Columns are grouped by task. Each block compares three baselines and our method for a given Target / Draft pair. All results use a temperature of 0.1 and a draft length

N=5

. Our Diversed enjoys a higher acceptance rate and also maintains comparable quality to baselines.

Figure 7: Average number of tokens accepted per verification round (temperature

=0.1

, draft length

N=5

) across three target/draft model pairs. Our Diversed consistently surpasses all baselines by achieving a higher number of accepted tokens.

Figure 8: Results on the CNNDM, XSUM, MBPP, and GSM8k datasets for three target/draft pairs: normalized total time (lower is better) versus acceptance rate (higher is better) under speculative decoding with

N{=}3

. Model pairs with higher acceptance achieve lower decoding time.

Figure 9: Results on the CNNDM, XSUM, MBPP, and GSM8k datasets for three target/draft pairs: normalized total time (lower is better) versus acceptance rate (higher is better) under speculative decoding with

N{=}7

. Model pairs with higher acceptance achieve lower decoding time.

Figure 10: Our Diversed outperforms the Pareto front of the static ensemble and other baselines. Results are reported on the CNNDM dataset using the target/draft model pair: Qwen3-8B/Qwen3-0.6B and Gemma-3-12b-it/Gemma-3-4b-it, with temperature set to

0

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 291 papers

Speculative Speculative Decoding
Mar 3, 2026Tanishq Kumar, Tri Dao, Avner MaySpeculative Decoding LLM Inference Acceleration
Department of Computer Science, Stanford University · Department of Computer Science, Princeton University · Together AI
LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization
Aug 9, 2026Zexun Lin, Yuan Feng, Junlin Lv +2Inference-Time Optimization Speculative Decoding for Diffusion Language Models
Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, Jiangsu, China
D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models
Jun 3, 2026Liyuan Zhang, Jiarui Zhang, Jinwei Yao +6Speculative Decoding for Diffusion Language Models Speculative Decoding
Peking University · Tsinghua University · HKUST +2

Mar 3, 2026J/K move · Enter open · S save

Speculative Speculative Decoding

Tanishq Kumar, Tri Dao, Avner May

Department of Computer Science, Stanford University · Department of Computer Science, Princeton University · Together AI

Speculative Decoding LLM Inference Acceleration Parallel Speculative Decoding Autoregressive Decoding

Open paper AskPDF arXiv

Autoregressive decoding is bottlenecked by its sequential nature. Speculative decoding has become a standard way to accelerate inference by using a fast draft model to predict upcoming tokens from a slower target model, and then verifying them in parallel with a single target model forward pass. However, speculative decoding itself relies on a sequential dependence between speculation and verification. We introduce speculative speculative decoding (SSD) to parallelize these operations. While a verification is ongoing, the draft model predicts likely verification outcomes and prepares speculations pre-emptively for them. If the actual verification outcome is then in the predicted set, a speculation can be returned immediately, eliminating drafting overhead entirely. We identify three key challenges presented by speculative speculative decoding, and suggest principled methods to solve each. The result is Saguaro, an optimized SSD algorithm. Our implementation is on average 30% faster than optimized speculative decoding baselines and up to 5x faster than autoregressive decoding with open source inference engines.

Abstract

Speculative Decoding

Explore similar work

Speculative Speculative Decoding

LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization

D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models