Acceptance Is Not Speed: Length Inflation in Training-Free Lossy Speculative Decoding

Chiatzen Wang; Bill Xu; Haochen Gu; Hongyang Zhang

cs.CLOct 16, 2026

Acceptance Is Not Speed: Length Inflation in Training-Free Lossy Speculative Decoding

Authors: Chiatzen Wang, Bill Xu, Haochen Gu, Hongyang Zhang

Organizations: University of Waterloo

Abstract

Speculative decoding typically characterizes speedup by draft tokens accepted per verification round. Lossless verification preserves the target model's output distribution and thus expected completion length, making this metric a meaningful efficiency proxy. Relaxed (lossy) verification accepts tokens that lossless verification would reject, potentially altering that distribution. We identify a length inflation phenomenon: relaxed verification lengthens completions, primarily by extending reasoning traces, largely offsetting apparent gains in accepted tokens per round. We evaluate existing training-free relaxed verification rules on twelve target-drafter pairs against lossless verification on identical prompts, measuring completion length, verifier rounds, wall-clock time, and task accuracy. At each rule's most relaxed evaluated setting, relaxed verification on average accepts $1.34\times$ as many tokens per round but produces $1.54\times$ longer completions. For rules with higher acceptance lengths, tokens accepted only under relaxed verification tend to have low target-model probability, contributing to length inflation. For rules that reduce verifier rounds, accuracy typically drops substantially. Further analysis traces some failures to continued reasoning that exhausts the budget needed to deliver a final answer. Overall, accepted tokens per round alone is an unreliable measure of decoding speed under relaxed verification. Completion length and verifier rounds must also be reported to assess actual efficiency.

Figures & tables

Dataset	mentored	cactus	casc-opt	r-fuzzy	casc-tok	mentored	cactus	casc-opt	r-fuzzy	casc-tok
	GPT-OSS-20B + EAGLE-3 (nebius)					Qwen3-8B + EAGLE-3 (RedHatAI)
GSM8K	1.16 (.93)	1.61 (.91)	1.46 (.94)	1.70 (.87)	0.96 ∘ (.97)	1.03 ∘ (.77)	1.04 ∘ (.77)	1.33 (.49)	0.95 ∘ (.55)	0.99 ∘ (.79)
AIME24	1.69 (.63)	1.78 (.60)	2.48 (.37)	2.13 (.50)	1.48 (.70)	1.01 ∘ (.73)	1.26 (.23)	1.42 (.30)	1.29 (.40)	0.98 ∘ (.70)
HumanEval	1.30 (.96)	1.79 (.87)	1.73 (.82)	1.64 (.63)	1.05 ∘ (.94)	1.08 (.85)	1.14 (.75)	1.72 (.49)	1.65 (.16)	1.08 (.85)
LiveCodeBench	1.25 (.84)	1.54 (.61)	1.64 (.49)	1.65 (.34)	1.16 (.88)	1.05 (.63)	1.15 (.44)	1.27 (.33)	1.07 ∘ (.02)	1.00 ∘ (.71)
MT-Bench	1.10 ∘ (6.41)	1.18 (4.51)	1.09 ∘ (5.74)	1.20 (4.58)	0.96 ∘ (7.49)	1.04 ∘ (6.40)	1.16 (2.99)	1.22 (3.95)	1.12 (3.05)	1.04 ∘ (7.21)

Table 1: Length inflation on the primary pairs (Section 3.1 ) at each method’s most relaxed tested setting: completion length relative to lossless (

\lambda

;

>1

is inflation), with the accuracy at that setting in parentheses (MT-Bench: judge score out of 10). ∘ : the 95% interval for

\lambda

includes

1

(Table B5 ). All twelve pairs: Table D1 .

Dataset	ment.	cactus	c-opt	r-fuzzy	c-tok	ment.	cactus	c-opt	r-fuzzy	c-tok
	GPT-OSS-20B + EAGLE-3 (nebius)					Qwen3-8B + EAGLE-3 (RedHatAI)
GSM8K	0.83	0.99	1.07	1.14	0.83	0.93	0.90	1.05	0.76	0.95
AIME24	1.18	1.07	1.70	1.37	1.22	0.94	0.55	1.06	1.16	0.93
HumanEval	0.97	1.12	1.29	1.16	0.93	0.99	0.96	1.36	1.33	1.04
LiveCodeBench	0.89	0.94	1.16	1.11	0.98	0.93	0.72	1.03	0.87	0.94
MT-Bench	0.75	0.62	0.76	0.76	0.88	0.89	0.64	0.94	0.85	0.96

Table 2: Verifier rounds relative to lossless (

R/R^{*}

) on the primary pairs at each method’s most relaxed tested setting (ment., c-: mentored, casc-). Bold: the 95% interval for

R/R^{*}

lies entirely above

1

(more rounds than lossless); underline: entirely below

1

. All twelve pairs: Table D2 .

Panel of Figure 1 — Figure 1: Every tested setting of the primary pairs (151 runs; filled markers GPT-OSS-20B, hollow Qwen3-8B): length inflation against acceptance gain. A relaxed rule saves verifier rounds only below the dashed line $\lambda=(\bar{\ell}+1)/(\bar{\ell}^{*}+1)$ (Equation ( 5 )). All twelve pairs, one panel per rule: Figure 2 .

Panel of Figure 2 — Figure 2: All 360 cells at each rule’s most relaxed tested setting, one panel per rule (log scales; outlined: the primary pairs). A cell saves verifier rounds only below the dashed line $\lambda=(\bar{\ell}+1)/(\bar{\ell}^{*}+1)$ (Equation ( 5 )). Above it: 21, 31, 31, 38, and 11 of each rule’s 72 cells (mean gain 1.21, 1.74, 1.41, 1.22, 1.11), and 111 of 210 cells with an EAGLE-3, EAGLE-1, or P-EAGLE head, 3 of 30 with DSpark, and 18 of 120 with a standalone drafter.

GPT-OSS-20B
			mean $\lambda$						mean $R/R^{*}$
Target, drafter	$\bar{\ell}^{*}$	gain	ment.	cactus	c-opt	r-fuzzy	c-tok	all	ment.	cactus	c-opt	r-fuzzy	c-tok	all	acc. loss
EAGLE-3 (nebius)	2.28	1.38	1.35	1.61	1.70	1.67	1.12	1.49	0.96	0.93	1.22	1.10	0.97	1.04	22/25
EAGLE-3 (RedHatAI)	1.49	1.35	1.31	2.32	1.36	1.84	1.07	1.58	1.14	1.01	1.16	1.52	1.05	1.18	17/25
Qwen3-8B
EAGLE-3 (RedHatAI)	1.20	1.27	1.05	1.31	1.35	1.20	1.01	1.18	0.94	0.76	1.04	0.99	0.96	0.94	18/25

Table 3: Twelve target-drafter pairs at each rule’s most relaxed tested setting: for each rule, the mean of

\lambda

and of

R/R^{*}

over the pair’s datasets; all: over all the pair’s cells. Bold (underline): the 95% interval lies above (below)

1

on every dataset.

\bar{\ell}^{*}

: mean lossless accepted length; gain: mean

(\bar{\ell}+1)/(\bar{\ell}^{*}+1)

; acc. loss: graded cells two or more points below lossless. Every cell: Tables D1 and D2 . ment., c-: mentored, casc-.

Dataset	forced	correct		$+$	$-$	forced	correct		$+$	$-$
	Qwen3-8B					GPT-OSS-20B
GSM8K	307	56 $\to$	225	171	2	32	4 $\to$	10	6	0
AIME24	60	2 $\to$	6	4	0	36	3 $\to$	7	7	3
HumanEval	205	16 $\to$	52	37	1	9	2 $\to$	0	0	2
LiveCodeBench	226	11 $\to$	81	72	2	62	7 $\to$	16	13	4
LongBench-v2	157	17 $\to$	33	18	2	38	8 $\to$	9	7	6

Table 4: Forcing relaxed runs to close their reasoning trace at a threshold near the generation budget, summed over the five rules at their loosest tested settings. forced : runs still reasoning at the threshold; correct : how many of them are correct without

\to

with forcing;

+

: forced runs that go from wrong to correct;

-

: from correct to wrong. Per rule and thresholds: Table F1 .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Method (arm)	$\alpha$ controls	$\pi^{\mathrm{rej}}[q,p,x,\alpha](v)$	$\pi^{\mathrm{res}}$	$\pi^{\mathrm{bonus}}$	$\alpha$ swept
CACTUS ( cactus ) ( Hao and Mou, 2026 )	reverse-KL budget $D_{\mathrm{KL}}(\pi\,\\|\,p)\leq\alpha$	$\gamma_{x}$ if $v=x$ , else $\frac{1-\gamma_{x}}{1-p(x)}\,p(v)$ ; $\gamma_{x}=\min\{p(x)+\sqrt{2\alpha\,p(x)(1-p(x))},\,1\}$	$\pi^{\mathrm{rej}}$	$p$	0.03, 0.08, 0.18, 0.35
Mentored decoding ( mentored_dec ) ( Tran-Thien, 2023 )	forward-KL budget $D_{\mathrm{KL}}(p\,\\|\,\pi)\leq\alpha$	$\min\{p(v)/(1-\alpha),\,1\}$	$p$	$p$	0.15, 0.35, 0.55, 0.75
Fuzzy, reducible variant ( r_fuzzy ) ( Holsman et al., 2025 )	divergence threshold	$q(v)$ if $\Div(p,q)<\alpha$ , else $p(v)$	$\pi^{\mathrm{rej}}$	$p$	0.03, 0.08, 0.15, 0.25
Speculative cascade, optimal rule ( spec_casc_opt ) ( Narasimhan et al., 2025 )	deferral cost	$q(v)$ if $\max_{u}q(u)\geq\max_{u}p(u)-\alpha\,\mathrm{TV}(p,q)$ , else $p(v)$	$\pi^{\mathrm{rej}}$	$p$	$-0.3$ , $-0.1$ , $-0.02$ , $0.05$
Speculative cascade, token rule ( spec_casc_tok ) ( Narasimhan et al., 2025 )	trusted-set width	$q(v)+\eta_{\alpha}\,p(v)$ if $v\in A_{\alpha}$ , else $\eta_{\alpha}\,p(v)$ ; $A_{\alpha}=\{u:p(u)\geq(1-\alpha)\max_{w}p(w)\}$ , $\eta_{\alpha}=1-\sum_{u\in A_{\alpha}}q(u)$	$\pi^{\mathrm{rej}}$	$p$	0.15, 0.35, 0.55, 0.8
Ensemble ( ens ) ( Wang et al., 2026b )	mixture weight	$\alpha\,q(v)+(1-\alpha)\,p(v)$	$\pi^{\mathrm{rej}}$	$p$	not swept

Appendix

Table A1: Training-free relaxed verification rules as instances of Algorithm 1 . Strict speculative decoding is

\pi=p

throughout. Every rule except the two cascade rules recovers strict verification at

\alpha=0

; spec_casc_opt recovers it for

\alpha<-1

(since

\max_{u}q(u)-\max_{u}p(u)\leq\mathrm{TV}(p,q)

) and spec_casc_tok for

\alpha<0

. On the second sampler path (Section 3.1 ), cactus and spec_casc_tok use

\pi^{\mathrm{res}}=p

. Last column: the four

\alpha

values of the pilot (Section 3.2 ), used for every pair.

Method	Models and tasks	Speed metric	Length	Rounds	Long-form
CACTUS ( Hao and Mou, 2026 )	Qwen3, Gemma, R1-Distill, Llama pairs; GSM8K, IFEval, GPQA	$\mathrm{AL}_{m}$ , rejected-token count, vLLM wall-clock	(✓)	(✓)	(✓)
Mentored decoding ( Tran-Thien, 2023 )	T5-large/small; WMT15 En–Fr	tokens per second	$\times$	$\times$	$\times$
Fuzzy ( Holsman et al., 2025 )	Llama-3.1, Gemma-2, Qwen2.5 pairs; GSM8K, CSQA, MMLU, HumanEval	tokens per second, accepted length	$\times$	$\times$	$\times$
Speculative cascades ( Narasimhan et al., 2025 )	T5, Gemma pairs; translation, summarization, GSM8K, MBPP, QA	batch-one relative latency, rejection rate	$\times$	$\times$	$\times$
Ensemble ( Wang et al., 2026b )	Llama-3.1, Qwen3, Gemma-3 pairs; GSM8K, CNN/DM, XSum, MBPP	acceptance rate, wall-clock at fixed token count	$\times$	$\times$	$\times$
Spec. contrastive ( Yuan et al., 2024 )	Llama-2 7B/70B; WikiText, HumanEval, AlpacaEval, GSM8K	expected acceleration from acceptance rate	$\times$	$\times$	$\times$

Appendix

Table A2: Results reported by the source papers (Appendix A.1 ). Length: completion length reported as a measured quantity; Rounds: verifier rounds or target forward passes reported; Long-form: evaluation on long reasoning traces. (✓) marks indirect or partial coverage: CACTUS reports rejected-token counts, which grow with length and with rounds ending in rejection, on reasoning-mode models, and discusses length only as a caveat and one qualitative case.

Dataset	$n$	$\bar{\ell}^{*}$	$\bar{L}^{*}$	$R^{*}$	Acc.	cut off
GPT-OSS-20B
GSM8K	150	2.57	322	94	0.960	2
AIME24	30	2.19	8 717	2 953	0.767	7
HumanEval	150	2.53	912	272	0.960	0
LiveCodeBench	90	2.16	3 456	1 151	0.889	2
MT-Bench	80	2.34	1 264	414	7.29	4

Appendix

Table B1: Lossless (strict) speculative decoding on the evaluation cases, with no relaxation, for the two primary pairs (Section 3.1 ).

\bar{\ell}^{*}

: mean accepted draft tokens per verification round;

\bar{L}^{*}

: mean completion length in tokens;

R^{*}

: mean verifier rounds; Acc.: pass rate; for MT-Bench, the mean judge score out of 10 (Table B4 ); cut off: share of runs (%) that end at the generation budget. Generation budgets (tokens): GSM8K 2 048, AIME24 32 768, HumanEval 9 000, LiveCodeBench 12 000, MT-Bench 4 096, LongBench-v2 8 192. Every comparison in Section 3 is against the lossless run of the same pair, not against autoregressive decoding.

Method	only	only	both	only	only	both
		target prob. (median)		rank	entropy (mean)
mentored_dec	16%	0.09	0.93	37	1.8	0.8
cactus	24%	0.01	0.78	234	2.1	1.1
spec_casc_opt	16%	0.07	0.92	88	1.8	0.8
r_fuzzy	21%	0.01	0.83	134	1.8	1.0
spec_casc_tok	7%	0.29	0.99	0.9	1.5	0.5

Appendix

Table B2: What relaxation admits (GPT-OSS-20B, traced runs, loosest settings): committed draft tokens that only the relaxed rule accepts (“only”) against those lossless verification would also accept (“both”). Share of committed draft tokens, median target probability, mean target rank (

0

is the argmax), mean target entropy (nats).

	GPT-OSS-20B		Qwen3-8B
Method	cells	best $R/R^{*}$	cells	best $R/R^{*}$
mentored_dec	5/5	0.86	5/5	0.88
cactus	1/5	0.98	2/5	0.97
spec_casc_opt	2/5	0.93	2/5	0.97
r_fuzzy	2/5	1.13	0/5	—
spec_casc_tok	5/5	0.83	5/5	0.93

Appendix

Table B3: The best accuracy-preserving setting. For each graded (dataset, method) cell we take, among the method’s tested

\alpha

values, the one with the fewest verifier rounds whose accuracy is within two points of lossless. “cells” counts the graded datasets (of five) on which any tested setting qualifies; “best

R/R^{*}

” is the lowest round ratio among them. The median best round ratio over the 29 qualifying cells is

0.96

Method	$\alpha$	score [95% CI]	no ans.	answered
GPT-OSS-20B
lossless	—	7.29 [6.75, 7.80]	0	7.29
mentored_dec	0.75	6.41 [5.73, 7.09]	1	6.48
cactus	0.35	4.51 [3.70, 5.34]	6	4.80
spec_casc_opt	0.05	5.74 [5.04, 6.45]	2	5.86
r_fuzzy	0.25	4.58 [3.94, 5.23]	3	4.71

Appendix

Table B4: MT-Bench quality of the primary pairs at the loosest settings, judged by Claude ( claude-fable-5-1 ) with FastChat’s single-answer grading ( Zheng et al., 2023 ) of the first turn (the GPT-4 reference answer is supplied for math, reasoning, and coding questions). score: mean over the 80 questions, 1 to 10, with a bootstrap interval; no ans.: runs that never produced an answer (scored 1 without a judge call); answered: mean over the other runs.

Dataset	Method	$\alpha$	$\lambda$ [95% CI]	time [95% CI]	cut off	$\alpha$	$\lambda$ [95% CI]	time [95% CI]	cut off
		GPT-OSS-20B + EAGLE-3 (nebius)				Qwen3-8B + EAGLE-3 (RedHatAI)
GSM8K	mentored_dec	0.75	1.16 [1.02, 1.33]	0.96 [0.86, 1.08]	4	0.75	1.03 [0.98, 1.07]	0.95 [0.91, 1.00]	26
	cactus	0.35	1.61 [1.41, 1.83]	1.19 [1.06, 1.33]	5	0.35	1.04 [1.00, 1.08]	0.94 [0.90, 0.97]	27
	spec_casc_opt	0.05	1.46 [1.28, 1.65]	1.18 [1.06, 1.31]	5	0.05	1.33 [1.26, 1.40]	1.12 [1.07, 1.18]	58
	r_fuzzy	0.25	1.70 [1.49, 1.94]	1.29 [1.15, 1.45]	5	0.25	0.95 [0.86, 1.05]	0.81 [0.74, 0.90]	35
	spec_casc_tok	0.8	0.96 [0.88, 1.04]	0.91 [0.85, 0.98]	1	0.8	0.99 [0.95, 1.02]	0.96 [0.92, 1.00]	23

Appendix

Table B5: The primary pairs’ cells at each method’s most relaxed tested setting:

\alpha

; length ratio

\lambda

and per-request time ratio to lossless, each with its 95% paired bootstrap interval over cases (10 000 resamples); cut off: share of runs (%) that end at the generation budget (lossless: Table B1 ). Round ratios: Table 2 ; accuracy: Table 1 ; acceptance gains and round-ratio intervals: released tables.

Dataset	Method	$\bar{\ell}$	$\lambda$ [95% CI]	$R/R^{*}$ [95% CI]	Acc.	cut off
GSM8K	mentored_dec	1.85	1.01 [0.97, 1.04]	0.94 [0.91, 0.98]	0.81	23%
	cactus	1.81	1.01 [0.97, 1.05]	0.96 [0.92, 1.00]	0.81	23%
	spec_casc_opt	3.34	1.52 [1.43, 1.62]	0.95 [0.89, 1.02]	0.20	87%
	r_fuzzy	2.05	1.01 [0.92, 1.09]	0.89 [0.81, 0.96]	0.64	39%
	spec_casc_tok	1.80	1.01 [0.97, 1.04]	0.96 [0.92, 1.00]	0.78	26%
LiveCodeBench	mentored_dec	1.57	1.04 [1.00, 1.09]	0.94 [0.90, 0.99]	0.68	34%

Appendix

Table C1: Qwen3-8B at temperature

0.6

, top-

p

0.95

, top-

k

20 (seed 0, third server), loosest settings, against lossless at the same sampler and cases: lossless

\bar{\ell}^{*}

1.68

(GSM8K) and

1.36

(LiveCodeBench), accuracy

0.79

and

0.67

, cut off

25\%

and

33\%

Dataset	mentored	cactus	casc-opt	r-fuzzy	casc-tok	mentored	cactus	casc-opt	r-fuzzy	casc-tok
	GPT-OSS-20B + EAGLE-3 (nebius)					Qwen3-8B + EAGLE-3 (RedHatAI)
GSM8K	1.16 (.93)	1.61 (.91)	1.46 (.94)	1.70 (.87)	0.96 ∘ (.97)	1.03 ∘ (.77)	1.04 ∘ (.77)	1.33 (.49)	0.95 ∘ (.55)	0.99 ∘ (.79)
AIME24	1.69 (.63)	1.78 (.60)	2.48 (.37)	2.13 (.50)	1.48 (.70)	1.01 ∘ (.73)	1.26 (.23)	1.42 (.30)	1.29 (.40)	0.98 ∘ (.70)
HumanEval	1.30 (.96)	1.79 (.87)	1.73 (.82)	1.64 (.63)	1.05 ∘ (.94)	1.08 (.85)	1.14 (.75)	1.72 (.49)	1.65 (.16)	1.08 (.85)
LiveCodeBench	1.25 (.84)	1.54 (.61)	1.64 (.49)	1.65 (.34)	1.16 (.88)	1.05 (.63)	1.15 (.44)	1.27 (.33)	1.07 ∘ (.02)	1.00 ∘ (.71)
MT-Bench	1.10 ∘ (6.41)	1.18 (4.51)	1.09 ∘ (5.74)	1.20 (4.58)	0.96 ∘ (7.49)	1.04 ∘ (6.40)	1.16 (2.99)	1.22 (3.95)	1.12 (3.05)	1.04 ∘ (7.21)

Appendix

Table D1: Length inflation at each method’s most relaxed tested setting, for all twelve pairs (first block: the primary pairs of Table 1 ): completion length relative to lossless (

\lambda

;

>1

is inflation), with the accuracy at that setting in parentheses (MT-Bench: judge score out of 10). ∘ : the 95% interval for

\lambda

includes

1

(primary pairs: Table B5 ; other pairs: released tables).

Dataset	ment.	cactus	c-opt	r-fuzzy	c-tok	ment.	cactus	c-opt	r-fuzzy	c-tok	ment.	cactus	c-opt	r-fuzzy	c-tok
	GPT-OSS-20B + EAGLE-3 (nebius)					Qwen3-8B + EAGLE-3 (RH)					GPT-OSS-20B + EAGLE-3 (RH)
GSM8K	0.83	0.99	1.07	1.14	0.83	0.93	0.90	1.05	0.76	0.95	0.98	1.38	1.08	1.59	0.83
AIME24	1.18	1.07	1.70	1.37	1.22	0.94	0.55	1.06	1.16	0.93	1.23	0.46	1.18	1.67	1.33
HumanEval	0.97	1.12	1.29	1.16	0.93	0.99	0.96	1.36	1.33	1.04	1.30	1.80	1.43	2.70	1.02
LiveCodeBench	0.89	0.94	1.16	1.11	0.98	0.93	0.72	1.03	0.87	0.94	1.34	1.08	1.38	1.28	1.06
MT-Bench	0.75	0.62	0.76	0.76	0.88	0.89	0.64	0.94	0.85	0.96	0.95	0.74	0.99	0.93	1.03

Appendix

Table D2: Verifier rounds relative to lossless (

R/R^{*}

) at each method’s most relaxed tested setting, for all twelve pairs (first two: the primary pairs of Table 2 ; RH: RedHatAI; Llama-3.1-8B-I.: Llama-3.1-8B-Instruct; ment., c-: mentored, casc-). Bold: the 95% interval for

R/R^{*}

lies entirely above

1

(more rounds than lossless); underline: entirely below

1

; intervals are in the released tables.

	GSM8K			AIME24			HumanEval			LiveCodeBench			MT-Bench			LongBench-v2
Target, drafter	$\bar{\ell}^{*}$	$\bar{L}^{*}$	acc.	$\bar{\ell}^{*}$	$\bar{L}^{*}$	acc.	$\bar{\ell}^{*}$	$\bar{L}^{*}$	acc.	$\bar{\ell}^{*}$	$\bar{L}^{*}$	acc.	$\bar{\ell}^{*}$	$\bar{L}^{*}$	score	$\bar{\ell}^{*}$	$\bar{L}^{*}$	acc.
GPT-OSS-20B
EAGLE-3 (RedHatAI)	2.13	353	.97	1.34	9 650	.77	2.04	911	.98	1.57	3 445	.83	1.82	1 157	7.31	0.05	1 403	.54
Qwen3-8B
P-EAGLE	2.16	1 303	.81	2.26	18 758	.73	2.05	3 863	.85	1.84	8 072	.73	1.57	2 112	6.96	1.38	3 023	.55
DSpark	2.77	1 335	.75	2.51	19 550	.70	2.50	3 756	.84	2.31	8 063	.74	2.12	2 133	6.74	1.88	2 948	.47

Appendix

Table D3: Lossless references of the ten pairs beyond the primary ones (primary pairs: Table B1 ; same cases and budgets), seed 0: mean accepted length

\bar{\ell}^{*}

, mean completion tokens

\bar{L}^{*}

, and accuracy (MT-Bench: judge score out of 10).

Pair	same	$\lambda$	$R/R^{*}$	$R/R^{*}>1$	acc. loss	$\lambda$	$R/R^{*}$	$R/R^{*}>1$	acc. loss
		gentlest tested setting				most relaxed tested setting
GPT-OSS-20B + EAGLE-3 (nebius)	0	1.07	0.98	9	14	1.57	1.03	15	22
GPT-OSS-20B + EAGLE-3 (RedHatAI)	14	1.23	1.09	22	15	1.35	1.07	20	17
Qwen3-8B + EAGLE-3 (RedHatAI)	0	1.04	0.99	12	13	1.08	0.94	7	18
Qwen3-8B + DSpark	8	1.01	0.89	1	12	1.06	0.85	2	18
Llama-3.1-8B-Instruct + EAGLE-3	7	1.84	1.57	24	18	1.69	1.35	21	18

Appendix

Table D4: Gentlest against most relaxed tested setting on the seven pairs run with the matched protocol (30 cells each; the settings with the lowest and the highest measured

\bar{\ell}

): median

\lambda

and median

R/R^{*}

over the pair’s cells, cells that need more rounds than lossless (

R/R^{*}>1

), and graded cells two or more points below lossless (acc. loss, of 25; All: of 175). same: cells whose two settings are the same run.

	GPT-OSS-20B				Qwen3-8B
Rule	$\lambda$	gain	$R/R^{*}$	$T$	$\lambda$	gain	$R/R^{*}$	$T$
mentored_dec	1.15	1.44	0.76	0.75	1.07	1.14	0.93	0.91 †
cactus	1.26	1.89	0.63	0.62	1.35	1.91	0.67	0.68 ‡
spec_casc_opt	1.23	1.41	0.82	0.82	1.40	1.32	1.02	1.06 †
r_fuzzy	1.20	1.52	0.77	0.75	1.20	1.28	0.95	0.96
spec_casc_tok	1.09	1.13	0.95	0.92	1.01	1.05	0.95	0.95 †

Appendix

Table E1: SPEED-Bench over all 672 prompts, loosest settings, seed 0 (GPT-OSS-20B on the second server; Qwen3-8B’s first 40 prompts per arm on the second, the rest on the third). Lossless: 1 266 and 2 332 tokens on average, 1% and 5% cut off.

\lambda

R/R^{*}

T

: ratios of completion length, verifier rounds, and per-request time to lossless; gain:

(\bar{\ell}+1)/(\bar{\ell}^{*}+1)

. Bold (underline): 95% paired interval above (below)

1

. † : over the prompt pairs that share a node; ‡ : fewer than ten do, all-pair ratio.

	Qwen3-8B						GPT-OSS-20B
Dataset, rule	forced	correct		$+$	$-$	cut off	forced	correct		$+$	$-$	cut off
GSM8K
mentored_dec	44	6 $\to$	38	32	0	4	6	2 $\to$	2	0	0	3
cactus	56	15 $\to$	45	30	0	11	9	1 $\to$	2	1	0	6
spec_casc_opt	102	18 $\to$	60	44	2	27	7	1 $\to$	2	1	0	4
r_fuzzy	62	8 $\to$	47	39	0	13	8	0 $\to$	3	3	0	3

Appendix

Table F1: Table 4 by dataset and rule, each rule at its loosest tested setting. Columns as in Table 4 , plus cut off : forced runs that still end at the budget. Thresholds in tokens (Qwen3-8B / GPT-OSS-20B): GSM8K 1 684 / 2 002; AIME24 31 260 / 31 605; HumanEval 8 150 / 8 497; LiveCodeBench 11 004 / 9 294; LongBench-v2 6 894 / 8 133.

Target	Dataset	Pairs	lossless	relaxed	lossless	relaxed	after answer (%)	relaxed	lossless
			First answer (token)		Length (tokens)		Extra tokens	Cut-off runs with the answer
GPT-OSS-20B	GSM8K	486	102	126	183	264	50 [37, 59]	15/26	0/2
Qwen3-8B	GSM8K	304	325	350	877	989	82 [65, 96]	147/201	28/33
GPT-OSS-20B	AIME24	36	1 721	3 865	3 822	7 020	41 [28, 54]	8/33	1/2
Qwen3-8B	AIME24	33	4 221	6 031	11 160	14 230	$-16$ [ $-177$ , 49]	9/53	1/5

Appendix

Table F2: Where the extra tokens sit (Section 4 ): the four rules other than spec_casc_tok , each at its loosest setting. Pairs: (case, rule) pairs in which both arms answer correctly and neither is cut off; first answer and length: medians over pairs; extra tokens after answer: share of the summed extra length that falls after the point where each run first writes the answer, with a 95% bootstrap interval over cases (negative: the part after the answer shrinks while the part before it grows); cut-off runs with the answer: runs that end at the budget without a correct answer, and how many of them contain it.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 308 papers

Jul 29, 2026

Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes

Speculative Decoding (SD) accelerates large language model inference by allowing a lightweight draft model to propose tokens that are subsequently verified in parallel by a larger target model. Recent approaches introduce lossy verification schemes to further improve efficiency by relaxing strict distributional matching. Yet such relaxation silently rewrites the decoding distribution, and the resulting acceleration can come at the cost of unstable, sometimes severely degraded generation quality. In this work, we present a principled analysis of the distributions induced by lossy verification methods. We show that many seemingly distinct approaches differ only superficially and can be classified into two categories: truncation-based verification and collaborative verification. We further construct a diagnostic evaluation framework across curated benchmarks. For truncation-based methods, we identify a fundamental pitfall: performance can degrade significantly compared to the true truncation sampling baseline due to distributional distortion. For collaborative verification, we uncover a key principles: controlling the overshoot of draft probabilities relative to target probabilities is essential to prevent low-quality outputs. Our code is available at https://github.com/ZhouYuxuanYX/Fast-HSD.

Tianyu Wang, Yuxuan Zhou, Wenbin Wang +3

1Independent Researcher · 2Baidu Inc. · 3Zhejiang University

Speculative Decoding LLM Inference Acceleration

Ask PDF

Aug 31, 2026

Verification-Aware Training for Speculative Decoding

Speculative decoding accelerates large language model inference by using a draft model to generate candidate tokens, which are verified by the target model in a single forward pass. Verification proceeds sequentially and discards every position from the first rejection onward, yet existing draft training relies on token-level imitation of the target with a fixed per-position weighting that reflects neither property. We introduce Verification-Aware Training (VAT), a plug-in framework that simulates verification at every training step and turns the resulting accept and reject patterns into supervision. VAT consists of two components: (i) a verification head, a lightweight jointly trained binary classifier that supervises the draft model on whether each position survives sequential verification; (ii) verification-adaptive weighting, which replaces the fixed weighting schedule by keeping full weight up to each sample's first rejection point and re-anchoring the decay to start there. VAT modifies only the training objective, so it can be layered on top of existing methods without changing the draft architecture, the target model, or the inference procedure. Applied to EAGLE-3 and DFlash on Qwen3-4B, Qwen3-8B, and LLaMA-3.1-8B, VAT improves average acceptance length by up to 11.4% and wall-clock speedup by up to 8.7%, with consistent gains across math, code, and chat benchmarks. Code will be available at https://github.com/naver-ai/vat

Geonmo Gu, Byeongho Heo, HeeJae Jun +4

NAVER AI Lab · NAVER AI Search Platform · Korea University

Speculative Decoding LLM Inference Acceleration

Ask PDF

Apr 8, 2026

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification

Speculative decoding is an effective technique for accelerating large language model inference by drafting multiple tokens in parallel. In practice, its speedup is often bottlenecked by a rigid verification step that strictly enforces the accepted token distribution to exactly match the target model. This constraint leads to the rejection of many plausible tokens, lowering the acceptance rate and limiting overall time speedup. To overcome this limitation, we propose Dynamic Verification Relaxed Speculative Decoding (DIVERSED), a relaxed verification framework that improves time efficiency while preserving generation quality. DIVERSED learns an ensemble-based verifier that blends the draft and target model distributions with a task-dependent and context-dependent weight. We provide theoretical justification for our approach and demonstrate empirically that DIVERSED achieves substantially higher inference efficiency compared to standard speculative decoding methods. Code is available at: https://github.com/comeusr/diversed.

Ziyi Wang, Siva Rajesh Kasa, Ankith M S +6

Purdue University · Amazon Inc. · University of Illinois Urbana-Champaign +1

Speculative Decoding LLM Inference Acceleration Parallel Speculative Decoding

Ask PDF

Abstract

Speculative Decoding

Explore similar work

Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes

Verification-Aware Training for Speculative Decoding

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification