Give the Drafter a Second Chance to Guess: Parallel Reconsideration for Speculative Decoding

Hoang Anh Duy Le; Huibing Dong; Alireza Khadem; Ziteng Sun; Xiang Song; Yizheng Huang; Soroush Radpour; Anshumali Shrivastava

cs.LGNov 3, 2026

Give the Drafter a Second Chance to Guess: Parallel Reconsideration for Speculative Decoding

Authors: Hoang Anh Duy Le, Huibing Dong, Alireza Khadem, Ziteng Sun, Xiang Song, Yizheng Huang, Soroush Radpour, Anshumali Shrivastava,

Organizations: Google · Rice University · Google Research · Google DeepMind

Abstract

Speculative decoding accelerates large language model inference with a small drafter, whose per-guess acceptance rate determines the achievable speedup. Feature-level drafters share a design choice: every token, easy or hard, is drafted under a single forward pass. Many of the tokens the target rejects are ones the drafter is itself unsure about yet the target answers decisively. To address this, we introduce Parallel Reconsideration: guided by its own uncertainty, the drafter reconsiders its hesitant guesses with extra passes that run in parallel with the one-pass draft, adding negligible latency. A co-designed training recipe and serving-time drafting algorithm append these reconsidered guesses to the original draft tree as refined candidates. Across models and benchmarks, on average, Parallel Reconsideration improves over the baselines by 5.3% to 10.2% on acceptance length and by 3.5% to 9.2% on speedup.

Figures & tables

Panel of Figure 1 — Figure 1: Two observations on where an EAGLE-style drafter loses acceptance. ( 1(a) ) Acceptance rate versus the drafter’s own top- $1$ confidence, which falls from $95.6\%$ on confident positions to $7.3\%$ on the least confident. ( 1(b) ) Drafter confidence against target confidence over the rejected tokens (% of the reject pool): the drafter is unsure on $62.1\%$ of them, and $17.0\%$ of the pool is unsure at a decisive target ( $c^{\mathrm{t}}_{p}\geq 0.9$ ).

Figure 2: Co-forwarding reconsideration on the draft tree. Each token produces a feature (diamond) from which the next token is sampled ( ${\sim}$ ); the one-pass draft (gray, level $0$ ) forms the main branches, and tree width comes free because a node’s top- $k$ children are all sampled from its single feature. The reconsideration pass (blue, level $1$ ) deepens a node’s feature within the same batched forward as the next draft step so it proposes a corrected token (here $x_{2}^{\prime}$ ) with no extra sequential draft step. The one-pass draft still reaches $x_{4}$ while the reconsidered branch grows in parallel.

Temperature $=0$
		Math						Code						Chat
		GSM8K		MATH-500		AIME25		HumanEval		MBPP		LCB		MT-Bench		Avg.
Model	Method	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup
Llama-3.1-8B	EAGLE-3.1 (24)	4.41	1.90 $\times$	4.70	2.14 $\times$	5.41	2.29 $\times$	5.15	2.25 $\times$	4.97	2.24 $\times$	4.31	1.78 $\times$	4.20	1.69 $\times$	4.74	2.04 $\times$
	PR (24)	4.73	2.03 $\times$	4.77	2.16 $\times$	5.36	2.28 $\times$	5.45	2.38 $\times$	5.28	2.37 $\times$	4.78	1.95 $\times$	4.65	1.85 $\times$	5.00	2.15 $\times$
	$\Delta$	$+7.3\%$	$+6.8\%$	$+1.5\%$	$+0.9\%$	$-0.9\%$	$-0.4\%$	$+5.8\%$	$+5.8\%$	$+6.2\%$	$+5.8\%$	$+10.9\%$	$+9.6\%$	$+10.7\%$	$+9.5\%$	$\mathbf{+5.5\%}$	$\mathbf{+5.4\%}$

Table 1: Acceptance length

\tau

and decoding speedup over autoregressive decoding. Parenthesized values give the total draft budget

B

(number of draft nodes verified per round), held equal for both methods along with the offer width, grow width, and number of draft steps.

\Delta

rows give the relative gain of PR over EAGLE-3.1 at the same budget.

			Batch size $n$
			1	2	4	8	16
Llama	GSM8K	EAGLE-3.1	2.05 $\times$	1.68 $\times$	1.62 $\times$	1.26 $\times$	1.12 $\times$
		PR	2.26 $\times$	1.91 $\times$	1.91 $\times$	1.46 $\times$	1.22 $\times$
	MT-Bench	EAGLE-3.1	2.03 $\times$	1.71 $\times$	1.71 $\times$	1.50 $\times$	1.41 $\times$
		PR	2.05 $\times$	1.79 $\times$	1.77 $\times$	1.56 $\times$	1.51 $\times$
	LCB	EAGLE-3.1	1.95 $\times$	1.61 $\times$	1.60 $\times$	1.40 $\times$	1.31 $\times$

Table 2: Speedup as requests are batched. Wall-clock speedup of EAGLE-3.1 and Parallel Reconsideration (PR) over autoregressive decoding at a matched batch size

n

, measured on the same serving engine (

B=48

, temperature

0

	GSM8K		LiveCodeBench		MBPP		Avg.
Drafter	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup
EAGLE-3.1	4.84	2.05 $\times$	4.70	1.95 $\times$	5.32	2.33 $\times$	4.95	2.11 $\times$
Parallel Reconsideration	5.40	2.26 $\times$	5.11	2.11 $\times$	5.57	2.41 $\times$	5.36	2.26 $\times$
no balance term ( $\beta=0$ )	5.17	2.14 $\times$	5.03	2.04 $\times$	5.58	2.41 $\times$	5.26	2.20 $\times$
$\Delta$	$-4.3\%$	$-5.3\%$	$-1.6\%$	$-3.3\%$	$+0.2\%$	$+0.0\%$	$-1.9\%$	$-2.8\%$
no self-query ( $Q\to e(t_{p})$ )	5.03	2.10 $\times$	4.92	2.02 $\times$	5.47	2.35 $\times$	5.14	2.16 $\times$

Table 3: Removing each design choice, and spending the compute on depth instead. Llama-3.1-8B at

B=48

, temperature

0

, reporting

\tau

/ speedup. The first two rows remove one component of Parallel Reconsideration and retrain; the third is EAGLE-3.1 with a second decoder layer and no reconsideration, spending its additional capacity on every position rather than on the uncertain ones.

\Delta

is relative to Parallel Reconsideration.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Reconsidering in parallel. Diamonds are features produced by a forward pass of the shared block $M$ and circles are the tokens sampled from them; gray marks ordinary one-pass drafting (feature $f_{i}$ , token $x_{i}$ ) and blue marks reconsideration (deepened feature $f_{i}^{\prime}$ , reconsidered token $x_{i}^{\prime}$ ). Top (sequential): running reconsideration in series spends a forward of $M$ that advances no new position, so within a fixed draft budget the drafter reaches fewer tokens. Bottom (co-forward): reconsideration of a position depends only on its one-pass feature, which is already available, so it is co-forwarded with the ordinary draft step in a single batched pass of $M$ . The one-pass chain still reaches $x_{3}$ and the reconsidered token $x_{2}^{\prime}$ is obtained for negligible additional latency.

	GSM8K		LiveCodeBench		MT-Bench		Avg.
Drafter	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup
EAGLE-3.1 ( $B_{w}=6$ )	5.44	2.81 $\times$	5.54	3.13 $\times$	3.40	1.64 $\times$	4.79	2.53 $\times$
+ advance $3$ more	5.51	2.83 $\times$	5.68	3.17 $\times$	3.43	1.64 $\times$	4.87	2.55 $\times$
$\Delta$	$+1.3\%$	$+0.7\%$	$+2.5\%$	$+1.3\%$	$+0.9\%$	$+0.0\%$	$+1.7\%$	$+0.8\%$
widen to $B_{w}=9$	5.49	2.82 $\times$	5.61	3.15 $\times$	3.42	1.64 $\times$	4.84	2.54 $\times$
$\Delta$	$+0.9\%$	$+0.4\%$	$+1.3\%$	$+0.6\%$	$+0.6\%$	$+0.0\%$	$+1.0\%$	$+0.4\%$

Appendix

Table 4: Reconsidering the additional nodes against drafting wider with them. Gemma-3-12B at

B=48

, temperature

1

, reporting

\tau

/ speedup. The two EAGLE-3.1 controls batch

9

nodes per draft step: the first takes

3

nodes of the previous step’s batch, matching where reconsideration draws from, and forwards each one’s best not-yet-forwarded one-pass child (a new sibling branch); the second widens the grow width outright, so all

9

nodes extend the current frontier, which is more drafting breadth than Parallel Reconsideration itself gets. Parallel Reconsideration with

m=0

runs our full serving path with no node ever reconsidered and batches the same

6

nodes as EAGLE-3.1; adding

3

reconsidered nodes brings it to

9

\Delta

is relative to EAGLE-3.1.

			$\tau$			Speedup
$B$	$B_{w}$	$k$	EAGLE-3.1	PR	$\Delta\tau$	EAGLE-3.1	PR
Total budget
24	6	6	4.39	4.83	$\mathbf{+0.44}$	1.94 $\times$	2.11 $\times$
32	6	6	4.606	4.869	$+0.263$	2.077 $\times$	2.208 $\times$
48 ∗	6	6	4.835	5.127	$+0.292$	2.184 $\times$	2.291 $\times$
64	6	6	4.961	5.222	$+0.261$	2.195 $\times$	2.280 $\times$

Appendix

Table 5: Sensitivity to the tree configuration. Acceptance length

\tau

and speedup over autoregressive decoding for EAGLE-3.1 and Parallel Reconsideration (PR) under identical budgets, varying one of the total budget

B

, grow width

B_{w}

and offer width

k

at a time around our default (

B=48

B_{w}=6

k=6

, marked

\ast

and repeated in each block);

\gamma=8

and

m=3

throughout, on Qwen3-8B, GSM8K, temperature

0

\Delta\tau

is PR minus EAGLE-3.1. Bold marks the best value per column.

	Math						Code						Chat
	GSM8K		MATH-500		AIME25		HumanEval		MBPP		LCB		MT-Bench		Avg.
Method	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup
HASS	4.81	2.03 $\times$	4.84	2.11 $\times$	5.21	2.27 $\times$	5.59	2.38 $\times$	5.34	2.38 $\times$	4.68	1.90 $\times$	4.64	1.97 $\times$	5.02	2.15 $\times$
HASS + PR	5.17	2.12 $\times$	5.21	2.21 $\times$	5.88	2.49 $\times$	5.86	2.45 $\times$	5.67	2.44 $\times$	4.99	1.98 $\times$	4.97	2.02 $\times$	5.39	2.24 $\times$
$\Delta$	$+7.6\%$	$+4.6\%$	$+7.7\%$	$+4.6\%$	$+12.7\%$	$+9.6\%$	$+4.7\%$	$+2.9\%$	$+6.0\%$	$+2.4\%$	$+6.6\%$	$+3.9\%$	$+7.2\%$	$+2.6\%$	$\mathbf{+7.5\%}$	$\mathbf{+4.4\%}$

Appendix

Table 6: Parallel Reconsideration on top of HASS. Acceptance length

\tau

and speedup over autoregressive decoding on Llama-3.1-8B at temperature

0

, both drafters taken at the same training checkpoint and served under the tree configuration of Table 1 (

B=48

\Delta

is the relative gain of adding reconsideration to HASS.

recovered, % of bucket (bucket’s share of pool)
	target confidence $c^{\mathrm{t}}_{p}$
drafter $c^{\mathrm{d}}_{p}$	$<0.5$	$0.5$ – $0.9$	$\geq 0.9$	all
$<0.5$	17.7% (18.5%)	29.5% (26.6%)	39.7% (17.0%)	28.8% (62.1%)
$0.5$ – $0.9$	11.3% (6.7%)	23.4% (13.6%)	42.0% (15.1%)	29.0% (35.4%)
$\geq 0.9$	3.7% (0.4%)	6.9% (1.3%)	27.1% (0.8%)	12.9% (2.5%)
all	16.4% (25.6%)	26.7% (41.5%)	39.9% (32.9%)	28.5% (100%)

Appendix

Table 7: What reconsideration recovers, by bucket. Top: percentage of each bucket’s rejected tokens that Parallel Reconsideration gets right, with the bucket’s share of the reject pool in parentheses. Buckets are the

c^{\mathrm{d}}_{p}\times c^{\mathrm{t}}_{p}

bands of Section 2.2 . Bottom: mean change in top-

1

probability against the position’s own one-pass distribution, on the tokens reconsideration recovers and on those it does not.

Category	Benchmark	Prompts	Source
Chat	MT-Bench	80	FastChat mt_bench
Math	GSM8K	1,319	openai/gsm8k (main)
	MATH-500	500	HuggingFaceH4/MATH-500
	AIME25	30	math-ai/aime25
Code	HumanEval	164	openai/openai_humaneval
	MBPP	500	google-research-datasets/mbpp (full)

Appendix

Table 8: Evaluation benchmarks. Each is evaluated on its full set.

$B$	$B_{w}$	$N$	EAGLE-3.1	PR	measured	serial drafting
			$t_{\mathrm{cycle}}$	$t_{\mathrm{cycle}}$	whole-cycle ratio	$(\gamma{+}N)/\gamma$
48	6	3	26.03	26.60	1.022 $\times$	1.375 $\times$
24	6	3	25.55	25.94	1.015 $\times$	1.375 $\times$

Appendix

Table 9: Per-cycle cost of co-forwarding on Qwen3-8B. Measured ratio of

t_{\mathrm{cycle}}=\tau/\text{throughput}

(time per verification step), Parallel Reconsideration over EAGLE-3.1 run for run at temperature

0

; this is a whole-cycle ratio (drafting plus verification). With

\gamma=8

draft steps and

N

reconsidered positions per draft, serializing would stretch drafting by

(\gamma+N)/\gamma

(Section 3.4 ), an upper bound on the serial whole-cycle ratio. Co-forwarding keeps the measured ratio near

1

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 331 papers

Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds
Oct 7, 2026Yunxiao Zhao, Changxiao CaiSpeculative Decoding Parallel Speculative Decoding
Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, MI, USA.
Carryover Drafting: Recycling Rejected States for Speculative Decoding
Sep 13, 2026Jahyun Koo, Sunghyeon Woo, Jaeeun Kil +4Speculative Decoding LLM Inference Acceleration
Seoul National University · NAVER Cloud
Draft-OPD: On-Policy Distillation for Speculative Draft Models
May 28, 2026Haodi Lei, Yafu Li, Haoran Zhang +8Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Shanghai AI Laboratory · The Chinese University of Hong Kong +3

Oct 7, 2026J/K move · Enter open · S save

Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds

Yunxiao Zhao, Changxiao Cai

Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, MI, USA.

Speculative Decoding Parallel Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Speculative decoding accelerates large language model inference by using a low-cost draft model to propose tokens that the full-size target model verifies in parallel. Parallel and semi-autoregressive (semi- AR) drafters improve drafting efficiency by proposing an entire block in a single forward pass, but training them raises a new difficulty: the draft distribution for a given position depends on where the decoding round starts, and where rounds start depends on how many tokens earlier rounds accepted. Existing training objectives typically rely on block-local surrogates that ignore this cross-round coupling, and therefore do not directly optimize the global decoding efficiency. In this work, we develop a theoretical framework for training and evaluating these drafters by representing speculative decoding as a Markov reward process. This formulation yields the Expected Decoding Rounds (EDR) objective, which weights local rejection costs by state occupancies and exactly equals the expected number of decoding rounds. Unlike prior surrogate objectives, EDR introduces no auxiliary hyperparameters. We then derive an exact temporal-difference gradient that supports unbiased stochastic optimization from target-model rollouts. The same framework also yields an exact offline evaluator for round counts, enabling paired drafter comparisons on shared target rollouts without running speculative decoding. Finetuning two state-of-the- art drafters, DSpark and DFly, with EDR consistently improves mean accepted length and outperforms existing training objectives across nine benchmarks spanning math reasoning, code generation, and chat.

Abstract

Speculative Decoding

Explore similar work

Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds

Carryover Drafting: Recycling Rejected States for Speculative Decoding

Draft-OPD: On-Policy Distillation for Speculative Draft Models