Carryover Drafting: Recycling Rejected States for Speculative Decoding

Jahyun Koo; Sunghyeon Woo; Jaeeun Kil; Jeongtae Lee; Sungjae Lee; Kyomin Jung; Minsub Kim

cs.LGSep 13, 2026

Carryover Drafting: Recycling Rejected States for Speculative Decoding

Authors: Jahyun Koo, Sunghyeon Woo, Jaeeun Kil, Jeongtae Lee, Sungjae Lee, Kyomin Jung, Minsub Kim

Organizations: Seoul National University · NAVER Cloud

Abstract

Speculative decoding accelerates LLM inference by verifying multiple drafted tokens in parallel, allowing a single target forward pass to accept several tokens. By construction, verification computes representations for both accepted and rejected tokens. Yet, conventional drafters retain only the representations of accepted tokens, leaving the substantial verifier computation spent on rejected tokens effectively wasted. We find that these discarded hidden states generated during target forward retain useful information about future tokens that can improve subsequent drafts. However, realizing this opportunity poses two distinct challenges. At inference, recycling overhead can increase drafting latency, diminishing the speedup gained from increased acceptance length. During training, standard parallel drafter training does not produce inference-aligned rejected states, while obtaining them through sequential rollouts would sacrifice parallelism across training positions. We introduce Carryover Drafting, which addresses both challenges. Carryover recycles rejected target hidden states as temporary KV context, allowing the drafter to selectively attend to them. It reuses the drafter's existing interface and adds only a single learned embedding to distinguish rejected states from committed context. The additional KV context is replaced each drafting round, keeping its length bounded by one proposal block. We introduce parallel draft--verify--draft training that exposes the drafter to inference-aligned rejected states while preserving parallelism across training positions. Experiments with DFlash and a DSpark-derived semi-autoregressive drafter across two target models show that this simple Carryover mechanism improves average acceptance length by 6.5--14.7% and end-to-end vLLM speedup by 7.9--14.4% over the corresponding baselines, with speedup gains reaching 28.8% on translation.

Figures & tables

Figure 1: Carryover Drafting. Standard block diffusion drafting retains accepted target hidden states and discards rejected states. Carryover recycles rejected states as temporary drafter KV context for the next proposal, allowing the drafter to selectively attend to them through its existing attention layers.

Figure 2: Parallel draft–verify–draft training. Randomly sampled anchors share a single recorded target response. Detached drafter proposals are verified by a frozen target; the second draft uses rejected states and recorded supervision at shifted anchors.

Target	Drafter	Dataset	Baseline $\|$	+Carryover	Baseline $\|$	+Carryover	Baseline $\|$	+Carryover	Baseline $\|$	+Carryover
			$T{=}0$				$T{=}1$
			$\tau$		Speedup		$\tau$		Speedup
Qwen3-4B	DFlash	HumanEval	4.89 $\|$	5.33 (+9.0%)	4.74 $\times$ $\|$	5.17 $\times$ (+9.0%)	4.22 $\|$	4.53 (+7.4%)	3.97 $\times$ $\|$	4.53 $\times$ (+14.0%)
		Math	5.96 $\|$	6.51 (+9.2%)	5.58 $\times$ $\|$	6.10 $\times$ (+9.4%)	5.31 $\|$	5.64 (+6.1%)	5.09 $\times$ $\|$	5.34 $\times$ (+4.9%)
		MT-Bench	3.45 $\|$	3.77 (+9.1%)	3.40 $\times$ $\|$	3.72 $\times$ (+9.5%)	2.98 $\|$	3.18 (+6.5%)	2.72 $\times$ $\|$	2.95 $\times$ (+8.6%)
		QA	3.03 $\|$	3.26 (+7.7%)	2.96 $\times$ $\|$	3.15 $\times$ (+6.5%)	2.61 $\|$	2.84 (+8.9%)	2.60 $\times$ $\|$	2.64 $\times$ (+1.5%)

Table 1: Acceptance and speedup over autoregressive decoding. Per-dataset speedup uses offline-vLLM, and Avg. speedup is a separate online-vLLM measurement. Avg. acceptance is micro average over 844 prompts.

Figure 3: Acceptance length at

T=0

. Baseline and Carryover for DFlash and Markov with Qwen3-4B and Gemma 4 12B across 8 datasets. The full measurements are in Table 1 .

Target	$c$	AR	DFlash		Markov
		Throughput (tok/s)
			Baseline $\|$	+Carryover	Baseline $\|$	+Carryover
Qwen3-4B	1	251	837 $\|$	903 (+7.9%)	861 $\|$	961 (+11.5%)
	2	491	1,609 $\|$	1,739 (+8.1%)	1,652 $\|$	1,840 (+11.4%)
	4	946	3,015 $\|$	3,275 (+8.6%)	3,097 $\|$	3,443 (+11.2%)
	8	1,800	5,328 $\|$	5,741 (+7.7%)	5,337 $\|$	5,879 (+10.1%)

Table 2: Throughput across concurrency. Output tokens/s on the mixed 844-prompt workload at

T=0

on a single H200 GPU.

(a) Carried-over information
Variant	HumanEval	Math	MT-Bench
Baseline (nothing carried)	3.10	3.72	2.46
Draft-emitted tokens	3.05 (-1.8%)	3.76 (+1.2%)	2.41 (-2.4%)
Target-emitted tokens	3.16 (+1.8%)	3.93 (+5.8%)	2.52 (+2.3%)
Target top-5 weighted	3.18 (+2.7%)	3.92 (+5.4%)	2.56 (+4.0%)
Target hidden states (ours)	3.41 (+10.0%)	4.27 (+14.8%)	2.70 (+9.6%)

Table 3: Carryover-interface ablation. Qwen3-4B Markov diagnostic probe at

T=0

. Cells report

\tau

(gain over baseline, %).

Figure 4: Target-token recall from identical decoding anchors. Qwen3-4B Markov on 844 prompts at

T=0

: Baseline, Carryover with states (ON), and the same Carryover weights without them (OFF). Appendix B.2 details the sampling and error bars.

Figure 5: Rejected-state statistics and Carryover gains. Final Qwen3-4B Markov at

T=0

. Left: local replay gain grouped by rejected-branch statistics. Center: re-converged positions per rejected round. Right: normalized re-convergence versus final dataset-level acceptance gain.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Qwen3-4B	Gemma 4 12B
Target feature layers	1, 9, 17, 25, 33	5, 17, 29, 41, 46
Drafter layers / proposed tokens	5 / 15	5 / 15
Markov correction rank	256	256
Initial anchors / response	512	512
Maximum training sequence length	4,096	4,096
Global batch size	512	512

Appendix

Table 4: Training settings. Shared architecture and optimizer settings in the final release configuration; Carryover continues the corresponding Baseline-10 checkpoint. Target-layer IDs are zero-based transformer-block indices.

	Dataset results	Average speedup	Concurrency sweep
Engine	Offline vLLM	Online vLLM	Online vLLM
Workload	Each dataset	Mixed 844 prompts	Mixed 844 prompts
Batch / client concurrency	Batch size 1	Concurrency 1	1, 2, 4, 8, 16, 32
Output cap	2,048	1,024	1,024
Temperature	0 and 1	0 and 1	0
Reported quantity	$\tau$ , speedup vs. AR	Speedup vs. AR	Output tokens/s

Appendix

Table 5: Evaluation protocols. Offline and online measurements answer different questions and are not averaged together. All output caps count generated tokens.

Figure 6: Attention diagnostic. Visualized attention for DFlash checkpoints on MT-Bench: baseline, and Carryover rounds with 8 or 15 rejected states. Probabilities are averaged across heads, drafter layers, and rounds in each subset. The display crops context positions without renormalizing the probabilities.

Figure 7: Agreement with the eventual continuation versus acceptance after rejection. 8-dataset associations for final Qwen3-4B Markov. The left statistic counts re-converged positions; the right counts all local proposal–verifier matches after rejection. Both are normalized by the dataset’s Baseline acceptance length.

Figure 8: Rejected-branch structure across drafters. Final Qwen3-4B DFlash and Markov pairs at

T=0

. Left: re-converged positions per rejected round. Right: the normalized statistic and final acceptance gain for each dataset and drafter.

Figure 9: Acceptance length at

T=1

. Baseline and Carryover for both targets and drafting regimes. These bars complement the greedy results in Figure 3 ; the numerical results are in Table 1 .

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 291 papers

Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds
Oct 7, 2026Yunxiao Zhao, Changxiao CaiSpeculative Decoding Parallel Speculative Decoding
Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, MI, USA.
Draft-OPD: On-Policy Distillation for Speculative Draft Models
May 28, 2026Haodi Lei, Yafu Li, Haoran Zhang +8Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Shanghai AI Laboratory · The Chinese University of Hong Kong +3
ReAnchor: Reusing Rejected Verifier States for Diffusion Drafting
Oct 19, 2026Sanghyun Lee, Wonjin Nam, Seungho Jang +4Speculative Decoding Speculative Decoding for Diffusion Language Models
KAIST · Seoul National University

Oct 7, 2026J/K move · Enter open · S save

Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds

Yunxiao Zhao, Changxiao Cai

Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, MI, USA.

Speculative Decoding Parallel Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Speculative decoding accelerates large language model inference by using a low-cost draft model to propose tokens that the full-size target model verifies in parallel. Parallel and semi-autoregressive (semi- AR) drafters improve drafting efficiency by proposing an entire block in a single forward pass, but training them raises a new difficulty: the draft distribution for a given position depends on where the decoding round starts, and where rounds start depends on how many tokens earlier rounds accepted. Existing training objectives typically rely on block-local surrogates that ignore this cross-round coupling, and therefore do not directly optimize the global decoding efficiency. In this work, we develop a theoretical framework for training and evaluating these drafters by representing speculative decoding as a Markov reward process. This formulation yields the Expected Decoding Rounds (EDR) objective, which weights local rejection costs by state occupancies and exactly equals the expected number of decoding rounds. Unlike prior surrogate objectives, EDR introduces no auxiliary hyperparameters. We then derive an exact temporal-difference gradient that supports unbiased stochastic optimization from target-model rollouts. The same framework also yields an exact offline evaluator for round counts, enabling paired drafter comparisons on shared target rollouts without running speculative decoding. Finetuning two state-of-the- art drafters, DSpark and DFly, with EDR consistently improves mean accepted length and outperforms existing training objectives across nine benchmarks spanning math reasoning, code generation, and chat.

Abstract

Speculative Decoding

Explore similar work

Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds

Draft-OPD: On-Policy Distillation for Speculative Draft Models

ReAnchor: Reusing Rejected Verifier States for Diffusion Drafting