Speculative decoding accelerates LLM inference by verifying multiple drafted tokens in parallel, allowing a single target forward pass to accept several tokens. By construction, verification computes representations for both accepted and rejected tokens. Yet, conventional drafters retain only the representations of accepted tokens, leaving the substantial verifier computation spent on rejected tokens effectively wasted. We find that these discarded hidden states generated during target forward retain useful information about future tokens that can improve subsequent drafts. However, realizing this opportunity poses two distinct challenges. At inference, recycling overhead can increase drafting latency, diminishing the speedup gained from increased acceptance length. During training, standard parallel drafter training does not produce inference-aligned rejected states, while obtaining them through sequential rollouts would sacrifice parallelism across training positions. We introduce Carryover Drafting, which addresses both challenges. Carryover recycles rejected target hidden states as temporary KV context, allowing the drafter to selectively attend to them. It reuses the drafter's existing interface and adds only a single learned embedding to distinguish rejected states from committed context. The additional KV context is replaced each drafting round, keeping its length bounded by one proposal block. We introduce parallel draft--verify--draft training that exposes the drafter to inference-aligned rejected states while preserving parallelism across training positions. Experiments with DFlash and a DSpark-derived semi-autoregressive drafter across two target models show that this simple Carryover mechanism improves average acceptance length by 6.5--14.7% and end-to-end vLLM speedup by 7.9--14.4% over the corresponding baselines, with speedup gains reaching 28.8% on translation.
Figures & tables
Figure 1: Carryover Drafting. Standard block diffusion drafting retains accepted target hidden states and discards rejected states. Carryover recycles rejected states as temporary drafter KV context for the next proposal, allowing the drafter to selectively attend to them through its existing attention layers.
Figure 2: Parallel draft–verify–draft training. Randomly sampled anchors share a single recorded target response. Detached drafter proposals are verified by a frozen target; the second draft uses rejected states and recorded supervision at shifted anchors.
T=0
T=1
τ
Speedup
τ
Speedup
Target
Drafter
Dataset
Baseline ∣
+Carryover
Baseline ∣
+Carryover
Baseline ∣
+Carryover
Baseline ∣
+Carryover
Qwen3-4B
DFlash
HumanEval
4.89 ∣
5.33 (+9.0%)
4.74 ×∣
5.17 × (+9.0%)
4.22 ∣
4.53 (+7.4%)
3.97 ×∣
4.53 × (+14.0%)
Math
5.96 ∣
6.51 (+9.2%)
5.58 ×∣
6.10 × (+9.4%)
5.31 ∣
5.64 (+6.1%)
5.09 ×∣
5.34 × (+4.9%)
MT-Bench
3.45 ∣
3.77 (+9.1%)
3.40 ×∣
3.72 × (+9.5%)
2.98 ∣
3.18 (+6.5%)
2.72 ×∣
2.95 × (+8.6%)
QA
3.03 ∣
3.26 (+7.7%)
2.96 ×∣
3.15 × (+6.5%)
2.61 ∣
2.84 (+8.9%)
2.60 ×∣
2.64 × (+1.5%)
Table 1: Acceptance and speedup over autoregressive decoding. Per-dataset speedup uses offline-vLLM, and Avg. speedup is a separate online-vLLM measurement. Avg. acceptance is micro average over 844 prompts.
Figure 3: Acceptance length at T=0 . Baseline and Carryover for DFlash and Markov with Qwen3-4B and Gemma 4 12B across 8 datasets. The full measurements are in Table 1 .
Throughput (tok/s)
Target
c
AR
DFlash
Markov
Baseline ∣
+Carryover
Baseline ∣
+Carryover
Qwen3-4B
1
251
837 ∣
903 (+7.9%)
861 ∣
961 (+11.5%)
2
491
1,609 ∣
1,739 (+8.1%)
1,652 ∣
1,840 (+11.4%)
4
946
3,015 ∣
3,275 (+8.6%)
3,097 ∣
3,443 (+11.2%)
8
1,800
5,328 ∣
5,741 (+7.7%)
5,337 ∣
5,879 (+10.1%)
Table 2: Throughput across concurrency. Output tokens/s on the mixed 844-prompt workload at T=0 on a single H200 GPU.
Variant
HumanEval
Math
MT-Bench
Baseline (nothing carried)
3.10
3.72
2.46
(a) Carried-over information
Draft-emitted tokens
3.05 (-1.8%)
3.76 (+1.2%)
2.41 (-2.4%)
Target-emitted tokens
3.16 (+1.8%)
3.93 (+5.8%)
2.52 (+2.3%)
Target top-5 weighted
3.18 (+2.7%)
3.92 (+5.4%)
2.56 (+4.0%)
Target hidden states (ours)
3.41 (+10.0%)
4.27 (+14.8%)
2.70 (+9.6%)
Table 3: Carryover-interface ablation. Qwen3-4B Markov diagnostic probe at T=0 . Cells report τ (gain over baseline, %).
Figure 4: Target-token recall from identical decoding anchors. Qwen3-4B Markov on 844 prompts at T=0 : Baseline, Carryover with states (ON), and the same Carryover weights without them (OFF). Appendix B.2 details the sampling and error bars.
Figure 5: Rejected-state statistics and Carryover gains. Final Qwen3-4B Markov at T=0 . Left: local replay gain grouped by rejected-branch statistics. Center: re-converged positions per rejected round. Right: normalized re-convergence versus final dataset-level acceptance gain.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Qwen3-4B
Gemma 4 12B
Target feature layers
1, 9, 17, 25, 33
5, 17, 29, 41, 46
Drafter layers / proposed tokens
5 / 15
5 / 15
Markov correction rank
256
256
Initial anchors / response
512
512
Maximum training sequence length
4,096
4,096
Global batch size
512
512
Appendix
Table 4: Training settings. Shared architecture and optimizer settings in the final release configuration; Carryover continues the corresponding Baseline-10 checkpoint. Target-layer IDs are zero-based transformer-block indices.
Dataset results
Average speedup
Concurrency sweep
Engine
Offline vLLM
Online vLLM
Online vLLM
Workload
Each dataset
Mixed 844 prompts
Mixed 844 prompts
Batch / client concurrency
Batch size 1
Concurrency 1
1, 2, 4, 8, 16, 32
Output cap
2,048
1,024
1,024
Temperature
0 and 1
0 and 1
0
Reported quantity
τ , speedup vs. AR
Speedup vs. AR
Output tokens/s
Appendix
Table 5: Evaluation protocols. Offline and online measurements answer different questions and are not averaged together. All output caps count generated tokens.
Figure 6: Attention diagnostic. Visualized attention for DFlash checkpoints on MT-Bench: baseline, and Carryover rounds with 8 or 15 rejected states. Probabilities are averaged across heads, drafter layers, and rounds in each subset. The display crops context positions without renormalizing the probabilities.
Figure 7: Agreement with the eventual continuation versus acceptance after rejection. 8-dataset associations for final Qwen3-4B Markov. The left statistic counts re-converged positions; the right counts all local proposal–verifier matches after rejection. Both are normalized by the dataset’s Baseline acceptance length.
Figure 8: Rejected-branch structure across drafters. Final Qwen3-4B DFlash and Markov pairs at T=0 . Left: re-converged positions per rejected round. Right: the normalized statistic and final acceptance gain for each dataset and drafter.
Figure 9: Acceptance length at T=1 . Baseline and Carryover for both targets and drafting regimes. These bars complement the greedy results in Figure 3 ; the numerical results are in Table 1 .
Speculative decoding accelerates large language model inference by using a low-cost draft model to propose tokens that the full-size target model verifies in parallel. Parallel and semi-autoregressive (semi- AR) drafters improve drafting efficiency by proposing an entire block in a single forward pass, but training them raises a new difficulty: the draft distribution for a given position depends on where the decoding round starts, and where rounds start depends on how many tokens earlier rounds accepted. Existing training objectives typically rely on block-local surrogates that ignore this cross-round coupling, and therefore do not directly optimize the global decoding efficiency. In this work, we develop a theoretical framework for training and evaluating these drafters by representing speculative decoding as a Markov reward process. This formulation yields the Expected Decoding Rounds (EDR) objective, which weights local rejection costs by state occupancies and exactly equals the expected number of decoding rounds. Unlike prior surrogate objectives, EDR introduces no auxiliary hyperparameters. We then derive an exact temporal-difference gradient that supports unbiased stochastic optimization from target-model rollouts. The same framework also yields an exact offline evaluator for round counts, enabling paired drafter comparisons on shared target rollouts without running speculative decoding. Finetuning two state-of-the- art drafters, DSpark and DFly, with EDR consistently improves mean accepted length and outperforms existing training objectives across nine benchmarks spanning math reasoning, code generation, and chat.
Yunxiao Zhao, Changxiao Cai
Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, MI, USA.
Speculative decoding accelerates large language model inference by pairing a target model with a lightweight draft model whose proposed tokens are verified in parallel. A common way to build draft models, like EAGLE3 or DFlash is supervised fine-tuning (SFT) on target-generated trajectories. However, we observe that SFT quickly plateaus: the draft model's acceptance length on test data stops improving. The reason is an offline-to-inference mismatch: In SFT, the drafter learns from fixed target-generated trajectories, whereas during speculative decoding it is evaluated on blocks proposed under its own policy. This motivates on-policy distillation (OPD), where the target model supervises the drafter on draft-induced states. Yet OPD remains difficult for draft models, as they cannot reliably roll out complete sequences independently, whereas target-assisted generation makes the collected sequences follow the target distribution and thus eliminates the on-policy signal. We therefore propose Draft-OPD, which uses target-assisted rollout for stable continuations and replays drafting from the verification-exposed error positions. This allows the drafter to learn from target feedback on both accepted and rejected proposals, focusing training on the draft-induced errors that limit speculative acceptance. Experiments show that Draft-OPD achieves over 5× lossless acceleration for thinking models across diverse tasks, improving over EAGLE-3 and DFlash by 23% and 13%.
Haodi Lei, Yafu Li, Haoran Zhang +8
Shanghai Jiao Tong University · Shanghai AI Laboratory · The Chinese University of Hong Kong +3
Block diffusion drafters accelerate speculative decoding by generating multiple draft tokens in a single forward pass. However, each pass produces only a fixed-length block from a single target-generated anchor, limiting how far the draft can extend. Meanwhile, target verification computes hidden states even at positions beyond the first rejection, but these states are typically discarded rather than reused. We propose ReAnchor, a training-free method that reanchors a pretrained drafter at shifted positions using discarded verifier states to generate parallel draft blocks. These blocks improve predictions within the original block and extend coverage beyond its endpoint. At sampling temperature of T=0, ReAnchor achieves its largest average gains on Gemma 4-12B, improving acceptance length by 19.4% and decoding speed by 17.9% across eight benchmarks. Averaged equally over four target models and eight benchmarks, it improves acceptance length by 15.2% and decoding speed by 13.4% with pretrained DFlash drafters.