Speculative decoding accelerates autoregressive large language model (LLM) inference by using a lightweight draft model to propose candidate tokens that are then verified in parallel by the target model. The speedup is significantly determined by the acceptance rate, yet standard training minimizes Kullback-Leibler (KL) divergence as a proxy objective. While KL divergence and acceptance rate share the same global optimum, small draft models, having limited capacity, typically converge to suboptimal solutions where minimizing KL does not guarantee maximizing acceptance rate. To address this issue, we propose LK losses, special training objectives that directly target acceptance rate. Comprehensive experiments across four draft architectures and six target models, ranging from 8B to 685B parameters, demonstrate consistent improvements in acceptance metrics across all configurations compared to the standard KL-based training. We evaluate our approach on general, coding and math domains and report gains of up to 8-10% in average acceptance length. LK losses are easy to implement, introduce no computational overhead and can be directly integrated into any existing speculator training framework, making them a compelling alternative to the existing draft training objectives.
Figures & tables
Figure 1: Acceptance length τ vs maximum length K for EAGLE-3 draft models, trained using different objectives with Qwen3-235B-A22B-Instruct as a target model. The values were obtained on the MT-Bench dataset with chain sampling at temperature = 1.
Figure 2: Fitting a single Gaussian to a Gaussian mixture under different objectives. Top: Loss landscapes (log-scale) over parameters μ and σ . Bottom: Resulting distributions and overlap (green). KL divergence produces a mass-covering solution ( α=50.2% ), reverse KL exhibits mode-seeking behavior ( α=50.8% ), while TV maximizes overlap ( α=60.2% ).
MT-Bench
HumanEval
GSM8K
Method
Loss
τ
τ
τ
Temperature=0
EAGLE-3
KL
3.75
4.82
4.50
TV
2.81
3.42
3.34
LLKα
3.77
4.82
4.55
LLKλ,λ=0.5
3.78
4.84
4.53
Table 1: Average acceptance length τ for LLaMA-3.1-8B-Instruct with EAGLE-3, MEDUSA, and MLP draft models.
Temperature = 0
Temperature = 1
Model
Method/Loss
MT ( τ )
HE ( τ )
GSM ( τ )
Mean ( Δ% )
MT ( τ )
HE ( τ )
GSM ( τ )
Mean ( Δ% )
LLaMA 3.1
EAGLE-3 KL
3.75
4.82
4.50
4.36
3.39
4.31
3.88
3.86
8B Instruct
EAGLE-3 LLKλ
3.84
4.89
4.57
4.43 (+1.6)
3.48
4.52
4.02
4.01 (+3.9)
LLaMA 3.3
EAGLE-3 KL
4.01
5.18
5.16
4.78
3.76
4.86
4.89
4.50
70B Instruct
EAGLE-3 LLKλ
4.00
5.21
5.21
4.81 (+0.5)
3.89
5.08
5.01
4.66 (+3.5)
GPT-OSS 20B
EAGLE-3 KL
3.31
3.07
4.01
3.46
3.12
2.89
3.51
3.17
Table 2: Average acceptance length τ for other target models under different speculative decoding methods. Values in parentheses denote relative improvement (%) of LLKλ ( η=3 ) over KL in average acceptance length. MT: MT-Bench; HE: HumanEval; GSM: GSM8K.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Loss
Gradient on S
Gradient off S
KL
−1/k
+1/V
TV
−1/V
≈0
LLKα
−1/k
+1/V
Appendix
Table 3: Gradient components for different losses in the diffuse- q , concentrated- p regime.
Model
Method
Type
Setup
Temperature = 0
Temperature = 1
MT-Bench
HumanEval
GSM8K
MT-Bench
HumanEval
GSM8K
τ / speedup
τ / speedup
τ / speedup
τ / speedup
τ / speedup
τ / speedup
LLaMA 3.1
EAGLE-3
Ours
KL
3.75 / 2.60
4.82 / 3.30
4.50 / 3.05
3.39 / 2.29
4.31 / 3.00
3.88 / 2.63
8B Instruct
TV
2.81 / 1.96
3.42 / 2.39
3.34 / 2.28
2.67 / 1.83
3.25 / 2.26
3.12 / 2.11
LLKα
3.77 / 2.61
4.82 / 3.28
4.55 / 3.07
3.50 / 2.30
4.48 / 3.03
3.98 / 2.72
LLK , λ=0.5
3.78 / 2.63
4.84 / 3.32
4.53 / 3.07
3.35 / 2.36
4.36 / 3.03
3.95 / 2.68
Appendix
Table 4: Average acceptance length τ and end-to-end speedup relative to the baseline without speculative decoding. We evaluate speculative decoding methods trained with different objectives on MT-Bench, HumanEval, and GSM8K in a low-latency setting with batch size 1 , and compare them against public Hugging Face checkpoints at temperatures T=0 and T=1 . Type: Ours = models trained from scratch with the specified objective; HF = public HuggingFace checkpoints evaluated with the same inference pipeline. 7 7 7 RH= RedHatAI , YH= yuhuili , ZK= zhuyksir , DS= deepseek-ai . RH-8B= Llama-3.1-8B-Instruct-speculator.eagle3 ; YH-8B= EAGLE3-LLaMA3.1-Instruct-8B ; ZK-8B= EAGLE3-Llama-3.1-8B-Instruct ; RH-70B= Llama-3.3-70B-Instruct-speculator.eagle3 ; YH-70B= EAGLE3-LLaMA3.3-Instruct-70B ; RH-20B= gpt-oss-20b-speculator.eagle3 ; RH-235B= Qwen3-235B-A22B-Instruct-2507-speculator.eagle3 ; ZK-235B= EAGLE3-Qwen3-235B-A22B-Instruct-2507-FP8 ; DS-685B= DeepSeek-V3-0324 .
Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to generate multiple candidate tokens that are verified by the target model in a single forward pass. Its speedup is largely determined by the acceptance length, yet existing draft-model training methods mainly optimize cross-entropy or Kullback-Leibler (KL) divergence as proxies. These objectives encourage distribution matching but do not directly optimize acceptance length, and the acceptance mechanism also differs between greedy and sampling-based decoding. In this work, we propose acceptance-length-aware training losses that directly optimize the expected number of accepted tokens within a speculative window. For greedy verification, we derive an expected accepted length (EAL) loss that explicitly maximizes expected acceptance length. For sampling-based decoding, we introduce a window total variation (WTV) loss that optimizes the overlap between temperature-scaled draft and target distributions while accounting for sequential acceptance dependencies. Both objectives can be further combined with a group-relative reinforcement learning stage (GRPO) using simulated acceptance length as the reward. Experiments across different target and draft models, tasks, and decoding settings show that our losses consistently improve acceptance length over KL-based training. WTV provides particularly strong gains under sampling-based decoding, while EAL better matches greedy verification. These results show that directly optimizing the acceptance objective, with losses tailored to the decoding mode, is more effective than conventional distribution-matching objectives.
Tianhua Xia, Mugilan Ganesan, Yifei Feng +3
New York University · University of Waterloo · Independent Researcher
Speculative decoding is a technique that accelerates the inference speed of autoregressive large language models (LLMs) by sampling from a faster, lower quality draft model. Traditionally, a fundamental characteristic of speculative decoding is that it is lossless with respect to the verifier distribution, guaranteeing speed without sacrificing quality. This differs from recently developed diffusion LLMs (dLLMs), which are notably able to achieve substantial speedups by degrading the quality of their outputs with less diffusion effort. Because diffusion models are able to sample faster than autoregressive models by generating tokens in parallel, they are considered strong candidates for draft models in speculative diffusion. While it is widely understood that diffusion models can sacrifice quality for speed, speculative decoding itself can also be generalized to generate lossy samples faster using more lenient acceptance criteria with respect to the verifier distribution. Presented with the two parameters of diffusion effort and speculative decoding leniency, this paper explores the full achievable Pareto frontier via joint variations of these parameters. Our experiments show that jointly varying diffusion effort and acceptance leniency can extend the speed-quality Pareto frontier over single step diffusion drafting.
Aaron Zorel, Guy Van den Broeck, Aditya Grover +1
Independent Researcher · University of California, Los Angeles
Speculative decoding accelerates large language model inference by employing a compact draft model to propose candidate tokens that are verified in parallel by a target model. Conventional training relies on soft-label distillation, which forces the draft model to mimic the target's full output distribution, including high-entropy regions that degrade acceptance rates and increase early rejection risk. We propose RCSD (Reinforcement and Continual Learning for Speculative Decoding), a training framework that aligns draft model optimization with the system-level objective of maximizing acceptance length. RCSD leverages naturally occurring accepted and rejected sequences during speculative inference to construct implicit preference pairs, with masked loss to avoid penalizing target-aligned tokens. Evaluated across multiple benchmarks, RCSD improves average acceptance length by 35% over HASS and achieves 1.35x faster end-to-end inference than HASS on practical workloads, without compromising generation quality.