Speculative decoding accelerates large language model inference by employing a compact draft model to propose candidate tokens that are verified in parallel by a target model. Conventional training relies on soft-label distillation, which forces the draft model to mimic the target's full output distribution, including high-entropy regions that degrade acceptance rates and increase early rejection risk. We propose RCSD (Reinforcement and Continual Learning for Speculative Decoding), a training framework that aligns draft model optimization with the system-level objective of maximizing acceptance length. RCSD leverages naturally occurring accepted and rejected sequences during speculative inference to construct implicit preference pairs, with masked loss to avoid penalizing target-aligned tokens. Evaluated across multiple benchmarks, RCSD improves average acceptance length by 35% over HASS and achieves 1.35x faster end-to-end inference than HASS on practical workloads, without compromising generation quality.
Figures & tables
Figure 1: Misalignment between KL divergence and acceptance rate. Due to limited parameters, the draft model cannot perfectly replicate the target distribution. However, low KL loss may still lead to token rejection. This discrepancy arises because speculative decoding acceptance depends on the relative ranking of top candidate tokens rather than exact distribution matching.
Figure 2: Output Throughput (TPS) and Mean Accept Length across model configurations. Evaluated on Llama-3.1-8B-Instruct with SGLang 5.5.0, the results demonstrate that SFT+DPO and SFT+GRPO consistently outperform the baseline SFT approach on both metrics.
Figure 3: Overview of the RCSD framework. Given requests and dataset samples, a draft model πθ first proposes k tokens via speculative decoding, and a target model πT performs parallel verification. The verification stage partitions outputs into accepted tokens (forming the chosen sequence y+ ) and early-rejected tokens (forming the rejected sequence y− ), from which preference or reward signals are constructed. The resulting pairs (x,y+,y−) or response groups G are then used to update πθ via DPO- or GRPO-based optimization. In DPO-based optimization, target-model preferences between y+ and y− guide parameter updates. In GRPO-based optimization, grouped responses are optimized using an acceptance-rate reward.
Dataset
Model
Step 3
Step 4
Step 5
Step 6
HumanEval
HASS
2.322
2.446
2.510
2.533
HASS + DPO
2.616
2.780
2.922
3.014
HASS + GRPO
2.629
2.807
2.928
3.015
MT-Bench
HASS
1.986
2.047
2.072
2.092
HASS + DPO
2.333
2.442
2.537
2.571
HASS + GRPO
2.346
2.467
2.549
2.601
Table 1: Qwen3-1.7B: results at Steps 3–6. Baseline HASS, HASS+DPO, and HASS+GRPO are compared. Higher values indicate better acceptance length. HASS+GRPO generally achieves better performance.
Dataset
Model
Step 3
Step 4
Step 5
Step 6
HumanEval
HASS
2.731
2.950
3.103
3.202
HASS + DPO
2.896
3.185
3.391
3.540
HASS + GRPO
2.897
3.189
3.402
3.558
MT-Bench
HASS
2.282
2.397
2.438
2.497
HASS + DPO
2.420
2.595
2.674
2.731
HASS + GRPO
2.426
2.596
2.692
2.736
Table 2: Llama-3.1-8B: results at Steps 3–6. HASS+GRPO generally achieves better performance.
Speculative decoding (SD) accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens and a stronger target model to verify them. However, standard SD is mainly designed for acceleration, and its output quality is typically constrained by the target model. In this work, we propose Entropy-Aware Speculative Decoding (EASD), a lightweight and training-free extension of SD that improves reasoning quality through token-level entropy-guided rejection. EASD detects cases where both draft and target models exhibit high uncertainty while strongly overlapping in their top predictions. In such uncertain-agreement cases, EASD rejects the aligned token and resamples from the target distribution, preventing low-confidence errors from propagating. Experiments on challenging reasoning benchmarks show that EASD consistently improves accuracy over standard SD and reward-guided variants while maintaining comparable inference efficiency. Notably, EASD can surpass the standalone performance of the target model, suggesting that speculative decoding can serve not only as an acceleration method but also as an effective mechanism for improving reasoning quality. The code is available at https://github.com/ECNU-Text-Computing/EASD.
Tiancheng Su, Meicong Zhang, Guoxiu He
School of Economics and Management East China Normal University
Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to generate multiple candidate tokens that are verified by the target model in a single forward pass. Its speedup is largely determined by the acceptance length, yet existing draft-model training methods mainly optimize cross-entropy or Kullback-Leibler (KL) divergence as proxies. These objectives encourage distribution matching but do not directly optimize acceptance length, and the acceptance mechanism also differs between greedy and sampling-based decoding. In this work, we propose acceptance-length-aware training losses that directly optimize the expected number of accepted tokens within a speculative window. For greedy verification, we derive an expected accepted length (EAL) loss that explicitly maximizes expected acceptance length. For sampling-based decoding, we introduce a window total variation (WTV) loss that optimizes the overlap between temperature-scaled draft and target distributions while accounting for sequential acceptance dependencies. Both objectives can be further combined with a group-relative reinforcement learning stage (GRPO) using simulated acceptance length as the reward. Experiments across different target and draft models, tasks, and decoding settings show that our losses consistently improve acceptance length over KL-based training. WTV provides particularly strong gains under sampling-based decoding, while EAL better matches greedy verification. These results show that directly optimizing the acceptance objective, with losses tailored to the decoding mode, is more effective than conventional distribution-matching objectives.
Tianhua Xia, Mugilan Ganesan, Yifei Feng +3
New York University · University of Waterloo · Independent Researcher
Speculative decoding (SPD) accelerates large language model (LLM) inference by letting a smaller draft model propose multiple future tokens that are verified in parallel by a larger target model. The dominant SPD paradigm treats the target model as the sole reliable teacher, accepting a draft token only when it exactly matches the target prediction. This design implicitly assumes that the target is always the better choice at every position. In practice, this assumption does not hold. Although the draft is the weaker model overall, it is not uniformly inferior at the token level. In a meaningful fraction of cases where draft and target disagree, the draft's choice is the one that leads to the correct final answer. Inspired by this, we introduce Collaborative Speculative Decoding (CoSpec), a generalization of SPD that no longer treats the target model as the sole token-level authority. CoSpec trains an arbitration policy via reinforcement learning to decide whether to accept tokens from the draft or target model, selectively accepting draft tokens at mismatches when doing so is likely to yield a correct final answer. Experimental results show that CoSpec maintains substantial speedups while surpassing target-only performance. By shifting the emphasis from imitation to collaboration, CoSpec suggests a new perspective on speculative decoding.
Jinze Li, Yixing Xu, Guanchen Li +7
Advanced Micro Devices, Inc., Beijing, China · The University of Hong Kong · University of North Texas