Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning
Authors: Jiebin Zhang, Zhenghan Yu, Liang Wang, Nan Yang, Eugene J. Yu, Zheng Li, Yifan Song, Dawei Zhu, Xingxing Zhang, Furu Wei, Sujian Li
Organizations: National Key Laboratory for Multimedia Information Processing, Peking University · School of Computer Science, Peking University · Microsoft Research Asia
Speculative decoding accelerates large language model (LLM) inference by using a small draft model to generate candidate tokens for a larger target model to verify. The efficacy of this technique hinges on the trade-off between the time spent on drafting candidates and verifying them. However, current state-of-the-art methods rely on a static time allocation, while recent dynamic approaches optimize for proxy metrics like acceptance length, often neglecting the true time cost and treating the drafting and verification phases in isolation. To address these limitations, we introduce Learning to Draft (LTD), a novel method that directly optimizes for throughput of each draft-and-verify cycle. We formulate the problem as a reinforcement learning environment and train two co-adaptive policies to dynamically coordinate the draft and verification phases. This encourages the policies to adapt to each other and explicitly maximize decoding efficiency. We conducted extensive evaluations on five diverse LLMs and four distinct tasks. Our results show that LTD achieves speedup ratios ranging from 2.24x to 4.32x, outperforming the state-of-the-art method Eagle3 up to 36.4%.
Figures & tables
Figure 1: The overview of LTD method. We formulate the interaction between draft model and target model as an RL environment. We employ two policies to dynamically manage the allocation between draft generation and target verification by controlling the draft tree’s depth and verification size. These policies are optimized together using the reward signal based on the throughput of each draft-and-verify cycle.
Figure 2: Analysis of Draft Time and Verification Time. Left: The average draft and verification latency of Vicuna-13B as a function of draft depth D and verification size V on benchmarks. Right: The composition of the total inference time for Eagle3 and LTD on the benchmarks, with Eagle3’s total time normalized to 100% as baselines.
Figure 3: We report the training reward curves for the Size Policy and Depth Policy of the Vicuna-13B model, as well as the evolution of the speedup ratio on the validation set.
MT-bench
Gsm8k
Alpaca
Qa
Mean
Model
Method
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Temperature=0
L31 8B
Eagle3
3.70
6.39
3.58
6.44
4.26
6.93
3.17
5.35
3.68
6.28
Eagle3+DDD
3.67
5.41
3.55
5.48
3.97
5.58
3.07
4.21
3.57
5.17
Eagle3+Svip
3.65
6.21
3.47
6.21
4.04
6.73
3.12
4.78
3.57
5.98
Eagle3+GT
3.79
5.86
3.64
5.77
4.37
6.49
3.38
4.92
3.80
5.76
Table 1: Speedup ratios and average acceptance lengths τ of different methods during greedy decoding. L31 8B represents Llama-3.1-8B-Instruct, V 13B represents Vicuna-13B-v1.3, Dpsk 8B represents DeepSeek-R1-Distill-LLaMA 8B. For the Qwen3-14B and Qwen3-32B models, their corresponding Eagle3 models lack default settings. Therefore, only the results from Grid Search are presented. GT, Spec++, and Gs stand for Gammatune, SpecDec++, and Grid Search, respectively. The results of Temperature=1 are presented in Table 5 .
Figure 4: Left: The effectiveness of our iterative training strategy on Llama3-8B and Dpsk-8B models. Right: The contribution of each component on Vicuna-13B model.
Reward
MT-bench
Gsm8k
Alpaca
Qa
Mean
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Acceptance Length
3.77
7.70
3.81
7.20
3.71
7.05
3.06
5.61
3.59
6.89
Time Cost
3.89
6.49
3.54
5.80
3.48
5.79
2.87
4.54
3.45
5.66
Throughput
4.40
7.68
4.43
7.10
4.23
7.00
3.47
5.56
4.13
6.84
Table 2: The effectiveness of reward signal on size policy, evaluated on Vicuna-13B model.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Method Name
Predict Policy
Observation Space
Dynamic Size or Depth
Training Methods
Objective
DDD ( Brown et al., 2024 )
Heuristic
Probability
Depth
-
LA
SVIP ( Zhang et al., 2024c )
Heuristic
Entropy
Depth
-
LA
Gammatune ( Gautam et al., 2025 )
Heuristic
Token Acceptance Rate
Depth
-
LA
Disco ( Mamou et al., 2024 )
FFN
Probability, Entropy, Draft Depth
Depth
FT
LA
SpecDec++ ( Huang et al., 2025a )
FFN
Hidden States
Depth
FT
LA
C2T ( Huo et al., 2025 )
FFN
Probability, Entropy, Draft Depth
Size
FT
LA
Appendix
Table 3: Comparison of Different Dynamic Draft Tree Methods
MT-bench
Gsm8k
Alpaca
QA
Mean
Method
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Eagle3
4.48
7.22
4.36
6.58
4.16
6.47
3.43
5.12
4.11
6.35
-Depth
4.40
7.68
4.43
7.10
4.23
7.00
3.47
5.56
4.13
6.84
-Size
4.77
7.75
4.46
6.82
4.36
6.59
3.55
5.12
4.29
6.57
-Iterative Training
4.54
8.29
4.48
7.41
4.28
7.08
3.51
5.45
4.20
7.06
LTD
4.64
8.24
4.59
7.38
4.40
7.01
3.66
5.37
4.32
7.00
Appendix
Table 4: Contribution of each components of LTD, evaluated on Vicuna-13B.
MT-bench
Gsm8k
Alpaca
Qa
Mean
Model
Method
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Temperature=1
L31 8B
Eagle3
2.85
4.11
3.43
5.24
3.38
5.47
2.27
3.06
2.98
4.47
Eagle3+DDD
2.89
3.40
3.35
4.09
3.38
4.59
2.36
2.95
3.00
3.76
Eagle3+Svip
2.74
3.41
3.21
4.92
3.19
4.95
2.31
3.20
2.86
4.12
Eagle3+GT
2.75
3.54
3.28
4.27
3.51
5.12
2.46
3.24
3.00
4.04
Appendix
Table 5: Speedup ratios and average acceptance lengths τ of different methods during Temperature=1. L31 8B represents Llama-3.1-8B-Instruct, V 13B represents Vicuna-13B-v1.3, Dpsk 8B represents DeepSeek-R1-Distill-LLaMA 8B. For the Qwen3-14B and Qwen3-32B models, their corresponding Eagle3 models lack default settings. Therefore, only the results from Grid Search are presented. GT, Spec++, and Gs stand for Gammatune, SpecDec++, and Grid Search, respectively.
Reward
MT-bench
Gsm8k
Alpaca
Qa
Mean
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
P+L
4.35
7.68
4.30
7.10
4.18
7.00
3.35
5.55
4.05
6.83
P+D
4.31
7.68
4.35
7.10
4.22
7.00
3.41
5.56
4.07
6.84
P+D+L (LTD)
4.40
7.68
4.43
7.10
4.23
7.00
3.47
5.56
4.13
6.84
P+D+L+H
4.40
7.53
4.32
6.97
4.22
6.94
3.44
5.51
4.10
6.74
P+D+L+E
4.36
7.68
4.36
7.10
4.28
7.00
3.43
5.56
4.11
6.84
Appendix
Table 6: Ablation Study for Observation Space on size policy evaluated on Vicuna-13B.
Figure 5: The average acceptance length for Llama-3.1-8B-Instruct using Eagle3 on HumanEval dataset. Left: Change the draft depth with verification size fixed to 60. Right: Change the verification size with draft depth fixed to 8.
MT-bench
Gsm8k
Alpaca
Qa
Mean
Method
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Eagle3
3.70
6.39
3.58
6.44
4.26
6.93
3.17
5.35
3.68
6.28
Iter0
3.88
6.46
3.76
6.40
4.34
6.82
3.50
5.12
3.87
6.20
Iter1
3.95
6.50
3.80
6.47
4.36
6.89
3.50
5.18
3.90
6.26
Iter2
3.96
6.74
3.79
6.73
4.53
7.31
3.41
5.44
3.92
6.56
Iter3
3.93
6.74
3.73
6.73
4.57
7.31
3.44
5.44
3.92
6.56
Appendix
Table 7: Llama3.1-8B Iterative Optimizing
MT-bench
Gsm8k
Alpaca
Qa
Mean
Method
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Eagle3
3.67
5.82
4.73
7.39
3.65
5.63
3.14
5.02
3.80
5.53
Iter0
3.88
5.95
4.69
8.03
3.68
5.63
3.37
5.07
3.91
5.72
Iter1
3.96
5.98
4.92
8.14
3.86
5.67
3.46
5.08
4.05
5.78
Iter2
4.02
6.22
4.94
8.36
3.71
5.91
3.52
5.29
4.05
5.97
Iter3
4.05
6.32
4.98
8.53
3.98
6.00
3.62
5.37
4.16
6.08
Appendix
Table 8: Dpsk-8B Iterative Optimizing
Figure 6: We report the interaction between depth policy and size policy on Llama3-8B for Iter0 and LTD.
Time per Sample
MT-bench
Alpaca
Qa
Gsm8K
Mean
Llama-3.1-8B-Instruct
Size Policy Latency (ms)
21.4
6.22
6.36
5.83
9.96
Depth Policy Latency (ms)
40.8
12.4
10.2
11.9
18.8
Total Latency per Sample (s)
4.48
1.28
1.27
1.24
2.07
Size Policy Overhead (%)
0.478
0.485
0.500
0.469
0.483
Depth Policy Overhead (%)
0.910
0.969
0.803
0.955
0.909
Appendix
Table 9: Latency and overhead analysis of dynamic policies on Vicuna-13B and Llama31-8B. Policy latency is measured in milliseconds (ms), total generation latency in seconds (s), and overhead in percentage (%).
Configuration
MT-bench
Gsm8k
Alpaca
Qa
Mean
[1024]
3.84
3.99
4.39
3.22
3.86
[1024, 256]
3.95
4.03
4.44
3.24
3.92
[1024, 512, 256]
3.96
4.07
4.43
3.25
3.93
Appendix
Table 10: Speedup of the Size Policy across three architecture configurations on Llama-3.1-8B-Instruct.
Figure 7: Speedup comparison between LTD and Eagle3 on the Llama-3.1-8B-Instruct model. We evaluated performance across 57 subtasks from MMLU(e.g., mathematics, history, law), using 80 sampled examples per subtask. Domain levels are abbreviated as H (High School), C (College), and P (Professional). Left: The relative speedup of LTD normalized against Eagle3; values greater than 100% indicate LTD outperforms Eagle3. Right: The absolute speedup ratio of the Eagle3 baseline across different subtasks.
MT-bench
Gsm8k
Alpaca
Qa
Mean
Method
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Griffin
1.83
4.82
3.18
5.16
2.98
4.79
2.20
3.96
2.55
4.68
Griffin+GS
2.81
5.11
3.19
5.47
2.95
5.04
2.18
4.10
2.78
4.93
Griffin+LTD
2.88
4.98
3.13
5.45
2.98
5.01
2.23
4.10
2.81
4.89
Appendix
Table 11: Speedup ratios and average acceptance lengths τ of different methods on Griffin using Meta-Llama-3-8B-Instruct during greedy decoding.
Speculative decoding has emerged as a widely adopted paradigm for accelerating large language model inference, where a lightweight draft model rapidly generates candidate tokens that are then verified in parallel by a larger target model. However, due to limited model capacity, drafts often struggle to approximate the target distribution, resulting in shorter acceptance lengths and diminished speedup. A key yet under-explored observation is that speculative decoding inherently provides verification feedback that quantifies the deviation between the draft and target models at no additional cost. This process naturally forms an iterative "draft commits-feedback provides-draft adapts" evolving loop, which precisely matches the online learning paradigm. Motivated by this connection, we propose OnlineSPEC, a unified framework that systematically leverages interactive feedback to continuously evolve draft models. Grounded in dynamic regret minimization, we establish a formal link between online learning performance and speculative system's acceleration rate, and develop novel algorithms via modern online learning techniques, including optimistic online learning that adaptively reuses historical gradients as predictive update hints, and online ensemble learning that dynamically maintains multiple draft models. Our algorithms are equipped with theoretical justifications and improved acceleration rates, achieving up to 24% speedup over seven benchmarks and five foundation models.
Yu-Yang Qian, Hao-Cong Wu, Yichao Fu +2
State Key Laboratory for Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · University of California, San Diego
Speculative decoding (SD) accelerates large language model inference by using a smaller draft model to propose draft tokens that are subsequently verified by a larger target model. However, the performance of standard SD is often limited by the strictly sequential execution of these drafting and verification stages. To address this, this paper proposes MineDraft, a batch parallel speculative decoding (PSD) framework designed to effectively hide drafting latency by overlapping it with verification. Our theoretical analysis shows that PSD is substantially more efficient than standard SD. MineDraft realizes the PSD through a novel batch-parallel design that maintains two batches of requests, overlapping drafting for one batch with verification for the other. Our experimental results show significant improvements of \alg{} in both throughput (up to 75%) and end-to-end latency (up to 39%) over standard SD. Furthermore, we have implemented MineDraft as a plugin for vLLM, demonstrating its practicality for production-ready inference systems.
Zhenwei Tang, Arun Verma, Zijian Zhou +4
College of Computing and Data Science, Nanyang Technological University, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre · Department of Computer Science, National University of Singapore, Republic of Singapore +1
Speculative decoding (SD) accelerates large language model (LLM) inference by using a smaller model to draft future tokens, which are then verified by the target LLM. This preserves generation quality by accepting only aligned tokens. However, individual drafters, often trained for specific tasks or domains, exhibit limited effectiveness across diverse applications. To address this, we introduce MetaSD, a unified framework that integrates multiple drafters into the SD process. MetaSD dynamically allocates computational resources to heterogeneous drafters by leveraging alignment feedback and framing drafter selection as a multi-armed bandit problem. Extensive experiments show MetaSD consistently outperforms single-drafter approaches.