Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning

Jiebin Zhang; Zhenghan Yu; Liang Wang; Nan Yang; Eugene J. Yu; Zheng Li; Yifan Song; Dawei Zhu

cs.CLMar 2, 2026

Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning

Authors: Jiebin Zhang, Zhenghan Yu, Liang Wang, Nan Yang, Eugene J. Yu, Zheng Li, Yifan Song, Dawei Zhu,

Organizations: National Key Laboratory for Multimedia Information Processing, Peking University · School of Computer Science, Peking University · Microsoft Research Asia

Abstract

Speculative decoding accelerates large language model (LLM) inference by using a small draft model to generate candidate tokens for a larger target model to verify. The efficacy of this technique hinges on the trade-off between the time spent on drafting candidates and verifying them. However, current state-of-the-art methods rely on a static time allocation, while recent dynamic approaches optimize for proxy metrics like acceptance length, often neglecting the true time cost and treating the drafting and verification phases in isolation. To address these limitations, we introduce Learning to Draft (LTD), a novel method that directly optimizes for throughput of each draft-and-verify cycle. We formulate the problem as a reinforcement learning environment and train two co-adaptive policies to dynamically coordinate the draft and verification phases. This encourages the policies to adapt to each other and explicitly maximize decoding efficiency. We conducted extensive evaluations on five diverse LLMs and four distinct tasks. Our results show that LTD achieves speedup ratios ranging from 2.24x to 4.32x, outperforming the state-of-the-art method Eagle3 up to 36.4%.

Figures & tables

Figure 1: The overview of LTD method. We formulate the interaction between draft model and target model as an RL environment. We employ two policies to dynamically manage the allocation between draft generation and target verification by controlling the draft tree’s depth and verification size. These policies are optimized together using the reward signal based on the throughput of each draft-and-verify cycle.

Figure 2: Analysis of Draft Time and Verification Time. Left: The average draft and verification latency of Vicuna-13B as a function of draft depth D and verification size V on benchmarks. Right: The composition of the total inference time for Eagle3 and LTD on the benchmarks, with Eagle3’s total time normalized to 100% as baselines.

Figure 3: We report the training reward curves for the Size Policy and Depth Policy of the Vicuna-13B model, as well as the evolution of the speedup ratio on the validation set.

Temperature=0
		MT-bench		Gsm8k		Alpaca		Qa		Mean
Model	Method	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$
L31 8B	Eagle3	3.70	6.39	3.58	6.44	4.26	6.93	3.17	5.35	3.68	6.28
	Eagle3+DDD	3.67	5.41	3.55	5.48	3.97	5.58	3.07	4.21	3.57	5.17
	Eagle3+Svip	3.65	6.21	3.47	6.21	4.04	6.73	3.12	4.78	3.57	5.98
	Eagle3+GT	3.79	5.86	3.64	5.77	4.37	6.49	3.38	4.92	3.80	5.76

Table 1: Speedup ratios and average acceptance lengths

\bm{\tau}

of different methods during greedy decoding. L31 8B represents Llama-3.1-8B-Instruct, V 13B represents Vicuna-13B-v1.3, Dpsk 8B represents DeepSeek-R1-Distill-LLaMA 8B. For the Qwen3-14B and Qwen3-32B models, their corresponding Eagle3 models lack default settings. Therefore, only the results from Grid Search are presented. GT, Spec++, and Gs stand for Gammatune, SpecDec++, and Grid Search, respectively. The results of Temperature=1 are presented in Table 5 .

Figure 4: Left: The effectiveness of our iterative training strategy on Llama3-8B and Dpsk-8B models. Right: The contribution of each component on Vicuna-13B model.

Reward	MT-bench		Gsm8k		Alpaca		Qa		Mean
Reward	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$
Acceptance Length	3.77	7.70	3.81	7.20	3.71	7.05	3.06	5.61	3.59	6.89
Time Cost	3.89	6.49	3.54	5.80	3.48	5.79	2.87	4.54	3.45	5.66
Throughput	4.40	7.68	4.43	7.10	4.23	7.00	3.47	5.56	4.13	6.84

Table 2: The effectiveness of reward signal on size policy, evaluated on Vicuna-13B model.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Method Name	Predict Policy	Observation Space	Dynamic Size or Depth	Training Methods	Objective
DDD ( Brown et al., 2024 )	Heuristic	Probability	Depth	-	$L_{A}$
SVIP ( Zhang et al., 2024c )	Heuristic	Entropy	Depth	-	$L_{A}$
Gammatune ( Gautam et al., 2025 )	Heuristic	Token Acceptance Rate	Depth	-	$L_{A}$
Disco ( Mamou et al., 2024 )	FFN	Probability, Entropy, Draft Depth	Depth	FT	$L_{A}$
SpecDec++ ( Huang et al., 2025a )	FFN	Hidden States	Depth	FT	$L_{A}$
C2T ( Huo et al., 2025 )	FFN	Probability, Entropy, Draft Depth	Size	FT	$L_{A}$

Appendix

Table 3: Comparison of Different Dynamic Draft Tree Methods

	MT-bench		Gsm8k		Alpaca		QA		Mean
Method	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$
Eagle3	4.48	7.22	4.36	6.58	4.16	6.47	3.43	5.12	4.11	6.35
-Depth	4.40	7.68	4.43	7.10	4.23	7.00	3.47	5.56	4.13	6.84
-Size	4.77	7.75	4.46	6.82	4.36	6.59	3.55	5.12	4.29	6.57
-Iterative Training	4.54	8.29	4.48	7.41	4.28	7.08	3.51	5.45	4.20	7.06
LTD	4.64	8.24	4.59	7.38	4.40	7.01	3.66	5.37	4.32	7.00

Appendix

Table 4: Contribution of each components of LTD, evaluated on Vicuna-13B.

Temperature=1
		MT-bench		Gsm8k		Alpaca		Qa		Mean
Model	Method	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$
L31 8B	Eagle3	2.85	4.11	3.43	5.24	3.38	5.47	2.27	3.06	2.98	4.47
	Eagle3+DDD	2.89	3.40	3.35	4.09	3.38	4.59	2.36	2.95	3.00	3.76
	Eagle3+Svip	2.74	3.41	3.21	4.92	3.19	4.95	2.31	3.20	2.86	4.12
	Eagle3+GT	2.75	3.54	3.28	4.27	3.51	5.12	2.46	3.24	3.00	4.04

Appendix

Table 5: Speedup ratios and average acceptance lengths

\bm{\tau}

of different methods during Temperature=1. L31 8B represents Llama-3.1-8B-Instruct, V 13B represents Vicuna-13B-v1.3, Dpsk 8B represents DeepSeek-R1-Distill-LLaMA 8B. For the Qwen3-14B and Qwen3-32B models, their corresponding Eagle3 models lack default settings. Therefore, only the results from Grid Search are presented. GT, Spec++, and Gs stand for Gammatune, SpecDec++, and Grid Search, respectively.

Reward	MT-bench		Gsm8k		Alpaca		Qa		Mean
Reward	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$
P+L	4.35	7.68	4.30	7.10	4.18	7.00	3.35	5.55	4.05	6.83
P+D	4.31	7.68	4.35	7.10	4.22	7.00	3.41	5.56	4.07	6.84
P+D+L (LTD)	4.40	7.68	4.43	7.10	4.23	7.00	3.47	5.56	4.13	6.84
P+D+L+H	4.40	7.53	4.32	6.97	4.22	6.94	3.44	5.51	4.10	6.74
P+D+L+E	4.36	7.68	4.36	7.10	4.28	7.00	3.43	5.56	4.11	6.84

Appendix

Table 6: Ablation Study for Observation Space on size policy evaluated on Vicuna-13B.

Figure 5: The average acceptance length for Llama-3.1-8B-Instruct using Eagle3 on HumanEval dataset. Left: Change the draft depth with verification size fixed to 60. Right: Change the verification size with draft depth fixed to 8.

	MT-bench		Gsm8k		Alpaca		Qa		Mean
Method	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$
Eagle3	3.70	6.39	3.58	6.44	4.26	6.93	3.17	5.35	3.68	6.28
Iter0	3.88	6.46	3.76	6.40	4.34	6.82	3.50	5.12	3.87	6.20
Iter1	3.95	6.50	3.80	6.47	4.36	6.89	3.50	5.18	3.90	6.26
Iter2	3.96	6.74	3.79	6.73	4.53	7.31	3.41	5.44	3.92	6.56
Iter3	3.93	6.74	3.73	6.73	4.57	7.31	3.44	5.44	3.92	6.56

Appendix

Table 7: Llama3.1-8B Iterative Optimizing

	MT-bench		Gsm8k		Alpaca		Qa		Mean
Method	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$
Eagle3	3.67	5.82	4.73	7.39	3.65	5.63	3.14	5.02	3.80	5.53
Iter0	3.88	5.95	4.69	8.03	3.68	5.63	3.37	5.07	3.91	5.72
Iter1	3.96	5.98	4.92	8.14	3.86	5.67	3.46	5.08	4.05	5.78
Iter2	4.02	6.22	4.94	8.36	3.71	5.91	3.52	5.29	4.05	5.97
Iter3	4.05	6.32	4.98	8.53	3.98	6.00	3.62	5.37	4.16	6.08

Appendix

Table 8: Dpsk-8B Iterative Optimizing

Figure 6: We report the interaction between depth policy and size policy on Llama3-8B for Iter0 and LTD.

Time per Sample	MT-bench	Alpaca	Qa	Gsm8K	Mean
Llama-3.1-8B-Instruct
Size Policy Latency (ms)	21.4	6.22	6.36	5.83	9.96
Depth Policy Latency (ms)	40.8	12.4	10.2	11.9	18.8
Total Latency per Sample (s)	4.48	1.28	1.27	1.24	2.07
Size Policy Overhead (%)	0.478	0.485	0.500	0.469	0.483
Depth Policy Overhead (%)	0.910	0.969	0.803	0.955	0.909

Appendix

Table 9: Latency and overhead analysis of dynamic policies on Vicuna-13B and Llama31-8B. Policy latency is measured in milliseconds (ms), total generation latency in seconds (s), and overhead in percentage (%).

Configuration	MT-bench	Gsm8k	Alpaca	Qa	Mean
[1024]	3.84	3.99	4.39	3.22	3.86
[1024, 256]	3.95	4.03	4.44	3.24	3.92
[1024, 512, 256]	3.96	4.07	4.43	3.25	3.93

Appendix

Table 10: Speedup of the Size Policy across three architecture configurations on Llama-3.1-8B-Instruct.

Figure 7: Speedup comparison between LTD and Eagle3 on the Llama-3.1-8B-Instruct model. We evaluated performance across 57 subtasks from MMLU(e.g., mathematics, history, law), using 80 sampled examples per subtask. Domain levels are abbreviated as H (High School), C (College), and P (Professional). Left: The relative speedup of LTD normalized against Eagle3; values greater than 100% indicate LTD outperforms Eagle3. Right: The absolute speedup ratio of the Eagle3 baseline across different subtasks.

	MT-bench		Gsm8k		Alpaca		Qa		Mean
Method	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$	Speedup	$\bm{\tau}$
Griffin	1.83	4.82	3.18	5.16	2.98	4.79	2.20	3.96	2.55	4.68
Griffin+GS	2.81	5.11	3.19	5.47	2.95	5.04	2.18	4.10	2.78	4.93
Griffin+LTD	2.88	4.98	3.13	5.45	2.98	5.01	2.23	4.10	2.81	4.89

Appendix

Table 11: Speedup ratios and average acceptance lengths

\bm{\tau}

of different methods on Griffin using Meta-Llama-3-8B-Instruct during greedy decoding.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 309 papers

When Drafts Evolve: Speculative Decoding Meets Online Learning
Mar 13, 2026Yu-Yang Qian, Hao-Cong Wu, Yichao Fu +2Speculative Decoding LLM Inference Acceleration
State Key Laboratory for Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · University of California, San Diego
MineDraft: A Framework for Batch Parallel Speculative Decoding
Feb 24, 2026Zhenwei Tang, Arun Verma, Zijian Zhou +4Speculative Decoding LLM Inference Acceleration
College of Computing and Data Science, Nanyang Technological University, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre · Department of Computer Science, National University of Singapore, Republic of Singapore +1
Multi-Drafter Speculative Decoding with Alignment Feedback
Apr 7, 2026Taehyeon Kim, Hojung Jung, Se-Young YunSpeculative Decoding LLM Inference Acceleration
LG AI Research · KAIST AI

Mar 13, 2026J/K move · Enter open · S save

When Drafts Evolve: Speculative Decoding Meets Online Learning

Yu-Yang Qian, Hao-Cong Wu, Yichao Fu, Hao Zhang+1

State Key Laboratory for Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · University of California, San Diego

Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Speculative decoding has emerged as a widely adopted paradigm for accelerating large language model inference, where a lightweight draft model rapidly generates candidate tokens that are then verified in parallel by a larger target model. However, due to limited model capacity, drafts often struggle to approximate the target distribution, resulting in shorter acceptance lengths and diminished speedup. A key yet under-explored observation is that speculative decoding inherently provides verification feedback that quantifies the deviation between the draft and target models at no additional cost. This process naturally forms an iterative "draft commits-feedback provides-draft adapts" evolving loop, which precisely matches the online learning paradigm. Motivated by this connection, we propose OnlineSPEC, a unified framework that systematically leverages interactive feedback to continuously evolve draft models. Grounded in dynamic regret minimization, we establish a formal link between online learning performance and speculative system's acceleration rate, and develop novel algorithms via modern online learning techniques, including optimistic online learning that adaptively reuses historical gradients as predictive update hints, and online ensemble learning that dynamically maintains multiple draft models. Our algorithms are equipped with theoretical justifications and improved acceleration rates, achieving up to 24% speedup over seven benchmarks and five foundation models.

Abstract

Speculative Decoding

Explore similar work

When Drafts Evolve: Speculative Decoding Meets Online Learning

MineDraft: A Framework for Batch Parallel Speculative Decoding

Multi-Drafter Speculative Decoding with Alignment Feedback