When Drafts Evolve: Speculative Decoding Meets Online Learning

Yu-Yang Qian; Hao-Cong Wu; Yichao Fu; Hao Zhang; Peng Zhao

cs.LGMar 13, 2026

When Drafts Evolve: Speculative Decoding Meets Online Learning

Authors: Yu-Yang Qian, Hao-Cong Wu, Yichao Fu, Hao Zhang, Peng Zhao

Organizations: State Key Laboratory for Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · University of California, San Diego

Abstract

Speculative decoding has emerged as a widely adopted paradigm for accelerating large language model inference, where a lightweight draft model rapidly generates candidate tokens that are then verified in parallel by a larger target model. However, due to limited model capacity, drafts often struggle to approximate the target distribution, resulting in shorter acceptance lengths and diminished speedup. A key yet under-explored observation is that speculative decoding inherently provides verification feedback that quantifies the deviation between the draft and target models at no additional cost. This process naturally forms an iterative "draft commits-feedback provides-draft adapts" evolving loop, which precisely matches the online learning paradigm. Motivated by this connection, we propose OnlineSPEC, a unified framework that systematically leverages interactive feedback to continuously evolve draft models. Grounded in dynamic regret minimization, we establish a formal link between online learning performance and speculative system's acceleration rate, and develop novel algorithms via modern online learning techniques, including optimistic online learning that adaptively reuses historical gradients as predictive update hints, and online ensemble learning that dynamically maintains multiple draft models. Our algorithms are equipped with theoretical justifications and improved acceleration rates, achieving up to 24% speedup over seven benchmarks and five foundation models.

Figures & tables

Figure 1: Illustration of Generation-refinement framework . A draft sequence is first generated rapidly by a small draft model, and then verified by the target model, which naturally forms an iterative “draft commits–feedback provides–draft adapts” evolving loop.

Figure 2: A comprehensive 3D-visualization illustrates generation-refinement approaches across three dimensions: draft level, token level, and incorporating interactive feedback. Our Online Spec framework provides a unified perspective for integrating interactive feedback and can be seamlessly combined with existing methods to further enhance the acceleration rate.

	GSM8K		Spider		Code-Search		Alpaca-Finance
	AvgLen $\uparrow$	SpeedUp $\uparrow$	AvgLen $\uparrow$	SpeedUp $\uparrow$	AvgLen $\uparrow$	SpeedUp $\uparrow$	AvgLen $\uparrow$	SpeedUp $\uparrow$
	Target model: lmsys / Vicuna-7B-v1.3
Vanilla SD	1.25 $\pm$ 0.01	1.00 $\times$	1.20 $\pm$ 0.03	1.00 $\times$	1.22 $\pm$ 0.02	1.00 $\times$	1.20 $\pm$ 0.01	1.00 $\times$
OSD	1.52 $\pm$ 0.03	1.23 $\times$	1.36 $\pm$ 0.02	1.12 $\times$	1.37 $\pm$ 0.02	1.09 $\times$	1.30 $\pm$ 0.01	1.08 $\times$
Hydra	2.14 $\pm$ 0.05	1.00 $\times$	2.65 $\pm$ 0.31	1.00 $\times$	1.82 $\pm$ 0.04	1.00 $\times$	1.78 $\pm$ 0.01	1.00 $\times$
OSD-Hydra	2.56 $\pm$ 0.06	1.19 $\times$	3.11 $\pm$ 0.31	1.11 $\times$	2.11 $\pm$ 0.05	1.16 $\times$	2.19 $\pm$ 0.03	1.25 $\times$

Table 1: Comparison of generation-refinement acceleration methods across different benchmark datasets. For each method, we report the average accepted length ( AvgLen

\uparrow

) and wall-clock speedup ratio ( SpeedUp

\uparrow

). Results are averaged over three runs with standard deviations. The best results are highlighted in bold.

	GSM8K		MBPP		MATH		MMLU
	AvgLen $\uparrow$	Acc (%) $\uparrow$	AvgLen $\uparrow$	Acc (%) $\uparrow$	AvgLen $\uparrow$	Acc (%) $\uparrow$	AvgLen $\uparrow$	Acc (%) $\uparrow$
	Target model: Qwen / Qwen3-8B , Draft model: Qwen / Qwen3-0.6B-Base
Target Model	1.00 (1.00 $\times$ )	94.32	1.00 (1.00 $\times$ )	53.56	1.00 (1.00 $\times$ )	91.54	1.00 (1.00 $\times$ )	83.81
Draft Model	1.00 (3.60 $\times$ )	53.84	1.00 (3.56 $\times$ )	14.15	1.00 (3.54 $\times$ )	60.66	1.00 (3.56 $\times$ )	55.71
LR	13.25 (1.26 $\times$ )	91.04	5.76 (1.09 $\times$ )	50.65	9.56 (1.20 $\times$ )	92.84	9.40 (1.21 $\times$ )	82.86
OSD-LR	12.57 (1.20 $\times$ )	93.84	5.66 (1.09 $\times$ )	50.54	6.21 (1.10 $\times$ )	89.87	6.67 (1.14 $\times$ )	82.14

Table 2: Evaluation of online learning-based generation-refinement methods on reasoning benchmarks. We pair each target model with a corresponding smaller draft model and report the average accepted length ( AvgLen

\uparrow

) with wall-clock speedup ratio in parentheses, and the accuracy ( Acc (%)

\uparrow

). Results are averaged over three runs. The best results are highlighted in bold.

Panel of Figure 3 — Figure 3: Evolution of tokens per second ( TPS ) on the GSM8K dataset: (a) Opt-Hydra with lmsys/Vicuna-7B-v1.3 , (b) Ens-EAGLE with lmsys/Vicuna-7B-v1.3 , (c) Ens-EAGLE-3 with lmsys/Vicuna-7B-v1.3 , and (d) Online-LR with Qwen/Qwen3-8B . This demonstrates consistent performance improvements via online learning, validating the effectiveness of our OnlineSpec during deployment.

Model	Method	SpeedUp $\uparrow$	AvgLen $\uparrow$	Acc (%)
Vicuna-13B	Standard AR	1.00 (40.40)	1.00	–
	Vanilla SD	1.20 (48.52)	1.93	–
	OSD-EAGLE-3	1.37 (55.52)	2.24	–
	Ens-EAGLE-3	1.46 (59.03)	2.35	–
Vicuna-13B	Standard AR	1.00 (40.40)	1.00	–
	Vanilla SD	1.50 (60.61)	2.22	–

Table 3: Results on larger target models. We report SpeedUp

\uparrow

(with TPS ), AvgLen

\uparrow

, and the accuracy ( Acc ).

Method	AvgLen $\uparrow$	SpeedUp $\uparrow$
Hydra (baseline)	2.22	1.00 $\times$ (60.61)
OSD-Hydra	2.50	1.13 $\times$ (68.36)
Ens-Hydra (ensemble)	2.61	1.15 $\times$ (70.64)
Opt-Hydra (optimistic)	2.73	1.23 $\times$ (74.37)

Table 4: Comparison of different online learning strategies applied to Hydra on GSM8K with Vicuna-13B-v1.3 .

Method	AvgLen $\uparrow$	SpeedUp $\uparrow$
(i) Hydra: Varying Learning Rate vs. Opt-Hydra
OSD-Hydra ( $\eta\!=\!0.5$ )	2.53 $\pm$ 0.05	1.17 $\pm$ 0.02
OSD-Hydra ( $\eta\!=\!0.8$ )	2.56 $\pm$ 0.06	1.19 $\pm$ 0.02
OSD-Hydra ( $\eta\!=\!1.0$ )	2.51 $\pm$ 0.05	1.16 $\pm$ 0.02
OSD-Hydra ( $\eta\!=\!1.5$ )	2.27 $\pm$ 0.03	1.05 $\pm$ 0.01
Opt-Hydra	2.69 $\pm$ 0.08	1.26 $\pm$ 0.03

Table 5: Hyperparameter analysis on GSM8K . We report the average accepted length ( AvgLen

\uparrow

), and the wall-clock speedup ratio ( SpeedUp

\uparrow

). (i) Varying learning rate on OSD-Hydra vs. Opt-Hydra . (ii) Varying learning rate on OSD-EAGLE vs Ens-EAGLE . (iii) Varying total steps

T

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Performance comparison of Hydra , OSD-Hydra , and Opt-Hydra on (a) GSM8K , (b) Spider , (c) Code-Search , and (d) Alpaca-Finance using lmsys/Vicuna-7B-v1.3 as the foundation model. We report the average accepted length ( AvgLen , top row) and tokens per second ( TPS , bottom row) as inference evolves over time.

Panel of Figure 5 — Figure 5: Performance comparison of EAGLE , OSD-EAGLE , and Ens-EAGLE on (a) GSM8K , (b) Spider , (c) Code-Search , and (d) Alpaca-Finance using lmsys/Vicuna-7B-v1.3 as the foundation model. We report the average accepted length ( AvgLen , top row) and tokens per second ( TPS , bottom row) as inference evolves over time.

Panel of Figure 6 — Figure 6: Performance comparison of EAGLE-3 , OSD-EAGLE-3 , and Ens-EAGLE-3 on (a) GSM8K , (b) Spider , (c) Code-Search , and (d) Alpaca-Finance using lmsys/Vicuna-7B-v1.3 as the foundation model. We report the average accepted length ( AvgLen , top row) and tokens per second ( TPS , bottom row) as inference evolves over time.

Panel of Figure 7 — Figure 7: Performance comparison of Hydra , OSD-Hydra , and Opt-Hydra on (a) GSM8K , (b) Spider , (c) Code-Search , and (d) Alpaca-Finance using meta-llama/Llama-2-7B-Chat as the foundation model. We report the average accepted length ( AvgLen , top row) and tokens per second ( TPS , bottom row) as inference evolves over time.

Panel of Figure 8 — Figure 8: Performance comparison of EAGLE , OSD-EAGLE , and Ens-EAGLE on (a) GSM8K , (b) Spider , (c) Code-Search , and (d) Alpaca-Finance using meta-llama/Llama-2-7B-Chat as the foundation model. We report the average accepted length ( AvgLen , top row) and tokens per second ( TPS , bottom row) as inference evolves over time.

Panel of Figure 10 — Figure 10: Performance comparison of LR , OSD-LR , and Online-LR on (a) GSM8K , (b) MBPP , (c) MATH , and (d) MMLU using Qwen/Qwen3-8B paired with Qwen3-0.6B-Base as the draft model. We report the average accepted length ( AvgLen , top row) and tokens per second ( TPS , bottom row) as inference evolves over time.

Method	Code-Search		Alpaca-Finance
Method	Time (s) $\downarrow$	SpeedUp $\uparrow$	Time (s) $\downarrow$	SpeedUp $\uparrow$
EAGLE-3	300.83	1.00	347.57	1.00
Ens-EAGLE-3 (+train)	280.23	1.07	306.31	1.13
Hydra	527.46	1.00	393.18	1.00
Opt-Hydra (+train)	515.00	1.02	371.37	1.05

Appendix

Table 6: Comparison of inference time vs. our online training + inference time. Our methods (Opt-Hydra and Ens-EAGLE-3) achieve better speedup even when accounting for training overhead.

Method	Vicuna-7B	Llama-2-7B	Vicuna-13B
Standard AR	54.11	53.76	40.40
Ens-EAGLE-3	73.96 (+36.7%)	79.30 (+47.5%)	59.03 (+46.1%)
Opt-Hydra	107.17 (+98.1%)	122.61 (+128.1%)	74.37 (+84.1%)

Appendix

Table 7: Comparison against standard AR decoding on GSM8K . We report TPS

\uparrow

and relative speedup.

Figure 11: Theoretical trend vs. observed speedup for Ens-EAGLE on GSM8K with Vicuna-7B-v1.3 .

Warm-up	AvgLen $\uparrow$		TPS $\uparrow$
Warm-up	OSD-EAGLE	Ens-EAGLE	OSD-EAGLE	Ens-EAGLE
500	1.75	1.78	63.28	65.52
1000	1.88	1.91	69.66	70.82
1500	2.03	2.17	76.13	80.07

Appendix

Table 8: Sensitivity to the number of offline warm-up samples on GSM8K dataset with Vicuna-7B-v1.3 .

Method	AvgLen $\uparrow$	SpeedUp $\uparrow$
Hydra (baseline)	2.22	1.00 $\times$ (60.61)
OSD-Hydra	2.50	1.13 $\times$ (68.36)
Ens-Hydra (ensemble)	2.61	1.15 $\times$ (70.64)
Opt-Hydra (optimistic)	2.73	1.23 $\times$ (74.37)

Appendix

Table 9: Comparison of different online learning strategies applied to Hydra on GSM8K with Vicuna-13B-v1.3 .

System Prompt:

You are Qwen, created by Alibaba Cloud. You are a helpful assistant.

User Prompt:

Evaluate whether the following two reasoning steps (s1 and s2) convey exactly the same meaning. Focus on semantic similarity rather than exact wording. Compare the main ideas, key points, overall message, logical structure, and numerical calculations/results of both reasoning steps. If the reasoning steps convey essentially the same meaning and generate same calculation results, respond with [aligned]. If the reasoning steps express different meanings, respond with [unaligned]. If it is too hard to determine, respond with [unaligned] Please directly provide the final result in [aligned] or [unaligned]. Reasoning step 1 (s1):

<start_s1>

{draft_output}

<end_s1>

Appendix

Table 10: Prompt template used for semantic alignment verification in reasoning. We use Qwen2.5-7B-Instruct as judge (temperature=0.0).

User Prompt:

{question}

Please reason step by step, and put your final answer within \boxed{} . Once a full round of reasoning is completed, do not check again, immediately cease reasoning, and output the answer.

Appendix

Table 11: Problem-solving prompt template for math reasoning tasks.

Context: {preceding prompt and reasoning trace} Instruction: Considering the limited time by the user, I have to give the solution based on the thinking directly now. </think>

Appendix

Table 12: Thinking-to-Output mode transition instruction.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 309 papers

Mar 2, 2026

Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning

Speculative decoding accelerates large language model (LLM) inference by using a small draft model to generate candidate tokens for a larger target model to verify. The efficacy of this technique hinges on the trade-off between the time spent on drafting candidates and verifying them. However, current state-of-the-art methods rely on a static time allocation, while recent dynamic approaches optimize for proxy metrics like acceptance length, often neglecting the true time cost and treating the drafting and verification phases in isolation. To address these limitations, we introduce Learning to Draft (LTD), a novel method that directly optimizes for throughput of each draft-and-verify cycle. We formulate the problem as a reinforcement learning environment and train two co-adaptive policies to dynamically coordinate the draft and verification phases. This encourages the policies to adapt to each other and explicitly maximize decoding efficiency. We conducted extensive evaluations on five diverse LLMs and four distinct tasks. Our results show that LTD achieves speedup ratios ranging from 2.24x to 4.32x, outperforming the state-of-the-art method Eagle3 up to 36.4%.

Jiebin Zhang, Zhenghan Yu, Liang Wang +8

National Key Laboratory for Multimedia Information Processing, Peking University · School of Computer Science, Peking University · Microsoft Research Asia

Speculative Decoding LLM Inference Acceleration

Ask PDF

May 28, 2026

Draft-OPD: On-Policy Distillation for Speculative Draft Models

Speculative decoding accelerates large language model inference by pairing a target model with a lightweight draft model whose proposed tokens are verified in parallel. A common way to build draft models, like EAGLE3 or DFlash is supervised fine-tuning (SFT) on target-generated trajectories. However, we observe that SFT quickly plateaus: the draft model's acceptance length on test data stops improving. The reason is an offline-to-inference mismatch: In SFT, the drafter learns from fixed target-generated trajectories, whereas during speculative decoding it is evaluated on blocks proposed under its own policy. This motivates on-policy distillation (OPD), where the target model supervises the drafter on draft-induced states. Yet OPD remains difficult for draft models, as they cannot reliably roll out complete sequences independently, whereas target-assisted generation makes the collected sequences follow the target distribution and thus eliminates the on-policy signal. We therefore propose Draft-OPD, which uses target-assisted rollout for stable continuations and replays drafting from the verification-exposed error positions. This allows the drafter to learn from target feedback on both accepted and rejected proposals, focusing training on the draft-induced errors that limit speculative acceptance. Experiments show that Draft-OPD achieves over

5\times

lossless acceleration for thinking models across diverse tasks, improving over EAGLE-3 and DFlash by 23% and 13%.

Haodi Lei, Yafu Li, Haoran Zhang +8

Shanghai Jiao Tong University · Shanghai AI Laboratory · The Chinese University of Hong Kong +3

Speculative Decoding LLM Inference Acceleration Language Model Distillation On-Policy Distillation

Ask PDF

Feb 24, 2026

MineDraft: A Framework for Batch Parallel Speculative Decoding

Speculative decoding (SD) accelerates large language model inference by using a smaller draft model to propose draft tokens that are subsequently verified by a larger target model. However, the performance of standard SD is often limited by the strictly sequential execution of these drafting and verification stages. To address this, this paper proposes MineDraft, a batch parallel speculative decoding (PSD) framework designed to effectively hide drafting latency by overlapping it with verification. Our theoretical analysis shows that PSD is substantially more efficient than standard SD. MineDraft realizes the PSD through a novel batch-parallel design that maintains two batches of requests, overlapping drafting for one batch with verification for the other. Our experimental results show significant improvements of \alg{} in both throughput (up to 75%) and end-to-end latency (up to 39%) over standard SD. Furthermore, we have implemented MineDraft as a plugin for vLLM, demonstrating its practicality for production-ready inference systems.

Zhenwei Tang, Arun Verma, Zijian Zhou +4

College of Computing and Data Science, Nanyang Technological University, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre · Department of Computer Science, National University of Singapore, Republic of Singapore +1

Speculative Decoding LLM Inference Acceleration Parallel Speculative Decoding Parallel LLM Inference

Ask PDF

Abstract

Speculative Decoding

Explore similar work

Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning

Draft-OPD: On-Policy Distillation for Speculative Draft Models

MineDraft: A Framework for Batch Parallel Speculative Decoding