Efficiently Aligning Draft Models via Parameter- and Data-Efficient Adaptation

Luxi Lin; Zhihang Lin; Zhanpeng Zeng; Yuhao Chen; Qingyu Zhang; Jixiang Luo; Xuelong Li; Rongrong Ji

cs.LGMar 10, 2026

Efficiently Aligning Draft Models via Parameter- and Data-Efficient Adaptation

Authors: Luxi Lin, Zhihang Lin, Zhanpeng Zeng, Yuhao Chen, Qingyu Zhang, Jixiang Luo, Xuelong Li, Rongrong Ji

Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Innovation Institute · University of Science and Technology of China (USTC)

Abstract

Speculative decoding accelerates LLM inference but suffers from performance degradation when target models are fine-tuned for specific domains. A naive solution is to retrain draft models for every target model, which is costly and inefficient. To address this, we introduce a parameter- and data-efficient framework named Efficient Draft Adaptation, abbreviated as EDA, for efficiently adapting draft models. EDA introduces three innovations: (1) a decoupled architecture that utilizes shared and private components to model the shared and target-specific output distributions separately, enabling parameter-efficient adaptation by updating only the lightweight private component;(2) a data regeneration strategy that utilizes the fine-tuned target model to regenerate training data, thereby improving the alignment between training and speculative decoding, leading to higher average acceptance length;(3) a sample selection mechanism that prioritizes high-value data for efficient adaptation. Our experiments show that EDA effectively restores speculative performance on fine-tuned models, achieving superior average acceptance lengths with significantly reduced training costs compared to full retraining. Code is available at https://github.com/Lyn-Lucy/Efficient-Draft-Adaptation.

Figures & tables

Figure 1: The comparison of average acceptance length on GSM8K under different draft-target pairings. We refer to the target models as

T_{Base}

(Qwen2.5-7B) and

T_{Math}

(Qwen2.5-Math-7B). The target model

T_{Base}

’s draft model

D_{Base}

achieves high average acceptance length when paired together (

D_{Base}\rightarrow T_{Base}

), while suffers a substantial acceptance drop when paired with

T_{Math}

(

D_{Base}\rightarrow T_{Math}

). Our EDA framework can restore performance through efficient, lightweight adaptation.

Figure 2: The EDA framework for efficient draft model adaptation, combining shared–private draft decomposition, domain-specific self-generation, and representation-shift–based data selection to restore speculative decoding performance with minimal training cost.

GSM8K

AIME_2024

SVAMP

Hendrycks-MATH

MathQA

Avg

Method

\tau

Speedup

\tau

Speedup

\tau

Speedup

\tau

Speedup

\tau

Speedup

\tau

Speedup

Temperature

T=0

Training-Free

1.17

0.85

\times

1.18

0.76

\times

1.19

0.85

\times

1.20

0.82

\times

1.23

0.91

\times

1.19

0.84

\times

Full-FT

4.37

2.88

\times

5.05

3.14

\times

4.40

2.91

\times

4.98

3.23

\times

4.66

3.19

\times

4.69

3.07

\times

LoRA

4.32

2.84

\times

4.90

2.93

\times

4.36

2.75

\times

4.92

3.18

\times

4.55

3.00

\times

4.61

2.94

\times

EDA (Base)

4.40

2.89

\times

5.02

3.12

\times

4.42

2.92

\times

5.00

3.24

\times

4.70

3.16

\times

4.71

3.07

\times

Table 1: Main results on five math tasks (GSM8K, AIME_2024, SVAMP, Hendrycks-MATH, MathQA) on Qwen2.5-Math-7B target model [ 39 ] . We report average acceptance length

\tau

and decoding speedup.

HumanEval

APPS

BigCodeBench

HumanEval+

MBPP

Avg

Method

\tau

Speedup

\tau

Speedup

\tau

Speedup

\tau

Speedup

\tau

Speedup

\tau

Speedup

Temperature

T=0

Training-Free

1.75

1.21

\times

1.69

1.14

\times

1.74

1.24

\times

1.74

1.08

\times

1.85

1.21

\times

1.75

1.18

\times

Full-FT

4.79

3.10

\times

4.87

2.91

\times

3.66

2.38

\times

4.70

2.62

\times

4.93

2.82

\times

4.59

2.76

\times

LoRA

4.70

3.05

\times

4.80

2.88

\times

3.60

2.35

\times

4.62

2.58

\times

4.78

2.76

\times

4.50

2.72

\times

EDA (Base)

4.75

3.08

\times

4.92

2.90

\times

3.68

2.40

\times

4.72

2.63

\times

4.90

2.80

\times

4.59

2.76

\times

Table 2: Main results on five code tasks (HumanEval, APPS, BigCodeBench, HumanEval+, MBPP) on Qwen2.5-Coder-7B target model [ 14 ] . We report average acceptance length

\tau

and decoding speedup.

MedMCQA

MedQA-USMLE

PubMedQA

MedQA

MMLU

Avg

Method

\tau

Speedup

\tau

Speedup

\tau

Speedup

\tau

Speedup

\tau

Speedup

\tau

Speedup

Temperature

T=0

Training-Free

2.20

1.47

\times

2.17

1.40

\times

1.92

1.29

\times

2.14

1.29

\times

2.33

1.49

\times

2.15

1.39

\times

Full-FT

3.97

2.35

\times

3.97

2.28

\times

3.86

2.34

\times

3.97

2.35

\times

3.67

3.03

\times

3.89

2.47

\times

LoRA

3.85

2.30

\times

3.90

2.25

\times

3.78

2.18

\times

3.92

2.28

\times

3.50

2.80

\times

3.79

2.36

\times

EDA (Base)

3.95

2.33

\times

3.94

2.26

\times

3.85

2.32

\times

3.96

2.34

\times

3.66

3.00

\times

3.87

2.45

\times

Table 3: Main results on five medical tasks (MedMCQA, MedQA-USMLE, PubMedQA, MedQA, MMLU) on Meditron3-Qwen2.5-7B target model [ 27 ] . We report average acceptance length

\tau

and decoding speedup.

Figure 3: Comparison of different data selection strategies under varying data budgets. Draft adaptation from Qwen2.5-7B to Qwen2.5-Math-7B , measured on 4

\times

NVIDIA H200 GPUs;

\tau

is evaluated on GSM8K under this setup.

Method	Trainable Size (MB)	Training Time (h)	Avg. Acceptance ( $\tau$ )
Re-Train	462	5.1	4.22
EDA	127 (27.5%)	2.0 (39.2%)	4.79 (+13.5%)

Table 4: Overhead analysis on math transfer. We follow the same experimental setup as in Sec. 5.3 .

Figure 4: Qualitative example comparing direct transfer and EDA-adapted drafts on Qwen2.5-Coder-7B. Red tokens indicate incorrect predictions, while black tokens indicate correct predictions.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 308 papers

Draft-OPD: On-Policy Distillation for Speculative Draft Models
May 28, 2026Haodi Lei, Yafu Li, Haoran Zhang +8Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Shanghai AI Laboratory · The Chinese University of Hong Kong +3
Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds
Oct 7, 2026Yunxiao Zhao, Changxiao CaiSpeculative Decoding Parallel Speculative Decoding
Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, MI, USA.
AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters
Jul 21, 2026Yu-Yang Qian, Hao-Cong Wu, Chen Chen +4Speculative Decoding for Diffusion Language Models Speculative Decoding
State Key Laboratory for Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · Huawei Foundation Model Dept

May 28, 2026J/K move · Enter open · S save

Draft-OPD: On-Policy Distillation for Speculative Draft Models

Haodi Lei, Yafu Li, Haoran Zhang, Shunkai Zhang+7

Shanghai Jiao Tong University · Shanghai AI Laboratory · The Chinese University of Hong Kong+3

Speculative Decoding LLM Inference Acceleration Language Model Distillation On-Policy Distillation

Open paper AskPDF arXiv

Speculative decoding accelerates large language model inference by pairing a target model with a lightweight draft model whose proposed tokens are verified in parallel. A common way to build draft models, like EAGLE3 or DFlash is supervised fine-tuning (SFT) on target-generated trajectories. However, we observe that SFT quickly plateaus: the draft model's acceptance length on test data stops improving. The reason is an offline-to-inference mismatch: In SFT, the drafter learns from fixed target-generated trajectories, whereas during speculative decoding it is evaluated on blocks proposed under its own policy. This motivates on-policy distillation (OPD), where the target model supervises the drafter on draft-induced states. Yet OPD remains difficult for draft models, as they cannot reliably roll out complete sequences independently, whereas target-assisted generation makes the collected sequences follow the target distribution and thus eliminates the on-policy signal. We therefore propose Draft-OPD, which uses target-assisted rollout for stable continuations and replays drafting from the verification-exposed error positions. This allows the drafter to learn from target feedback on both accepted and rejected proposals, focusing training on the draft-induced errors that limit speculative acceptance. Experiments show that Draft-OPD achieves over

5\times

lossless acceleration for thinking models across diverse tasks, improving over EAGLE-3 and DFlash by 23% and 13%.

Abstract

Speculative Decoding

Explore similar work

Draft-OPD: On-Policy Distillation for Speculative Draft Models

Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds

AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters