T2D: Target-to-Draft Model Pruning & Adaptation for Efficient Speculative Decoding

Shahriar Kabir Nahin; Xiao Ye; Muhao Chen; Ben Zhou; Anshuman Chhabra

cs.CLOct 25, 2026

T2D: Target-to-Draft Model Pruning & Adaptation for Efficient Speculative Decoding

Authors: Shahriar Kabir Nahin, Xiao Ye, Muhao Chen, Ben Zhou, Anshuman Chhabra

Organizations: Bellini College of AI, Cybersecurity, and Computing, University of South Florida · School of Computing and Augmented Intelligence, Arizona State University · Department of Computer Science, University of California, Davis

Abstract

Speculative decoding significantly accelerates autoregressive text generation in Large Language Models (LLMs), but its efficacy depends largely on two key factors: the distributional alignment between the draft and the target model, and the draft model's latency during inference. However, existing approaches typically utilize independently pretrained small draft models or seek to prune the target model to a manually chosen architecture, thereby failing to explicitly balance the trade-off between draft model alignment and token generation efficiency. To address this gap, we formulate draft model design as an optimization problem and propose T2D, a framework that constructs draft models directly from the target model using latency-aware structured pruning and knowledge distillation. In this manner, T2D obtains draft models that are jointly optimized for both measured inference latency and cross-entropy loss on the target model's responses, resulting in superlative speculative decoding performance. Through extensive experiments on multi-domain speculative decoding benchmarks and across several target model families of differing scale, we demonstrate how T2D prunes up to 92% of the target model to obtain highly performant draft models compared to competitive baselines. Moreover, draft models obtained via T2D can also be utilized by other speculative decoding token-alignment and knowledge distillation methods in a plug-and-play fashion, leading to further gains. Our findings underscore the importance of jointly navigating the draft model architecture-alignment frontier to unlock the full potential of speculative decoding for LLMs.

Figures & tables

Figure 1: Example speculative-decoding traces for the same output sequence using draft models with different latencies and acceptance ratios (Accepted-to-draft token length ratio). The top row illustrates that a more accurate but slower draft model can result in longer overall decoding time, highlighting the trade-off between draft-model latency and acceptance ratio.

Figure 2: Overview of the T2D training pipeline. Learnable gates mask layers, attention heads, and FFN channels to jointly optimize the pruned architecture and model weights using a frozen latency predictor, target-generated cross-entropy loss, and a reference-guided penalty.

Target Model	Draft	Code	Hum.	Math	Multilin.	QA	RAG	Reason.	Role.	Stem	Summ.	Writing	Overall
Llama-2-7B-Chat	ShortGPT	1.72	1.30	1.57	1.46	1.06	1.57	1.66	1.20	1.43	1.41	1.41	1.45
	Wide	1.82	1.46	1.74	1.83	1.34	1.63	1.73	1.25	1.58	1.43	1.50	1.59
	T2D	1.98	1.59	1.92	2.03	1.46	1.79	1.90	1.35	1.72	1.60	1.62	1.74
Vicuna-13B-v1.5	ShortGPT	1.95	1.66	1.82	2.22	1.40	1.74	1.86	1.30	1.70	1.73	1.62	1.74
	Wide	2.06	1.67	1.92	2.22	1.44	1.79	1.91	1.33	1.72	1.74	1.66	1.78
	T2D	2.13	1.79	1.99	2.46	1.47	1.97	2.04	1.39	1.83	1.88	1.73	1.89

Table 1: Speculative decoding speedup across different evaluation domains. For each target model, we compare draft models constructed by ShortGPT, Wide , and T2D . The highest speedup (

\times

) for each target model and domain is shown in bold .

Figure 3: Trade-off curve of acceptance ratio and latency across pruning ratio and speedup of the draft model across pruning ratio of Llama-2-7B-Chat model when pruned with ShortGPT.

Figure 4: Pruning ratio of the Llama-2-7B-Chat model over the training period along with the kept percentage (ratio) of the Layers, Heads, and FFN. We also plot latency and CE loss.

Draft Model	Speedup
ShortGPT	1.43
Wide	1.59
T2D	1.79

Table 2: Speculative decoding performance comparison of draft models under the AdaSpec distillation method (best in bold )

Draft Model	Size (B)	Speedup
Sheared-Llama-1.3B-ft	1.27	1.18
T2D	0.86	1.74

Table 3: Speculative decoding speedup (

\times

) comparison between finetuned Sheared-Llama and T2D with Llama-2-7B-Chat as the target model. The highest speedup is shown in bold .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Code	Hum.	Math	Multilin.	QA	RAG	Reason.	Role.	STEM	Summ.	Writing	Total
80	80	80	80	80	71	51	48	80	78	70	798

Appendix

Table 4: Number of evaluation samples from SPEED Bench used in our experiments.

Target Model	Total Response Tokens (B)	Avg. Response Length
Llama-2-7B-Chat	0.147	588
Vicuna-13B-v1.5	0.144	577
Qwen2.5-7B-Instruct	0.142	567
Qwen3-14B	0.144	577
Nemotron-32B	0.147	588

Appendix

Table 5: Statistics of the generated training data. Both models are prompted with the same set of 250K instruction-tuning queries.

Hyperparameter	Value
Target modules	Layers, attention heads, FFN channels
Reference CE loss ( $\mathcal{L}^{\prime}_{CE}$ )	$0.5$
Cross-entropy loss weight ( $\mu_{CE}$ )	$1.0$
Latency loss weight ( $\mu_{lat}$ )	$200$
Joint optimization epochs	$2$
Continual fine-tuning epochs	$1$

Appendix

Table 6: Hyperparameters used for pruning and fine-tuning in T2D framework for Llama-2-7B-Chat.

T2D Draft Model	Params (B)	Layers	Hidden Size	Num Heads	FFN Size
Llama-2-7B-Chat	0.86	3	4096	32	10752
Vicuna-13B-v1.5	1.21	4	4480	35	11264
Qwen2.5-7B-Instruct	1.54	2	3584	28	18432
Qwen3-14B	2.20	2	5120	40	16896
Nemotron-32	2.51	2	5120	40	27136

Appendix

Table 7: Architectural specifications of the T2D -derived draft models for all T2D models.

Figure 5: Pruning ratio of the target model (Vicuna-13B-v1.5) over the training period, along with the kept percentage (ratio) of the Layers, Heads, and FFN. We also plot latency and CE loss.

Figure 6: Pruning ratio of the target model (Qwen2.5-7B-Instruct) over the training period, along with the kept percentage (ratio) of the Layers, Heads, and FFN. We also plot latency and CE loss.

Figure 7: Pruning ratio of the target model (Qwen3-14B) over the training period, along with the kept percentage (ratio) of the Layers, Heads, and FFN. We also plot latency and CE loss.

Figure 8: Pruning ratio of the target model (Nemotron-32B) over the training period, along with the kept percentage (ratio) of the Layers, Heads, and FFN. We also plot latency and CE loss.

Draft Model	Size (B)	Code	Hum.	Math	Multilin.	QA	RAG	Reason.	Role.	Stem	Summ.	Writing	Overall
Sheared-Llama-1.3B-ft	1.27	1.32	1.11	1.29	1.16	0.90	1.36	1.40	1.12	1.20	0.73	1.27	1.18
T2D	0.86	1.98	1.59	1.92	2.03	1.46	1.79	1.90	1.35	1.72	1.60	1.62	1.74

Appendix

Table 8: Speculative decoding speedup (

\times

) of Sheared-Llama-1.3B and T2D . The highest speedup for each domain is shown in bold .

Prompt	T2D Draft Model Response (excerpt)	Observation
How are you?	I’m just an AI, I don’t have personal experiences or emotions, so I don’t have a physical body or a specific situation. However, I can provide information on a wide range of topics, including but not limited to: 1. Travel and exploration: I can provide information on different cultures, languages, and landscapes, and I can also provide information on different cultures and landscapes. 2. Culture and traditions: I can provide information on different cultures and traditions […]	Relevant and appropriate framing (correctly disclaims lacking emotions) repetition.
What is the capital of France?	The capital of France is Paris.	Fully correct and relevant; concise factual answer.
Can you help me in my study?	Of course! I’d be happy to help you in your study. Here are some steps you can take to help me improve my study skills: 1. Identify the type of study: Determine which type of study you want to focus on, such as studies that you want to study for a specific purpose or subject. 2. Research the topic: Identify the topic or subject you want to focus on, such as the impact of social media on mental health […]	Relevant and largely coherent, giving actionable study advice.
What is a computer?	A computer is a type of computer that is designed to perform specific tasks, such as image processing, data analysis, and machine learning. It is typically used in a variety of applications, including: 1. Image processing: A computer can process images, such as images of objects, scenes, or time-series data, and can be used to perform tasks such as image compression, feature extraction, and image processing […]	Topically relevant (stays within computing/AI applications) but the core definition is circular.
What is Artificial Intelligence?	Artificial Intelligence (AI) is a rapidly growing field that has the potential to revolutionize various industries, including healthcare, finance, transportation, and education. Here are some of the key aspects of AI: 1. Machine Learning: AI systems can be used to analyze large datasets, identify patterns, and make predictions, and optimize them for better decision-making. 2. Natural Language Processing (NLP): AI systems can be used to analyze large amounts of text data, such as customer reviews, marketing campaigns, and customer feedback, to understand their context and make more informed decisions […]	Correct and directly relevant; accurately defines AI and enumerates coherent, describes the subfields (ML, NLP), demonstrating strong topical grounding.

Appendix

Table 9: Example response generated by the T2D Llama-2 draft model, with observations on response quality.

Panel of Figure 9 — Figure 9: Examples of speculative decoding outputs across different domains.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 308 papers

May 9, 2026

PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decoding

Speculative decoding accelerates Large Language Models (LLMs) inference by using a lightweight draft model to propose candidate tokens that are verified in parallel by the target model. However, existing draft model training objectives are not directly aligned with the inference-time goal of maximizing consecutive token acceptance. To address this issue, we reformulate the draft model optimization objective, shifting the focus from token prediction accuracy to the overall acceptance length. In this paper, we build upon PARD to propose PARD-2, a dual-mode speculative decoding framework with Confidence-Adaptive Token (CAT) optimization. This approach adaptively reweights each token to better align with the verification process. Notably, PARD-2 enables a single draft model to support both target-dependent and target-independent modes. Experiments across diverse models and tasks demonstrate that PARD-2 achieves up to 6.94

\times

lossless acceleration, surpassing EAGLE-3 by 1.9

\times

and PARD by 1.3

\times

on Llama3.1-8B. Our code is available at https://github.com/AMD-AGI/PARD.

Zihao An, Taichi Liu, Ziqiong Liu +3

Advanced Micro Devices, Inc. · Rutgers University · Michigan State University

Speculative Decoding LLM Inference Acceleration

Ask PDF

Jul 12, 2026

Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting

Speculative decoding has significantly accelerated Large Language Model (LLM) inference by alleviating memory-bound bottlenecks. However, traditional speculative decoding typically relies on auxiliary draft modules, incurring significant training and communication overhead. Although recent methods attempt to generate drafts within the target model itself, they often fail to fully exploit its latent parallel capacity due to a lack of structural coordination. In this paper, we propose Progressive Tree Drafting (PTD), which employs a structured, guided parallel drafting strategy to harness the model's parallel potential. By coupling a progressive tree structure with a stepwise pruning mechanism, PTD actively guides the LLM to explore multiple semantic paths in a single forward pass, ensuring both draft diversity and coherence. Experiments demonstrate that PTD achieves up to

2\times

decoding speedup across various benchmarks while remaining training-free and model-agnostic. Our code is available at: https://github.com/MINE-USTC/PTD.

Zipeng Gao, Zhi Zheng, Qingrong Xia +5

1State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · 2Huawei Technologies Co., Ltd.

Speculative Decoding LLM Inference Acceleration Autoregressive Decoding Parallel Speculative Decoding

Ask PDF

Mar 10, 2026

Efficiently Aligning Draft Models via Parameter- and Data-Efficient Adaptation

Speculative decoding accelerates LLM inference but suffers from performance degradation when target models are fine-tuned for specific domains. A naive solution is to retrain draft models for every target model, which is costly and inefficient. To address this, we introduce a parameter- and data-efficient framework named Efficient Draft Adaptation, abbreviated as EDA, for efficiently adapting draft models. EDA introduces three innovations: (1) a decoupled architecture that utilizes shared and private components to model the shared and target-specific output distributions separately, enabling parameter-efficient adaptation by updating only the lightweight private component;(2) a data regeneration strategy that utilizes the fine-tuned target model to regenerate training data, thereby improving the alignment between training and speculative decoding, leading to higher average acceptance length;(3) a sample selection mechanism that prioritizes high-value data for efficient adaptation. Our experiments show that EDA effectively restores speculative performance on fine-tuned models, achieving superior average acceptance lengths with significantly reduced training costs compared to full retraining. Code is available at https://github.com/Lyn-Lucy/Efficient-Draft-Adaptation.

Luxi Lin, Zhihang Lin, Zhanpeng Zeng +5

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Innovation Institute +1

Speculative Decoding LLM Inference Acceleration Domain Adaptation for LLMs Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

Speculative Decoding

Explore similar work

PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decoding

Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting

Efficiently Aligning Draft Models via Parameter- and Data-Efficient Adaptation