Multi-Drafter Speculative Decoding with Alignment Feedback

Taehyeon Kim; Hojung Jung; Se-Young Yun

cs.CLApr 7, 2026

Multi-Drafter Speculative Decoding with Alignment Feedback

Authors: Taehyeon Kim, Hojung Jung, Se-Young Yun

Organizations: LG AI Research · KAIST AI

Abstract

Speculative decoding (SD) accelerates large language model (LLM) inference by using a smaller model to draft future tokens, which are then verified by the target LLM. This preserves generation quality by accepting only aligned tokens. However, individual drafters, often trained for specific tasks or domains, exhibit limited effectiveness across diverse applications. To address this, we introduce MetaSD, a unified framework that integrates multiple drafters into the SD process. MetaSD dynamically allocates computational resources to heterogeneous drafters by leveraging alignment feedback and framing drafter selection as a multi-armed bandit problem. Extensive experiments show MetaSD consistently outperforms single-drafter approaches.

Figures & tables

Method	Multi-drafter	Training-free	No extra compute	Regret bound	RNS
Standard SD	✗	✓	—	—	—
Static Ensemble SD	✓	✓	✗	✗	✓
Learned Routing	✓	✗	✓	✗	✗
MetaSD (Ours)	✓	✓	✓	✓	✓

Table 1: Comparison of MetaSD with existing single- and multi-drafter SD approaches. MetaSD is training-free without extra compute for drafting and verification cost than standard, offers theoretical regret guarantees and Robustness to Non-Stationarity (RNS).

Figure 1: Overview of speculative decoding with multiple drafters in multi-armed bandit (MAB) framework. Drafters are selected and updated based on alignment feedback through the block divergence reward. The example in this figure is from an instance in MT-Bench dataset ( Zheng et al., 2024 ) .

Drafter	Mean reward		variance		Zero-reward ratio †
Drafter	BE	BD	BE	BD	BE only
Ja-drafter	0.232	0.488	0.093	0.044	0.503
Ru-drafter	0.099	0.294	0.032	0.026	0.678
De-drafter	0.081	0.317	0.024	0.032	0.721
Fr-drafter	0.074	0.288	0.023	0.029	0.743
Zh-drafter	0.106	0.326	0.037	0.034	0.681

Table 2: Comparison of reward statistics on the Japanese queries with multilingual specialized drafters. Detailed descriptions are in Table 9 .

Speedup	SpS with specialized drafters					SpS	Other methods		Bandit in MetaSpS
Speedup	Drafter1	Drafter2	Drafter3	Drafter4	Drafter5	OFA	PLD	Lookahead	Rand	EXP3	SH	UCB
Code	2.437 $\CIRCLE$	1.224	1.565	1.814	1.687	2.435 $\CIRCLE$	1.923	1.542	1.640	1.919	2.148	2.300
Trans	0.991	2.076 $\CIRCLE$	1.000	1.019	0.950	1.032	1.076	1.133	1.150	1.217	1.422	1.587 $\CIRCLE$
Sum	1.513	1.087	2.133 $\CIRCLE$	1.510	1.387	1.526	2.501 $\CIRCLE$	1.275	1.429	1.606	1.812	1.971
QA	1.332	1.200	1.343	1.960 $\CIRCLE$	1.252	1.267	1.178	1.208	1.294	1.437	1.599	1.711 $\CIRCLE$
Math	1.483	1.228	1.378	1.486	2.454 $\CIRCLE$	1.571	1.653	1.533	1.471	1.690	2.144	2.280 $\CIRCLE$

Table 3: (Black-box SD) Speedup ratio relative to standard autoregressive greedy decoding on various datasets, comparing single specialized independent drafters, other methods (PLD ( Saxena, 2023 ) and Lookahead ( Fu et al., 2024 ) ), and bandit-based drafter selection (Rand (uniformly random), EXP3 ( Auer et al., 2002 ) , SH ( Karnin et al., 2013 ) , UCB). Evaluations are conducted with a single NVIDIA A6000 GPU under greedy decoding settings. Drafter specializations: 1: Code, 2: Translation, 3: Summarization, 4: QA, 5: Math.

Speedup	Specialized Eagle drafters					Eagle	Other methods				Bandit in MetaEagle
Speedup	Eagle1	Eagle2	Eagle3	Eagle4	Eagle5	OFA	BPD	R-BPD	Medusa	R-Medusa	Rand	EXP3	SH	UCB
Code	3.934 $\CIRCLE$	1.303	1.776	2.150	2.427	3.776 $\CIRCLE$	1.963	2.146	2.661	2.822	2.310	2.858	3.650	3.724
Trans	1.750	2.496 $\CIRCLE$	2.281	2.131	1.714	2.143	1.626	1.442	1.909	2.056	2.036	2.171	2.225	2.318 $\CIRCLE$
Sum	1.707	1.507	3.382 $\CIRCLE$	2.005	1.589	2.640	1.509	1.455	1.723	2.136	2.261	2.261	2.801	3.057 $\CIRCLE$
QA	1.842	1.579	2.181	2.916 $\CIRCLE$	1.783	2.446	1.489	1.468	1.817	2.154	2.006	2.128	2.466	2.641 $\CIRCLE$
Math	2.584	1.618	2.337	2.433	3.903 $\CIRCLE$	3.049	1.696	1.696	2.142	2.519	2.449	2.811	3.339	3.520 $\CIRCLE$

Table 4: (White-box SD) Speedup ratio relative to standard autoregressive greedy decoding on various datasets, comparing single specialized drafters, other methods (blockwise parallel decoding (BPD) ( Stern et al., 2018 ) , Medusa, Rescored-BPD (R-BPD) and Rescored-Medusa ( Kim et al., 2024 ) ), and bandit-based drafter selection. Evaluations are conducted with a single NVIDIA A100 GPU under greedy decoding settings.

Speedup	Sps with specialized drafters					Bandit in MetaSps
Speedup	Drafter1	Drafter2	Drafter3	Drafter4	Drafter5	DraftAll	EXP3	UCB
Ja →En	1.757 $\CIRCLE$	1.109	1.012	1.018	1.154	0.352	1.260 $\CIRCLE$	1.161
Ru →En	1.055	1.817 $\CIRCLE$	0.995	0.963	1.036	0.396	1.259	1.503 $\CIRCLE$
De →En	1.098	1.369	2.360 $\CIRCLE$	1.036	1.099	0.544	1.472	1.693 $\CIRCLE$
Fr →En	1.106	1.445	1.108	2.135 $\CIRCLE$	1.122	0.484	1.506	1.775 $\CIRCLE$
Zh →En	1.198	1.086	1.021	1.023	1.516 $\CIRCLE$	0.367	1.204	1.369 $\CIRCLE$

Table 5: Speedup ratio relative to standard AR greedy decoding on various multilingual datasets, comparing single specialized drafters, all drafting method, and our bandit-based algorithm (EXP3, UCB). Evaluations are conducted with a single NVIDIA A5000 GPU under greedy decoding settings. Drafter specializations: 1: Ja →En, 2: Ru →En, 3: De →En, 4: Fr →En, 5: Zh →En.

Figure 2: Ablations on

N_{max}

. ‘Optimal’ represents the optimal drafter and UCB denotes MetaSps-UCB with BD.

Routing Method	In-domain		Perturbed prompts		Out-of-Domain (Sps/Eagle)
Routing Method	SpS	Eagle	SpS	Eagle	RAG	Finance
MOE routing (BERT)
Finetuned BERT-Base	1.772	2.759	1.456	2.245	1.645 / 2.132	1.410 / 2.413
Finetuned BERT-Large	1.741	2.797	1.567	2.323	1.638 / 2.110	1.398 /2.426
MetaSD (Bandit-based policy routing)
MetaSpS-UCB	1.912 $\CIRCLE$	—	1.820 $\CIRCLE$	—	1.799 $\CIRCLE$	1.436 $\CIRCLE$

Table 6: Speedup ratio comparison of MoE routing and MetaSD across different scenarios.

Task	MetaSpS-UCB		MetaEagle-UCB
Task	BE	BD	BE	BD
Code	$2.052_{\pm 0.004}$	$2.231_{\pm 0.006}$ $\CIRCLE$	$3.590_{\pm 0.017}$	$3.661_{\pm 0.003}$ $\CIRCLE$
Trans	$1.465_{\pm 0.004}$	$1.554_{\pm 0.001}$ $\CIRCLE$	$2.228_{\pm 0.009}$ $\CIRCLE$	$2.201_{\pm 0.001}$
Sum	$1.770_{\pm 0.002}$	$1.929_{\pm 0.001}$ $\CIRCLE$	$3.038_{\pm 0.005}$	$3.043_{\pm 0.001}$ $\CIRCLE$
QA	$1.591_{\pm 0.003}$	$1.698_{\pm 0.001}$ $\CIRCLE$	$2.629_{\pm 0.003}$ $\CIRCLE$	$2.608_{\pm 0.001}$
Math	$1.992_{\pm 0.003}$	$2.238_{\pm 0.002}$ $\CIRCLE$	$3.461_{\pm 0.009}$	$3.515_{\pm 0.001}$ $\CIRCLE$

Table 7: Average of speedup ratio comparing the BE and BD rewards for MetaSD-UCB with both SpS and EAGLE drafters over 3 different runs.

Figure 3: Best arm ratio over rounds for various configurations. (Left) MetaSpS (black-box SD) with BE and BD rewards. (Right) MetaEagle (white-box SD) with BE and BD rewards.

Task	Eagle1	Eagle2	Eagle3	Eagle4	Eagle5	OFA	MetaSD-UCB
Code	3.748	1.335	1.697	2.030	2.451	3.626 $\CIRCLE$	3.563
Trans	1.757	2.553	2.161	2.035	1.677	2.293	2.375 $\CIRCLE$
Sum	1.671	1.529	3.084	1.939	1.639	2.648	2.742 $\CIRCLE$
QA	1.837	1.616	2.094	2.932	1.722	2.439	2.516 $\CIRCLE$
Math	2.511	1.664	2.207	2.913	3.844	3.136	3.366 $\CIRCLE$

Table 8: White-box performance with perturbed prompts (speedup relative to greedy decoding on A100).

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Comparison of average speedup ratios by various methods relative to standard autoregressive greedy decoding on a single NVIDIA A100 GPU. The target model is Vicuna 7B v1.3. (a) Results for black-box methods. (b) Results for white-box methods. Detailed settings are in section 4 .

Dataset	Ja-drafter	Ru-drafter	De-drafter	Fr-drafter	Zh-drafter
Ja →En	1.757 $\CIRCLE$	1.109	1.012	1.018	1.154
Ru →En	1.055	1.817 $\CIRCLE$	0.995	0.963	1.036
De →En	1.098	1.369	2.360 $\CIRCLE$	1.036	1.099
Fr →En	1.106	1.445	1.108	2.135 $\CIRCLE$	1.122
Zh →En	1.198	1.086	1.021	1.023	1.516 $\CIRCLE$

Appendix

Table 9: Speedup ratio relative to the standard autoregressive greedy decoding on various multilingual datasets following Yi et al. (2024) where target model is Vicuna 7B v1.3 and the drafter is decoder-only 68M language model: Japanese (Ja)→English (En) ( Morishita et al., 2022 ) , Russian (Ru)→En, German (De)→En ( Bojar et al., 2016 ) , French (Fr)→En ( Bojar et al., 2014 ) , and Chinese (Zh)→En ( Barrault et al., 2019 ) . Evaluations are conducted with a NVIDIA A5000 GPU.

Panel of Figure 5 — Figure 5: Comparison of rewards on the Ja→En dataset across different drafters in two scenarios: (a) BE and (b) BD. Box plots show the distribution of rewards, with whiskers extending to the 5th and 95th percentiles. Drafter specializations: 1: Ja →En, 2: Ru →En, 3: De →En, 4: Fr →En, 5: Zh →En.

Dataset	Drafter1	Drafter2	Drafter3	Drafter4	Drafter5	UCB
Long De→En	1.238	1.316	2.044	0.970	1.187	2.031

Appendix

Table 10: Speedup ratio on long-context De→En translation with the same settings in Table 5 .

Dataset	Drafter1	Drafter2	Drafter3	Drafter4	Drafter5	OFA Drafter	MetaSpS-UCB	EAGLE1	EAGLE2	EAGLE3	EAGLE4	EAGLE5	OFA Eagle	MetaEagle-UCB
RAG	1.720	1.373	1.752	1.944	1.552	1.638	1.799	1.844	1.568	2.566	2.535	1.793	2.175	2.238
Finance	1.416	1.284	1.414	1.550	1.397	1.367	1.436	2.432	2.175	2.494	2.826	2.175	2.435	2.517

Appendix

Table 11: Performance of MetaSpS, MetaEagle, and baselines on out-of-domain datasets (measured on A100 GPU).

Task	Drafter1	Drafter2	Drafter3	Drafter4	Drafter5	OFA Drafter	MetaSpS - UCB
Code	2.368	1.158	1.521	1.763	1.633	1.937	2.139
Translation	0.997	1.986	0.973	1.036	0.935	0.969	1.422
CNN	1.458	1.016	1.895	1.458	1.318	1.521	1.779
NQA	1.297	1.158	1.285	1.907	1.237	1.387	1.610
MathQA	1.482	1.184	1.357	1.470	2.346	1.895	2.149

Appendix

Table 12: Black-box performance with perturbed prompts (speedup relative to greedy decoding, measured on A100 GPU).

Task	MetaEagle-UCB (Recomputing KV)	MetaEagle-UCB with StreamingLLM
Code	3.724	3.624
Trans	2.318	2.352
Sum	3.057	2.986
NQA	2.641	2.654
Math	3.520	3.338

Appendix

Table 13: Performance comparison of MetaSD-UCB with different KV cache strategies (speedup relative to standard greedy decoding, measured on A100 GPU).

Dataset	SpS with specialized drafters					Bandit
Dataset	Drafter1	Drafter2	Drafter3	Drafter4	Drafter5	UCB
Code	2.250	1.215	1.379	1.532	1.513	1.896
Trans	1.086	1.886	1.096	1.130	1.078	1.431
Sum	1.461	1.165	1.874	1.463	1.353	1.744
QA	1.316	1.193	1.324	1.776	1.272	1.534
Math	1.450	1.258	1.355	1.616	2.379	2.046

Appendix

Table 14: Speedup ratio with temperature sampling as temperature is set to 0.7 over a NVIDIA A6000 GPU.

Task	Drafter1	Drafter2	Drafter3	Drafter4	Drafter5	OFA	MetaSpS-UCB
Code	1.781	0.963	1.168	1.260	1.178	1.501	1.596
Translation	0.856	1.695	0.897	0.880	0.838	0.861	1.197
CNN	1.201	0.918	1.629	1.223	1.092	1.230	1.439
NQA	1.073	0.961	1.132	1.510	1.031	1.123	1.322
MathQA	1.220	1.026	1.200	1.360	1.968	1.512	1.673

Appendix

Table 15: Performance of MetaSpS under stochastic decoding (temperature = 1.0). Speedup ratios are reported relative to standard autoregressive greedy decoding.

Task	EAGLE1	EAGLE2	EAGLE3	EAGLE4	EAGLE5	OFA Eagle	MetaEagle-UCB
Code	3.019	0.872	1.012	1.348	1.809	2.926	2.765
Translation	1.030	1.817	1.373	1.278	0.997	1.578	1.668
CNN	0.998	0.834	2.289	1.267	0.864	1.749	1.935
NQA	1.179	0.922	1.269	2.181	1.011	1.680	1.756
MathQA	1.739	0.966	1.462	2.141	3.099	2.289	2.539

Appendix

Table 16: Performance of MetaEagle under stochastic decoding (temperature = 1.0). Speedup ratios are reported relative to standard autoregressive greedy decoding.

Task / Batch Size	Single Batch Throughput		Heterogeneous Batch Throughput
Task / Batch Size	OFA Eagle	MetaEagle-UCB	Single OFA Eagle	Homogeneous (MetaEagle-UCB)	Heterogeneous (MetaEagle-UCB)
Physics	2.424	2.573	—	—	—
Hotpot QA	2.262	2.270	—	—	—
MMLU-CoT (Avg. 57 tasks)	2.466	2.529	—	—	—
Batch Size = 1	—	—	2.803	3.045	3.045
Batch Size = 2	—	—	2.751	2.933	2.518

Appendix

Table 17: Throughput in single-batch and heterogeneous batch settings.

Notation	Descriptions
$K$	Number of drafters
$[K]$	For a given integer $K$ , denotes the set $\{1,...,K\}$
$i$	Drafter index $i\in[K]$
$\alpha_{i}$	True mean of acceptance rate when using drafter $i$
$i^{\star}$	Drafter index with the highest $\alpha_{i}$
$t$	Number of current round

Appendix

Table 18: Mathematical terms and notations in our work.

Panel of Figure 6 — Figure 6: Empirical measurement of BD reward statistics along speculation rounds in greedy decoding ( $T=0$ ) scenario.

Panel of Figure 7 — Figure 7: Empirical measurement of BD reward statistics along speculation rounds in temperature sampling $T=1$ .

\hat{r}_{j,t}=\begin{cases}r_{j,t}/p_{t}(j)&\text{if }j=a_{t}\\ 0&\text{otherwise},\end{cases}

Appendix

Algorithm 7 MetaSD-EXP3 ( Auer et al., 2002 )

	Drafter1	Drafter2	Drafter3	Drafter4	Drafter5	Upper Bound	MetaSpS-UCB
Block Efficiency	1.668	1.722	1.485	1.759	1.803	1.803	1.951
Speedup Ratio	1.429	1.492	1.289	1.539	1.581	1.581	1.722

Appendix

Table 19: Results of the non-stationary translation task.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 309 papers

MineDraft: A Framework for Batch Parallel Speculative Decoding
Feb 24, 2026Zhenwei Tang, Arun Verma, Zijian Zhou +4Speculative Decoding LLM Inference Acceleration
College of Computing and Data Science, Nanyang Technological University, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre · Department of Computer Science, National University of Singapore, Republic of Singapore +1
Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning
Mar 2, 2026Jiebin Zhang, Zhenghan Yu, Liang Wang +8Speculative Decoding LLM Inference Acceleration
National Key Laboratory for Multimedia Information Processing, Peking University · School of Computer Science, Peking University · Microsoft Research Asia
Beyond the Target: From Imitation to Collaboration in Speculative Decoding
May 24, 2026Jinze Li, Yixing Xu, Guanchen Li +7Speculative Decoding LLM Inference Acceleration
Advanced Micro Devices, Inc., Beijing, China · The University of Hong Kong · University of North Texas

Feb 24, 2026J/K move · Enter open · S save

MineDraft: A Framework for Batch Parallel Speculative Decoding

Zhenwei Tang, Arun Verma, Zijian Zhou, Zhaoxuan Wu+3

College of Computing and Data Science, Nanyang Technological University, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre · Department of Computer Science, National University of Singapore, Republic of Singapore+1

Speculative Decoding LLM Inference Acceleration Parallel Speculative Decoding Parallel LLM Inference

Open paper AskPDF arXiv

Speculative decoding (SD) accelerates large language model inference by using a smaller draft model to propose draft tokens that are subsequently verified by a larger target model. However, the performance of standard SD is often limited by the strictly sequential execution of these drafting and verification stages. To address this, this paper proposes MineDraft, a batch parallel speculative decoding (PSD) framework designed to effectively hide drafting latency by overlapping it with verification. Our theoretical analysis shows that PSD is substantially more efficient than standard SD. MineDraft realizes the PSD through a novel batch-parallel design that maintains two batches of requests, overlapping drafting for one batch with verification for the other. Our experimental results show significant improvements of \alg{} in both throughput (up to 75%) and end-to-end latency (up to 39%) over standard SD. Furthermore, we have implemented MineDraft as a plugin for vLLM, demonstrating its practicality for production-ready inference systems.

Abstract

Speculative Decoding

Explore similar work

MineDraft: A Framework for Batch Parallel Speculative Decoding

Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning

Beyond the Target: From Imitation to Collaboration in Speculative Decoding