SP$^2$ec: Adaptive Self-Speculative Decoding for Vision-Language Models

Yuqi Huang; Xingyao Li; Yunlong Hou; Fengzhuo Zhang; Jiachun Pan; Vincent Y. F. Tan

cs.LGOct 13, 2026

SP $^2$ ec: Adaptive Self-Speculative Decoding for Vision-Language Models

Authors: Yuqi Huang, Xingyao Li, Yunlong Hou, Fengzhuo Zhang, Jiachun Pan, Vincent Y. F. Tan

Organizations: Department of Mathematics, National University of Singapore, Singapore · Yale University, Center for Algorithms, Data, and Market Design, USA · Department of Electrical and Computer Engineering, National University of Singapore, Singapore

Abstract

Speculative Decoding (SD) has demonstrated its efficacy in accelerating large language model inference, but for Vision-Language Models (VLMs), the diverse input modalities make it difficult for a small trained drafter to achieve consistently high throughput. Self-speculative decoding reuses parts of the target model as a drafter which retains the multi-modal capability of the VLMs, yet its efficiency depends on choosing a suitable draft configuration for each prompt. We propose Single-Peak SPeculative decoding (SP $^2$ ec), a training-free bandit algorithm that adapts the draft configuration during generation. SP $^2$ ec exploits the single-peak throughput pattern observed in existing self-speculative methods, including SWIFT, KnapSpec, SpecVLM, and our simple cosSim. Our algorithmic design explicitly minimizes a novel notion of wall-time regret for each given query, corresponding to the empirical wall-time latency. Theoretical guarantees on the wall-time regret show that SP $^2$ ec can exploit the optimal draft configuration, with negligible exploration cost. Extensive experiments on Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-Mistral-7B across visual and textual tasks show consistent improvements of up to 11% over the original self-speculative methods, demonstrating the wide applicability of the proposed SP $^2$ ec algorithm.

Figures & tables

Figure 1: Prompt-level adaptive drafter selection in SP 2 ec . Each arm corresponds to a drafter with different layer-skipping configuration and wall-time cost. SP 2 ec uses accepted-token and time feedback from the target model to exploit the single-peak throughput structure and adaptively select a drafter during generation. Different prompts can favor different arms, motivating per-query adaptation for VLM speculative decoding.

Panel of Figure 2 — Figure 2: Empirical validation of the layer-skipping candidate arms with cosSim on Qwen2.5-VL-7B-Instruct. (a) Average acceptance rate shows a diminishing return pattern and time per speculative decoding step is an affine function in skip ratio, supporting Assumptions 3.2 and 3.3 . (b) Tokens per second (TPS) illustrate the single-peak structure of throughput. (c) Percentage of prompts for which the arm at each skip ratio is oracle best, i.e., attains the highest measured TPS. The best arm varies across prompts, motivating prompt-adaptive arm selection.

\displaystyle T_{i}(t)=\sum_{s=1}^{t}\mathbbm{1}\{I_{s}=i\},\;\;\hat{\mu}_{i}(t)=\frac{\sum_{s=1}^{t}\textrm{len}(X_{i,s})\mathbbm{1}\{I_{s}=i\}}{T_{i}(t)},\;\;L_{t}=\argmax_{i}\frac{\hat{\mu}_{i}(t)}{c_{i}},\;

(11)

Algorithm 2 Single-Peak Speculative Decoding ( SP 2 ec )

Method	Video-MME		MMBench-Video		ARKitScenes		AIME25		Average
Method	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$
cosSim + UCBSpec	2.53	1.21 $\times$	2.84	1.16 $\times$	4.33	1.15 $\times$	4.08	1.13 $\times$	3.45	1.16 $\times$
cosSim + SP 2 ec	2.86	1.28 $\times$	3.38	1.27 $\times$	3.87	1.25 $\times$	3.35	1.22 $\times$	3.37	1.26 $\times$
SWIFT	2.43	1.14 $\times$	2.60	1.16 $\times$	3.22	1.22 $\times$	2.92	1.10 $\times$	2.79	1.16 $\times$
SWIFT + UCBSpec	2.76	1.16 $\times$	2.87	1.13 $\times$	3.80	1.20 $\times$	3.45	1.15 $\times$	3.22	1.16 $\times$
SWIFT + SP 2 ec	2.51	1.23 $\times$	2.65	1.19 $\times$	3.55	1.24 $\times$	3.29	1.22 $\times$	3.00	1.22 $\times$

Table 1: Main results on Qwen2.5-VL-7B-Instruct. MAT denotes mean accepted tokens, and SU denotes normalized speedup. SP 2 ec achieves consistent wall-time speedup.

Method	Video-MME		MMBench-Video		ARKitScenes		AIME25		Average
Method	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$
cosSim + UCBSpec	3.30	1.19 $\times$	3.19	1.20 $\times$	3.09	1.18 $\times$	4.56	1.25 $\times$	3.54	1.21 $\times$
cosSim + SP 2 ec	3.00	1.26 $\times$	2.87	1.26 $\times$	2.80	1.23 $\times$	4.28	1.40 $\times$	3.24	1.29 $\times$
SWIFT	3.00	1.12 $\times$	3.05	1.22 $\times$	3.11	1.12 $\times$	4.11	1.21 $\times$	3.32	1.17 $\times$
SWIFT + UCBSpec	3.01	1.14 $\times$	3.33	1.22 $\times$	3.15	1.15 $\times$	4.35	1.26 $\times$	3.46	1.19 $\times$
SWIFT + SP 2 ec	2.87	1.19 $\times$	3.24	1.28 $\times$	3.04	1.22 $\times$	4.23	1.32 $\times$	3.35	1.25 $\times$

Table 2: Main results on LLaVA-NeXT-Mistral-7B. SP 2 ec demonstrates consistent wall-time speedup.

Figure 3: Arm-pull allocation on (a) Qwen2.5-VL-7B-Instruct and (b) LLaVA-NeXT-Mistral-7B across Video-MME, MMBench-Video, ARKitScenes, and AIME25. Each dataset group contains two stacked bars: SP 2 ec (left) and UCBSpec (right). Segments show the percentage of pulls allocated to the best, second-best, third-best, and other arms. Bold labels highlight the percentages allocated to the top three arms; dashed outlines identify the other arms.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Average statistics across AIME25 prompts with cosSim on Qwen2.5-VL-7B-Instruct. The horizontal axis is the fraction of skipped layers. (a) average acceptance rate and time per speculative decoding step. (b) Tokens per Second (TPS). (c) Percentage of prompts for which the arm at each skip ratio is oracle best, i.e., attains the highest measured TPS. Error bars in (a) and (b) show $\pm 1$ standard error (SE) for the corresponding metric.

Panel of Figure 5 — Figure 5: Prompt-level statistics for three randomly selected AIME25 prompts with cosSim on Qwen2.5-VL-7B-Instruct. The figures are for (a) acceptance rate, (b) time per speculative decoding step, and (c) TPS, respectively. Colors identify the same prompts across all three figures. Error bars show $\pm 1$ SE specific to each prompt and metric at each skip ratio.

Panel of Figure 6 — Figure 6: Average statistics across Video-MME prompts with cosSim on LLaVA-NeXT-Mistral-7B. The horizontal axis is the fraction of skipped layers. (a) average acceptance rate and time per speculative decoding step. (b) Tokens per Second (TPS). (c) Percentage of prompts for which the arm at each skip ratio is oracle best, i.e., attains the highest measured TPS. Error bars in (a) and (b) show $\pm 1$ standard error (SE) for the corresponding metric.

Panel of Figure 7 — Figure 7: Prompt-level statistics for three randomly selected Video-MME prompts with cosSim on LLaVA-NeXT-Mistral-7B. The figures are for (a) acceptance rate, (b) time per speculative decoding step, and (c) TPS, respectively. Colors identify the same prompts across all three figures. Error bars show $\pm 1$ SE specific to each prompt and metric at each skip ratio.

Panel of Figure 8 — Figure 8: Average statistics across AIME25 prompts with SWIFT on Qwen2.5-VL-7B-Instruct. The horizontal axis is the fraction of skipped layers. (a) average acceptance rate and time per speculative decoding step. (b) Tokens per Second (TPS). (c) Percentage of prompts for which the arm at each skip ratio is oracle best, i.e., attains the highest measured TPS. Error bars in (a) and (b) show $\pm 1$ standard error (SE) for the corresponding metric.

Panel of Figure 9 — Figure 9: Prompt-level statistics for three randomly selected AIME25 prompts with SWIFT on Qwen2.5-VL-7B-Instruct. The figures are for (a) acceptance rate, (b) time per speculative decoding step, and (c) TPS, respectively. Colors identify the same prompts across all three figures. Error bars show $\pm 1$ SE specific to each prompt and metric at each skip ratio.

Panel of Figure 10 — Figure 10: Average statistics across Video-MME prompts with SWIFT on LLaVA-NeXT-Mistral-7B. The horizontal axis is the fraction of skipped layers. (a) average acceptance rate and time per speculative decoding step. (b) Tokens per Second (TPS). (c) Percentage of prompts for which the arm at each skip ratio is oracle best, i.e., attains the highest measured TPS. Error bars in (a) and (b) show $\pm 1$ standard error (SE) for the corresponding metric.

Panel of Figure 11 — Figure 11: Prompt-level statistics for three randomly selected Video-MME prompts with SWIFT on LLaVA-NeXT-Mistral-7B. The figures are for (a) acceptance rate, (b) time per speculative decoding step, and (c) TPS, respectively. Colors identify the same prompts across all three figures. Error bars show $\pm 1$ SE specific to each prompt and metric at each skip ratio.

Panel of Figure 12 — Figure 12: Average statistics across AIME25 prompts with KnapSpec on Qwen2.5-VL-7B-Instruct. The horizontal axis is the budget skip ratio. (a) average acceptance rate and time per speculative decoding step. (b) Tokens per Second (TPS). (c) Percentage of prompts for which the arm at each budget skip ratio is oracle best, i.e., attains the highest measured TPS. Error bars in (a) and (b) show $\pm 1$ standard error (SE) for the corresponding metric.

Panel of Figure 13 — Figure 13: Prompt-level statistics for three randomly selected AIME25 prompts with KnapSpec on Qwen2.5-VL-7B-Instruct. The figures are for (a) acceptance rate, (b) time per speculative decoding step, and (c) TPS, respectively. Colors identify the same prompts across all three figures. Error bars show $\pm 1$ SE specific to each prompt and metric at each budget skip ratio.

Panel of Figure 14 — Figure 14: Average statistics across Video-MME prompts with KnapSpec on LLaVA-NeXT-Mistral-7B. The horizontal axis is the budget skip ratio. (a) average acceptance rate and time per speculative decoding step. (b) Tokens per Second (TPS). (c) Percentage of prompts for which the arm at each budget skip ratio is oracle best, i.e., attains the highest measured TPS. Error bars in (a) and (b) show $\pm 1$ standard error (SE) for the corresponding metric.

Panel of Figure 15 — Figure 15: Prompt-level statistics for three randomly selected Video-MME prompts with KnapSpec on LLaVA-NeXT-Mistral-7B. The figures are for (a) acceptance rate, (b) time per speculative decoding step, and (c) TPS, respectively. Colors identify the same prompts across all three figures. Error bars show $\pm 1$ SE specific to each prompt and metric at each budget ratio.

Panel of Figure 16 — Figure 16: Average statistics across Video-MME prompts with SpecVLM on Qwen2.5-VL-7B-Instruct. The horizontal axis in (c) is the pruning ratio. (a) average acceptance rate and time per speculative decoding step. (b) Tokens per Second (TPS). (c) Percentage of prompts for which the arm at each pruning ratio is oracle best, i.e., attains the highest measured TPS. Error bars in (a) and (b) show $\pm 1$ standard error (SE) for the corresponding metric.

Panel of Figure 17 — Figure 17: Prompt-level statistics for three randomly selected Video-MME prompts with SpecVLM on Qwen2.5-VL-7B-Instruct. The figures are for (a) acceptance rate, (b) time per speculative decoding step, and (c) TPS, respectively. Colors identify the same prompts across all three figures. Error bars show $\pm 1$ SE specific to each prompt and metric at each pruning ratio.

Panel of Figure 18 — Figure 18: Average statistics across Video-MME prompts with SpecVLM on LLaVA-NeXT-Mistral-7B. The horizontal axis is the pruning ratio. (a) average acceptance rate and time per speculative decoding step. (b) Tokens per Second (TPS). (c) Percentage of prompts for which the arm at each pruning ratio is oracle best, i.e., attains the highest measured TPS. Error bars in (a) and (b) show $\pm 1$ standard error (SE) for the corresponding metric.

Panel of Figure 19 — Figure 19: Prompt-level statistics for three randomly selected Video-MME prompts with SpecVLM on LLaVA-NeXT-Mistral-7B. The figures are for (a) acceptance rate, (b) time per speculative decoding step, and (c) TPS, respectively. Colors identify the same prompts across all three figures. Error bars show $\pm 1$ SE specific to each prompt and metric at each pruning ratio.

Panel of Figure 20 — Figure 20: Effect of draft length on average speedup with Qwen2.5-VL-7B-Instruct. The panels compare UCBSpec and SP 2 EC applied to (a) cosSim , (b) KnapSpec , and (c) SpecVLM . The original methods are also shown for KnapSpec and SpecVLM . Speedup is normalized to autoregressive decoding, indicated by the dashed horizontal line at $1\times$ .

Panel of Figure 21 — Figure 21: Effect of draft length on average speedup with LLaVA-NeXT-Mistral-7B. The panels compare UCBSpec and SP 2 EC applied to (a) cosSim , (b) KnapSpec , and (c) SpecVLM . The original methods are also shown for KnapSpec and SpecVLM . Speedup is normalized to autoregressive decoding, indicated by the dashed horizontal line at $1\times$ .

Panel of Figure 22 — Figure 22: Effect of output length on average speedup with Qwen2.5-VL-7B-Instruct. The panels compare UCBSpec and SP 2 EC applied to (a) cosSim , (b) KnapSpec , and (c) SpecVLM . The original methods are also shown for KnapSpec and SpecVLM . The horizontal axis shows output length in tokens on a base-2 logarithmic scale, with $1\mathrm{k}=1024$ . Speedup is normalized to autoregressive decoding, indicated by the dashed horizontal line at $1\times$ .

Panel of Figure 23 — Figure 23: Effect of output length on average speedup with LLaVA-NeXT-Mistral-7B. The panels compare UCBSpec and SP 2 EC applied to (a) cosSim , (b) KnapSpec , and (c) SpecVLM . The original methods are also shown for KnapSpec and SpecVLM . The horizontal axis shows output length in tokens on a base-2 logarithmic scale, with $1\mathrm{k}=1024$ . Speedup is normalized to autoregressive decoding, indicated by the dashed horizontal line at $1\times$ .

Panel of Figure 24 — Figure 24: Effect of input size on average speedup with Qwen2.5-VL-7B-Instruct. The panels compare UCBSpec and SP 2 EC applied to (a) cosSim , (b) KnapSpec , and (c) SpecVLM . The original methods are also shown for KnapSpec and SpecVLM . The horizontal axis shows the visual token budget on a base-2 logarithmic scale, with $1\mathrm{k}=1024$ tokens. Speedup is normalized to autoregressive decoding, indicated by the dashed horizontal line at $1\times$ . AIME25 is excluded when running this comparison.

Panel of Figure 25 — Figure 25: Effect of input size on average speedup with LLaVA-NeXT-Mistral-7B. The panels compare UCBSpec and SP 2 EC applied to (a) cosSim , (b) KnapSpec , and (c) SpecVLM . The original methods are also shown for KnapSpec and SpecVLM . The horizontal axis shows the mean number of input video tokens on a base-2 logarithmic scale, with $1\mathrm{k}=1000$ tokens. Speedup is normalized to autoregressive decoding, indicated by the dashed horizontal line at $1\times$ .

Model	Method	Average SU Reset every 512 speculative steps	Average SU No within-prompt reset
Qwen2.5-VL-7B-Instruct	cosSim + SP 2 ec	1.26	1.25
	SWIFT + SP 2 ec	1.22	1.22
	KnapSpec + SP 2 ec	1.35	1.31
	SpecVLM + SP 2 ec	1.56	1.55
LLaVA-NeXT-Mistral-7B	cosSim + SP 2 ec	1.29	1.27
	SWIFT + SP 2 ec	1.25	1.24

Appendix

Table 3: Average speedup (SU) with and without within-prompt resets. The periodic-reset variant clears the online arm-pull and reward history every 512 speculative decoding steps, whereas the no-reset variant resets only when a new prompt starts.

Figure 26: Runtime breakdown on Qwen2.5-VL-7B-Instruct with (a) cosSim and (b) KnapSpec . Each dataset compares the baseline (left) with SP 2 ec (right). The baseline is UCBSpec for cosSim and the original method for KnapSpec . Numbers indicate percentages of recorded generation time.

Figure 27: Runtime breakdown for SpecVLM on Qwen2.5-VL-7B-Instruct. Each dataset compares original SpecVLM (left) with SpecVLM augmented by SP 2 ec (right). Numbers indicate percentages of recorded generation time.

Method	Training-free	MAT $\uparrow$	SU $\uparrow$
cosSim + SP 2 ec	Yes	4.5	1.31 $\times$
KnapSpec + SP 2 ec	Yes	4.6	1.27 $\times$
EAGLE3	No	1.9	1.20 $\times$

Appendix

Table 4: Comparison with EAGLE3 on MVBench. MAT denotes mean accepted tokens, including one verification bonus token; SU denotes speedup over vanilla autoregressive decoding. Training-free indicates that no auxiliary draft-model training is required.

Method	Video-MME		MMBench-Video		ARKitScenes		WritingBench		Average
Method	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$
cosSim + UCBSpec	3.98	1.13 $\times$	3.64	1.10 $\times$	3.50	1.09 $\times$	3.65	1.08 $\times$	3.69	1.10 $\times$
cosSim + SP 2 ec	3.88	1.22 $\times$	3.57	1.21 $\times$	3.33	1.16 $\times$	3.69	1.17 $\times$	3.62	1.19 $\times$

Appendix

Table 5: Additional results on Qwen3-VL-4B-Instruct. MAT denotes mean accepted tokens, and SU denotes speedup over vanilla autoregressive decoding. Bold marks the higher speedup.

Method	Video-MME		MMBench-Video		ARKitScenes		WritingBench		Average
Method	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$
cosSim + UCBSpec	4.23	1.16 $\times$	3.35	1.10 $\times$	3.40	1.16 $\times$	3.64	1.06 $\times$	3.66	1.12 $\times$
cosSim + SP 2 ec	4.19	1.24 $\times$	3.36	1.17 $\times$	3.43	1.21 $\times$	3.65	1.10 $\times$	3.66	1.18 $\times$

Appendix

Table 6: Additional results on LLaVA-1.5-7B. MAT denotes mean accepted tokens, and SU denotes speedup over vanilla autoregressive decoding. Bold marks the higher speedup.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 308 papers

DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation
May 30, 2026Zining Liu, Yunhai Hu, Tianhua Xia +4Efficient VLM Inference Speculative Decoding
New York University · Cerebras Systems Inc.
MMSpec: Benchmarking Speculative Decoding for Vision-Language Models
Mar 16, 2026Hui Shen, Xin Wang, Ping Zhang +11Speculative Decoding Efficient VLM Inference
University of Michigan · The Ohio State University · Independent +5
See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs
Apr 7, 2026Yicheng Ji, Jun Zhang, Jinpeng Chen +4Speculative Decoding LLM Inference Acceleration
ZJU · BUPT

May 30, 2026J/K move · Enter open · S save

DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation

Zining Liu, Yunhai Hu, Tianhua Xia, Bo Bao+3

New York University · Cerebras Systems Inc.

Efficient VLM Inference Speculative Decoding

Open paper AskPDF arXiv

Speculative decoding (SD) has proven to be an effective technique for accelerating autoregressive generation in large language models (LLMs) however, its application to vision-language models (VLMs) remains relatively unexplored. We propose~DREAM-S, a novel SD framework designed specifically for fast and efficient decoding in VLMs. DREAM-S leverages a neural architecture search (NAS) framework with target-aware supernet training to automatically identify both the optimal interaction strategy between the draft and target models, and the most suitable draft model architecture for the underlying hardware implementation platform. DREAM-S additionally incorporates adaptive intermediate feature distillation, guided by attention entropy, to enable efficient draft training. Experiments on a range of well-established VLMs show that DREAM-S achieves up to a

3.85\times

speedup compared to standard decoding approaches and significantly outperforms existing SD baselines. The code is publicly available at: https://github.com/SAI-Lab-NYU/DREAM-S .

Abstract

Speculative Decoding

Explore similar work

DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation

MMSpec: Benchmarking Speculative Decoding for Vision-Language Models

See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs