SP$^2$ec: Adaptive Self-Speculative Decoding for Vision-Language Models

Yuqi Huang; Xingyao Li; Yunlong Hou; Fengzhuo Zhang; Jiachun Pan; Vincent Y. F. Tan

cs.LGOct 13, 2026

SP $^2$ ec: Adaptive Self-Speculative Decoding for Vision-Language Models

Authors: Yuqi Huang, Xingyao Li, Yunlong Hou, Fengzhuo Zhang, Jiachun Pan, Vincent Y. F. Tan

Organizations: Department of Mathematics, National University of Singapore, Singapore · Yale University, Center for Algorithms, Data, and Market Design, USA · Department of Electrical and Computer Engineering, National University of Singapore, Singapore

Abstract

Speculative Decoding (SD) has demonstrated its efficacy in accelerating large language model inference, but for Vision-Language Models (VLMs), the diverse input modalities make it difficult for a small trained drafter to achieve consistently high throughput. Self-speculative decoding reuses parts of the target model as a drafter which retains the multi-modal capability of the VLMs, yet its efficiency depends on choosing a suitable draft configuration for each prompt. We propose Single-Peak SPeculative decoding (SP $^2$ ec), a training-free bandit algorithm that adapts the draft configuration during generation. SP $^2$ ec exploits the single-peak throughput pattern observed in existing self-speculative methods, including SWIFT, KnapSpec, SpecVLM, and our simple cosSim. Our algorithmic design explicitly minimizes a novel notion of wall-time regret for each given query, corresponding to the empirical wall-time latency. Theoretical guarantees on the wall-time regret show that SP $^2$ ec can exploit the optimal draft configuration, with negligible exploration cost. Extensive experiments on Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-Mistral-7B across visual and textual tasks show consistent improvements of up to 11% over the original self-speculative methods, demonstrating the wide applicability of the proposed SP $^2$ ec algorithm.

Figures & tables

Figure 1: Prompt-level adaptive drafter selection in SP 2 ec . Each arm corresponds to a drafter with different layer-skipping configuration and wall-time cost. SP 2 ec uses accepted-token and time feedback from the target model to exploit the single-peak throughput structure and adaptively select a drafter during generation. Different prompts can favor different arms, motivating per-query adaptation for VLM speculative decoding.

Panel of Figure 2 — Figure 2: Empirical validation of the layer-skipping candidate arms with cosSim on Qwen2.5-VL-7B-Instruct. (a) Average acceptance rate shows a diminishing return pattern and time per speculative decoding step is an affine function in skip ratio, supporting Assumptions 3.2 and 3.3 . (b) Tokens per second (TPS) illustrate the single-peak structure of throughput. (c) Percentage of prompts for which the arm at each skip ratio is oracle best, i.e., attains the highest measured TPS. The best arm varies across prompts, motivating prompt-adaptive arm selection.

\displaystyle T_{i}(t)=\sum_{s=1}^{t}\mathbbm{1}\{I_{s}=i\},\;\;\hat{\mu}_{i}(t)=\frac{\sum_{s=1}^{t}\textrm{len}(X_{i,s})\mathbbm{1}\{I_{s}=i\}}{T_{i}(t)},\;\;L_{t}=\argmax_{i}\frac{\hat{\mu}_{i}(t)}{c_{i}},\;

(11)

Algorithm 2 Single-Peak Speculative Decoding ( SP 2 ec )

Method	Video-MME		MMBench-Video		ARKitScenes		AIME25		Average
Method	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$
cosSim + UCBSpec	2.53	1.21 $\times$	2.84	1.16 $\times$	4.33	1.15 $\times$	4.08	1.13 $\times$	3.45	1.16 $\times$
cosSim + SP 2 ec	2.86	1.28 $\times$	3.38	1.27 $\times$	3.87	1.25 $\times$	3.35	1.22 $\times$	3.37	1.26 $\times$
SWIFT	2.43	1.14 $\times$	2.60	1.16 $\times$	3.22	1.22 $\times$	2.92	1.10 $\times$	2.79	1.16 $\times$
SWIFT + UCBSpec	2.76	1.16 $\times$	2.87	1.13 $\times$	3.80	1.20 $\times$	3.45	1.15 $\times$	3.22	1.16 $\times$
SWIFT + SP 2 ec	2.51	1.23 $\times$	2.65	1.19 $\times$	3.55	1.24 $\times$	3.29	1.22 $\times$	3.00	1.22 $\times$

Table 1: Main results on Qwen2.5-VL-7B-Instruct. MAT denotes mean accepted tokens, and SU denotes normalized speedup. SP 2 ec achieves consistent wall-time speedup.

Method	Video-MME		MMBench-Video		ARKitScenes		AIME25		Average
Method	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$
cosSim + UCBSpec	3.30	1.19 $\times$	3.19	1.20 $\times$	3.09	1.18 $\times$	4.56	1.25 $\times$	3.54	1.21 $\times$
cosSim + SP 2 ec	3.00	1.26 $\times$	2.87	1.26 $\times$	2.80	1.23 $\times$	4.28	1.40 $\times$	3.24	1.29 $\times$
SWIFT	3.00	1.12 $\times$	3.05	1.22 $\times$	3.11	1.12 $\times$	4.11	1.21 $\times$	3.32	1.17 $\times$
SWIFT + UCBSpec	3.01	1.14 $\times$	3.33	1.22 $\times$	3.15	1.15 $\times$	4.35	1.26 $\times$	3.46	1.19 $\times$
SWIFT + SP 2 ec	2.87	1.19 $\times$	3.24	1.28 $\times$	3.04	1.22 $\times$	4.23	1.32 $\times$	3.35	1.25 $\times$

Table 2: Main results on LLaVA-NeXT-Mistral-7B. SP 2 ec demonstrates consistent wall-time speedup.

Figure 3: Arm-pull allocation on (a) Qwen2.5-VL-7B-Instruct and (b) LLaVA-NeXT-Mistral-7B across Video-MME, MMBench-Video, ARKitScenes, and AIME25. Each dataset group contains two stacked bars: SP 2 ec (left) and UCBSpec (right). Segments show the percentage of pulls allocated to the best, second-best, third-best, and other arms. Bold labels highlight the percentages allocated to the top three arms; dashed outlines identify the other arms.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Average statistics across AIME25 prompts with cosSim on Qwen2.5-VL-7B-Instruct. The horizontal axis is the fraction of skipped layers. (a) average acceptance rate and time per speculative decoding step. (b) Tokens per Second (TPS). (c) Percentage of prompts for which the arm at each skip ratio is oracle best, i.e., attains the highest measured TPS. Error bars in (a) and (b) show $\pm 1$ standard error (SE) for the corresponding metric.

Panel of Figure 5 — Figure 5: Prompt-level statistics for three randomly selected AIME25 prompts with cosSim on Qwen2.5-VL-7B-Instruct. The figures are for (a) acceptance rate, (b) time per speculative decoding step, and (c) TPS, respectively. Colors identify the same prompts across all three figures. Error bars show $\pm 1$ SE specific to each prompt and metric at each skip ratio.

Panel of Figure 6 — Figure 6: Average statistics across Video-MME prompts with cosSim on LLaVA-NeXT-Mistral-7B. The horizontal axis is the fraction of skipped layers. (a) average acceptance rate and time per speculative decoding step. (b) Tokens per Second (TPS). (c) Percentage of prompts for which the arm at each skip ratio is oracle best, i.e., attains the highest measured TPS. Error bars in (a) and (b) show $\pm 1$ standard error (SE) for the corresponding metric.

Panel of Figure 7 — Figure 7: Prompt-level statistics for three randomly selected Video-MME prompts with cosSim on LLaVA-NeXT-Mistral-7B. The figures are for (a) acceptance rate, (b) time per speculative decoding step, and (c) TPS, respectively. Colors identify the same prompts across all three figures. Error bars show $\pm 1$ SE specific to each prompt and metric at each skip ratio.

Panel of Figure 8 — Figure 8: Average statistics across AIME25 prompts with SWIFT on Qwen2.5-VL-7B-Instruct. The horizontal axis is the fraction of skipped layers. (a) average acceptance rate and time per speculative decoding step. (b) Tokens per Second (TPS). (c) Percentage of prompts for which the arm at each skip ratio is oracle best, i.e., attains the highest measured TPS. Error bars in (a) and (b) show $\pm 1$ standard error (SE) for the corresponding metric.

Panel of Figure 9 — Figure 9: Prompt-level statistics for three randomly selected AIME25 prompts with SWIFT on Qwen2.5-VL-7B-Instruct. The figures are for (a) acceptance rate, (b) time per speculative decoding step, and (c) TPS, respectively. Colors identify the same prompts across all three figures. Error bars show $\pm 1$ SE specific to each prompt and metric at each skip ratio.

Panel of Figure 10 — Figure 10: Average statistics across Video-MME prompts with SWIFT on LLaVA-NeXT-Mistral-7B. The horizontal axis is the fraction of skipped layers. (a) average acceptance rate and time per speculative decoding step. (b) Tokens per Second (TPS). (c) Percentage of prompts for which the arm at each skip ratio is oracle best, i.e., attains the highest measured TPS. Error bars in (a) and (b) show $\pm 1$ standard error (SE) for the corresponding metric.

Panel of Figure 11 — Figure 11: Prompt-level statistics for three randomly selected Video-MME prompts with SWIFT on LLaVA-NeXT-Mistral-7B. The figures are for (a) acceptance rate, (b) time per speculative decoding step, and (c) TPS, respectively. Colors identify the same prompts across all three figures. Error bars show $\pm 1$ SE specific to each prompt and metric at each skip ratio.

Panel of Figure 12 — Figure 12: Average statistics across AIME25 prompts with KnapSpec on Qwen2.5-VL-7B-Instruct. The horizontal axis is the budget skip ratio. (a) average acceptance rate and time per speculative decoding step. (b) Tokens per Second (TPS). (c) Percentage of prompts for which the arm at each budget skip ratio is oracle best, i.e., attains the highest measured TPS. Error bars in (a) and (b) show $\pm 1$ standard error (SE) for the corresponding metric.

Panel of Figure 13 — Figure 13: Prompt-level statistics for three randomly selected AIME25 prompts with KnapSpec on Qwen2.5-VL-7B-Instruct. The figures are for (a) acceptance rate, (b) time per speculative decoding step, and (c) TPS, respectively. Colors identify the same prompts across all three figures. Error bars show $\pm 1$ SE specific to each prompt and metric at each budget skip ratio.

Panel of Figure 14 — Figure 14: Average statistics across Video-MME prompts with KnapSpec on LLaVA-NeXT-Mistral-7B. The horizontal axis is the budget skip ratio. (a) average acceptance rate and time per speculative decoding step. (b) Tokens per Second (TPS). (c) Percentage of prompts for which the arm at each budget skip ratio is oracle best, i.e., attains the highest measured TPS. Error bars in (a) and (b) show $\pm 1$ standard error (SE) for the corresponding metric.

Panel of Figure 15 — Figure 15: Prompt-level statistics for three randomly selected Video-MME prompts with KnapSpec on LLaVA-NeXT-Mistral-7B. The figures are for (a) acceptance rate, (b) time per speculative decoding step, and (c) TPS, respectively. Colors identify the same prompts across all three figures. Error bars show $\pm 1$ SE specific to each prompt and metric at each budget ratio.

Panel of Figure 16 — Figure 16: Average statistics across Video-MME prompts with SpecVLM on Qwen2.5-VL-7B-Instruct. The horizontal axis in (c) is the pruning ratio. (a) average acceptance rate and time per speculative decoding step. (b) Tokens per Second (TPS). (c) Percentage of prompts for which the arm at each pruning ratio is oracle best, i.e., attains the highest measured TPS. Error bars in (a) and (b) show $\pm 1$ standard error (SE) for the corresponding metric.

Panel of Figure 17 — Figure 17: Prompt-level statistics for three randomly selected Video-MME prompts with SpecVLM on Qwen2.5-VL-7B-Instruct. The figures are for (a) acceptance rate, (b) time per speculative decoding step, and (c) TPS, respectively. Colors identify the same prompts across all three figures. Error bars show $\pm 1$ SE specific to each prompt and metric at each pruning ratio.

Panel of Figure 18 — Figure 18: Average statistics across Video-MME prompts with SpecVLM on LLaVA-NeXT-Mistral-7B. The horizontal axis is the pruning ratio. (a) average acceptance rate and time per speculative decoding step. (b) Tokens per Second (TPS). (c) Percentage of prompts for which the arm at each pruning ratio is oracle best, i.e., attains the highest measured TPS. Error bars in (a) and (b) show $\pm 1$ standard error (SE) for the corresponding metric.

Panel of Figure 19 — Figure 19: Prompt-level statistics for three randomly selected Video-MME prompts with SpecVLM on LLaVA-NeXT-Mistral-7B. The figures are for (a) acceptance rate, (b) time per speculative decoding step, and (c) TPS, respectively. Colors identify the same prompts across all three figures. Error bars show $\pm 1$ SE specific to each prompt and metric at each pruning ratio.

Panel of Figure 20 — Figure 20: Effect of draft length on average speedup with Qwen2.5-VL-7B-Instruct. The panels compare UCBSpec and SP 2 EC applied to (a) cosSim , (b) KnapSpec , and (c) SpecVLM . The original methods are also shown for KnapSpec and SpecVLM . Speedup is normalized to autoregressive decoding, indicated by the dashed horizontal line at $1\times$ .

Panel of Figure 21 — Figure 21: Effect of draft length on average speedup with LLaVA-NeXT-Mistral-7B. The panels compare UCBSpec and SP 2 EC applied to (a) cosSim , (b) KnapSpec , and (c) SpecVLM . The original methods are also shown for KnapSpec and SpecVLM . Speedup is normalized to autoregressive decoding, indicated by the dashed horizontal line at $1\times$ .

Panel of Figure 22 — Figure 22: Effect of output length on average speedup with Qwen2.5-VL-7B-Instruct. The panels compare UCBSpec and SP 2 EC applied to (a) cosSim , (b) KnapSpec , and (c) SpecVLM . The original methods are also shown for KnapSpec and SpecVLM . The horizontal axis shows output length in tokens on a base-2 logarithmic scale, with $1\mathrm{k}=1024$ . Speedup is normalized to autoregressive decoding, indicated by the dashed horizontal line at $1\times$ .

Panel of Figure 23 — Figure 23: Effect of output length on average speedup with LLaVA-NeXT-Mistral-7B. The panels compare UCBSpec and SP 2 EC applied to (a) cosSim , (b) KnapSpec , and (c) SpecVLM . The original methods are also shown for KnapSpec and SpecVLM . The horizontal axis shows output length in tokens on a base-2 logarithmic scale, with $1\mathrm{k}=1024$ . Speedup is normalized to autoregressive decoding, indicated by the dashed horizontal line at $1\times$ .

Panel of Figure 24 — Figure 24: Effect of input size on average speedup with Qwen2.5-VL-7B-Instruct. The panels compare UCBSpec and SP 2 EC applied to (a) cosSim , (b) KnapSpec , and (c) SpecVLM . The original methods are also shown for KnapSpec and SpecVLM . The horizontal axis shows the visual token budget on a base-2 logarithmic scale, with $1\mathrm{k}=1024$ tokens. Speedup is normalized to autoregressive decoding, indicated by the dashed horizontal line at $1\times$ . AIME25 is excluded when running this comparison.

Panel of Figure 25 — Figure 25: Effect of input size on average speedup with LLaVA-NeXT-Mistral-7B. The panels compare UCBSpec and SP 2 EC applied to (a) cosSim , (b) KnapSpec , and (c) SpecVLM . The original methods are also shown for KnapSpec and SpecVLM . The horizontal axis shows the mean number of input video tokens on a base-2 logarithmic scale, with $1\mathrm{k}=1000$ tokens. Speedup is normalized to autoregressive decoding, indicated by the dashed horizontal line at $1\times$ .

Model	Method	Average SU Reset every 512 speculative steps	Average SU No within-prompt reset
Qwen2.5-VL-7B-Instruct	cosSim + SP 2 ec	1.26	1.25
	SWIFT + SP 2 ec	1.22	1.22
	KnapSpec + SP 2 ec	1.35	1.31
	SpecVLM + SP 2 ec	1.56	1.55
LLaVA-NeXT-Mistral-7B	cosSim + SP 2 ec	1.29	1.27
	SWIFT + SP 2 ec	1.25	1.24

Appendix

Table 3: Average speedup (SU) with and without within-prompt resets. The periodic-reset variant clears the online arm-pull and reward history every 512 speculative decoding steps, whereas the no-reset variant resets only when a new prompt starts.

Figure 26: Runtime breakdown on Qwen2.5-VL-7B-Instruct with (a) cosSim and (b) KnapSpec . Each dataset compares the baseline (left) with SP 2 ec (right). The baseline is UCBSpec for cosSim and the original method for KnapSpec . Numbers indicate percentages of recorded generation time.

Figure 27: Runtime breakdown for SpecVLM on Qwen2.5-VL-7B-Instruct. Each dataset compares original SpecVLM (left) with SpecVLM augmented by SP 2 ec (right). Numbers indicate percentages of recorded generation time.

Method	Training-free	MAT $\uparrow$	SU $\uparrow$
cosSim + SP 2 ec	Yes	4.5	1.31 $\times$
KnapSpec + SP 2 ec	Yes	4.6	1.27 $\times$
EAGLE3	No	1.9	1.20 $\times$

Appendix

Table 4: Comparison with EAGLE3 on MVBench. MAT denotes mean accepted tokens, including one verification bonus token; SU denotes speedup over vanilla autoregressive decoding. Training-free indicates that no auxiliary draft-model training is required.

Method	Video-MME		MMBench-Video		ARKitScenes		WritingBench		Average
Method	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$
cosSim + UCBSpec	3.98	1.13 $\times$	3.64	1.10 $\times$	3.50	1.09 $\times$	3.65	1.08 $\times$	3.69	1.10 $\times$
cosSim + SP 2 ec	3.88	1.22 $\times$	3.57	1.21 $\times$	3.33	1.16 $\times$	3.69	1.17 $\times$	3.62	1.19 $\times$

Appendix

Table 5: Additional results on Qwen3-VL-4B-Instruct. MAT denotes mean accepted tokens, and SU denotes speedup over vanilla autoregressive decoding. Bold marks the higher speedup.

Method	Video-MME		MMBench-Video		ARKitScenes		WritingBench		Average
Method	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$	MAT $\uparrow$	SU $\uparrow$
cosSim + UCBSpec	4.23	1.16 $\times$	3.35	1.10 $\times$	3.40	1.16 $\times$	3.64	1.06 $\times$	3.66	1.12 $\times$
cosSim + SP 2 ec	4.19	1.24 $\times$	3.36	1.17 $\times$	3.43	1.21 $\times$	3.65	1.10 $\times$	3.66	1.18 $\times$

Appendix

Table 6: Additional results on LLaVA-1.5-7B. MAT denotes mean accepted tokens, and SU denotes speedup over vanilla autoregressive decoding. Bold marks the higher speedup.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 308 papers

May 30, 2026

DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation

Speculative decoding (SD) has proven to be an effective technique for accelerating autoregressive generation in large language models (LLMs) however, its application to vision-language models (VLMs) remains relatively unexplored. We propose~DREAM-S, a novel SD framework designed specifically for fast and efficient decoding in VLMs. DREAM-S leverages a neural architecture search (NAS) framework with target-aware supernet training to automatically identify both the optimal interaction strategy between the draft and target models, and the most suitable draft model architecture for the underlying hardware implementation platform. DREAM-S additionally incorporates adaptive intermediate feature distillation, guided by attention entropy, to enable efficient draft training. Experiments on a range of well-established VLMs show that DREAM-S achieves up to a

3.85\times

speedup compared to standard decoding approaches and significantly outperforms existing SD baselines. The code is publicly available at: https://github.com/SAI-Lab-NYU/DREAM-S .

Zining Liu, Yunhai Hu, Tianhua Xia +4

New York University · Cerebras Systems Inc.

Efficient VLM Inference Speculative Decoding

Ask PDF

Mar 16, 2026

MMSpec: Benchmarking Speculative Decoding for Vision-Language Models

Vision-language models (VLMs) achieve strong performance on multimodal tasks but suffer from high inference latency due to large model sizes and long multimodal contexts. Speculative decoding has recently emerged as an effective acceleration technique, yet its behavior in VLMs remains insufficiently understood. We introduce MMSpec, the first benchmark for evaluating speculative decoding in vision-language models. MMSpec contains 600 multimodal samples across six task categories and integrates ten representative speculative decoding algorithms under a unified evaluation framework. Our study reveals three key findings: (1) methods designed for text-only LLMs degrade in multimodal scenarios, (2) vision awareness becomes increasingly important at larger batch sizes, and (3) throughput speedup alone does not reliably reflect latency performance. Motivated by these findings, we propose ViSkip, a plug-and-play speculative decoding method that dynamically adapts speculation to vision tokens and achieves state-of-the-art performance.

Hui Shen, Xin Wang, Ping Zhang +11

University of Michigan · The Ohio State University · Independent +5

Speculative Decoding Efficient VLM Inference Large Vision-Language Models

Ask PDF

Apr 7, 2026

See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs

Video Large Language Models (Video-LLMs) excel in video understanding but suffer from high inference latency during autoregressive generation. Speculative Decoding (SD) mitigates this by applying a draft-and-verify paradigm, yet existing methods are constrained by rigid exact-match rules, severely limiting the acceleration potential. To bridge this gap, we propose LVSpec, the first training-free loosely SD framework tailored for Video-LLMs. Grounded in the insight that generation is governed by sparse visual-relevant anchors (mandating strictness) amidst abundant visual-irrelevant fillers (permitting loose verification), LVSpec employs a lightweight visual-relevant token identification scheme to accurately pinpoint the former. To further maximize acceptance, we augment this with a position-shift tolerant mechanism that effectively salvages positionally mismatched but semantically equivalent tokens. Experiments demonstrate that LVSpec achieves high fidelity and speed: it preserves >99.8 of target performance while accelerating Qwen2.5-VL-32B by 2.70x and LLaVA-OneVision-72B by 2.94x. Notably, it boosts the mean accepted length and speedup ratio by 136% and 35% compared to SOTA training-free SD methods for Video-LLMs.

Yicheng Ji, Jun Zhang, Jinpeng Chen +4

ZJU · BUPT

Speculative Decoding LLM Inference Acceleration Video-Language Models

Ask PDF

Abstract

Speculative Decoding

Explore similar work

DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation

MMSpec: Benchmarking Speculative Decoding for Vision-Language Models

See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs