cs.CVMar 16, 2026

MMSpec: Benchmarking Speculative Decoding for Vision-Language Models

Authors: Hui Shen, Xin Wang, Ping Zhang, Yunta Hsieh, Qi Han, Zhongwei Wan, Ziheng Zhang, Jingxuan Zhang, 

Organizations: University of Michigan · The Ohio State University · Independent · Indiana University · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision-language models (VLMs) achieve strong performance on multimodal tasks but suffer from high inference latency due to large model sizes and long multimodal contexts. Speculative decoding has recently emerged as an effective acceleration technique, yet its behavior in VLMs remains insufficiently understood. We introduce MMSpec, the first benchmark for evaluating speculative decoding in vision-language models. MMSpec contains 600 multimodal samples across six task categories and integrates ten representative speculative decoding algorithms under a unified evaluation framework. Our study reveals three key findings: (1) methods designed for text-only LLMs degrade in multimodal scenarios, (2) vision awareness becomes increasingly important at larger batch sizes, and (3) throughput speedup alone does not reliably reflect latency performance. Motivated by these findings, we propose ViSkip, a plug-and-play speculative decoding method that dynamically adapts speculation to vision tokens and achieves state-of-the-art performance.

Figures & tables

Figure 1: Performance comparison of speculative decoding methods with Qwen2.5-VL-7B on MMSpec benchmark.
Figure 2: Sampled data from MMSpec
TopicSamplesData source(s)Avg. output length (tokens)
General VQA100GQA [ 6 ]46.98
Text VQA100TextVQA [ 20 ]63.15
Image Captioning100COCO [ 15 ]191.90
Chart VQA100CharXiv [ 21 ]68.56
Complex Reasoning100MMMU-Pro [ 23 ]285.60
Multi-turn Conversation100ConvBench [ 16 ] MMMTBench [ 1 ]747.65
Table 1: MMSpec composition, avg. output length computed from Qwen2.5-VL-7B.
MethodsKey IdeaDraftingVision AwarenessCategory
 StructureReuse
ViSpec [ 7 ]Vision-token compression for efficient multimodal drafting.Linear✗Vision-awareTraining-based
MSD [ 14 ]Train a multimodal draft with staged VLM training.Linear✗Vision-awareTraining-based
EAGLE-1 [ 11 ]Feature-level drafting from target hidden states.Linear✗Vision-agnosticTraining-based
EAGLE-2 [ 9 ]Context-adaptive draft tree for higher acceptance.Tree✗Vision-agnosticTraining-based
EAGLE-3 [ 12 ]Token-level drafting with multi-layer feature fusion.Tree✗Vision-agnosticTraining-based
Table 2: Summary of different speculative decoding methods. The table outlines each method’s key idea, category ( training-based , Training-free , or vision-aware ), speculation structure ( Linear or Tree ), ability to reuse repetitive content during generation, and whether the method is vision-aware or vision-agnostic .
Figure 3: Overview of speculative decoding algorithms evaluated in MMSpec framework.
ModelSubtaskGQATVQAICCQACRMTCOverall
MethodMATSpeedMATSpeedMATSpeedMATSpeedMATSpeedMATSpeedMATSpeed
Qwen2.5-VL-7BAR Baseline-1 ×\times-1 ×\times-1 ×\times-1 ×\times-1 ×\times-1 ×\times-1 ×\times
Training-based Methods
EAGLE-12.411.81 ×\times2.041.28 ×\times2.462.59 ×\times2.241.93 ×\times2.462.22 ×\times2.332.14 ×\times2.362.11 ×\times
EAGLE-21.791.73 ×\times1.521.17 ×\times1.702.32 ×\times1.871.95 ×\times1.862.56 ×\times1.781.92 ×\times1.782.02 ×\times
EAGLE-30.431.02 ×\times0.220.83 ×\times0.381.17 ×\times0.221.09 ×\times0.160.85 ×\times0.250.97 ×\times0.240.96 ×\times
Table 3: Performance comparison of speculative decoding methods for Qwen2.5-VL-7B and LLaVA-1.5-7B. The highest and the second highest scores of methods are respectively highlighted in blue and red .
Figure 4: Speedup comparison of speculative decoding methods across different batch sizes. The speedup is measured relative to autoregressive decoding.
Figure 5: CDF of per-sample latency for different speculative decoding methods.
Figure 6: Evaluation of ViSkip and baseline methods on Qwen2.5-VL-7B, including (a) Speedup across different batch sizes. (b) CDF of wall time per sample. (c) Average latency breakdown of draft and target models.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

ModelSubtaskGQATVQAICCQACRMTCOverall
MethodMATSpeedMATSpeedMATSpeedMATSpeedMATSpeedMATSpeedMATSpeed
Qwen2.5-VL-7BAR Baseline-1 ×\times-1 ×\times-1 ×\times-1 ×\times-1 ×\times-1 ×\times-1 ×\times
Training-based Methods
EAGLE-12.291.56 ×\times1.801.09 ×\times2.222.11 ×\times2.302.01 ×\times2.262.08 ×\times1.991.65 ×\times2.091.76 ×\times
EAGLE-21.591.42 ×\times1.391.07 ×\times1.591.66 ×\times1.771.75 ×\times1.701.76 ×\times1.671.65 ×\times1.661.63 ×\times
EAGLE-30.030.71 ×\times0.010.57 ×\times0.081.11 ×\times0.030.98 ×\times0.130.72 ×\times0.031.41 ×\times0.071.02 ×\times
Appendix
Table 4: Performance comparison of speculative decoding methods for Qwen2.5-VL-7B and LLaVA-1.5-7B at temperature t=0.6t=0.6 . The highest and the second highest overall scores are respectively highlighted in blue and red .
ItemSetting
Evaluated modelsQwen/Qwen2.5-VL-7B-Instruct llava-hf/llava-1.5-7b-hf
Batch size1
Max new tokens1024
Decoding temperaturest∈{0.0,0.6}t\in\{0.0,0.6\}
Appendix
Table 5: Shared settings for the main MMSpec experiments.
MethodTypeHyperparameters used in main runs
EAGLE-1TBdepth=3 , top_k=8 , total_token=30
EAGLE-2TBdepth=3 , top_k=8 , total_token=30 , threshold=0.3
EAGLE-3TBdepth=3 , top_k=8 , total_token=30
MedusaTBdepth=3 , top_k=8 , total_token=30
MSDTBdepth=5 , top_k=10
ViSpecTBdepth=3 , top_k=8 , total_token=30 , num_q=2
Appendix
Table 6: Method-specific parameters used in the main experiments. “TB” and “TF” denote training-based and training-free methods, respectively.
MethodMeasured stepsHigh-visual stepsAvg. accept (high-visual)Avg. accept (low-visual)
MSD2170332 (15.3%)2.122.27
ViSpec1874170 (9.1%)2.122.39
SAM2469176 (7.1%)0.140.44
Appendix
Table 7: Step-level acceptance statistics on Qwen2.5-VL-7B from the visual-attention probe runs on MMSpec testmini set. “Measured steps” are steps with a valid visual-attention ratio.
More in this topic

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 309 papers

Explore similar work

CardsList

  1. SP2^2ec: Adaptive Self-Speculative Decoding for Vision-Language Models

    Oct 13, 2026Yuqi Huang, Xingyao Li, Yunlong Hou +3Speculative DecodingEfficient VLM Inference

    Department of Mathematics, National University of Singapore, Singapore · Yale University, Center for Algorithms, Data, and Market Design, USA · Department of Electrical and Computer Engineering, National University of Singapore, Singapore

  2. CoVSpec: Efficient Device-Edge Co-Inference for Vision-Language Models via Speculative Decoding

    May 4, 2026Yuanyuan Jia, Shunpu Tang, Qianqian YangEfficient VLM InferenceEdge Computing

    College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, China

  3. TIGER: Text-Conditioned Visual Gated Routing with Acceptance Alignment for Multimodal Speculative Decoding

    Jul 13, 2026Quynh Vo, Cong-Duy Nguyen, Ponhvoan Srey +2Efficient VLM InferenceSpeculative Decoding

    National University of Singapore · Nanyang Technological University · Center of AI Research, VinUniversity