Speculative Decoding Reimagined for Multimodal Large Language Models
Authors: Luxi Lin, Zhihang Lin, Zhanpeng Zeng, Rongrong Ji
Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Shanghai Innovation Institute
Multimodal Large Language Models (MLLMs) deliver strong performance across vision-language tasks but suffer from high autoregressive decoding latency. Although speculative decoding can accelerate Large Language Models (LLMs) without changing their output distribution, directly applying existing approaches to MLLMs yields limited speedups because text and visual tokens exhibit distinct characteristics. We present Multimodal Speculative Decoding (MSD), a framework specifically designed for efficient MLLM inference. MSD is motivated by two key observations: text and visual tokens should be handled separately during drafting, and an effective draft model requires both strong language modeling and visual perception capabilities. Accordingly, MSD decouples text and visual token processing in the draft model and applies token-specific strategies to improve drafting efficiency. We further introduce a two-stage training strategy. The draft model is first trained on text-only instruction-tuning data to strengthen its language modeling ability and is then progressively exposed to multimodal data to develop visual perception while preserving its language competence. Extensive experiments across multimodal benchmarks demonstrate that MSD achieves up to 2.29× inference speedup for LLaVA-1.5-7B and 2.46× for LLaVA-1.5-13B without sacrificing generation quality. These results establish MSD as an effective and general approach for accelerating multimodal autoregressive generation. Our code is available at https://github.com/Lyn-Lucy/MSD
Figures & tables
Type
ChartQA
TextVQA
Hallusion
Average
Token-level
2.01
2.42
2.12
2.18
Feature-level
3.45
3.81
3.72
3.66
TABLE I: Average acceptance length of draft models with different predictive objectives on three multimodal tasks: ChartQA [ 18 ] , TextVQA [ 19 ] , and HallusionBench [ 20 ] , using LLaVA-1.5-7B [ 1 ] as the target model. Both draft models use the same architecture and number of parameters. “Hallusion” denotes HallusionBench in all tables.
Fig. 1: Input illustration of MLLMs. The input to MLLMs typically comprises two distinct types of tokens: textual tokens and visual tokens
Fig. 2: The framework of Multimodal Speculative Decoding (MSD). The left illustrates the draft phase, while the right illustrates the training phase. e represents token embeddings, and f denotes features. Red-bordered tokens are predicted by the draft model. MSD concatenates text token hidden states with the next token embeddings, while directly inputting visual tokens embedding without concatenation during drafting, and applies a lightweight clustering-based module to compress the visual tokens seen by the draft model. MSD trains the draft model using a text-only instruction dataset in Stage 1, and then gradually introduces a multimodal instruction dataset in Stage 2.
Method
ChartQA
TextVQA
Hallusion
Average
Text-only instruction dataset
3.26
3.62
3.50
3.46
Vision instruction dataset
3.45
3.81
3.72
3.66
TABLE II: Average acceptance length ( τ ) of the draft model trained on different datasets across three multimodal tasks [ 18 , 19 , 20 ] . The evaluation uses LLaVA-1.5-7B as the target model.
Fig. 3: Comparison of models trained on text-only and vision instruction data, using LLaVA-1.6-7B as the target model for evaluation. Black tokens are judged as correct predictions, while red and blue tokens indicate vision-related errors and visually unrelated errors, respectively.
Draft Model Training
MT-Bench
GSM8K
Natural Ques.
Average
Stage 1 (Text-only)
5.25
4.69
3.78
4.57
+ Direct visual fine-tuning
4.45 (-0.80)
3.85 (-0.84)
3.17 (-0.61)
3.82 (-0.75)
+ Progressive visual fine-tuning
5.10 (-0.15)
4.48 (-0.21)
3.72 (-0.06)
4.43 (-0.14)
TABLE III: Impact of draft-model training strategies on average acceptance length ( τ ) across three text benchmarks [ 36 , 35 , 37 ] , evaluated with LLaVA-1.6-7B as the target model.
VQAv2
AI2D
SQA_Image
ChartQA
TextVQA
Hallusion
MMB_EN
MME
Model
Method
SR
τ
SR
τ
SR
τ
SR
τ
SR
τ
SR
τ
SR
τ
SR
τ
Temperature = 0
LLaVA-1.5-7B
Baseline
2.45 ×
4.23
2.23 ×
3.46
2.18 ×
3.47
2.21 ×
3.45
1.99 ×
3.81
2.36 ×
3.72
2.28 ×
3.58
2.37 ×
3.67
Lookahead
1.28 ×
1.33
1.37 ×
1.45
1.35 ×
1.45
1.32 ×
1.42
1.22 ×
1.31
1.41 ×
1.50
1.32 ×
1.41
1.34 ×
1.40
Medusa
1.32 ×
2.28
1.39 ×
1.79
1.34 ×
1.79
1.36 ×
1.80
1.22 ×
1.89
1.26 ×
1.91
1.40 ×
1.86
1.28 ×
1.83
Text_Only
2.23 ×
4.01
1.98 ×
3.14
1.94 ×
3.30
2.02 ×
3.21
1.65 ×
3.55
2.11 ×
3.45
2.01 ×
3.29
2.18 ×
3.34
TABLE IV: Speedup ratios SR and average acceptance lengths τ of different speculative decoding methods on VQAv2 [ 5 ] , AI2D [ 38 ] , SQA_Image [ 39 ] , ChartQA [ 18 ] , TextVQA [ 19 ] , Hallusion [ 20 ] , MMB_EN [ 24 ] , and MME [ 40 ] with temperature T∈{0,1} using LLaVA-1.5-7B as the target model.
VQAv2
AI2D
SQA_Image
ChartQA
TextVQA
Hallusion
MMB_EN
MME
Model
Method
SR
τ
SR
τ
SR
τ
SR
τ
SR
τ
SR
τ
SR
τ
SR
τ
Temperature = 0
LLaVA-1.6-7B
Baseline
1.70 ×
3.82
2.03 ×
3.54
2.01 ×
3.65
2.11 ×
3.52
1.71 ×
3.99
2.23 ×
3.86
2.08 ×
3.79
2.14 ×
3.69
ViSpec
2.12 ×
4.45
2.64 ×
4.22
2.44 ×
4.25
2.56 ×
4.29
2.16 ×
4.25
2.81 ×
4.70
2.77 ×
4.56
2.66 ×
4.46
MSD
2.21 ×
4.67
2.79 ×
4.41
2.59 ×
4.41
2.72 ×
4.49
2.27 ×
4.39
2.91 ×
4.79
2.81 ×
4.66
2.81 ×
4.67
LLaVA-1.6-13B
Baseline
2.04 ×
4.22
2.49 ×
3.79
2.50 ×
3.86
2.51 ×
3.94
2.00 ×
4.14
2.58 ×
4.14
2.38 ×
3.90
2.66 ×
3.86
TABLE V: Speedup ratios SR and average acceptance lengths τ on the same benchmarks with temperature T∈{0,1} using LLaVA-1.6-7B, LLaVA-1.6-13B [ 3 ] , and Qwen2.5-VL-7B [ 4 ] as target models.
Method
Common Stages
Drafting & Verification
Total
Vision
Projector
Prefill
Reduction
Draft
Verify
Rounds
τ
LLaVA-1.6-7B
16.10
0.67
308.39
–
–
–
–
–
6416.34
+ Baseline
15.89
0.66
308.43
–
25.88
28.81
62.75
3.54
2806.74
+ MSD
15.93
0.66
307.94
1.28
9.16
28.59
48.54
4.41
2266.73
TABLE VI: System-level latency breakdown on AI2D. Draft is the first draft forward pass, Verify is measured per round, Rounds and τ denote the average verification-round count and acceptance length, respectively, and Total is the complete decoding latency per sample. All latency values are in ms.
Model
ChartQA ↑
AI2D ↑
MMB ↑
Hallusion ↑
Average
Baseline
3.52
3.54
3.79
3.86
3.68
+ Input Tokens Decoupling
3.93 (+0.41)
3.96 (+0.42)
4.25 (+0.46)
4.33 (+0.47)
4.12 (+0.44)
+ Two-Stage (Direct)
4.16 (+0.64)
4.05 (+0.51)
4.32 (+0.53)
4.42 (+0.56)
4.24 (+0.56)
+ Two-Stage (Progressive)
4.27 (+0.75)
4.24 (+0.70)
4.51 (+0.72)
4.56 (+0.70)
4.40 (+0.72)
MSD
4.49 (+0.97)
4.41 (+0.87)
4.66 (+0.87)
4.79 (+0.93)
4.59 (+0.91)
TABLE VII: Ablation study on the contributions of MSD components, evaluated by average acceptance length.
Fig. 4: Impact of visual token reduction on the draft model’s forward latency and the overall inference speedup, evaluated on the ChartQA benchmark [ 18 ] .
Method
ChartQA
AI2D
MMB EN
Hallusion
Average
K-Means
4.46 / 2.58 ×
4.40 / 2.91 ×
4.59 / 2.95 ×
4.90 / 2.93 ×
4.59 / 2.84 ×
FasterVLM
4.40 / 2.53 ×
4.35 / 2.84 ×
4.51 / 2.87 ×
4.80 / 2.86 ×
4.52 / 2.78 ×
ToMe
4.45 / 2.48 ×
4.32 / 2.77 ×
4.54 / 2.80 ×
4.81 / 2.79 ×
4.53 / 2.71 ×
Resampler
4.48 / 2.42 ×
4.44 / 2.69 ×
4.63 / 2.72 ×
4.92 / 2.70 ×
4.62 / 2.63 ×
TABLE VIII: Comparison of visual token reduction methods on LLaVA-1.6-7B at T=0 . All training and inference are conducted on an NVIDIA A100 GPU. Each entry reports τ / end-to-end speedup.
Method
ChartQA
AI2D
MMB
Hallusion
Baseline
3.52
3.54
3.79
3.86
Visual Feature
3.91 ( +0.39 )
3.92 ( +0.38 )
4.22 ( +0.43 )
4.30 ( +0.44 )
Visual Embedding
3.93 ( +0.41 )
3.96 ( +0.42 )
4.25 ( +0.46 )
4.33 ( +0.47 )
TABLE IX: Ablation study on different strategies for handling visual tokens, evaluated in terms of the average acceptance length across validation benchmarks.
Fig. 5: Single-layer draft-model attention for the first output token and draft-token comparison with and without input token decoupling on LLaVA-1.6-7B. The prompt is “Provide a one-sentence caption for the provided image.” Percentages show visual-token attention; red words mark draft tokens rejected by the target model.
Task
K=32
K=512
K=2048
DocVQA
4.40 / 2.47 ×
4.45 / 2.45 ×
4.52 / 2.40 ×
VQAv2
4.67 / 2.22 ×
4.64 / 2.16 ×
4.70 / 2.09 ×
TABLE X: Task-specific sensitivity to visual compression under different retained tokens K . Each entry reports τ / speedup ratio.
Strategy
τ
SR
Global Clustering
3.63
1.73 ×
Per-Image Clustering
4.08
2.01 ×
TABLE XI: Comparison of multi-image token reduction strategies evaluated on MuirBench.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Target Model
Hidden Dim.
FFN Dim.
Attention Heads
LLaVA-1.5/1.6-7B
4096
11008
32
LLaVA-1.6-13B
5120
13824
40
Qwen2.5-VL-7B
3584
18944
28
Appendix
TABLE XII: Hyperparameters of the single-layer draft Transformer.
Random Seed
0
1
2
42
1234
Average Acceptance Length
4.66
4.67
4.66
4.68
4.67
Appendix
TABLE XIII: Sensitivity of average acceptance length to K-Means initialization seeds on VQAv2 with LLaVA-1.6-7B.
Task
Input Tokens
Output Tokens
τ
Throughput (tok/s)
Speedup
AI2D
1923.7
213.2
4.41
88.23
2.82 ×
SQA_Image
1865.8
141.9
4.41
84.44
2.77 ×
TextVQA
2336.9
81.6
4.39
66.43
2.35 ×
Appendix
TABLE XIV: Effect of input and output lengths on MSD efficiency. All values are averages over evaluation samples. All throughput and speedup measurements are conducted on an NVIDIA A100 GPU.
Fig. 6: Comparison of acceptance rates between Baseline and MSD across different temperatures on TextVQA with LLaVA-1.6-7B.
# Visual Tokens
Clustering (ms)
Total Drafting (ms/sample)
Overhead (%)
576
0.93
620.4
0.15%
1917
1.28
1163.6
0.11%
Appendix
TABLE XV: Latency overhead of the visual token reduction module at different visual token resolutions, averaged over the ChartQA benchmark on an NVIDIA H200 GPU.
Fig. 7: Impact of draft model training strategies evaluated on ChartQA with LLaVA-1.6-7B.
Method
ChartQA
AI2D
Hallusion
VQAv2
LLaVA-1.6-7B
0.36
0.68
0.40
0.81
LLaVA-1.6-7B + MSD
0.36
0.68
0.40
0.81
Appendix
TABLE XVI: Task scores of LLaVA-1.6-7B with vanilla decoding and MSD under greedy decoding ( T=0 ).