Organizations: National Key Laboratory of Parallel and Distributed Computing College of Computer Science and Technology National University of Defense Technology, Changsha, China.
Although speculative decoding is widely used to accelerate Vision-Language Models (VLMs) inference, it faces severe performance collapse when applied to Video Large Language Models (Vid-LLMs). The draft model typically falls into the trap of attention dilution and negative visual gain due to key-value cache explosion and context window mismatches. We observe a visual semantic internalization phenomenon in Vid-LLMs, indicating that critical visual semantics are implicitly encoded into text hidden states during deep-layer interactions, which renders raw visual inputs structurally redundant during deep inference. To address this, we propose the Sparrow framework, which first utilizes visually-aware text-anchored window attention via hidden state reuse to fully offload visual computation to the target model, and leverages intermediate-layer visual state bridging to train the draft model with semantic-rich intermediate states, thereby filtering out low-level visual noise. Additionally, a multi-token prediction strategy is introduced to bridge the training-inference distribution shift. Experiments show that Sparrow achieves an average speedup of 2.82x even with 25k visual tokens, effectively resolving the performance degradation in long sequences and offering a practical solution for real-time long video tasks.
Figures & tables
Figure 1: Analysis of the impact of visual token length on performance. (a) Comparison of average accepted length and drafting latency between MSD and ViSpec on the VideoDetailCaption dataset across varying visual sequence lengths. (b) Performance on image and video tasks with MSD at different visual retention rates. The Last Instr. and All Text strategies retain the top-x% visual tokens based on the attention scores of the final instruction token and the average attention scores of all text tokens towards visual regions, respectively. ViSpec and MSD employ Qwen2.5-VL-7B and Qwen2-VL-7B as target models, respectively, with both utilizing the official released weights for their draft models.
Figure 2: (a) & (b) Visualization of the average attention weight distribution from the last instruction token to preceding Instruction, Visual, and Text tokens in short (0.4k) and long (3k) sequence tasks.
Figure 3: Layer-wise Importance Analysis of Visual Tokens in Qwen2.5-VL-7B. (a) Comparison of task accuracy (solid lines) after removing visual tokens starting from layer x versus the native baseline (dashed lines). (b) Sum of attention received by all visual tokens from the last instruction token across different attention heads and model layers.
Figure 4: Illustration of the Sparrow framework. The left and right panels illustrate the training and inference phases, respectively. Initially, the target model performs multi-stage fusion on visual embeddings vi and text embeddings ei , yielding a noise-filtered visual hidden state hvim and a visually-infused text hidden state heih . In the training phase, hvim and heih are concatenated and fed into the draft model to produce hei . Subsequently, hvim is concatenated with hei and re-input into the draft model. Losses from both stages are jointly computed to mitigate the discrepancy between training and inference. In the inference phase, the draft model directly takes heih as input.
Training & Inference Input
ChartQA
AI2D
Visual + Text
4.25
4.28
0% Visual + Text
4.23
4.25
50% Visual + Text
4.21
4.24
Table 1: Impact of visual information injection on draft model performance during training.
Method
0.5k
1.5k
17k
25k
τ
DSR
τ
DSR
τ
DSR
τ
DSR
MSD
4.12
2.96x
3.97
2.87x
1.11
0.48x
1.04
0.42x
ViSpec (orig.)
3.45
2.67x
3.35
2.52x
2.96
1.31x
2.81
1.20x
ViSpec
4.29
3.22x
4.18
3.10x
3.74
1.64x
3.42
1.48x
MSD+VATA
4.11
2.98x
4.01
2.92x
3.94
1.72x
4.01
1.68x
Ours
4.51
3.30x
4.36
3.18x
4.32
1.86x
4.37
1.82x
Table 4: Performance of Qwen2.5-VL-7B on VideoDetailCaption across varying visual token lengths, evaluated on A800 GPUs, in terms of average accepted length ( τ ) and decoding speedup ratio (DSR).
Visual Input Length
4.5K
15K
25K
Average Accepted Length
3.07
2.78
2.69
Generated Tokens
418.86
450.53
455.10
Target Model Calls
136.23
161.58
168.69
Wall (End-to-end) Time (s)
8.71
19.28
29.59
Total Decode Time (s)
7.51
13.52
18.13
Prefill Time (s)
1.20
5.76
11.46
Table 5: Performance breakdown of LLaVA-OneVision-7B on VideoDetailedCaption across varying visual input lengths using L20 GPUs.
Method Components
MME
VideoDetailedCaption
MTP
IVSB
VATA
0.4k
0.5k
1.5k
6k
17k
25k
X
X
X
3.40
3.95
3.87
3.29
2.97
2.95
X
X
✓
3.41
3.95
3.85
3.83
3.77
3.86
✓
X
X
3.75
4.44
4.31
3.87
3.58
3.51
✓
X
✓
3.75
4.45
4.34
4.24
4.24
4.29
✓
✓
X
3.82
4.53
4.38
3.08
1.91
1.21
Table 6: We conduct an ablation study on MTP, intermediate-layer visual state bridging(IVSB), and visually-aware text-anchored window attention(VATA) using Qwen2.5-VL-7B deployed on A800 GPUs as the base model on the MME and VideoDetailedCaption benchmarks, reporting the average accepted length across varying visual token lengths.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Draft Model Input
ChartQA
AI2D
Visual + Text
4.25
4.28
50% Visual + Text
4.26
4.30
12.5% Visual + Text
4.25
4.30
0% Visual + Text
4.27
4.31
Appendix
Table 7: Impact of visual information injection on draft model performance during inference.
Figure 5: Layer-wise importance analysis of visual tokens in LLaVA-OneVision-7B. The figure compares task accuracy after removing visual tokens starting from layer x (solid lines) with the native baseline performance (dashed lines).
Figure 6: Layer-wise information retention analysis of visual and text modalities in Qwen2.5-VL-7B. The curves illustrate the retention rates calculated based on similarity, with red denoting visual features and blue denoting text features. The vertical dashed line marks the critical Layer 20.
Video Large Language Models (Video-LLMs) excel in video understanding but suffer from high inference latency during autoregressive generation. Speculative Decoding (SD) mitigates this by applying a draft-and-verify paradigm, yet existing methods are constrained by rigid exact-match rules, severely limiting the acceleration potential. To bridge this gap, we propose LVSpec, the first training-free loosely SD framework tailored for Video-LLMs. Grounded in the insight that generation is governed by sparse visual-relevant anchors (mandating strictness) amidst abundant visual-irrelevant fillers (permitting loose verification), LVSpec employs a lightweight visual-relevant token identification scheme to accurately pinpoint the former. To further maximize acceptance, we augment this with a position-shift tolerant mechanism that effectively salvages positionally mismatched but semantically equivalent tokens. Experiments demonstrate that LVSpec achieves high fidelity and speed: it preserves >99.8 of target performance while accelerating Qwen2.5-VL-32B by 2.70x and LLaVA-OneVision-72B by 2.94x. Notably, it boosts the mean accepted length and speedup ratio by 136% and 35% compared to SOTA training-free SD methods for Video-LLMs.
Although current Video-LLMs achieve impressive performance in video understanding tasks, their autoregressive decoding efficiency remains constrained by the massive number of video tokens. Visual token pruning can partially ease this bottleneck, yet existing approaches still suffer from information loss and yield only modest acceleration in decoding. In this paper, we propose ParallelVLM, a training-free draft-then-verify speculative decoding framework that overcomes both mutual waiting and limited speedup-ratio problems between draft and target models in long-video settings. ParallelVLM features two parallelized stages that maximize hardware utilization and incorporate an Unbiased Verifier-Guided Pruning strategy to better align the draft and target models by eliminating the positional bias in attention-guided pruning. Extensive experiments demonstrate that ParallelVLM effectively expands the draft window by 1.6∼1.8× with high accepted lengths, and accelerates various video understanding benchmarks by 3.36× on LLaVA-Onevision-72B and 2.42× on Qwen2.5-VL-32B compared with vanilla autoregressive decoding.
Vision-language models (VLMs) achieve strong performance on multimodal tasks but suffer from high inference latency due to large model sizes and long multimodal contexts. Speculative decoding has recently emerged as an effective acceleration technique, yet its behavior in VLMs remains insufficiently understood. We introduce MMSpec, the first benchmark for evaluating speculative decoding in vision-language models. MMSpec contains 600 multimodal samples across six task categories and integrates ten representative speculative decoding algorithms under a unified evaluation framework. Our study reveals three key findings: (1) methods designed for text-only LLMs degrade in multimodal scenarios, (2) vision awareness becomes increasingly important at larger batch sizes, and (3) throughput speedup alone does not reliably reflect latency performance. Motivated by these findings, we propose ViSkip, a plug-and-play speculative decoding method that dynamically adapts speculation to vision tokens and achieves state-of-the-art performance.
Hui Shen, Xin Wang, Ping Zhang +11
University of Michigan · The Ohio State University · Independent +5