Organizations: National Key Laboratory of Parallel and Distributed Computing College of Computer Science and Technology National University of Defense Technology, Changsha, China.
Although speculative decoding is widely used to accelerate Vision-Language Models (VLMs) inference, it faces severe performance collapse when applied to Video Large Language Models (Vid-LLMs). The draft model typically falls into the trap of attention dilution and negative visual gain due to key-value cache explosion and context window mismatches. We observe a visual semantic internalization phenomenon in Vid-LLMs, indicating that critical visual semantics are implicitly encoded into text hidden states during deep-layer interactions, which renders raw visual inputs structurally redundant during deep inference. To address this, we propose the Sparrow framework, which first utilizes visually-aware text-anchored window attention via hidden state reuse to fully offload visual computation to the target model, and leverages intermediate-layer visual state bridging to train the draft model with semantic-rich intermediate states, thereby filtering out low-level visual noise. Additionally, a multi-token prediction strategy is introduced to bridge the training-inference distribution shift. Experiments show that Sparrow achieves an average speedup of 2.82x even with 25k visual tokens, effectively resolving the performance degradation in long sequences and offering a practical solution for real-time long video tasks.
Figures & tables
Figure 1: Analysis of the impact of visual token length on performance. (a) Comparison of average accepted length and drafting latency between MSD and ViSpec on the VideoDetailCaption dataset across varying visual sequence lengths. (b) Performance on image and video tasks with MSD at different visual retention rates. The Last Instr. and All Text strategies retain the top-x% visual tokens based on the attention scores of the final instruction token and the average attention scores of all text tokens towards visual regions, respectively. ViSpec and MSD employ Qwen2.5-VL-7B and Qwen2-VL-7B as target models, respectively, with both utilizing the official released weights for their draft models.
Figure 2: (a) & (b) Visualization of the average attention weight distribution from the last instruction token to preceding Instruction, Visual, and Text tokens in short (0.4k) and long (3k) sequence tasks.
Figure 3: Layer-wise Importance Analysis of Visual Tokens in Qwen2.5-VL-7B. (a) Comparison of task accuracy (solid lines) after removing visual tokens starting from layer x versus the native baseline (dashed lines). (b) Sum of attention received by all visual tokens from the last instruction token across different attention heads and model layers.
Figure 4: Illustration of the Sparrow framework. The left and right panels illustrate the training and inference phases, respectively. Initially, the target model performs multi-stage fusion on visual embeddings vi and text embeddings ei , yielding a noise-filtered visual hidden state hvim and a visually-infused text hidden state heih . In the training phase, hvim and heih are concatenated and fed into the draft model to produce hei . Subsequently, hvim is concatenated with hei and re-input into the draft model. Losses from both stages are jointly computed to mitigate the discrepancy between training and inference. In the inference phase, the draft model directly takes heih as input.
Training & Inference Input
ChartQA
AI2D
Visual + Text
4.25
4.28
0% Visual + Text
4.23
4.25
50% Visual + Text
4.21
4.24
Table 1: Impact of visual information injection on draft model performance during training.
Method
0.5k
1.5k
17k
25k
τ
DSR
τ
DSR
τ
DSR
τ
DSR
MSD
4.12
2.96x
3.97
2.87x
1.11
0.48x
1.04
0.42x
ViSpec (orig.)
3.45
2.67x
3.35
2.52x
2.96
1.31x
2.81
1.20x
ViSpec
4.29
3.22x
4.18
3.10x
3.74
1.64x
3.42
1.48x
MSD+VATA
4.11
2.98x
4.01
2.92x
3.94
1.72x
4.01
1.68x
Ours
4.51
3.30x
4.36
3.18x
4.32
1.86x
4.37
1.82x
Table 4: Performance of Qwen2.5-VL-7B on VideoDetailCaption across varying visual token lengths, evaluated on A800 GPUs, in terms of average accepted length ( τ ) and decoding speedup ratio (DSR).
Visual Input Length
4.5K
15K
25K
Average Accepted Length
3.07
2.78
2.69
Generated Tokens
418.86
450.53
455.10
Target Model Calls
136.23
161.58
168.69
Wall (End-to-end) Time (s)
8.71
19.28
29.59
Total Decode Time (s)
7.51
13.52
18.13
Prefill Time (s)
1.20
5.76
11.46
Table 5: Performance breakdown of LLaVA-OneVision-7B on VideoDetailedCaption across varying visual input lengths using L20 GPUs.
Method Components
MME
VideoDetailedCaption
MTP
IVSB
VATA
0.4k
0.5k
1.5k
6k
17k
25k
X
X
X
3.40
3.95
3.87
3.29
2.97
2.95
X
X
✓
3.41
3.95
3.85
3.83
3.77
3.86
✓
X
X
3.75
4.44
4.31
3.87
3.58
3.51
✓
X
✓
3.75
4.45
4.34
4.24
4.24
4.29
✓
✓
X
3.82
4.53
4.38
3.08
1.91
1.21
Table 6: We conduct an ablation study on MTP, intermediate-layer visual state bridging(IVSB), and visually-aware text-anchored window attention(VATA) using Qwen2.5-VL-7B deployed on A800 GPUs as the base model on the MME and VideoDetailedCaption benchmarks, reporting the average accepted length across varying visual token lengths.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Draft Model Input
ChartQA
AI2D
Visual + Text
4.25
4.28
50% Visual + Text
4.26
4.30
12.5% Visual + Text
4.25
4.30
0% Visual + Text
4.27
4.31
Appendix
Table 7: Impact of visual information injection on draft model performance during inference.
Figure 5: Layer-wise importance analysis of visual tokens in LLaVA-OneVision-7B. The figure compares task accuracy after removing visual tokens starting from layer x (solid lines) with the native baseline performance (dashed lines).
Figure 6: Layer-wise information retention analysis of visual and text modalities in Qwen2.5-VL-7B. The curves illustrate the retention rates calculated based on similarity, with red denoting visual features and blue denoting text features. The vertical dashed line marks the critical Layer 20.