Organizations: School of Artificial Intelligence, Wuhan University, Wuhan, China · Department of Computer Science, University of Exeter, Exeter, United Kingdom · Chinese Academy of Sciences, Beijing, China; Wuhan AI Research, Wuhan, China · Hangzhou Institute of Technology, Xidian University, Hangzhou, China
Token pruning has emerged as an effective approach to reduce the substantial computational overhead of Large Vision-Language Models (LVLMs) by discarding less informative visual tokens while preserving performance. However, existing methods typically rely on individual attention sources from different LVLM components, resulting in incomplete and suboptimal pruning decisions due to biased attention distributions. To address this problem, we propose DeSAP, a novel Decoupled Similarity-Aware Pruning method for precise, task-aware token pruning within the visual encoder. Specifically, DeSAP introduces a decoupled similarity to capture fine-grained cross-modal relevance between visual features and text tokens, providing explicit task-related guidance for pruning. By integrating decoupled similarity with visual saliency signals derived from visual attention, DeSAP performs token pruning under the guidance of both task-related and visual cues, enabling robust pruning even under aggressive pruning ratios. Extensive experiments across diverse benchmarks and architectures show that DeSAP consistently outperforms SOTA methods in both accuracy and efficiency. On LLaVA-1.5-7B, DeSAP achieves a 10 times FLOPs reduction and a 2.3 times prefill speedup by retaining only 11.1% of visual tokens, while maintaining 98.1% of the original performance.
Figures & tables
Figure 1. Advantages of the proposed DeSAP method for token pruning. (a) Visual comparisons on token pruning area: yellow - salient semantics, red - task-aware semantics, blue - bias. (b) Quantitative comparison with existing SOTA token pruning methods on LLaVA-1.5-7B ( Liu et al., 2024a ) at an 88.9% pruning ratio. DeSAP consistently demonstrates superior accuracy and efficiency.
Figure 2. Attention analysis in LVLMs. (a) [CLS] attention and cross-attention for pruning are biased toward global semantics and text-adjacent regions, respectively. (b) We separate detail-preserving attention from global semantic residuals by decomposing the hidden state. (c) Vanilla similarity is dominated by global semantics, while our decoupled similarity accurately identifies task-relevant local regions.
Figure 3. Decoupled Similarity for Token Pruning. (a) Global bias in visual-centric pruning methods and limitations of two variant strategies. (b) Current task-centric pruning methods fail to leverage textual guidance. (c) Our decoupled similarity better localizes task-related regions and effectively exploits dual information for token pruning. (d) Evaluation results validate the effectiveness of decoupled similarity and its ability to integrate visual cues for dual-source guided token pruning.
Figure 4. Overall Architecture of DeSAP, a novel Decoupled Similarity-Aware Pruning method. DeSAP obtains task guidance through fine-grained alignment between decoupled attention and text tokens, and combines it with salient semantic attention under equal token budgets for dual-source pruning. Discarded tokens are averaged into their most similar retained tokens.
Methods
GQA
MMB
MME
POPE
SQA IMG
VQA V2
VQA Text
VizWiz
Avg.
UpperBound, 576 Tokens(100%)
LLaVA-1.5-7B (CVPR’24)
61.9
64.7
1862
85.9
69.5
78.5
58.2
50.0
100.0%
Retain 64 Tokens on average ( ↓ 88.9%)
LLaVA-PruMerge (ICCV’25)
51.9
55.3
1549
65.3
68.1
67.4
54.0
50.1
88.2%
VisionZip (CVPR’25)
55.1
60.1
1690
77.0
69.0
72.4
55.5
52.8
94.3%
SparseVLM (ICML’25)
52.7
56.2
1505
75.1
62.2
68.2
51.8
50.1
88.2%
Table 1. Performance comparison of LLaVA-1.5-7B ( Liu et al., 2024a ) under different pruning ratios. The best and second-best results are bolded and underlined , respectively, and the last column shows the average accuracy relative to the upper bound.
Methods
GQA
MMB
MME
POPE
SQA IMG
VQA V2
VQA Text
VizWiz
Avg.
Upper Bound, 2880 Tokens (100%)
LLaVA-NeXT-7B (CVPR’24)
64.2
67.4
1851
86.5
70.1
81.8
61.3
57.6
100.0%
Retain 320 Tokens ( ↓ 88.9%)
SparseVLM (ICML’25)
56.1
60.6
1533
82.4
66.1
71.5
58.4
52.0
90.3%
PyramidDrop (CVPR’25)
56.4
63.4
1663
77.6
67.5
73.5
54.4
54.1
91.3%
LLaVA-PruMerge (ICCV’25)
53.6
61.3
1534
60.8
66.4
69.7
50.6
54.0
85.5%
Table 2. Performance comparison of LLaVA-NeXT-7B ( Liu et al., 2024b ) . The best and second-best results are bolded and underlined , respectively, and the last column shows the average accuracy relative to the upper bound.
Figure 5. Performance comparison of various methods on LLaVA-1.5-7B across multiple benchmarks at different pruning ratios.
Figure 6. Visualization on GQA comparing SparseVLM, HoloV, and Ours. The original image and the corresponding pruned results are shown at pruning ratios of 88.9%, 77.8%, and 66.7%. Gray areas indicate discarded tokens, while red regions highlight the primary task-relevant semantic tokens. Compared with other methods, our method well preserves critical tokens.
Methods
GQA
MME
POPE
SQA
VQA Text
Avg.
Upper Bound (100%)
Qwen2.5-VL-7B
60.6
2304
86.1
84.7
84.8
100.0%
Token Pruning Rate = 88.9% ↓
FastV (ECCV’24)
-
1940
78.6
77.4
60.3
84.5%
HoloV (NeurIPS’25)
-
2006
80.7
79.5
61.8
86.9%
DeSAP (Ours)
57.5
2056
83.5
80.3
63.2
90.1%
Table 3. Comparative Experiments on Qwen2.5-VL-7B ( Bai et al., 2025 ) . The best results are bolded .
Table 4. Ablation experiment on LLaVA-1.5-7B ( Liu et al., 2024a ) at a pruning ratio of 88.9%, analyzing (a) Retention rates T and S for task-related and salient semantic attention; (b) Decoupled attention encoding strategy E , with all other settings fixed.
Methods
GQA
MME
POPE
SQA
VQA T
Avg.
Upper Bound, 576 Tokens (100%)
LLaVA-1.5-7B
61.9
1862
85.9
69.5
58.2
100.0%
Retain 64 Tokens ( ↓ 88.9%)
Vanilla-Simi
50.2
1355
72.1
66.5
46.2
81.8%
+ alignment
50.4
1362
72.5
65.7
45.4
81.7%
De-Simi
58.0
1667
85.2
68.5
52.3
94.0%
Table 5. Ablation on fine-grained cross-modal alignment. All variants retain 64 tokens under identical settings.
Methods
Total Time ↓
Prefill Time ↓
FLOPs ↓
Acc. ↑
LLaVA-1.5-7B
42:20 (1.0×)
52 ms (1.0×)
3.82 T
85.9
+SparseVLM
35:38 (1.2×)
43 ms (1.2×)
1.31 T
75.1
+HoloV
29:12 (1.4×)
35 ms (1.5×)
0.42 T
80.3
+Ours
26:35 (1.6×)
22 ms (2.3×)
0.42 T
85.8
Table 6. Efficiency comparisons on the POPE benchmark. We report the practical total running time (min:sec), prefilling time, TFLOPs and the achieved accuracy.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Methods
MSVD-QA
MSRVTT-QA
Average
Acc.
Score
Acc.
Score
Acc.
Score
Video-LLaVA-7B
70.2
3.9
57.3
3.5
63.8
3.7
FastV ECCV’24
71.0
3.9
55.0
3.5
63.0
3.7
HoloV NeurIPS’25
71.0
4.0
56.5
3.6
63.7
3.8
Ours
71.2
4.0
57.0
3.6
64.1
3.8
Appendix
Table 7. Performance comparisons on Video-LLaVA-7B ( Lin et al., 2024 ) across 2 video understanding tasks with a 50% reduction rate.
Figure 7. Qualitative comparison of activation maps from [CLS] attention, cross-attention, vanilla similarity, and our decoupled similarity. While the first three exhibit noticeable biases, our method consistently achieves precise localization aligned with the text-query semantics , demonstrating its effectiveness for encoder-side semantic alignment in token pruning.
Figure 8. Qualitative comparison of SparseVLM ( Zhang et al., 2025b ) , HoloV ( Zou et al., 2025 ) , FlowCut ( Tong et al., 2025 ) , and our proposed method DeSAP on the GQA dataset ( Hudson and Manning, 2019 ) . The figure presents original images alongside their pruned versions at pruning ratios of 88.9%, 77.8%, and 66.7%. Bounding boxes are used to highlight key semantic regions aligned with the text. Our method demonstrates superior preservation of key semantics, especially under high pruning ratios (e.g., 88.9%).
Figure 9. Qualitative comparison of SparseVLM ( Zhang et al., 2025b ) , HoloV ( Zou et al., 2025 ) , FlowCut ( Tong et al., 2025 ) , and our proposed method DeSAP on the POPE dataset ( Li et al., 2023 ) . The figure presents original images alongside their pruned versions at pruning ratios of 88.9%, 77.8%, and 66.7%. Bounding boxes are used to highlight key semantic regions aligned with the text. Our method demonstrates superior preservation of key semantics, especially under high pruning ratios (e.g., 88.9%).