Large Vision-Language Models (LVLMs) have adopted visual token pruning strategies to mitigate substantial computational overhead incurred by extensive visual token sequences. While prior works primarily focus on either attention-based or diversity-based pruning methods, in-depth analysis of these approaches' characteristics and limitations remains largely unexplored. In this work, we conduct thorough empirical analysis using effective rank (erank) as a measure of feature diversity and attention score entropy to investigate visual token processing mechanisms and analyze the strengths and weaknesses of each approach. Our analysis reveals two insights: (1) Our erank-based quantitative analysis shows that many diversity-oriented pruning methods preserve substantially less feature diversity than intended; moreover, analysis using the CHAIR dataset reveals that the diversity they do retain is closely tied to increased hallucination frequency compared to attention-based pruning. (2) We further observe that attention-based approaches are more effective on simple images where visual evidence is concentrated, while diversity-based methods better handle complex images with distributed features. Building on these empirical insights, we show that incorporating image-aware adjustments into existing hybrid pruning strategies consistently improves their performance. We also provide a minimal instantiation of our empirical findings through a simple adaptive pruning mechanism, which achieves strong and reliable performance across standard benchmarks as well as hallucination-specific evaluations. Our project page available at https://cvsp-lab.github.io/AgilePruner.
Figures & tables
Figure 1: Overview of attention entropy and erank.
PruMerge+
VisionZip
VisPruner
DivPrune
Erank
10.91
14.02
14.35
21.84
Table 1: Mean erank of retained 64 tokens on POPE.
Method
Cs↓
Ci↓
Recall ↑
Len
LLaVA-1.5-7B
51.0
13.9
78.7
101.4
Attention-based methods
FasterVLM (arXiv’24)
45.4
13.5
69.3
94.0
PruMerge+ (ICCV’25)
45.2
15.6
66.7
91.4
Vispruner (ICCV’25)
49.8
15.0
72.6
96.7
Diversity-based methods
Table 2: Comparison on CHAIR. † FPSPruner is based on farthest point sampling (FPS), which iteratively selects the farthest token to guarantee diversity.
Figure 2: Response patterns of DivPrune (diversity-based) vs. FasterVLM (attention-based). DivPrune’s responses are more comprehensive but risk hallucination, whereas FasterVLM produces safer, more focused descriptions. In the annotations, GT Obj. and Hallucinated Obj. label object words; marks DivPrune-specific phrasing; red text indicates incorrect phrases.
Method
Cs↓
Ci↓
Recall ↑
Len
Mean erank
Mean attn.
Retain 64 Tokens
R=0
57.4
18.0
76.4
101.1
21.14
0.0035
R=0.25
50.8
16.8
74.5
97.6
14.98
0.0065
R=0.50
46.2
14.5
73.7
95.5
14.38
0.0072
R=0.75
45.2
14.1
70.5
94.0
13.58
0.0076
Table 3: Effect of attention-based selection ratio R on CHAIR metrics. Higher attention-based selection reduces hallucination ( CS , CI ) but lowers recall.
Figure 3: Diversity vs. attention in pruning across datasets and image complexities. (a) High-erank methods perform better on complex datasets (POPE), while low-erank methods excel on simple datasets (ScienceQA). (b) Simple images show low entropy and erank, leading to concentrated attention suitable for attention-based pruning. Complex images show high entropy and erank, where diversity-based pruning becomes more effective.
Table 4: Attention entropy and erank on simple and complex image datasets. Simple images exhibit lower entropy and erank, while complex images show higher values, and the two pruning methods show contrasting performance between simple and complex images.
Table 8
Figure 4: Effect of similarity threshold τ on token selection. A low ( strict ) τ prioritizes high-attention tokens, while a high ( loose ) τ increases the diversity of the selected tokens.
Method
VQA v2
GQA
VizWiz
SQA IMG
TextVQA
POPE
MME
MMB
MMB CN
Average
Vanila 576 Tokens
LLaVA-1.5-7B
78.5
61.9
50.1
69.5
58.2
85.9
1862
64.7
58.1
100.00%
Retain 128 Tokens
FastV (ECCV’24)
71.0
54.0
51.9
69.2
56.4
68.2
1490
63.0
55.9
92.31%
PDrop (CVPR’25)
74.3
57.1
49.4
70.1
56.7
77.5
1696
62.3
55.3
95.17%
SparseVLM (ICML’25)
75.1
57.3
49.7
69.0
56.3
83.1
1761
62.6
56.9
96.61%
Table 7: Results of different token pruning methods on 9 multimodal benchmarks. Average is normalized to the full-token LLaVA-1.5-7B (set to 100%). MME is reported in its original score units.
Method
Retain 64 Tokens
Retain 128 Tokens
Cs↓
Ci↓
Recall ↑
Len
Cs↓
Ci↓
Recall ↑
Len
LLaVA-1.5-7B
51.0
13.9
78.7
101.4
51.0
13.9
78.7
101.4
Attention-based methods
FasterVLM (arXiv’24)
45.4
13.5
69.3
94.0
45.8
13.3
75.4
97.0
PruMerge+ (ICCV’25)
45.2
15.6
66.7
91.4
46.8
14.4
71.5
95.2
Vispruner (ICCV’25)
49.8
15.0
72.6
96.7
52.8
15.4
77.1
98.7
Table 8: CHAIR evaluation (64 /128 tokens).
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Retain Tokens
FLOPs (T)
Latency (ms/sample)
GPU Memory (GB)
Accuracy
Vanilla (LLaVA-1.5-7B)
576
3.14
172
13.60
58.2
PDrop (CVPR’25)
64
0.51
128
13.30
55.0
SparseVLM (ICML’25)
64
0.52
129
16.26
55.2
DivPrune (CVPR’25)
64
0.48
110
13.30
55.8
VisPruner (ICCV’25)
64
0.48
115
13.30
55.4
Ours
64
0.48
115
13.30
56.0
Appendix
Table 9: Efficiency and accuracy comparison on single RTX 4090 at TextVQA dataset. All models are evaluated under identical settings.
Batch Size
1
3
5
10
erank overhead (ms)
3.65
10.21
16.84
33.40
Appendix
Table 10: erank computation time across different batch sizes on single RTX 4090.
Method
GQA
SQA IMG
POPE
MME
TextVQA
Rel.
All 576 Tokens
LLaVA-1.5-7B
61.9
69.5
85.9
1862
58.2
100.0%
Retain 128 Tokens
Erank-based
59.4
68.6
87.4
1748
57.0
98.13%
Att. entropy-based
59.4
68.6
87.0
1721
56.9
98.06 %
Retain 64 Tokens
Appendix
Table 11: erank-based vs. entropy-based thresholding.
Figure 5: Attention entropy vs. erank on MME dataset.
Figure 6: Extended examples for simple vs. complex images. The same trend as in Fig. 3 is observed: attention-based methods work well on simple images, while diversity-based methods cover complex images more broadly.
Metric
Similarity Threshold ( τ )
0
0.01
0.05
0.1
0.15
0.20
0.25
Erank / Performance
High erank (complex images) datasets
MME (High)
13.85 / 1362
16.72 / 1358
17.44 / 1368
18.58 / 1379
19.55 / 1363
20.13 / 1381
19.60 / 1380
POPE
14.40 / 77.5
17.13 / 77.7
17.81 / 79.1
18.94 / 81.5
19.83 / 83.4
20.43 / 85.3
20.49 / 86.1
Low erank (simple images) datasets
Appendix
Table 12: Comparison of performance across different similarity thresholds. For each metric, the boldface indicates the best performance.
Method
Avg. tokens
GQA
SQA IMG
POPE
MME
MMBench
Rel.
LLaVA-1.5-7B
576
61.9
69.5
85.9
1510
64.3
100.0%
ATP-LLaVA (CVPR’25)
88
56.8
67.2
82.8
1401
64.7
95.1%
Our (fixed count)
88
58.1
68.0
84.2
1405
62.3
95.4 %
Our (Adaptive count)
85.5
58.4
68.2
85.2
1408
63.1
96.0%
Appendix
Table 13: Results of adaptive-count vs. fixed-count pruning
Method
VQA v2
GQA
VizWiz
SQA IMG
TextVQA
POPE
MME
MMB
MMB CN
Average
Vanilla 576 Tokens
LLaVA-1.5-13B
80.0
63.3
53.6
72.8
61.2
86.0
1531
68.5
63.5
100%
Retain 128 Tokens
FastV (ECCV’24)
75.3
58.3
54.6
74.2
58.6
75.5
1460
66.1
62.3
96.0%
PDrop (CVPR’25)
78.2
61.0
53.8
73.3
60.2
83.6
1489
67.5
62.8
98.4%
SparseVLM (ICML’25)
77.6
59.6
51.4
74.3
59.3
85.0
1488
68.4
62.6
97.8%
Appendix
Table 14: Results of different token pruning methods on 9 multimodal benchmarks. Average is normalized to the full-token LLaVA-1.5-13B (set to 100%). MME is reported in its original score units, and it is included only in the Perception section to enable broader comparison with existing methods.
Method
VQA v2
GQA
VizWiz
SQA IMG
TextVQA
POPE
MME
MMB
MMB CN
Average
Vanilla 2880 Tokens (Upper Bound)
LLaVA-NeXT-7B
81.3
62.5
55.2
67.5
60.3
86.8
1512
65.8
57.3
100%
Retain 640 Tokens
FastV (ECCV’24)
77.0
58.9
53.9
67.4
58.1
79.5
1412
63.1
53.5
95.2%
PDrop (CVPR’25)
79.1
60.0
53.8
66.7
57.8
83.8
1475
64.1
55.2
97.0%
SparseVLM (ICML’25)
79.2
61.2
53.6
67.6
59.7
85.3
1456
65.9
58.6
98.8%
Appendix
Table 15: Results of different token pruning methods on 9 multimodal benchmarks. Average is normalized to the full-token LLaVA-NeXT-7B (set to 100%). MME is reported in its original score units, and it is included only in the Perception section to enable broader comparison with existing methods.
Method
TextVQA
ChartQA
AI2D
OCRBench
MME (Perc.)
MME (Cogn.)
MMB-EN
MMB-CN
Rel.
Vanilla 1225 Tokens (100%)
Qwen2.5-VL-7B
82.1
77.5
83.0
84.1
1691
642
83.2
83.2
100.0%
Retain 512 Tokens (60.5% Reduction)
DivPrune(CVPR25)
79.7
72.0
81.9
78.5
1704
620
81.8
81.0
96.8%
Ours
79.8
71.5
82.1
78.5
1700
630
81.3
80.8
96.9%
Retain 256 Tokens (79.1% Reduction)
Appendix
Table 16: Performance comparison on Qwen2.5-VL-7B at different token retention ratios. Rel. is the relative performance normalized to the full-token Qwen2.5-VL-7B (set to 100%). MME scores are reported as separate Perception (Perc.) and Cognition (Cogn.).
Figure 7: Histograms of erank and attention entropy measured on the MME dataset.
Corruption Type
Severity = 1
Severity = 3
Mean erank diff.
Relative change
Mean erank diff.
Relative change
Zoom blur
5.85
6.14%
7.62
8.12%
Brightness
1.48
1.56%
2.52
2.70%
Contrast
2.38
2.54%
4.38
4.62%
Defocus blur
2.06
2.28%
3.74
4.16%
Elastic transform
2.18
2.54%
3.38
4.05%
Appendix
Table 17: Sensitivity of erank under 15 corruption types from COCO-C, evaluated on the MME dataset.
Figure 8: Visualization of the 15 COCO-C corruption types applied to MME images at severity levels 1 and 3.
Figure 9: Qualitative comparison of three pruning strategies—attention-based, diversity-based, and our adaptive method—on counting and spatial reasoning tasks. Diversity-based pruning retains a broader range of spatial cues, often producing detailed descriptions but occasionally introducing hallucinated objects or relationships. Attention-based pruning focuses on salient regions, offering stable but sometimes overly narrow predictions. The adaptive method balances both mechanisms, mitigating hallucinations while preserving essential visual cues for reliable reasoning.
Figure 10: Qualitative comparison on the Existence task. Adaptive pruning correctly preserves tokens needed to identify small objects, outperforming attention- and diversity-based methods in challenging cases.
Figure 11: Examples illustrating how image complexity governs the adaptive pruning method’s success and failure in fine-grained reasoning. For simple, low-erank images, the method performs well when key objects are spatially concentrated (a) but struggles when many similar objects are scattered across the scene (b). For complex, high-erank images, it succeeds when semantic cues are broadly distributed (c) but fails when crucial evidence for fine-grained reasoning is highly localized within a cluttered scene (d).
Figure 28Figure 29
Figure 12: CHAIR qualitative comparisons: FasterVLM vs. DivPrune (Set 1). In the annotations, GT Obj. and Hallucinated Obj. label object words; red text indicates incorrect phrases.
Figure 31Figure 32
Figure 13: CHAIR qualitative comparisons: FasterVLM vs. DivPrune (Set 2). In the annotations, GT Obj. and Hallucinated Obj. label object words; marks DivPrune’s phrasing; marks FasterVLM’s phrasing; red text indicates incorrect phrases.
Figure 14: Effect of varying the Attention-based selection ratio R under a 64-token budget. As R increases, hallucinated objects produced by DivPrune are progressively suppressed, and the responses shift from exploratory to fact-oriented descriptions. In the annotations, GT obj. and Hallucinated obj. label object words; denotes DivPrune-specific phrasing.
Large Vision-Language Models (LVLMs) face significant computational inefficiencies caused by the large number of visual tokens. Existing visual token pruning methods mainly focus on either retaining individually important tokens or selecting mutually diverse ones. In this work, we revisit visual token pruning from a coverage perspective and formulate it as a biased attention coverage maximization problem. The key idea is to select a compact token subset whose encoder-side outgoing attention can jointly cover the image while assigning higher coverage priority to more informative regions. From this perspective, we propose ACPruner, a training-free visual token pruning framework for efficient LVLM inference. ACPruner first estimates token importance by combining intra-modal saliency and inter-modal relevance, then derives token-wise coverage from attention patterns within the vision encoder, and finally performs greedy selection to maximize the proposed coverage objective. Extensive experiments across multiple LVLM backbones, including LLaVA-1.5-7B/13B, LLaVA-NeXT-7B/13B, Qwen2.5-VL-7B, and LLaVA-OneVision-7B, show that ACPruner consistently achieves strong performance retention while delivering substantial end-to-end inference speedups.
Xu Li, Yuxuan Liang, Yi Zheng +6
College of Computer Science and Artificial Intelligence Fudan University, Shanghai, China
Vision-Language Models (VLMs) have recently demonstrated remarkable capabilities in visual understanding and reasoning, but they also impose significant computational burdens due to long visual sequence inputs. Recent works address this issue by pruning unimportant visual tokens, achieving substantial computational reduction while maintaining model performance. The core of token pruning lies in determining token importance, with current approaches primarily relying on attention scores from vision encoders or Large Language Models (LLMs). In this paper, we analyze the effectiveness of attention mechanisms in both vision encoders and LLMs. We find that vision encoders suffer from attention sink, leading to poor focus on informative foreground regions, while in LLMs, although prior studies have identified attention bias toward token positions, text-to-vision attention demonstrates resistance to this bias and enables effective pruning guidance in middle layers. Based on these observations, we propose LearnPruner, a two-stage token pruning framework that first removes redundant vision tokens via a learnable pruning module after the vision encoder, then retains only task-relevant tokens in the LLM's middle layer. Experimental results show that our LearnPruner can preserve approximately 95% of the original performance while using only 5.5% of vision tokens, and achieve 3.2× inference acceleration, demonstrating a superior accuracy-efficiency trade-off.
Large Vision Language Models show impressive performance across image and video understanding tasks, yet their computational cost grows rapidly with the number of visual tokens. Existing token pruning methods mitigate this issue through empirical approaches while overlooking the internal mechanism of attention. In this paper, we propose a novel training free token pruning framework grounded in the dual form perspective of attention. We reformulate attention as an implicit linear layer whose weight matrix is the sum of rank 1 outer products, each generated by a single token's key value pair. Token pruning thus reduces to selecting an optimal subset of these rank 1 updates that best approximates the original dual weight matrix. Extending this perspective to standard softmax attention in LVLMs, we derive a novel metric quantifying both a token's information magnitude and information duplication. To efficiently select the subset with the proposed metric, we introduce Progressive Chunked Maximal Marginal Relevance. Extensive experiments demonstrate that our method achieves a better trade off between performance and efficiency, while providing another perspective on existing pruning approaches.