Vision-Language Models (VLMs) have exhibited impressive performance across diverse visual scenarios. However, this success comes at the cost of explosive growth in visual tokens, which imposes substantial memory and computational overhead during inference, ultimately increasing latency. To improve VLM inference efficiency, a typical class of visual token pruning methods estimates token importance by aggregating attention scores across all heads in the pruning layer of the Large Language Model (LLM) backbone and prunes tokens based on aggregated scores. However, in this paper, we reveal a compelling phenomenon: the capability to pinpoint critical visual tokens is concentrated within a small fraction of heads. Aggregation exclusively on these heads can improve task performance. Inspired by this observation, we propose ProViP, a training-free progressive visual token pruning framework. ProViP first removes redundant visual tokens based on the embedding similarity of input tokens before reasoning of the LLM backbone, and then further prunes tokens during reasoning via head-aware pruning. Experiments demonstrate that ProViP delivers outstanding task performance and inference efficiency. For instance, when applied to LLaVA-1.5-7B, ProViP retains 95.9% of the original performance and achieves 1.62x inference speedup under an 88.9% pruning ratio.
Figures & tables
Figure 1: Attention maps of two heads under the textual instruction “What is the word in the red part?” The left head focuses on task-relevant regions, while the right head attends to irrelevant areas. ProViP removes redundant visual tokens via head-aware pruning.
Figure 2: Task performance between All Attention Heads and Visual Heads Only under different pruning layers: (a) Total score of MME; (b) F1 score of POPE.
Figure 3: Overview of ProViP. ProViP consists of the progressive stages: critical textual token selection, pruning before the base LLM, and pruning during the base LLM (including shallow-layer and middle-layer pruning).
Method
GQA
MMB
MMB CN
MME
POPE
SQA
VQA Text
Average
LLaVA-1.5-7B
Original 576 Tokens
Vanilla
61.9
64.7
58.1
1,862
85.9
69.5
58.2
100%
LLaVA-1.5-7B
Retain 192 Tokens ( ↓ 66.7% )
FastV (ECCV24)
52.7
61.2
57.0
1,612
64.8
67.3
52.5
89.5%
LLaVA-PruMerge (ICCV25)
54.3
59.6
52.9
1,632
71.3
67.9
54.3
90.3%
MustDrop (2024.11)
58.2
62.3
55.8
1,787
82.6
69.2
56.5
96.5%
Table 1: Task performance on LLaVA-1.5-7B.
Method
GQA
MMB
MMB CN
MME
POPE
SQA
VQA Text
Average
LLaVA-NEXT-7B
Original Tokens
Vanilla
64.2
67.4
60.6
1,851
86.5
70.1
64.9
100%
LLaVA-NEXT-7B
Retain 320 Tokens
FastV (ECCV24)
55.9
61.6
51.9
1,661
71.7
62.8
55.7
87.4%
LLaVA-PruMerge (ICCV25)
53.6
61.3
55.3
1,534
60.8
66.4
50.6
84.5%
MustDrop (2024.11)
57.3
62.8
55.1
1,641
82.1
68.0
59.9
92.3%
Table 2: Task performance on LLaVA-NEXT-7B.
Method
Latency (ms)
Prefilling Time (ms)
Memory Peak (GB)
FLOPs (T)
LLaVA-1.5-7B
Original 576 Tokens
Vanilla
204
127
19.01
4.24
LLaVA-1.5-7B
Retain 192 Tokens ( ↓ 66.7% )
+SparseVLM
156
69
18.60
1.76
+ProViP
143
65
18.56
1.66
LLaVA-1.5-7B
Retain 128 Tokens ( ↓ 77.8% )
Table 3: Efficiency results on LLaVA-1.5-7B.
Method
GQA
MMB
MME
POPE
Average
Qwen2.5-VL-7B
Original Tokens
Vanilla
60.5
83.3
2,327
86.2
100%
Qwen2.5-VL-7B
Token Pruning Ratio = 80%
SparseVLM (ICML25)
54.7
76.0
2,063
73.6
88.9%
PDrop (CVPR25)
55.1
77.3
2,117
78.4
91.5%
ProViP (Ours)
55.1
76.0
2,092
87.0
93.3%
Table 4: Task performance on Qwen2.5-VL-7B.
Method
GQA
MMB
MMB CN
MME
POPE
SQA
VQA Text
Average
LLaVA-1.5-7B
Original 576 Tokens
Vanilla
61.9
64.7
58.1
1,862
85.9
69.5
58.2
100%
LLaVA-1.5-7B
Tokens Retained after Pruning: 219 (after Shallow), 60 (after Middle)
All Attention Heads
60.3
65.1
59.1
1,842
84.7
68.4
57.9
99.3%
Visual Heads Only
60.6
64.9
59.5
1,838
85.0
68.2
57.6
99.3%
LLaVA-1.5-7B
Tokens Retained after Pruning: 146 (after Shallow), 40 (after Middle)
Table 5: Task performance on LLaVA-1.5-7B with pruning before the LLM backbone disabled under two shallow-layer pruning configurations: Visual Heads Only and All Attention Heads .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Task performance of layer-wise experiments on LLaVA-1.5-7B across MME and POPE under different settings of visual head numbers. Each layer in the LLM backbone is treated as the pruning layer in turn.
Method
GQA
MMB
MMB CN
MME
POPE
SQA
VQA Text
Average
LLaVA-1.5-7B
Original 576 Tokens
Vanilla
61.9
64.7
58.1
1,862
85.9
69.5
58.2
100%
LLaVA-1.5-7B
Retain 192 Tokens ( ↓ 66.7% )
ProViP (Ours)
60.9
64.8
59.1
1,824
85.5
68.7
57.4
99.3%
Only Pruning before the Base LLM
59.4
62.2
54.7
1,803
86.8
69.0
55.9
97.0%
LLaVA-1.5-7B
Retain 128 Tokens ( ↓ 77.8% )
Appendix
Table 6: Task performance on LLaVA-1.5-7B between ProViP and only pruning before the LLM backbone.
Figure 5: Visualization examples on POPE when applying ProViP to LLaVA-1.5-7B.