Recent Vision-Language Models encode high-resolution images into long visual token sequences, incurring prohibitive prefill costs. To compress them, existing methods score each visual token by averaging text-to-visual attention uniformly across all heads, which assumes every head matches the query. However, our empirical analysis shows that misaligned heads dominate the average, amplifying background tokens and drowning out fine-grained cues. To address this, we propose PAQ (Prompt-Grounded Attention Quality), a metric quantifying how well each head aligns the prompt with image regions. Built on PAQ, our pruning proceeds in three stages. Given a target FLOPs budget, we first partition the transformer layers into groups and allocate a visual token budget to each. Within each group, we then aggregate per-head attention maps via PAQ-weighted softmax into a group-level matrix. Finally, we score visual tokens by this matrix's magnitude and retain the allocated budget per group. By weighting heads with PAQ, our method scores tokens by attention signals that more faithfully reflect prompt relevance, rather than diluting them through uniform averaging. Across 18 benchmarks, our method delivers state-of-the-art trade-offs. Specifically, on LLaVA-1.5-7B (9 tasks), retaining only 5.6% tokens preserves 99.1% of the original performance, surpassing the strongest baseline AutoPrune by 4.2 points. Code is available in https://github.com/baokou-fw2/HAP.
Figures & tables
Figure 1: Head-level cross-modal attention patterns. Layer 2 attention maps reveal sink , optimal , and dispersed cross-modal patterns across three VLMs.
Figure 2: (a) MME performance when retaining 25% of visual tokens at different layers across three VLMs. (b) RefCOCO IoU when retaining 50% of visual tokens at different layers. In both panels, solid curves denote average attention mass and dashed curves denote the best single head; average aggregation consistently trails the optimal head.
Figure 3: Overview of HAP. Left: FLOPs-driven budget allocation progressively reduces visual tokens across M layer groups. Right: unified PAQ-softmax aggregation. Head attention maps are softly fused within each layer and then aggregated within each group, yielding a reliable score for visual token pruning.
Method
Venue
SQA
GQA
MMB EN
MMB CN
POPE
Text
MME
VizWiz
MMVet
Rel. Score
FLOPs
Original
-
69.5
61.9
64.6
58.1
86.1
58.2
1864
50.0
30.9
100.0%
3.82T
Retain 128 Tokens (22% Retention)
FastV
ECCV’24
68.6
49.6
56.1
56.4
53.4
50.5
1490
51.3
26.3
86.6%
1.64T
SparseVLM
ICML’25
67.1
56.0
60.0
51.1
80.5
54.9
1696
51.4
29.6
93.9%
1.66T
PDrop
CVPR’25
69.9
56.0
61.1
56.6
82.3
55.1
1664
51.0
30.8
96.0%
1.53T
VisionZip
CVPR’25
68.9
57.6
62.0
53.4
83.2
56.8
1763
51.2
30.2
96.6%
1.44T
Table 1: Evaluation of our method on the LLaVA-1.5-7B model across nine datasets under three visual token compression levels (192, 128, and 64). The vanilla configuration uses 576 tokens and average 3.82T FLOPs. Relative score is the average ratio between the score and original score across all benchmarks.
Model
Method
Venue
MMB EN
MME
POPE
MVBench
AI2D
Seed2
RWQA
Relative Score(%)
FLOPs(T)
InternVL3-8B
original
-
83.4
2415
91.1
75.4
85.2
69.7
71.6
100%
8.15
FastV
ECCV’24
80.4
2235
87.6
72.9
82.2
62.5
62.6
91.5% (↓8.5%)
3.05
PDrop
CVPR’25
82.1
2298
88.2
73.4
83.1
63.8
63.8
94.2% (↓5.8%)
3.02
VisPruner
ICCV’25
81.3
2372
89.4
72.1
82.2
64.9
66.1
95.3% (↓4.7%)
2.98
HAP
-
83.8
2417
90.6
74.2
83.7
69.1
71.9
99.5% ( ↓0.5% )
2.90
Model
Method
Venue
MMB EN
MME
POPE
MMB CN
AI2D
MLVU
RWQA
Relative Score(%)
FLOPs(T)
Table 2: Performance comparison on InternVL3 and Qwen2.5-VL VLMs with 25% visual tokens retained.
Method
Venue
RefCOCO
RefCOCO+
RefCOCOg
Average
val
testA
testB
val
testA
testB
val
test
Qwen3-VL-8B
-
91.58
93.27
86.76
85.80
90.22
79.78
88.76
88.24
100.0%
FastV
ECCV’24
43.57
46.81
40.86
39.47
43.78
36.02
43.04
42.69
47.7%
PDrop
CVPR’25
46.46
53.83
37.23
42.29
47.76
32.81
45.32
44.91
49.8%
HAP
-
76.02
80.65
70.02
68.72
75.62
60.65
70.82
71.23
81.5%
Table 3: Performance comparisons on Qwen-3-VL-8B across 3 grounding benchmarks under 75% pruning ratio.
Method
KV Cache
Perf.
Speedup
(%)
(%)
( × )
FastV
52
92.5
1.54
VisPruner
37
95.8
1.87
HAP (Ours)
34
99.7
1.96
Table 4: Efficiency with InternVL3-8B at 25% tokens.
Figure 4: (a) PAQ vs. heuristic criteria on SQA. (b) Progressive integration of HAP components. (c) RefCOCO IoU of tokens selected by PAQ ranking across four VLMs. (d) Layer-wise retention stability (25% tokens): PAQ aggregation (solid) maintains robust performance across depths, while uniform aggregation (dashed) suffers from severe early-layer degradation caused by sink/dispersed heads.
Figure 5: Qualitative visualization of HAP across LLaVA, Qwen3-VL, InternVL3, and DeepSeek-VL. Columns from left: original image; attention maps before/after PAQ-weighted head aggregation and before/after PAQ-weighted layer aggregation; final pruned token masks; and corresponding QA pairs.
Benchmark
Standard
Detailed
Corrupt
Δ (%)
SQA
100.3
100.7
99.3
1.0
GQA
100.3
100.8
99.2
1.1
POPE
100.3
100.6
99.4
0.9
MME
100.4
100.9
99.3
1.2
Average
100.3
100.8
99.3
1.0
Table 5: Prompt robustness of HAP (LLaVA-1.5-7B, 128 tokens retained). All scores are relative performance (%) against the original prompt. Δ denotes the maximum relative variation across prompt styles.
Retained tokens
576
128
64
32
Performance (%)
100
102
99.7
99.1
KV Cache (%)
100
36
25
20
Latency (s)
0.48
0.22
0.19
0.17
Speedup ( × )
1.00
2.18
2.53
2.82
Table 6: Performance, latency, and KV cache usage analysis.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Scoring
Budget
Rel. Score (%)
PAQ
Pyramid (HAP)
99.1
Uniform
Pyramid
89.4
PAQ
Uniform
96.8
Uniform
Uniform
86.2
Appendix
Table 7: Factorial ablation on LLaVA-1.5-7B (32 tokens retained, 5.6%). Scoring rule and budget schedule are varied independently; relative score (%) is reported.
Model
Method
80%
60%
40%
20%
LLaVA-1.5-7B
HAP (Ours)
101.2 ± 0.3
100.5 ± 0.4
101.0 ± 0.4
102.0 ± 0.5
AutoPrune
99.8 ± 0.4
99.5 ± 0.3
99.0 ± 0.4
98.0 ± 0.5
Qwen2.5-VL-7B
HAP (Ours)
100.0 ± 0.3
100.2 ± 0.3
100.5 ± 0.4
99.0 ± 0.4
IVC-Prune
99.7 ± 0.4
99.3 ± 0.4
98.0 ± 0.5
93.5 ± 0.6
InternVL3-8B
HAP (Ours)
100.0 ± 0.3
100.2 ± 0.3
100.5 ± 0.4
99.5 ± 0.4
VisPruner
99.8 ± 0.4
99.4 ± 0.4
97.5 ± 0.5
93.0 ± 0.7
Appendix
Table 8: Retention-ratio sweep (relative score, %; mean ± std over 10 runs). HAP outperforms the strongest available baseline at every retention level, with small standard deviations confirming stable gains.
Method
Token ↓
SEED
MMB
MMS
RWQA
HallB
AI2D
Rel. Avg.
Vanilla
100%
76.9
79.2
57.7
70.3
43.8
82.0
100%
FastV
54%
75.6
78.2
55.9
69.0
42.7
81.0
98.1%
PDrop
57%
76.8
79.1
57.3
69.7
44.5
81.8
99.9%
IVC-Prune
52%
77.0
79.3
57.7
70.3
44.3
81.8
100.2%
HAP
50%
79.4
81.6
59.4
72.4
45.1
84.4
103.0%
Appendix
Table 9: Performance on DeepSeek-VL2 Small-16B across six benchmarks.
Method
VLM-Rob. Bench
MultiVerse
FLOPs (T)
Speedup ( × )
Original
100
100
6.10
1.00
CDPruner
82.0
84.1
2.60
1.88
VisPruner
93.8
95.9
2.62
1.86
AutoPrune
94.8
93.7
2.60
1.88
HAP (Ours)
97.2
96.8
2.55
1.91
Appendix
Table 10: Robustness under image corruptions (VLM-RobustBench) and multi-turn dialogues (MultiVerse) on Qwen2.5-VL-7B. Scores are relative performance (%) against the unpruned model.
Model
Prefill Saving
PAQ OH
S∗
LLaVA-1.5-7B
0.63T (33.5%)
6.3G ( < 0.5%)
∼ 65K
Qwen2.5-VL-7B
2.05T (33.6%)
20.2G ( < 0.5%)
∼ 217K
Appendix
Table 11: Gain limit analysis under 75% visual token pruning. Prefill saving is the absolute FLOPs reduction (relative share in parentheses); PAQ OH is the PAQ computation overhead; S∗ is the decode length at which the end-to-end FLOPs reduction drops to 1%.
Method
P
D
E2E
BS=1
BS=4
BS=8
PDrop
1.9
1.1
1.4
1.4
1.5
1.6
VisPruner
2.0
1.2
1.6
1.6
1.8
2.0
HAP (Ours)
2.4
1.3
1.9
1.9
2.2
2.4
Appendix
Table 12: Latency breakdown and batch-size scaling on Qwen3-VL-8B (25% tokens retained, RTX 3090). P, D, and E2E denote prefill, decoding, and end-to-end speedup ( × ); the last three columns report E2E speedup at batch sizes 1, 4, and 8.
Vision-Language Models (VLMs) have exhibited impressive performance across diverse visual scenarios. However, this success comes at the cost of explosive growth in visual tokens, which imposes substantial memory and computational overhead during inference, ultimately increasing latency. To improve VLM inference efficiency, a typical class of visual token pruning methods estimates token importance by aggregating attention scores across all heads in the pruning layer of the Large Language Model (LLM) backbone and prunes tokens based on aggregated scores. However, in this paper, we reveal a compelling phenomenon: the capability to pinpoint critical visual tokens is concentrated within a small fraction of heads. Aggregation exclusively on these heads can improve task performance. Inspired by this observation, we propose ProViP, a training-free progressive visual token pruning framework. ProViP first removes redundant visual tokens based on the embedding similarity of input tokens before reasoning of the LLM backbone, and then further prunes tokens during reasoning via head-aware pruning. Experiments demonstrate that ProViP delivers outstanding task performance and inference efficiency. For instance, when applied to LLaVA-1.5-7B, ProViP retains 95.9% of the original performance and achieves 1.62x inference speedup under an 88.9% pruning ratio.
Chaofang Ma, Lin Jiang, Carol Jingyi Li +4
The Hong Kong University of Science and Technology · Northeastern University · The Hong Kong University of Science and Technology (GZ)
In multimodal large language models (MLLMs), the surge of visual tokens significantly increases the inference time and computational overhead, making them impractical for real-time or resource-constrained applications. Visual token pruning is a promising strategy for reducing the cost of MLLM inference by removing redundant visual tokens. Existing research usually assumes that all attention heads contribute equally to the visual interpretation. However, our study reveals that different heads may capture distinct visual semantics and inherently play distinct roles in visual processing. In light of this observation, we propose HAWK, a head importance-aware visual token pruning method that perceives the varying importance of attention heads in visual tasks to maximize the retention of crucial tokens. By leveraging head importance weights and text-guided attention to assess visual token significance, HAWK effectively retains task-relevant visual tokens while removing redundant ones. The proposed HAWK is entirely training-free and can be seamlessly applied to various MLLMs. Extensive experiments on multiple mainstream vision-language benchmarks demonstrate that HAWK achieves state-of-the-art accuracy. When applied to Qwen2.5-VL, HAWK retains 96.0% of the original accuracy after pruning 80.2% of the visual tokens. Additionally, it reduces end-to-end latency to 74.4% of the original and further decreases GPU memory usage across the tested models. The code is available at https://github.com/peppery77/HAWK.git.
Qihui Zhu, Tao Zhang, Yuchen Wang +9
University of Science and Technology of China · ChangXin Memory Technologies, Inc · Huawei Noah’s Ark Lab
While Large Vision-Language Models (LVLMs) demonstrate exceptional multi-modal capabilities, the quadratic computational cost of processing high-resolution visual tokens remains a critical bottleneck. Though recent token reduction strategies attempt to accelerate inference, such methods inadequately exploit attention values and fail to address token redundancy. More critically, they overlook the ``attention shift'' phenomenon inherent in LVLMs, which skews token attention scores. In this work, we propose ASAP, a novel training-free, KV-Cache-compatible pruning recipe that comprehensively addresses these limitations. First, we mitigate the attention shift by utilizing a dynamic bidirectional soft attention mask, ensuring the selection of genuinely informative tokens rather than naive attention-based selection. Second, we posit that high semantic redundancy within the token set degrades performance. We therefore introduce a weighted soft merging component that merges semantically similar tokens, preserving only the most feature-dense visual patches for subsequent layers. ASAP achieves virtually lossless compression of visual context, retaining 99.02% of the original LLaVA-NeXT-7B performance while aggressively slashing computational FLOPs by ~80%.