In multimodal large language models (MLLMs), the surge of visual tokens significantly increases the inference time and computational overhead, making them impractical for real-time or resource-constrained applications. Visual token pruning is a promising strategy for reducing the cost of MLLM inference by removing redundant visual tokens. Existing research usually assumes that all attention heads contribute equally to the visual interpretation. However, our study reveals that different heads may capture distinct visual semantics and inherently play distinct roles in visual processing. In light of this observation, we propose HAWK, a head importance-aware visual token pruning method that perceives the varying importance of attention heads in visual tasks to maximize the retention of crucial tokens. By leveraging head importance weights and text-guided attention to assess visual token significance, HAWK effectively retains task-relevant visual tokens while removing redundant ones. The proposed HAWK is entirely training-free and can be seamlessly applied to various MLLMs. Extensive experiments on multiple mainstream vision-language benchmarks demonstrate that HAWK achieves state-of-the-art accuracy. When applied to Qwen2.5-VL, HAWK retains 96.0% of the original accuracy after pruning 80.2% of the visual tokens. Additionally, it reduces end-to-end latency to 74.4% of the original and further decreases GPU memory usage across the tested models. The code is available at https://github.com/peppery77/HAWK.git.
Figures & tables
Figure 1: Comparison of visual token pruning methods on multiple benchmarks for Qwen2.5-VL-7B, where the proposed approach consistently outperforms all baselines.
Figure 2: Left. The ablation process of the visual attention head. After ablating the attention heads as described above, the model’s visual comprehension ability noticeably declines. Right. Ablation results of visual attention heads. Different heads exhibit varying impacts on visual tasks, and their importance shows consistent trends across multiple datasets.
Figure 3: Overview of HAWK. We first employ a text-guided attention mechanism to assess the relevance of each visual embedding relative to the current textual instructions. Then, we integrate it with head importance weights for token pruning. A training-free method that reduces computation with minimal performance loss.
Method
HallBench
MME
TextVQA
ChartQA
AI2D
RealWorldQA
CCBench
OCRVQA
SQA-IMG
POPE
Rel.
Upper Bound, All 1296 Tokens (100%)
Qwen2.5-VL-7B
46.8
2322.0
85.1
86.3
80.3
66.3
58.0
70.7
72.2
85.3
100.0%
Retain 512 Tokens ( ↓ 60.5% )
DivPrune (CVPR25)
43.5
2258.0
81.8
80.3
78.7
63.8
55.1
69.0
71.0
84.4
96.4%
FastV (ECCV24)
42.4
2318.0
84.1
82.7
78.8
64.7
55.8
67.8
71.5
84.0
97.0%
CDPruner (NeurIPS25)
42.5
2327.0
84.2
82.8
78.9
64.8
55.7
67.1
71.3
82.6
96.8%
Table 1: Performance comparison of different pruning methods on Qwen2.5-VL-7B using fixed resolution input on image benchmarks. Rel. represents the average percentage of performance maintained. Similarity-based method is shown with a red background, attention-based method is with a yellow background, and similarity&attention-based method is with a purple background.
Method
HallBench
MME
TextVQA
ChartQA
AI2D
RealWorldQA
CCBench
OCRVQA
SQA-IMG
POPE
Rel.
Upper Bound,All 1280 Tokens (100%)
InternVL3-8B
49.8
2404.0
79.9
80.4
83.4
68.8
70.2
41.1
96.7
90.7
100.0%
Retain 512 Tokens ( ↓ 60% )
DivPrune (CVPR25)
43.7
2226.0
74.9
69.4
81.3
64.4
68.4
37.7
87.3
90.8
93.0%
CDPruner (NeurIPS25)
37.7
2136.0
55.8
52.9
76.6
59.5
66.9
33.3
84.5
88.6
84.0%
HAWK (Ours)
48.5
2421.0
79.3
76.5
82.6
67.2
69.8
39.5
95.2
90.8
98.2%
Table 2: Performance comparison of different pruning methods on InternVL3-8B using fixed resolution input on image benchmarks. Rel. represents the average percentage of performance maintained. Similarity-based method is shown with a red background, and similarity&attention-based method is with a purple background.
Method
HallBench
MME
TextVQA
ChartQA
AI2D
RealWorldQA
CCBench
OCRVQA
SQA-IMG
POPE
Rel.
Native-Resolution (100%)
Qwen2.5-VL-7B
46.5
2315.0
85.2
86.2
80.7
67.7
59.8
71
72.8
86.5
100.0%
Pruning Ratio ( ↓ 60% )
DivPrune (CVPR25)
45.8
2274.0
82.7
80.6
78.6
64.4
56.4
69.8
71.6
84.6
96.9%
FastV (ECCV24)
42.5
2283.0
84.1
82.5
78.2
64.7
55.9
67.8
71.6
84.1
96.1%
CDPruner (NeurIPS25)
39.3
2248.0
77.8
65.4
78.9
65.8
56.0
67.3
72.0
82.8
92.7%
Table 3: Performance comparison of different pruning methods on Qwen2.5-VL-7B using native resolution input on image benchmarks. Rel. represents the average percentage of performance maintained. Similarity-based method is shown with a red background, attention-based method is with a yellow background, and similarity&attention-based method is with a purple background.
Method
VideoMME
WorldSense
Rel.
Short
Medium
Long
Overall
<1 min
1∼2 min
2∼4 min
4∼6 min
6∼8 min
>8 min
Overall
Native-Resolution (100%)
Qwen2.5-VL-7B
61.9
51.2
46.4
53.2
38.7
34.9
30.7
35.3
31.6
33.6
35.1
100%
Pruning Ratio ( ↓ 60% )
DivPrune (CVPR25)
61.0
49.8
44.6
51.8
37.9
33.6
31.0
33.0
33.8
30.2
34.3
97.5%
FastV (ECCV24)
59.8
51.7
45.8
52.4
37.1
35.3
31.4
34.4
27.2
28.9
34.4
98.3%
Table 4: Performance comparison of different pruning methods on Qwen2.5-VL-7B using native resolution input on video benchmarks. Rel. represents the average percentage of performance maintained. Similarity-based method is shown with a red background, attention-based method is with a yellow background, and similarity&attention-based method is with a purple background.
Ratio 0.6
Ratio 0.8
Method
Score
E2E Latency
KV Cache (MB)
Mem. (GB)
Score
E2E Latency
KV Cache (MB)
Mem. (GB)
Original
2315.0
20m 15s
668
16.9
2315.0
20m 15s
668
16.9
HAWK
2313.0
16m 10s (x1.25)
276
16.1
2311.0
15m 04s (x1.34)
148
15.7
DivPrune
2274.0
18m 46s (x1.08)
276
16.1
2196.0
15m 31s (x1.31)
148
15.7
CDPruner
2248.0
19m 59s (x1.01)
276
16.1
2166.0
16m 33s (x1.22)
148
15.7
Table 5: Comparison results of HAWK and baselines on Qwen2.5-VL-7B across MME benchmark . Original denotes the original model without pruning. The computation of the KV cache is performed on individual samples. The memory refers to the GPU memory consumption during model inference. Since FastV does not support FlashAttention, it is excluded from the comparison.
Figure 4: Ablation study results under different pruning ratios.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Visual Head Ablation Study. Comparison of results across different benchmarks relative to the Base model. The red dashed line indicates the baseline performance (1.0).
Figure 6: Cross-model analysis of visual head ablation.
Method
Ratio
MME
ChartQA
TextVQA
AI2D
RealWorld
Avg. Rel.
Full Model
1.0
2315
86.2
85.2
80.7
67.7
100%
VisionZip
0.6
2308
78.0
75.3
79.9
69.8
96.1%
DART
0.6
2260
64.7
73.8
73.2
66.1
89.5%
HAWK
0.6
2313
83.6
85.0
79.9
67.6
99.1%
VisionZip
0.8
2182
62.1
70.1
77.9
68.2
89.2%
DART
0.8
2135
47.3
67.2
68.3
62.0
80.4%
Appendix
Table 6: Comparison with baselines at varying pruning ratios.
Dataset
Time (hours)
GPU Memory (GB)
MME
0.56
14.30
POPE
0.83
14.21
RealWorldQA
0.39
14.33
Appendix
Table 7: Overhead of head weight calculation on LLaVA-1.5-7B
Benchmark
Task Type
Description
Image-based Benchmarks
POPE [ 20 ]
Object Hallucination
Evaluates object hallucination ratios using random, popular, and adversarial sampling settings.
HallBench [ 12 ]
Visual Hallucination
Focuses on visual illusion and hallucination detection, requiring detailed image context reasoning.
MME [ 10 ]
Comprehensive Evaluation
A comprehensive suite covering 14 subtasks, including perception and cognition capabilities.
ScienceQA-IMG [ 25 ]
Science VQA
Contains multimodal science questions with annotated reasoning explanations and image contexts.
RealWorldQA [ 37 ]
Spatial Reasoning
Evaluates spatial perception and reasoning capabilities in diverse real-world environments.
Appendix
Table 8: Overview of Evaluation Benchmarks. We select ten image-based benchmarks (divided into general/reasoning and text/diagram categories) and two video-based benchmarks to comprehensively evaluate the model’s capabilities.
Figure 7: Qualitative Results: Heatmaps and Response Comparison. We present attention heatmaps to visualize the visual tokens retained by our HAWK, where redder regions indicate higher attention scores , alongside a qualitative comparison of the generated responses against other baseline methods. (Figure continued on the next page…)
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Nanyang Technological University · University of Electronic Science and Technology of China +1
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University · School of Information Engineering, Xiamen Ocean Vocational College · Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, Sino-Russian Research Center for Digital Economy