HAP: Head-Adaptive Visual Token Pruning via Cross-Modal Alignment

Yuanhao Sun; Huawei Ji; Yuan Jin; Cheng Deng; Luoyi Fu; Xinbing Wang

cs.CVAug 24, 2026

HAP: Head-Adaptive Visual Token Pruning via Cross-Modal Alignment

Authors: Yuanhao Sun, Huawei Ji, Yuan Jin, Cheng Deng, Luoyi Fu, Xinbing Wang

Organizations: Shanghai Jiao Tong University, Shanghai, China · University of Edinburgh, Edinburgh, UK

Abstract

Recent Vision-Language Models encode high-resolution images into long visual token sequences, incurring prohibitive prefill costs. To compress them, existing methods score each visual token by averaging text-to-visual attention uniformly across all heads, which assumes every head matches the query. However, our empirical analysis shows that misaligned heads dominate the average, amplifying background tokens and drowning out fine-grained cues. To address this, we propose PAQ (Prompt-Grounded Attention Quality), a metric quantifying how well each head aligns the prompt with image regions. Built on PAQ, our pruning proceeds in three stages. Given a target FLOPs budget, we first partition the transformer layers into groups and allocate a visual token budget to each. Within each group, we then aggregate per-head attention maps via PAQ-weighted softmax into a group-level matrix. Finally, we score visual tokens by this matrix's magnitude and retain the allocated budget per group. By weighting heads with PAQ, our method scores tokens by attention signals that more faithfully reflect prompt relevance, rather than diluting them through uniform averaging. Across 18 benchmarks, our method delivers state-of-the-art trade-offs. Specifically, on LLaVA-1.5-7B (9 tasks), retaining only 5.6% tokens preserves 99.1% of the original performance, surpassing the strongest baseline AutoPrune by 4.2 points. Code is available in https://github.com/baokou-fw2/HAP.

Figures & tables

Figure 1: Head-level cross-modal attention patterns. Layer 2 attention maps reveal sink , optimal , and dispersed cross-modal patterns across three VLMs.

Figure 2: (a) MME performance when retaining 25% of visual tokens at different layers across three VLMs. (b) RefCOCO IoU when retaining 50% of visual tokens at different layers. In both panels, solid curves denote average attention mass and dashed curves denote the best single head; average aggregation consistently trails the optimal head.

Figure 3: Overview of HAP. Left: FLOPs-driven budget allocation progressively reduces visual tokens across $M$ layer groups. Right: unified PAQ-softmax aggregation. Head attention maps are softly fused within each layer and then aggregated within each group, yielding a reliable score for visual token pruning.

Method	Venue	SQA	GQA	MMB ${}^{\text{EN}}$	MMB ${}^{\text{CN}}$	POPE	Text	MME	VizWiz	MMVet	Rel. Score	FLOPs
Original	-	69.5	61.9	64.6	58.1	86.1	58.2	1864	50.0	30.9	100.0%	3.82T
Retain 128 Tokens (22% Retention)
FastV	ECCV’24	68.6	49.6	56.1	56.4	53.4	50.5	1490	51.3	26.3	86.6%	1.64T
SparseVLM	ICML’25	67.1	56.0	60.0	51.1	80.5	54.9	1696	51.4	29.6	93.9%	1.66T
PDrop	CVPR’25	69.9	56.0	61.1	56.6	82.3	55.1	1664	51.0	30.8	96.0%	1.53T
VisionZip	CVPR’25	68.9	57.6	62.0	53.4	83.2	56.8	1763	51.2	30.2	96.6%	1.44T

Table 1: Evaluation of our method on the LLaVA-1.5-7B model across nine datasets under three visual token compression levels (192, 128, and 64). The vanilla configuration uses 576 tokens and average 3.82T FLOPs. Relative score is the average ratio between the score and original score across all benchmarks.

Model	Method	Venue	MMB EN	MME	POPE	MVBench	AI2D	Seed2	RWQA	Relative Score(%)	FLOPs(T)
InternVL3-8B	original	-	83.4	2415	91.1	75.4	85.2	69.7	71.6	100%	8.15
	FastV	ECCV’24	80.4	2235	87.6	72.9	82.2	62.5	62.6	91.5% (↓8.5%)	3.05
	PDrop	CVPR’25	82.1	2298	88.2	73.4	83.1	63.8	63.8	94.2% (↓5.8%)	3.02
	VisPruner	ICCV’25	81.3	2372	89.4	72.1	82.2	64.9	66.1	95.3% (↓4.7%)	2.98
	HAP	-	83.8	2417	90.6	74.2	83.7	69.1	71.9	99.5% ( ↓0.5% )	2.90
Model	Method	Venue	MMB EN	MME	POPE	MMB CN	AI2D	MLVU	RWQA	Relative Score(%)	FLOPs(T)

Table 2: Performance comparison on InternVL3 and Qwen2.5-VL VLMs with 25% visual tokens retained.

Method	Venue	RefCOCO			RefCOCO+			RefCOCOg		Average
Method	Venue	val	testA	testB	val	testA	testB	val	test	Average
Qwen3-VL-8B	-	91.58	93.27	86.76	85.80	90.22	79.78	88.76	88.24	100.0%
FastV	ECCV’24	43.57	46.81	40.86	39.47	43.78	36.02	43.04	42.69	47.7%
PDrop	CVPR’25	46.46	53.83	37.23	42.29	47.76	32.81	45.32	44.91	49.8%
HAP	-	76.02	80.65	70.02	68.72	75.62	60.65	70.82	71.23	81.5%

Table 3: Performance comparisons on Qwen-3-VL-8B across 3 grounding benchmarks under 75% pruning ratio.

Method	KV Cache	Perf.	Speedup
	(%)	(%)	( $\times$ )
FastV	52	92.5	1.54
VisPruner	37	95.8	1.87
HAP (Ours)	34	99.7	1.96

Table 4: Efficiency with InternVL3-8B at 25% tokens.

Figure 4: (a) PAQ vs. heuristic criteria on SQA. (b) Progressive integration of HAP components. (c) RefCOCO IoU of tokens selected by PAQ ranking across four VLMs. (d) Layer-wise retention stability (25% tokens): PAQ aggregation (solid) maintains robust performance across depths, while uniform aggregation (dashed) suffers from severe early-layer degradation caused by sink/dispersed heads.

Figure 5: Qualitative visualization of HAP across LLaVA, Qwen3-VL, InternVL3, and DeepSeek-VL. Columns from left: original image; attention maps before/after PAQ-weighted head aggregation and before/after PAQ-weighted layer aggregation; final pruned token masks; and corresponding QA pairs.

Benchmark	Standard	Detailed	Corrupt	$\mathbf{\Delta}$ (%)
SQA	100.3	100.7	99.3	1.0
GQA	100.3	100.8	99.2	1.1
POPE	100.3	100.6	99.4	0.9
MME	100.4	100.9	99.3	1.2
Average	100.3	100.8	99.3	1.0

Table 5: Prompt robustness of HAP (LLaVA-1.5-7B, 128 tokens retained). All scores are relative performance (%) against the original prompt.

\Delta

denotes the maximum relative variation across prompt styles.

Retained tokens	576	128	64	32
Performance (%)	100	102	99.7	99.1
KV Cache (%)	100	36	25	20
Latency (s)	0.48	0.22	0.19	0.17
Speedup ( $\times$ )	1.00	2.18	2.53	2.82

Table 6: Performance, latency, and KV cache usage analysis.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Scoring	Budget	Rel. Score (%)
PAQ	Pyramid (HAP)	99.1
Uniform	Pyramid	89.4
PAQ	Uniform	96.8
Uniform	Uniform	86.2

Appendix

Table 7: Factorial ablation on LLaVA-1.5-7B (32 tokens retained, 5.6%). Scoring rule and budget schedule are varied independently; relative score (%) is reported.

Model	Method	80%	60%	40%	20%
LLaVA-1.5-7B	HAP (Ours)	101.2 $\pm$ 0.3	100.5 $\pm$ 0.4	101.0 $\pm$ 0.4	102.0 $\pm$ 0.5
LLaVA-1.5-7B	AutoPrune	99.8 $\pm$ 0.4	99.5 $\pm$ 0.3	99.0 $\pm$ 0.4	98.0 $\pm$ 0.5
Qwen2.5-VL-7B	HAP (Ours)	100.0 $\pm$ 0.3	100.2 $\pm$ 0.3	100.5 $\pm$ 0.4	99.0 $\pm$ 0.4
Qwen2.5-VL-7B	IVC-Prune	99.7 $\pm$ 0.4	99.3 $\pm$ 0.4	98.0 $\pm$ 0.5	93.5 $\pm$ 0.6
InternVL3-8B	HAP (Ours)	100.0 $\pm$ 0.3	100.2 $\pm$ 0.3	100.5 $\pm$ 0.4	99.5 $\pm$ 0.4
InternVL3-8B	VisPruner	99.8 $\pm$ 0.4	99.4 $\pm$ 0.4	97.5 $\pm$ 0.5	93.0 $\pm$ 0.7

Appendix

Table 8: Retention-ratio sweep (relative score, %; mean

\pm

std over 10 runs). HAP outperforms the strongest available baseline at every retention level, with small standard deviations confirming stable gains.

Method	Token $\downarrow$	SEED	MMB	MMS	RWQA	HallB	AI2D	Rel. Avg.
Vanilla	100%	76.9	79.2	57.7	70.3	43.8	82.0	100%
FastV	54%	75.6	78.2	55.9	69.0	42.7	81.0	98.1%
PDrop	57%	76.8	79.1	57.3	69.7	44.5	81.8	99.9%
IVC-Prune	52%	77.0	79.3	57.7	70.3	44.3	81.8	100.2%
HAP	50%	79.4	81.6	59.4	72.4	45.1	84.4	103.0%

Appendix

Table 9: Performance on DeepSeek-VL2 Small-16B across six benchmarks.

Method	VLM-Rob. Bench	MultiVerse	FLOPs (T)	Speedup ( $\times$ )
Original	100	100	6.10	1.00
CDPruner	82.0	84.1	2.60	1.88
VisPruner	93.8	95.9	2.62	1.86
AutoPrune	94.8	93.7	2.60	1.88
HAP (Ours)	97.2	96.8	2.55	1.91

Appendix

Table 10: Robustness under image corruptions (VLM-RobustBench) and multi-turn dialogues (MultiVerse) on Qwen2.5-VL-7B. Scores are relative performance (%) against the unpruned model.

Model	Prefill Saving	PAQ OH	$\mathbf{S^{*}}$
LLaVA-1.5-7B	0.63T (33.5%)	6.3G ( $<$ 0.5%)	$\sim$ 65K
Qwen2.5-VL-7B	2.05T (33.6%)	20.2G ( $<$ 0.5%)	$\sim$ 217K

Appendix

Table 11: Gain limit analysis under 75% visual token pruning. Prefill saving is the absolute FLOPs reduction (relative share in parentheses); PAQ OH is the PAQ computation overhead;

S^{*}

is the decode length at which the end-to-end FLOPs reduction drops to 1%.

Method	P	D	E2E	BS=1	BS=4	BS=8
PDrop	1.9	1.1	1.4	1.4	1.5	1.6
VisPruner	2.0	1.2	1.6	1.6	1.8	2.0
HAP (Ours)	2.4	1.3	1.9	1.9	2.2	2.4

Appendix

Table 12: Latency breakdown and batch-size scaling on Qwen3-VL-8B (25% tokens retained, RTX 3090). P, D, and E2E denote prefill, decoding, and end-to-end speedup (

\times

); the last three columns report E2E speedup at batch sizes 1, 4, and 8.

Visual Token Pruning

Visual token pruning reduces the image and video patches processed by transformers to lower inference cost while preserving task-relevant information.

See all 276 papers

Not All Attention Heads Contribute to Critical Visual Token Selection: Head-Aware Pruning Matters More
Aug 26, 2026Chaofang Ma, Lin Jiang, Carol Jingyi Li +4Efficient VLM Inference Visual Token Pruning
The Hong Kong University of Science and Technology · Northeastern University · The Hong Kong University of Science and Technology (GZ)
HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Models
Apr 9, 2026Qihui Zhu, Tao Zhang, Yuchen Wang +9Visual Token Pruning Efficient VLM Inference
University of Science and Technology of China · ChangXin Memory Technologies, Inc · Huawei Noah’s Ark Lab
ASAP: Attention-Shift-Aware Pruning for Efficient LVLM Inference
Mar 15, 2026Surendra Pathak, Bo HanEfficient VLM Inference Visual Token Pruning
George Mason University, Fairfax, VA, USA

Aug 26, 2026J/K move · Enter open · S save

Not All Attention Heads Contribute to Critical Visual Token Selection: Head-Aware Pruning Matters More

Chaofang Ma, Lin Jiang, Carol Jingyi Li, Xingyu Liu+3

The Hong Kong University of Science and Technology · Northeastern University · The Hong Kong University of Science and Technology (GZ)

Efficient VLM Inference Visual Token Pruning Attention Head Specialization

Open paper AskPDF arXiv

Vision-Language Models (VLMs) have exhibited impressive performance across diverse visual scenarios. However, this success comes at the cost of explosive growth in visual tokens, which imposes substantial memory and computational overhead during inference, ultimately increasing latency. To improve VLM inference efficiency, a typical class of visual token pruning methods estimates token importance by aggregating attention scores across all heads in the pruning layer of the Large Language Model (LLM) backbone and prunes tokens based on aggregated scores. However, in this paper, we reveal a compelling phenomenon: the capability to pinpoint critical visual tokens is concentrated within a small fraction of heads. Aggregation exclusively on these heads can improve task performance. Inspired by this observation, we propose ProViP, a training-free progressive visual token pruning framework. ProViP first removes redundant visual tokens based on the embedding similarity of input tokens before reasoning of the LLM backbone, and then further prunes tokens during reasoning via head-aware pruning. Experiments demonstrate that ProViP delivers outstanding task performance and inference efficiency. For instance, when applied to LLaVA-1.5-7B, ProViP retains 95.9% of the original performance and achieves 1.62x inference speedup under an 88.9% pruning ratio.

Abstract

Visual Token Pruning

Explore similar work

Not All Attention Heads Contribute to Critical Visual Token Selection: Head-Aware Pruning Matters More

HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Models

ASAP: Attention-Shift-Aware Pruning for Efficient LVLM Inference