Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity

Zhengyao Fang; Pengyuan Lyu; Chengquan Zhang; Guangming Lu; Jun Yu; Wenjie Pei

cs.CVMar 10, 2026

Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity

Authors: Zhengyao Fang, Pengyuan Lyu, Chengquan Zhang, Guangming Lu, Jun Yu, Wenjie Pei

Organizations: Harbin Institute of Technology, Shenzhen · Independent Researcher · Peng Cheng Laboratory

Abstract

Vision-language models (VLMs) face significant computational inefficiencies caused by excessive generation of visual tokens. While prior work shows that a large fraction of visual tokens are redundant, existing compression methods struggle to balance importance preservation and information diversity. To address this, we propose PruneSID, a training-free Synergistic Importance-Diversity approach featuring a two-stage pipeline: (1) Principal Semantic Components Analysis (PSCA) for clustering tokens into semantically coherent groups, ensuring comprehensive concept coverage, and (2) Intra-group Non-Maximum Suppression (NMS) for pruning redundant tokens while preserving key representative tokens within each group. Additionally, PruneSID incorporates an information-aware dynamic compression ratio mechanism that optimizes token compression rates based on image complexity, enabling more effective average information preservation across diverse scenes. Extensive experiments demonstrate state-of-the-art performance, achieving 96.3% accuracy on LLaVA-1.5 with only 11.1% token retention, and 92.8% accuracy at extreme compression rates (5.6%) on LLaVA-NeXT, outperforming prior methods by 2.5% with 7.8 $\times$ faster prefilling speed compared to the original model. Our framework generalizes across diverse VLMs and both image and video modalities, showcasing strong cross-modal versatility. Code is available at https://github.com/ZhengyaoFang/PruneSID.

Figures & tables

Figure 1: Comparison of visual token reduction paradigms in VLMs. (a) Original input image. (b) Attention-guided methods preserve high-attention tokens but discard contextual background. (c) Duplication-aware methods remove redundant tokens via similarity pruning, yet may discard semantically important regions with high attention. (d) Our proposed semantically group-guided method balances semantic importance and information diversity.

Figure 2: Performance comparison of token reduction methods across multiple vision-language benchmarks on LLaVA-1.5.

Figure 3: Overview of our two-stage compression framework. PSCA first clusters visual tokens into semantically coherent groups via low-rank PCA decomposition. Then, intra-group NMS removes redundant tokens within each group using adaptive similarity thresholds $\tau$ , retaining the most informative representatives.

Method	Early Cmp.	GQA	MMB	MME	POPE	SQA	VQA ${}^{\text{v2}}$	VQA ${}^{\text{Text}}$	MMMU	SEED	VizWiz	LLaVa-B	Avg.
Upper Bound, 576 Tokens (100%)
Vanilla ${}_{\text{CVPR24}}$	$-$	61.9	64.7	1862	85.9	69.5	78.5	58.2	36.3	60.5	54.3	66.8	100%
Retain 192 Tokens ( $\downarrow$ 66.7%)
FastV ${}_{\text{ECCV24}}$	$\times$	52.7	61.2	1612	64.8	67.3	67.1	52.5	34.3	57.1	50.8	49.4	88.2%
SparseVLM ${}_{\text{24.10}}$	$\times$	57.6	62.5	1721	83.6	69.1	75.6	56.1	33.8	55.8	50.5	66.1	95.3%
MustDrop ${}_{\text{24.11}}$	$\times$	58.2	62.3	1787	82.6	69.2	76.0	56.5	$-$	$-$	51.4	$-$	96.3%

Table 1: Performance of PruneSID on LLaVA-1.5. Vanilla refers to the uncompressed baseline model using all 576 visual tokens, serving as the upper performance bound. Early Cmp. indicates whether the compression is applied prior to the LLM for improved efficiency. PruneSID -Dyn denotes the variant of our method augmented with the Dynamic Compression Ratio mechanism.

Table 5

Method	TGIF	MSVD	MSRVTT	A-Net	Avg.
Video-LLaVA	47.1	70.7	59.2	43.1	100%
FastV	23.1	38.0	19.3	30.6	52.1%
SparseVLM	44.7	68.2	31.0	42.6	86.5%
VisionZip	42.4	63.5	52.1	43.0	93.2%
PruneSID	45.8	67.1	53.3	43.1	95.5%

Table 4: Performance on Video-LLaVA.

Method	Retain 64				Retain 128				Retain 192				Avg.
Method	GQA	MME	POPE	SQA	GQA	MME	POPE	SQA	GQA	MME	POPE	SQA	Avg.
random	56.2	1707	79.6	67.5	57.8	1723	84.0	66.0	59.4	1743	85.6	67.5	94.8%
kmeans	56.5	1630	82.8	67.8	58.7	1714	86.3	67.8	60.0	1745	86.8	68.0	95.6%
PruneSID	57.1	1733	83.8	67.8	58.8	1749	86.5	68.3	60.1	1793	86.9	68.5	96.8%

Table 5: Ablation study of group method on LLaVA-1.5.

K	Retain 64					Retain 128					Retain 192
K	GQA	MME	POPE	SQA	Avg.	GQA	MME	POPE	SQA	Avg.	GQA	MME	POPE	SQA	Avg.
8	56.2	1684	81.4	67.1	93.1%	58.2	1720	85.3	68.3	96.0%	59.2	1763	83.0	67.9	96.2%
16	57.1	1733	83.8	67.8	95.1%	58.6	1692	86.4	68.2	96.1%	59.2	1755	85.4	68.3	96.9%
32	57.1	1700	83.7	68.0	94.7%	58.8	1749	86.5	68.3	97.0%	60.1	1763	86.8	68.4	97.9%
48	56.9	1668	83.7	67.6	94.1%	58.7	1720	85.8	68.3	96.3%	60.1	1793	86.9	68.5	98.3%
60	56.7	1657	83.6	67.6	93.8%	58.6	1715	85.7	68.0	96.1%	60.0	1774	86.9	68.5	98.0%

Table 6: Ablation study of group

K

on LLaVA-1.5.

Figure 4: Ablation study on ViT layer features for PSCA Grouping.

Methods	LLaVA-1.5						LLaVA-NeXT						Mini-Gemini
Methods	MME	SQA	MMMU	POPE	Avg.	$\Delta$	MME	SQA	MMMU	POPE	Avg.	$\Delta$	MME	SQA	MMMU	POPE	Avg.	$\Delta$
baseline	1862	69.5	36.3	85.9	100%		1842	70.2	35.1	86.4	100%		1841	70.7	36.1	85.8	100%
	Retain 192 Tokens ( $\downarrow$ 66.7%)						Retain 128 Tokens ( $\downarrow$ 77.8%)						Retain 192 Tokens ( $\downarrow$ 66.7%)
PruneSID	1791	68.5	36.1	86.9	98.8%		1795	68.3	37.9	86.3	100.7%		1842	71.1	36.1	84.4	99.7%
PruneSID -Dyn	1797	69.1	36.8	87.1	99.7%	+0.9	1798	68.9	38.2	86.5	101.2%	+0.5	1845	71.3	36.9	84.6	100.5%	+0.8
	Retain 128 Tokens ( $\downarrow$ 77.8%)						Retain 64 Tokens ( $\downarrow$ 88.9%)						Retain 128 Tokens ( $\downarrow$ 77.8%)

Table 7: Ablation study of dynamic compression ratio mechanism.

Figure 5: Histogram of Information Score distributions for the MMMU and GQA benchmarks. A higher Information Score indicates greater visual information content.

Method	Token	Inference	Prefilling	POPE
Method	Token	Time $\downarrow$	Time $\downarrow$	(F 1 ) $\uparrow$
LLaVA-NeXT	2880	254ms	218ms	86.4
FastV	160	199ms	119ms	50.5%
SparseVLM	160	211ms	128ms	80.2%
VisionZip	160	84ms	27.8ms	86.6%
PruneSID	160	89ms	27.8ms	89.0%

Table 8: Efficiency analysis and comparison. Inference and prefilling times represent the average per-sample latency.

Figure 6: Visualization of a limitation of our method. Left: input image and the corresponding question. Right: tokens retained by our method and the VLM’s response after compression. In fine-grained scenarios, our method may discard important local details due to redundancy in the relevant region, leading to incomplete answers.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Runtime comparison of different methods. Vanilla denotes the average runtime of the uncompressed LLaVA-NeXT 7B model on the POPE dataset and the upper performance bound. VisionZip and PruneSID represent the runtimes after applying the corresponding compression methods. While maintaining comparable efficiency to VisionZip, our method achieves better performance.

Method	GQA	MMB	MME	POPE	SQA	VQA ${}^{\text{v2}}$	VQA ${}^{\text{Text}}$	MMMU	SEED ${}^{\text{I}}$	VizWiz	Avg.
Upper Bound, 576 Tokens (100%)
Vanilla ${}_{\text{CVPR24}}$	63.2	67.7	1818	85.9	72.8	80.0	61.3	36.4	66.9	56.6	100%
Retain 192 Tokens ( $\downarrow$ 66.7%)
VisionZip	59.6	65.9	1770	86.4	72.8	78.0	58.6	36.9	65.2	54.9	97.8%
PruneSID	59.6	65.9	1770	86.4	72.8	78.0	58.6	36.9	65.2	56.0	98.0%
Retain 128 Tokens ( $\downarrow$ 77.8%)

Appendix

Table 9: Performance of PruneSID on LLaVA-1.5 13B. Vanilla refers to the uncompressed baseline model using all 576 visual tokens, serving as the upper performance bound.

Method	GQA	MMB	MME	POPE	SQA	VQA ${}^{\text{v2}}$	VQA ${}^{\text{Text}}$	MMMU	SEED ${}^{\text{I}}$	VizWiz	Avg.
Upper Bound, 2880 Tokens (100%)
Vanilla ${}_{\text{CVPR24}}$	65.4	70.0	1858	86.2	73.5	81.8	64.3	36.2	71.9	64.0	100%
Retain 640 Tokens ( $\downarrow$ 77.8%)
PruneSID	62.4	67.0	1817	85.6	70.1	79.1	60.2	36.2	68.5	60.2	96.7%
Retain 320 Tokens ( $\downarrow$ 88.9%)
PruneSID	61.5	65.4	1810	82.7	70.3	76.9	58.5	36.8	66.8	58.5	95.2%

Appendix

Table 10: Performance of PruneSID on LLaVA-NeXT 13B.

Method	GQA	MMB	MME	POPE	SQA	VQA ${}^{\text{v2}}$	MMMU	SEED ${}^{\text{I}}$	VizWiz	Avg.
Upper Bound (100%)
baseline	62.3	79.1	2318	87.9	84.7	81.2	51.1	76.5	68.2	100%
Token retention ratio 33.3% ( $\downarrow$ 66.7%)
PACT ${}_{\text{CVPR25}}$	59.1	75.8	2068	85.6	80.5	78.2	48.2	74.0	67.9	95.3%
PruneSID	60.5	72.1	2146	87.0	81.8	79.8	48.3	75.5	66.1	96.1%
Token retention ratio 22.2% ( $\downarrow$ 77.8%)

Appendix

Table 11: Performance of PruneSID on Qwen2-VL 7B. baseline refers to the uncompressed model using all the visual tokens, serving as the upper performance bound.

Method	GQA	MMB	MME	POPE	SQA	VQA ${}^{\text{v2}}$	HRB ${}^{\text{8k}}$	XLRS ${}^{\text{macro}}$	Avg.
Upper Bound (100%)
baseline	60.9	83.9	2310	86.3	88.9	82.9	68.1	47.1	100%
Token retention ratio 33.3% ( $\downarrow$ 66.7%)
VisionZip ${}_{\text{CVPR25}}$	56.6	78.9	2317	85.8	80.5	80.7	61.8	46.0	95.4%
PruneSID	59.8	80.9	2218	85.9	87.6	80.4	62.4	46.3	97.0%
Token retention ratio 22.2% ( $\downarrow$ 77.8%)

Appendix

Table 12: Performance of PruneSID on Qwen2.5-VL 7B. baseline refers to the uncompressed model using all the visual tokens, serving as the upper performance bound.

Upper Bound, 576 Tokens (100%)
Condition	GQA	MME	POPE	SQA	MMMU	SEED	MMB	Vizwiz	Avg.
Vanilla ${}_{\text{CVPR24}}$	61.9	1862	85.9	69.5	36.3	60.5	64.7	54.3	100%
Retain 192 Tokens ( $\downarrow$ 66.7%)
w/o PSCA	58.8	1743	84.9	68.2	35.7	58.8	62.7	54.1	97.2%
w/o NMS	59.1	1755	85.8	68.1	35.2	58.6	62.3	54.9	97.3%
Ours	60.2	1797	87.1	69.1	36.8	59.0	63.8	55.5	99.3%

Appendix

Table 13: Performance comparison of ablation variants on LLaVA-1.5 7B (relative to vanilla baseline, 100%).

Upper Bound, 576 Tokens (100%)
Condition	GQA	MME	POPE	SQA	MMMU	SEED	MMB	Vizwiz	Avg (%)
Vanilla ${}_{\text{CVPR24}}$	61.9	1862	85.9	69.5	36.3	60.5	64.7	54.3	100%
Retain 192 Tokens ( $\downarrow$ 66.7%)
Descend	57.3	1667	83.9	66.8	35.8	56.4	58.8	54.8	96.1%
Ascend	59.1	1763	85.8	68.1	36.2	58.6	63.3	54.9	97.9%
Ours	60.1	1791	86.9	68.5	36.1	59.0	63.7	55.4	98.8%

Appendix

Table 14: Performance comparison of importance and diversity preservation ablation variants on LLaVA-1.5 7B (relative to vanilla baseline, 100%).

$\alpha$	Retain 64					Retain 128					Retain 192
$\alpha$	GQA	MME	POPE	SQA	Avg.	GQA	MME	POPE	SQA	Avg.	GQA	MME	POPE	SQA	Avg.
24	56.7	1726	83.4	68.0	94.8%	58.4	1718	85.5	68.3	96.1%	59.7	1770	86.3	68.4	97.6%
28	57.0	1718	84.1	67.8	94.9%	58.8	1723	86.4	68.2	96.6%	59.9	1783	86.8	68.4	98.0%
32	57.1	1733	83.8	67.8	95.1%	58.8	1749	86.5	68.3	97.0%	60.1	1791	86.9	68.5	98.3%
36	57.3	1705	84.7	67.9	95.1%	58.7	1733	85.9	68.3	96.5%	59.7	1789	86.6	68.5	98.0%
40	57.3	1675	84.7	67.9	94.7%	58.6	1711	85.7	68.1	96.2%	59.6	1772	86.3	68.4	97.6%

Appendix

Table 15: Performance sensitivity to NMS threshold scaling factor

\lambda

on LLaVA-1.5 7B (relative to vanilla baseline, 100%).

Panel of Table 16 — Table 16: Token distribution across PSCA groups (Group 0–15) for different $\lambda$ (retain 64 tokens, MME benchmark).

Figure 8: Visualization of PSCA-based semantic token grouping and final retained tokens under an extreme compression ratio (retaining 32 tokens, 5.6%). Semantically Coherent Groups show partial grouping results from our PSCA stage, and Retained compares the selected tokens from our method and VisionZip. Our approach preserves representative tokens from semantically redundant groups, enabling broader information coverage than attention-only methods like VisionZip.

Figure 9: Visualization of PSCA-based token grouping and final retained tokens when retaining 64 tokens (11.1%). Similar to Fig. 8 , our method effectively selects representative tokens across diverse groups while filtering redundancy. This helps maintain semantic diversity under tight token budgets, outperforming redundancy-prone baselines like VisionZip.

Explore similar work

OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning
May 28, 2026Geng Li, Guohao Chen, Ting Chen +6Efficient VLM Inference Large Vision-Language Models
1Nanyang Technological University (NTU)
Balancing Saliency and Coverage: Semantic Prominence-Aware Budgeting for Visual Token Compression in VLMs
Mar 16, 2026Jaehoon Lee, Mingi Jung, Soohyuk Jang +3Efficient VLM Inference Visual Token Pruning
Interdisciplinary Program in AI, Seoul National University · Department of Electrical and Computer Engineering, Seoul National University · Department of Artificial Intelligence, Chung-Ang University
DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference
Feb 21, 2026Aditya Kumar Singh, Hitesh Kandala, Pratik Prabhanjan Brahma +2Efficient VLM Inference Multimodal Token Compression
Advanced Micro Devices, Inc. (AMD)

May 28, 2026J/K move · Enter open · S save

OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning

Geng Li, Guohao Chen, Ting Chen, Shilin Shan+5

1Nanyang Technological University (NTU)

Efficient VLM Inference Large Vision-Language Models Visual Token Pruning Visual Token Compression

Open paper AskPDF arXiv

Vision-language models (VLMs) rely on long visual token sequences for visual understanding, making the prefill stage expensive in both computation and memory. Most existing pruning methods follow an absolute-ranking paradigm, assigning importance scores to visual tokens and retaining a fixed top-K subset. In this work, we argue that this paradigm is fundamentally brittle: attention sinks distort token importance rankings, while image redundancy and query-dependent visual evidence make fixed token budgets unreliable across inputs. We propose OccamToken, a training-free framework that replaces absolute token ranking with register-anchored relative evidence testing. Instead of asking which tokens are globally important, OccamToken evaluates whether a visual token provides information beyond a register-based reference. Our key insight is that register tokens naturally absorb low-information attention patterns, making them a stable reference for identifying genuinely informative visual evidence. Based on this principle, OccamToken performs both image-adaptive redundancy pruning and query-adaptive relevance pruning through dynamic thresholds derived from register attention. Across LLaVA-NeXT, LLaVA-v1.5, and Qwen3-VL, OccamToken consistently improves the accuracy-efficiency trade-off without additional training. Notably, on LLaVA-NeXT, it reduces 2,880 visual tokens to approximately 40 while preserving over 93% of the original accuracy, enabling stable visual token compression even in the extreme 1.4% retention regime.

Abstract

Explore similar work

OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning

Balancing Saliency and Coverage: Semantic Prominence-Aware Budgeting for Visual Token Compression in VLMs

DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference