Organizations: State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences School of Computer Science and Technology, University of Chinese Academy of Sciences Beijing, China · School of Computer Science and Technology, University of Chinese Academy of Sciences Beijing, China · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences Beijing Academy of Artificial Intelligence Beijing, China
Visual token compression has emerged as a critical technique for reducing the inference cost of Vision-Language Models. Existing methods select tokens based on text relevance, visual diversity, or a fixed combination of both---treating the relevance-diversity balance as a universal constant independent of the input. We conduct a systematic analysis to challenge this assumption. At the dataset level, grid-searching the optimal relevance-diversity weight per benchmark reveals substantial cross-task variation. More critically, we further maximize per-sample answer likelihood to recover sample-level oracle weights, finding that their distributions not only differ markedly across benchmarks but also exhibit high within-benchmark variance. Based on these observations, we propose SAVER, a framework for Sample-Aware relevance-diversity balanced Visual tokEn Reduction. We first derive a Geometric Intent Prior (GIP) from the spatial concentration of text-relevant tokens to infer a sample-specific relevance-diversity weight, and then apply this weight to jointly select tokens via a Weighted Text Centroid (WTC) for relevance scoring and a Visual Marginal Coverage Gain (VMCG) for diversity. SAVER requires no additional training and achieves sample-aware balancing entirely at inference time. Extensive experiments demonstrate that SAVER consistently outperforms state-of-the-art baselines across diverse benchmarks, retaining 96.5% of full-model performance on LLaVA-1.5-7B with only 11.1% of the original visual tokens, and further generalizing to LLaVA-NeXT and Mini-Gemini.
Figures & tables
Figure 1. Dataset-level performance vs. θ (dashed line) and instance-level oracle weight distributions θi∗ (solid curve and histogram) for POPE, GQA, and MMBench. POPE favors high θ (diversity-dominant); MMBench favors low θ (relevance-dominant); GQA peaks at intermediate values. Per-sample distributions closely correspond to dataset-level optima.
Figure 2. Overview of the SAVER framework. Given a multimodal input, we compute the Geometric Intent Prior (GIP) from the spatial distribution of high-relevance visual tokens, which dynamically weights text relevance (via WTC) and visual diversity (via VMCG) in a unified greedy selection process.
Method
GQA
MMBench
MME
POPE
ScienceQA
TextVQA
OCRBench
Avg.
No compression (576 tokens)
LLaVA-1.5-7B
61.9
64.3
1862
85.9
69.5
58.2
297
100%
192 tokens (66.7% reduction)
FastV (ECCV’24) ( Chen et al., 2025 )
52.7
61.2
1612
64.8
67.3
52.5
291
89.6%
SparseVLM (ICML’25) ( Zhang et al., 2025b )
57.6
62.5
1721
83.6
69.1
56.1
292
96.3%
PyramidDrop (CVPR’25) ( Xing et al., 2024 )
57.1
63.2
1766
82.3
68.8
56.1
290
96.3%
Table 1. Comparison with state-of-the-art token reduction methods on LLaVA-1.5-7B across three compression budgets.
Method
#Tokens
GQA
MMBench
POPE
MME
ScienceQA
SEEDBench
Avg.
LLaVA-NeXT (Full)
2880
64.2
67.9
86.4
1842
70.2
70.2
100%
VisionZip (CVPR’25) ( Yang et al., 2025 )
640
61.3
66.3
86.3
1787
68.1
66.7
97.0%
PruneSID (ICLR’26) ( Fang et al., 2026 )
640
61.6
64.2
86.3
1795
68.3
67.3
96.8%
SAVER (Ours)
640
63.6
65.8
86.4
1802
69.8
68.5
98.5%
VisionZip (CVPR’25) ( Yang et al., 2025 )
320
59.3
63.1
82.1
1702
67.3
63.4
93.2%
PruneSID (ICLR’26) ( Fang et al., 2026 )
320
60.5
63.0
83.1
1754
67.3
65.0
94.5%
Table 2. Generalization results on LLaVA-NeXT.
Method
#Tokens
GQA
MMBench
POPE
MME
ScienceQA
SEEDBench
Avg.
Mini-Gemini (Full)
576
62.4
69.3
85.8
1841
70.7
69.7
100%
VisionZip (CVPR’25) ( Yang et al., 2025 )
128
58.7
68.1
78.5
1841
70.0
65.6
96.2%
PruneSID (ICLR’26) ( Fang et al., 2026 )
128
60.1
66.6
82.4
1821
70.7
66.5
97.1%
SAVER (Ours)
128
60.6
67.2
84.1
1831
71.0
67.2
98.1%
VisionZip (CVPR’25) ( Yang et al., 2025 )
64
55.8
65.9
69.6
1737
70.7
61.7
91.4%
PruneSID (ICLR’26) ( Fang et al., 2026 )
64
58.3
63.1
76.0
1735
70.6
63.6
93.1%
Table 3. Generalization results on Mini-Gemini.
Method
#Tok.
Peak GPU ↓
Throughput ↑
POPE ↑
LLaVA-1.5-7B (Full)
576
14.69
11.21
85.9
SparseVLM ( Zhang et al., 2025b )
64
18.84
12.09
75.1
DART ( Wen et al., 2025 )
64
14.33
17.17
73.9
SAVER (Ours)
64
14.33
17.11
83.8
Table 4. Efficiency comparison on LLaVA-1.5-7B.
Method
POPE
GQA
MMBench
Avg.
WTC relevance
73.7
56.3
61.9
91.0%
VMCG diversity
83.4
58.7
59.9
95.0%
WTC + VMCG(static)
84.0
59.5
60.4
95.9%
SAVER (Ours)
83.8
60.0
61.8
96.9%
Table 5. Ablation study on LLaVA-1.5-7B.
Figure 3. Ablation on γ across different token budgets on LLaVA-NeXT-7B (POPE F1). The effect is amplified under higher compression ratios.
Visual token pruning, which aims to compress and prune redundant visual tokens, plays a critical role in efficient inference with large vision-language models (LVLMs). However, existing methods fail to disentangle intra-modal visual redundancy from cross-modal redundancy between vision and language. We show that visual token diversity and task-specific token relevance are two crucial yet orthogonal factors that complement each other in conveying useful information and should therefore be treated separately for more effective visual token pruning. Building upon this insight, we design TODRE, a two-stage and training-free framework that incorporates Token Diversity and task RElevance for effective token compression and efficient LVLM inference. Instead of pruning redundant tokens, we introduce a greedy max-sum diversification algorithm that selects and retains a subset of diverse and representative visual tokens after the vision encoder. On top of that, ToDRE leverages an ``information migration'' mechanism to eliminate task-irrelevant visual tokens within certain decoder layers of the large language model (LLM), further improving token pruning and LVLM inference. Extensive experiments show that ToDRE prunes 90% of visual tokens after the vision encoder as well as all visual tokens in certain LLM decoder layers, leading to a 2.6x speed-up in total inference time while maintaining 95.0% model performance plus excellent model compatibility. The code is available at: this https URL.
Duo Li, Zuhao Yang, Xiaoqin Zhang +2
CCDS, NTU, Singapore · CCST, ZJUT, China · Terminus AI Lab, UCAS, China
Despite their remarkable performance, Vision Language Models (VLMs) incur substantial computational overhead due to the large number of visual tokens. While diversity maximization has become a dominant strategy for token reduction, existing methods rely on cosine-based normalized similarity that discards magnitude information, failing to faithfully approximate the original feature representation and leading to suboptimal performance, particularly on compositional multi-skill reasoning tasks. In this paper, we introduce SPARE, a subspace reconstruction method that reformulates token pruning as a column subset selection problem and explicitly minimizes reconstruction error. By iteratively selecting tokens with large projection residuals, SPARE performs reconstruction-driven pruning beyond angular diversity. Moreover, we reveal a counterintuitive anti-relevance phenomenon: tokens with lower image-text relevance score can better preserve contextual information. Based on this finding, we incorporate anti-relevance into SPARE as an additional selection criterion to promote context-aware token selection. Extensive experiments across multiple VLMs and benchmarks demonstrate that SPARE consistently achieves state-of-the-art performance, with strong gains on compositional tasks. When applied to LLaVA, SPARE removes up to 94% of visual tokens while retaining 95% of the baseline performance, all in a fully training-free manner.
Vision-language models (VLMs) face significant computational inefficiencies caused by excessive generation of visual tokens. While prior work shows that a large fraction of visual tokens are redundant, existing compression methods struggle to balance importance preservation and information diversity. To address this, we propose PruneSID, a training-free Synergistic Importance-Diversity approach featuring a two-stage pipeline: (1) Principal Semantic Components Analysis (PSCA) for clustering tokens into semantically coherent groups, ensuring comprehensive concept coverage, and (2) Intra-group Non-Maximum Suppression (NMS) for pruning redundant tokens while preserving key representative tokens within each group. Additionally, PruneSID incorporates an information-aware dynamic compression ratio mechanism that optimizes token compression rates based on image complexity, enabling more effective average information preservation across diverse scenes. Extensive experiments demonstrate state-of-the-art performance, achieving 96.3% accuracy on LLaVA-1.5 with only 11.1% token retention, and 92.8% accuracy at extreme compression rates (5.6%) on LLaVA-NeXT, outperforming prior methods by 2.5% with 7.8 × faster prefilling speed compared to the original model. Our framework generalizes across diverse VLMs and both image and video modalities, showcasing strong cross-modal versatility. Code is available at https://github.com/ZhengyaoFang/PruneSID.
Zhengyao Fang, Pengyuan Lyu, Chengquan Zhang +3
Harbin Institute of Technology, Shenzhen · Independent Researcher · Peng Cheng Laboratory