Organizations: Interdisciplinary Program in AI, Seoul National University · Department of Electrical and Computer Engineering, Seoul National University · Department of Artificial Intelligence, Chung-Ang University
Large Vision-Language Models (VLMs) achieve strong multimodal understanding capabilities by leveraging high-resolution visual inputs, but the resulting large number of visual tokens creates a major computational bottleneck. Recent work mitigates this issue through visual token compression, typically compressing tokens based on saliency, diversity, or a fixed combination of both. We observe that the distribution of semantic prominence varies substantially across samples, leading to different optimal trade-offs between local saliency preservation and global coverage. This observation suggests that applying a static compression strategy across all samples can be suboptimal. Motivated by this insight, we propose PromPrune, a sample-adaptive visual token selection framework composed of semantic prominence-aware budget allocation and a two-stage selection pipeline. Our method adaptively balances local saliency preservation and global coverage according to the semantic prominence distribution of each sample. By allocating token budgets between locally salient regions and globally diverse regions, our method maintains strong performance even under high compression ratios. On LLaVA-NeXT-7B, our approach reduces FLOPs by 88% and prefill latency by 22% while preserving 97.5% of the original accuracy.
Figures & tables
Figure 1: Saliency–Coverage Trade-Off across two distinct benchmarks. (a) Performance trends of POPE and SQA under different saliency–coverage allocations with a fixed token budget. (b) Example samples illustrating the contrasting semantic prominence distributions of the two benchmarks.
Figure 2: Sensitivity of entropy metrics to semantic prominence distribution. Mean normalized entropy values across semantic prominence distribution levels for feature norm entropy (Norm), attention entropy (Attn), and singular value spectral entropy (Spectral).
Figure 3: Overview of the proposed PromPrune framework. Given an input image, the projected visual tokens are first processed by entropy-guided budget allocation to determine the saliency and coverage budgets. A two-stage token selection pipeline then retains locally salient tokens and selects diverse tokens from the unselected token pool to ensure global coverage before passing the compressed tokens to the LLM.
Method
MME
POPE
MMBench
TextVQA
SQA
GQA
VQAv2
VizWiz
MMVet
Rel.
Upper Bound, All 576 Tokens 100.0%
LLaVA-1.5 7B
1507.0
87.7
63.9
58.1
68.6
61.9
78.6
50.1
31.2
100.0%
Retain 128 Tokens ↓77.8%
FastV [ 5 ]
1364.2
67.9
63.1
55.7
69.4
54.1
70.5
51.4
27.8
92.0%
SparseVLM [ 41 ]
1398.9
83.7
61.8
56.8
68.7
56.9
74.9
49.8
30.0
96.5%
VisionZip [ 37 ]
1436.3
69.2
61.5
57.4
68.7
58.0
75.5
51.8
31.5
94.7%
Table 1: Performance comparison with existing visual token compression methods on LLaVA-1.5 7B. Results are reported across nine image-text understanding benchmarks under different visual token budgets. Rel. denotes the relative performance compared to the full-token upper bound.
Table 5
Strategy
MMBench (33)
VQA Text (17)
GQA (13)
SQA (40)
POPE (12)
Rel.
LLaVA-1.5 7B
63.9
58.1
61.9
68.6
87.7
100.0%
Fixed Allocation
58.8
55.4
54.6
68.8
80.7
93.6%
Semantic Prominence-Aware
60.7
55.3
57.2
68.7
85.1
96.1%
Table 4: Effect of semantic prominence-aware budget allocation. We compare the proposed semantic prominence-aware budget allocation with a fixed allocation strategy on LLaVA-1.5 7B under a 64-token budget. The fixed allocation applies the average Tsal of each benchmark to all samples. Values in parentheses indicate the benchmark-wise average Tsal . Rel. denotes the relative performance compared to the full-token upper bound.
Figure 4: Comparison of proxy metrics for estimating the distribution of semantic prominence. We compare spectral entropy with two alternative proxy metrics, feature norm entropy and attention entropy on MME, POPE, GQA benchmarks.
Method
POPE
MME
SQA
MMBench
GQA
Avg.
FPS
84.4
1343.1
68.9
60.7
57.2
54.4
FL
83.1
1377.4
69.1
60.5
58.0
54.2
DPP
85.1
1390.9
68.8
60.7
57.2
54.5
Table 5: Ablation of diversity selection methods in Stage 2. We compare different diversity maximization strategies including Farthest Point Sampling (FPS), Facility Location (FL), and Determinantal Point Process (DPP).
Figure 5: Robustness of Sigmoidal Mapping Hyperparameters. We vary μ and τ in the proposed allocation function and evaluate the resulting performance on SQA, POPE, GQA benchmarks.
Method
Tokens (Budget)
FLOPs (T)
Total Prefill (ms/sample)
KV Cache (MB)
Memory (GB)
POPE (F1)
LLaVA-NeXT-7B
2880
42.6
516.3
1440.0
18.62
87.5
VisPruner
320
5.02
321.1
160.0
16.6
81.6
DivPrune
320
5.02
388.5
160.0
16.3
84.6
PromPrune
320
5.02
402.7
160.0
16.7
85.3
Table 6: Detailed efficiency analysis on LLaVA-NeXT-7B under a fixed token budget. Efficiency metrics are measured during POPE evaluation.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : Saliency–Coverage trade-off across additional benchmarks. We vary the allocation ratio between saliency-driven tokens (stage 1) and coverage-driven tokens (stage 2) while keeping the total token budget fixed.
Figure B1 : Prompt Used for GPT-4o-based Semantic Prominence Annotation. Prompt used for GPT-4o to annotate the semantic prominence distribution of an image. The model assigns a score from 1 (highly distributed) to 5 (highly concentrated) based on the spatial concentration of semantically meaningful visual units.
Figure B2 : Examples from different semantic prominence groups. Representative examples from the five semantic prominence groups annotated using the protocol in Fig. B1, ranging from broadly distributed scenes (top) to highly concentrated visual information (bottom).
Figure C1 : The distribution of spectral entropy for different benchmarks (LLaVA-1.5). Benchmark-wise distributions of normalized spectral entropy (left) and the resulting saliency allocation ratio (right) computed using the CLIP vision encoder of LLaVA-1.5. Different benchmarks exhibit different entropy distributions, which are translated into different average saliency–coverage splits by the same sigmoidal mapping without benchmark-specific tuning.
Figure E1 : Visualization of selected visual tokens ( T=64 ) using LLaVA-1.5-7B. Saliency-only applies stage-1 selection, Coverage-only applies stage-2 selection, and PromPrune adaptively combines both. Red and cyan tokens denote saliency-driven and coverage-driven selections, respectively.
Vision-language models (VLMs) face significant computational inefficiencies caused by excessive generation of visual tokens. While prior work shows that a large fraction of visual tokens are redundant, existing compression methods struggle to balance importance preservation and information diversity. To address this, we propose PruneSID, a training-free Synergistic Importance-Diversity approach featuring a two-stage pipeline: (1) Principal Semantic Components Analysis (PSCA) for clustering tokens into semantically coherent groups, ensuring comprehensive concept coverage, and (2) Intra-group Non-Maximum Suppression (NMS) for pruning redundant tokens while preserving key representative tokens within each group. Additionally, PruneSID incorporates an information-aware dynamic compression ratio mechanism that optimizes token compression rates based on image complexity, enabling more effective average information preservation across diverse scenes. Extensive experiments demonstrate state-of-the-art performance, achieving 96.3% accuracy on LLaVA-1.5 with only 11.1% token retention, and 92.8% accuracy at extreme compression rates (5.6%) on LLaVA-NeXT, outperforming prior methods by 2.5% with 7.8 × faster prefilling speed compared to the original model. Our framework generalizes across diverse VLMs and both image and video modalities, showcasing strong cross-modal versatility. Code is available at https://github.com/ZhengyaoFang/PruneSID.
Zhengyao Fang, Pengyuan Lyu, Chengquan Zhang +3
Harbin Institute of Technology, Shenzhen · Independent Researcher · Peng Cheng Laboratory
Vision-language models (VLMs) rely on long visual token sequences for visual understanding, making the prefill stage expensive in both computation and memory. Most existing pruning methods follow an absolute-ranking paradigm, assigning importance scores to visual tokens and retaining a fixed top-K subset. In this work, we argue that this paradigm is fundamentally brittle: attention sinks distort token importance rankings, while image redundancy and query-dependent visual evidence make fixed token budgets unreliable across inputs. We propose OccamToken, a training-free framework that replaces absolute token ranking with register-anchored relative evidence testing. Instead of asking which tokens are globally important, OccamToken evaluates whether a visual token provides information beyond a register-based reference. Our key insight is that register tokens naturally absorb low-information attention patterns, making them a stable reference for identifying genuinely informative visual evidence. Based on this principle, OccamToken performs both image-adaptive redundancy pruning and query-adaptive relevance pruning through dynamic thresholds derived from register attention. Across LLaVA-NeXT, LLaVA-v1.5, and Qwen3-VL, OccamToken consistently improves the accuracy-efficiency trade-off without additional training. Notably, on LLaVA-NeXT, it reduces 2,880 visual tokens to approximately 40 while preserving over 93% of the original accuracy, enabling stable visual token compression even in the extreme 1.4% retention regime.
While Large Vision-Language Models (LVLMs) demonstrate exceptional multi-modal capabilities, the quadratic computational cost of processing high-resolution visual tokens remains a critical bottleneck. Though recent token reduction strategies attempt to accelerate inference, such methods inadequately exploit attention values and fail to address token redundancy. More critically, they overlook the ``attention shift'' phenomenon inherent in LVLMs, which skews token attention scores. In this work, we propose ASAP, a novel training-free, KV-Cache-compatible pruning recipe that comprehensively addresses these limitations. First, we mitigate the attention shift by utilizing a dynamic bidirectional soft attention mask, ensuring the selection of genuinely informative tokens rather than naive attention-based selection. Second, we posit that high semantic redundancy within the token set degrades performance. We therefore introduce a weighted soft merging component that merges semantically similar tokens, preserving only the most feature-dense visual patches for subsequent layers. ASAP achieves virtually lossless compression of visual context, retaining 99.02% of the original LLaVA-NeXT-7B performance while aggressively slashing computational FLOPs by ~80%.