Organizations: Interdisciplinary Program in AI, Seoul National University · Department of Electrical and Computer Engineering, Seoul National University · Department of Artificial Intelligence, Chung-Ang University
Large Vision-Language Models (VLMs) achieve strong multimodal understanding capabilities by leveraging high-resolution visual inputs, but the resulting large number of visual tokens creates a major computational bottleneck. Recent work mitigates this issue through visual token compression, typically compressing tokens based on saliency, diversity, or a fixed combination of both. We observe that the distribution of semantic prominence varies substantially across samples, leading to different optimal trade-offs between local saliency preservation and global coverage. This observation suggests that applying a static compression strategy across all samples can be suboptimal. Motivated by this insight, we propose PromPrune, a sample-adaptive visual token selection framework composed of semantic prominence-aware budget allocation and a two-stage selection pipeline. Our method adaptively balances local saliency preservation and global coverage according to the semantic prominence distribution of each sample. By allocating token budgets between locally salient regions and globally diverse regions, our method maintains strong performance even under high compression ratios. On LLaVA-NeXT-7B, our approach reduces FLOPs by 88% and prefill latency by 22% while preserving 97.5% of the original accuracy.
Figures & tables
Figure 1: Saliency–Coverage Trade-Off across two distinct benchmarks. (a) Performance trends of POPE and SQA under different saliency–coverage allocations with a fixed token budget. (b) Example samples illustrating the contrasting semantic prominence distributions of the two benchmarks.
Figure 2: Sensitivity of entropy metrics to semantic prominence distribution. Mean normalized entropy values across semantic prominence distribution levels for feature norm entropy (Norm), attention entropy (Attn), and singular value spectral entropy (Spectral).
Figure 3: Overview of the proposed PromPrune framework. Given an input image, the projected visual tokens are first processed by entropy-guided budget allocation to determine the saliency and coverage budgets. A two-stage token selection pipeline then retains locally salient tokens and selects diverse tokens from the unselected token pool to ensure global coverage before passing the compressed tokens to the LLM.
Method
MME
POPE
MMBench
TextVQA
SQA
GQA
VQAv2
VizWiz
MMVet
Rel.
Upper Bound, All 576 Tokens 100.0%
LLaVA-1.5 7B
1507.0
87.7
63.9
58.1
68.6
61.9
78.6
50.1
31.2
100.0%
Retain 128 Tokens ↓77.8%
FastV [ 5 ]
1364.2
67.9
63.1
55.7
69.4
54.1
70.5
51.4
27.8
92.0%
SparseVLM [ 41 ]
1398.9
83.7
61.8
56.8
68.7
56.9
74.9
49.8
30.0
96.5%
VisionZip [ 37 ]
1436.3
69.2
61.5
57.4
68.7
58.0
75.5
51.8
31.5
94.7%
Table 1: Performance comparison with existing visual token compression methods on LLaVA-1.5 7B. Results are reported across nine image-text understanding benchmarks under different visual token budgets. Rel. denotes the relative performance compared to the full-token upper bound.
Table 5
Strategy
MMBench (33)
VQA Text (17)
GQA (13)
SQA (40)
POPE (12)
Rel.
LLaVA-1.5 7B
63.9
58.1
61.9
68.6
87.7
100.0%
Fixed Allocation
58.8
55.4
54.6
68.8
80.7
93.6%
Semantic Prominence-Aware
60.7
55.3
57.2
68.7
85.1
96.1%
Table 4: Effect of semantic prominence-aware budget allocation. We compare the proposed semantic prominence-aware budget allocation with a fixed allocation strategy on LLaVA-1.5 7B under a 64-token budget. The fixed allocation applies the average Tsal of each benchmark to all samples. Values in parentheses indicate the benchmark-wise average Tsal . Rel. denotes the relative performance compared to the full-token upper bound.
Figure 4: Comparison of proxy metrics for estimating the distribution of semantic prominence. We compare spectral entropy with two alternative proxy metrics, feature norm entropy and attention entropy on MME, POPE, GQA benchmarks.
Method
POPE
MME
SQA
MMBench
GQA
Avg.
FPS
84.4
1343.1
68.9
60.7
57.2
54.4
FL
83.1
1377.4
69.1
60.5
58.0
54.2
DPP
85.1
1390.9
68.8
60.7
57.2
54.5
Table 5: Ablation of diversity selection methods in Stage 2. We compare different diversity maximization strategies including Farthest Point Sampling (FPS), Facility Location (FL), and Determinantal Point Process (DPP).
Figure 5: Robustness of Sigmoidal Mapping Hyperparameters. We vary μ and τ in the proposed allocation function and evaluate the resulting performance on SQA, POPE, GQA benchmarks.
Method
Tokens (Budget)
FLOPs (T)
Total Prefill (ms/sample)
KV Cache (MB)
Memory (GB)
POPE (F1)
LLaVA-NeXT-7B
2880
42.6
516.3
1440.0
18.62
87.5
VisPruner
320
5.02
321.1
160.0
16.6
81.6
DivPrune
320
5.02
388.5
160.0
16.3
84.6
PromPrune
320
5.02
402.7
160.0
16.7
85.3
Table 6: Detailed efficiency analysis on LLaVA-NeXT-7B under a fixed token budget. Efficiency metrics are measured during POPE evaluation.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : Saliency–Coverage trade-off across additional benchmarks. We vary the allocation ratio between saliency-driven tokens (stage 1) and coverage-driven tokens (stage 2) while keeping the total token budget fixed.
Figure B1 : Prompt Used for GPT-4o-based Semantic Prominence Annotation. Prompt used for GPT-4o to annotate the semantic prominence distribution of an image. The model assigns a score from 1 (highly distributed) to 5 (highly concentrated) based on the spatial concentration of semantically meaningful visual units.
Figure B2 : Examples from different semantic prominence groups. Representative examples from the five semantic prominence groups annotated using the protocol in Fig. B1, ranging from broadly distributed scenes (top) to highly concentrated visual information (bottom).
Figure C1 : The distribution of spectral entropy for different benchmarks (LLaVA-1.5). Benchmark-wise distributions of normalized spectral entropy (left) and the resulting saliency allocation ratio (right) computed using the CLIP vision encoder of LLaVA-1.5. Different benchmarks exhibit different entropy distributions, which are translated into different average saliency–coverage splits by the same sigmoidal mapping without benchmark-specific tuning.
Figure E1 : Visualization of selected visual tokens ( T=64 ) using LLaVA-1.5-7B. Saliency-only applies stage-1 selection, Coverage-only applies stage-2 selection, and PromPrune adaptively combines both. Red and cyan tokens denote saliency-driven and coverage-driven selections, respectively.