Recent advances have explored visual token pruning to accelerate the inference of large vision-language models (LVLMs). However, existing methods often struggle to balance token importance and diversity: importance-based methods tend to retain redundant tokens, whereas diversity-based methods may overlook informative ones. This trade-off becomes especially problematic under high reduction ratios, where preserving only a small subset of visual tokens is critical. To address this issue, we propose ID-Selection, a simple yet effective token selection strategy for efficient LVLM inference. The key idea is to couple importance estimation with diversity-aware iterative selection: each token is first assigned an importance score, after which high-scoring tokens are selected one by one while the scores of similar tokens are progressively suppressed. In this way, ID-Selection preserves informative tokens while reducing redundancy in a unified selection process. Extensive experiments across 5 LVLM backbones and 16 main benchmarks demonstrate that ID-Selection consistently achieves superior performance and efficiency, especially under extreme pruning ratios. For example, on LLaVA-1.5-7B, ID-Selection prunes 97.2% of visual tokens, retaining only 16 tokens, while reducing inference FLOPs by over 97% and preserving 91.8% of the original performance, all without additional training.
Figures & tables
Figure 1: t-SNE visualization of the distribution of 16 selected tokens on POPE [ 18 ] using different methods. ”All Tokens” represents the original token distribution of LLaVA-1.5-7B [ 21 ] . Red dashed boxes highlight dispersed tokens that are harder to identify and are often informative for downstream understanding.
Figure 2: Overview of ID-Selection. Each token is assigned an importance score. For the LLaVA series [ 21 , 20 , 19 ] , we adopt a unified score (Eq. ( 6 )) that combines visual saliency ( [CLS] attention [ 41 ] ) and instruction relevance (Eq. ( 4 )). Tokens with high scores are iteratively selected while suppressing similar ones to maintain diversity, until the token budget is reached, and the remaining tokens are pruned.
Figure 3: Visualization of visual saliency ( [CLS] Attention [ 41 ] ), instruction relevance (Eq. ( 4 )), and unified importance scores (Eq. ( 6 )) of LLaVA-1.5-7B [ 21 ] on TextVQA [ 32 ] . Red indicates higher scores, while blue indicates lower scores.
Method
Venue
VQA V2
GQA
VizWiz
SQA IMG
VQA Text
POPE
MME
MMB
MMB CN
Average
Upper Bound, All 576 Tokens (100%)
LLaVA-1.5-7B [ 21 ]
CVPR’24
78.5
62.0
50.0
69.5
58.2
85.9
1862
64.3
58.3
100.0%
Retain 64 Tokens ( ↓ 88.9%)
FastV [ 6 ]
ECCV’24
55.0
46.1
50.8
51.1
47.8
48.0
1256
48.0
52.7
76.7%
FasterVLM [ 41 ]
ArXiv’24
72.6
55.0
53.2
68.9
55.3
76.8
1659
61.3
55.2
94.5%
SparseVLM [ 43 ]
ICML’25
68.2
52.7
50.1
62.2
51.8
75.1
1505
56.2
46.1
87.3%
Table 1: Comparison with previous approaches on LLaVA-1.5-7B [ 21 ] across 9 image understanding benchmarks. The highest score is denoted in bold . Average are computed by calculating the mean accuracy across all 9 datasets. † : importance from LLM second-layer cross-modal attention (Eq. ( 3 )); ‡ : importance from visual encoder [CLS] attention [ 30 ] ; \lx@sectionsign : importance from the unified score in Eq. ( 6 ).
Method
Venue
VQA V2
GQA
VizWiz
SQA IMG
VQA Text
POPE
MME
MMB
MMB CN
Average
Upper Bound, All 2880 Tokens (100%)
LLaVA-Next-7B [ 20 ]
CVPR’24
81.8
64.2
57.6
70.1
61.3
86.5
1851
67.4
60.6
100.0%
Retain 320 Tokens ( ↓ 88.9%)
FastV [ 6 ]
ECCV’24
72.9
55.9
53.1
62.8
55.7
71.7
1661
61.6
51.9
88.7%
FasterVLM [ 41 ]
ArXiv’24
75.8
58.5
57.0
68.3
57.9
80.4
1661
63.8
55.4
93.7%
SparseVLM [ 43 ]
ICML’25
74.6
57.9
54.2
67.2
56.5
76.9
1688
63.1
54.5
91.9%
Table 2: Comparison with previous approaches on LLaVA-Next-7B [ 20 ] across 9 image understanding benchmarks. The highest score is denoted in bold . Average are computed by calculating the mean accuracy across all 9 datasets. † : importance from LLM second-layer cross-modal attention (Eq. ( 3 )); ‡ : importance from visual encoder [CLS] attention [ 30 ] ; \lx@sectionsign : importance from the unified score in Eq. ( 6 ).
Method
TGIF
MSVD
MSRVTT
Average
Acc
Score
Acc
Score
Acc
Score
Acc
Score
Upper Bound, All 2048 Tokens (100%)
Video-LLaVA
19.8
2.53
70.5
3.93
57.5
3.50
100%
100%
Retain 256 Tokens ( ↓ 87.5%)
FastV [ 6 ]
15.7
2.51
67.3
3.92
52.9
3.38
88.9%
98.5%
FasterVLM [ 41 ]
15.5
2.41
69.0
3.92
55.3
3.42
90.8%
97.6%
Table 3: Comparison of Video-LLaVA’s performance [ 19 ] on 3 video understanding benchmarks [ 15 , 5 , 37 ] . The highest score is denoted in bold . ‡ : importance from visual encoder [CLS] attention [ 30 ] ; \lx@sectionsign : importance from the unified score in Eq. ( 6 ).
Method
VQA Text
POPE
MMB
SQA IMG
MME
Average
Qwen2.5-VL-7B
77.7
86.0
83.9
76.6
2330
100%
Token Reduction Ratio = 77.8%
FastV [ 6 ]
75.2
77.6
80.6
74.0
2000
93.1%
DART [ 36 ]
72.0
78.5
79.8
75.2
2193
94.3%
ID-Selection †
76.0
84.1
82.1
77.1
2217
97.9%
Token Reduction Ratio = 88.9%
Table 4: Comparison of Qwen2.5-VL-7B’s and InternVL2.5-8B’s performance [ 4 , 8 ] on 5 image understanding benchmarks [ 18 , 32 , 24 , 22 , 9 ] . The highest score is denoted in bold . † : importance from LLM second-layer cross-modal attention (Eq. ( 3 )).
Method
Chart
OCR
Doc
Info
Average
Qwen2.5-VL-7B
84.0
83.8
95
81
100.0%
Token Reduction Ratio = 77.8%
FastV [ 6 ]
65.4
58.7
79
54
74.4%
DART [ 36 ]
53.2
58.8
60
40
61.5%
ID-Selection †
72.4
72.8
86
60
84.4%
Token Reduction Ratio = 88.9%
Table 5: Comparison of Qwen2.5-VL-7B’s and InternVL2.5-8B’s performance [ 4 , 8 ] on 4 text-focused image understanding benchmarks [ 26 , 23 , 28 , 27 ] . The highest score is denoted in bold . † : importance from LLM second-layer cross-modal attention (Eq. ( 3 )).
Method
FLOPs (T) ↓
Total Time (s) ↓
KV Cache ↓
LLaVA-Next-7B
20.8 (1.0 × )
2294 (1.0 × )
1510 MB
FastV
3.3 (6.3 × )
1097 (2.1 × )
168 MB
DivPrune
2.1 (9.9 × )
992 (2.3 × )
160 MB
CDPruner
2.1 (9.9 × )
1010 (2.3 × )
160 MB
ID-Selection ‡
2.1 (9.9 × )
916 (2.5 × )
160 MB
ID-Selection §
2.1 (9.9 × )
910 (2.5 × )
160 MB
Table 6: Efficiency comparisons on the POPE benchmark [ 18 ] using LLaVA-Next-7B [ 20 ] under an 88.9% pruning ratio. We report the theoretical FLOPs, actual runtime, and the KV Cache. ‡ : importance from visual encoder [CLS] attention [ 30 ] ; \lx@sectionsign : importance from the unified score in Eq. ( 6 ).
Large Vision-Language Models (LVLMs) have achieved remarkable success across diverse multimodal tasks, yet their practical deployment remains constrained by the computational burden arising from lengthy visual tokens. While visual token pruning has emerged as a promising solution, existing methods suffer from a fundamental limitation: once tokens are pruned at a specific layer, they become inaccessible to all subsequent layers, leading to premature information loss that can compromise model performance. Through empirical studies, we observe that different layers exhibit distinct visual region focus, indicating a varying optimal token subset across layers. Motivated by this insight, we propose Adaptive Layer-wise Visual Token Selection (ALVTS), a novel framework that breaks away from the conventional static token pruning paradigm. ALVTS incorporates a lightweight token selector to identify and route important tokens for further processing, while allowing less important tokens to skip the layer, thus minimizing computational redundancy. These two streams of tokens are seamlessly reintegrated before being fed into subsequent layers, facilitating adaptive compression across the entire model. Grounded in our importance consistency constrained low-rank approximation, the proposed token selection module closely emulates the full attention mechanism, effectively capturing its essential patterns without requiring model retraining. Extensive experiments on LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL validate the effectiveness of our method. With an 89% token compression ratio, ALVTS retains 96.7% of the original model's accuracy, achieving a superior efficiency-accuracy trade-off for LVLM inference.
Yongru Chen, Kai Zhang, Zeliang Zong +4
Hikvision Research Institute · East China Normal University · Peking University
Visual token pruning, which aims to compress and prune redundant visual tokens, plays a critical role in efficient inference with large vision-language models (LVLMs). However, existing methods fail to disentangle intra-modal visual redundancy from cross-modal redundancy between vision and language. We show that visual token diversity and task-specific token relevance are two crucial yet orthogonal factors that complement each other in conveying useful information and should therefore be treated separately for more effective visual token pruning. Building upon this insight, we design TODRE, a two-stage and training-free framework that incorporates Token Diversity and task RElevance for effective token compression and efficient LVLM inference. Instead of pruning redundant tokens, we introduce a greedy max-sum diversification algorithm that selects and retains a subset of diverse and representative visual tokens after the vision encoder. On top of that, ToDRE leverages an ``information migration'' mechanism to eliminate task-irrelevant visual tokens within certain decoder layers of the large language model (LLM), further improving token pruning and LVLM inference. Extensive experiments show that ToDRE prunes 90% of visual tokens after the vision encoder as well as all visual tokens in certain LLM decoder layers, leading to a 2.6x speed-up in total inference time while maintaining 95.0% model performance plus excellent model compatibility. The code is available at: this https URL.
Duo Li, Zuhao Yang, Xiaoqin Zhang +2
CCDS, NTU, Singapore · CCST, ZJUT, China · Terminus AI Lab, UCAS, China
Visual tokens dominate inference cost in vision-language models (VLMs), yet many carry redundant information. Existing pruning methods alleviate this but typically rely on attention magnitude or similarity scores. We reformulate visual token pruning as capacity constrained communication: given a fixed budget K, the model must allocate limited bandwidth to maximally preserve visual information. We propose AutoSelect, which attaches a lightweight Scorer and Denoiser to a frozen VLM and trains with only the standard next token prediction loss, without auxiliary objectives or extra annotations. During training, a variance preserving noise gate modulates each token's information flow according to its predicted importance so that gradients propagate through all tokens; a diagonal attention Denoiser then recovers the perturbed representations. At inference, only the Scorer and a hard top-K selection remain, adding negligible latency. On ten VLM benchmarks, AutoSelect retains 96.5% of full model accuracy while accelerating LLM prefill by 2.85x with only 0.69 ms overhead, and transfers to different VLM backbones without architecture-specific tuning. Code is available at https://github.com/MedHK23/AutoSelect.