ID-Selection: Importance-Diversity Based Visual Token Selection for Efficient LVLM Inference

Zhaohong Huang; Wenjing Liu; Yuxin Zhang; Fei Chao; Rongrong Ji

cs.CVApr 7, 2026

ID-Selection: Importance-Diversity Based Visual Token Selection for Efficient LVLM Inference

Authors: Zhaohong Huang, Wenjing Liu, Yuxin Zhang, Fei Chao, Rongrong Ji

Organizations: Xiamen University, 361005, P.R. China.

Abstract

Recent advances have explored visual token pruning to accelerate the inference of large vision-language models (LVLMs). However, existing methods often struggle to balance token importance and diversity: importance-based methods tend to retain redundant tokens, whereas diversity-based methods may overlook informative ones. This trade-off becomes especially problematic under high reduction ratios, where preserving only a small subset of visual tokens is critical. To address this issue, we propose ID-Selection, a simple yet effective token selection strategy for efficient LVLM inference. The key idea is to couple importance estimation with diversity-aware iterative selection: each token is first assigned an importance score, after which high-scoring tokens are selected one by one while the scores of similar tokens are progressively suppressed. In this way, ID-Selection preserves informative tokens while reducing redundancy in a unified selection process. Extensive experiments across 5 LVLM backbones and 16 main benchmarks demonstrate that ID-Selection consistently achieves superior performance and efficiency, especially under extreme pruning ratios. For example, on LLaVA-1.5-7B, ID-Selection prunes 97.2% of visual tokens, retaining only 16 tokens, while reducing inference FLOPs by over 97% and preserving 91.8% of the original performance, all without additional training.

Figures & tables

Figure 1: t-SNE visualization of the distribution of 16 selected tokens on POPE [ 18 ] using different methods. ”All Tokens” represents the original token distribution of LLaVA-1.5-7B [ 21 ] . Red dashed boxes highlight dispersed tokens that are harder to identify and are often informative for downstream understanding.

Figure 2: Overview of ID-Selection. Each token is assigned an importance score. For the LLaVA series [ 21 , 20 , 19 ] , we adopt a unified score (Eq. ( 6 )) that combines visual saliency ( [CLS] attention [ 41 ] ) and instruction relevance (Eq. ( 4 )). Tokens with high scores are iteratively selected while suppressing similar ones to maintain diversity, until the token budget is reached, and the remaining tokens are pruned.

Figure 3: Visualization of visual saliency ( [CLS] Attention [ 41 ] ), instruction relevance (Eq. ( 4 )), and unified importance scores (Eq. ( 6 )) of LLaVA-1.5-7B [ 21 ] on TextVQA [ 32 ] . Red indicates higher scores, while blue indicates lower scores.

Method	Venue	VQA ${}^{\text{V2}}$	GQA	VizWiz	SQA ${}^{\text{IMG}}$	VQA ${}^{\text{Text}}$	POPE	MME	MMB	MMB ${}^{\text{CN}}$	Average
Upper Bound, All 576 Tokens (100%)
LLaVA-1.5-7B [ 21 ]	CVPR’24	78.5	62.0	50.0	69.5	58.2	85.9	1862	64.3	58.3	100.0%
Retain 64 Tokens ( $\downarrow$ 88.9%)
FastV [ 6 ]	ECCV’24	55.0	46.1	50.8	51.1	47.8	48.0	1256	48.0	52.7	76.7%
FasterVLM [ 41 ]	ArXiv’24	72.6	55.0	53.2	68.9	55.3	76.8	1659	61.3	55.2	94.5%
SparseVLM [ 43 ]	ICML’25	68.2	52.7	50.1	62.2	51.8	75.1	1505	56.2	46.1	87.3%

Table 1: Comparison with previous approaches on LLaVA-1.5-7B [ 21 ] across

9

image understanding benchmarks. The highest score is denoted in bold . Average are computed by calculating the mean accuracy across all

9

datasets.

{{\dagger}}

: importance from LLM second-layer cross-modal attention (Eq. ( 3 ));

{{\ddagger}}

: importance from visual encoder [CLS] attention [ 30 ] ;

{\lx@sectionsign}

: importance from the unified score in Eq. ( 6 ).

Method	Venue	VQA ${}^{\text{V2}}$	GQA	VizWiz	SQA ${}^{\text{IMG}}$	VQA ${}^{\text{Text}}$	POPE	MME	MMB	MMB ${}^{\text{CN}}$	Average
Upper Bound, All 2880 Tokens (100%)
LLaVA-Next-7B [ 20 ]	CVPR’24	81.8	64.2	57.6	70.1	61.3	86.5	1851	67.4	60.6	100.0%
Retain 320 Tokens ( $\downarrow$ 88.9%)
FastV [ 6 ]	ECCV’24	72.9	55.9	53.1	62.8	55.7	71.7	1661	61.6	51.9	88.7%
FasterVLM [ 41 ]	ArXiv’24	75.8	58.5	57.0	68.3	57.9	80.4	1661	63.8	55.4	93.7%
SparseVLM [ 43 ]	ICML’25	74.6	57.9	54.2	67.2	56.5	76.9	1688	63.1	54.5	91.9%

Table 2: Comparison with previous approaches on LLaVA-Next-7B [ 20 ] across

9

image understanding benchmarks. The highest score is denoted in bold . Average are computed by calculating the mean accuracy across all

9

datasets.

{{\dagger}}

: importance from LLM second-layer cross-modal attention (Eq. ( 3 ));

{{\ddagger}}

: importance from visual encoder [CLS] attention [ 30 ] ;

{\lx@sectionsign}

: importance from the unified score in Eq. ( 6 ).

Method	TGIF		MSVD		MSRVTT		Average
Method	Acc	Score	Acc	Score	Acc	Score	Acc	Score
Upper Bound, All 2048 Tokens (100%)
Video-LLaVA	19.8	2.53	70.5	3.93	57.5	3.50	100%	100%
Retain 256 Tokens ( $\downarrow$ 87.5%)
FastV [ 6 ]	15.7	2.51	67.3	3.92	52.9	3.38	88.9%	98.5%
FasterVLM [ 41 ]	15.5	2.41	69.0	3.92	55.3	3.42	90.8%	97.6%

Table 3: Comparison of Video-LLaVA’s performance [ 19 ] on 3 video understanding benchmarks [ 15 , 5 , 37 ] . The highest score is denoted in bold .

{{\ddagger}}

: importance from visual encoder [CLS] attention [ 30 ] ;

{\lx@sectionsign}

: importance from the unified score in Eq. ( 6 ).

Method	VQA ${}^{\text{Text}}$	POPE	MMB	SQA ${}^{\text{IMG}}$	MME	Average
Qwen2.5-VL-7B	77.7	86.0	83.9	76.6	2330	100%
Token Reduction Ratio = 77.8%
FastV [ 6 ]	75.2	77.6	80.6	74.0	2000	93.1%
DART [ 36 ]	72.0	78.5	79.8	75.2	2193	94.3%
ID-Selection †	76.0	84.1	82.1	77.1	2217	97.9%
Token Reduction Ratio = 88.9%

Table 4: Comparison of Qwen2.5-VL-7B’s and InternVL2.5-8B’s performance [ 4 , 8 ] on 5 image understanding benchmarks [ 18 , 32 , 24 , 22 , 9 ] . The highest score is denoted in bold .

{{\dagger}}

: importance from LLM second-layer cross-modal attention (Eq. ( 3 )).

Method	Chart	OCR	Doc	Info	Average
Qwen2.5-VL-7B	84.0	83.8	95	81	100.0%
Token Reduction Ratio = 77.8%
FastV [ 6 ]	65.4	58.7	79	54	74.4%
DART [ 36 ]	53.2	58.8	60	40	61.5%
ID-Selection †	72.4	72.8	86	60	84.4%
Token Reduction Ratio = 88.9%

Table 5: Comparison of Qwen2.5-VL-7B’s and InternVL2.5-8B’s performance [ 4 , 8 ] on 4 text-focused image understanding benchmarks [ 26 , 23 , 28 , 27 ] . The highest score is denoted in bold .

{{\dagger}}

: importance from LLM second-layer cross-modal attention (Eq. ( 3 )).

Method	FLOPs (T) $\downarrow$	Total Time (s) $\downarrow$	KV Cache $\downarrow$
LLaVA-Next-7B	20.8 (1.0 $\times$ )	2294 (1.0 $\times$ )	1510 MB
FastV	3.3 (6.3 $\times$ )	1097 (2.1 $\times$ )	168 MB
DivPrune	2.1 (9.9 $\times$ )	992 (2.3 $\times$ )	160 MB
CDPruner	2.1 (9.9 $\times$ )	1010 (2.3 $\times$ )	160 MB
ID-Selection ‡	2.1 (9.9 $\times$ )	916 (2.5 $\times$ )	160 MB
ID-Selection §	2.1 (9.9 $\times$ )	910 (2.5 $\times$ )	160 MB

Table 6: Efficiency comparisons on the POPE benchmark [ 18 ] using LLaVA-Next-7B [ 20 ] under an 88.9% pruning ratio. We report the theoretical FLOPs, actual runtime, and the KV Cache.

{{\ddagger}}

: importance from visual encoder [CLS] attention [ 30 ] ;

{\lx@sectionsign}

: importance from the unified score in Eq. ( 6 ).

Figure 4: Ablation of Unified Importance.

Figure 5: Ablation study on

\gamma

in Eq. ( 8 )

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs
Jun 12, 2026Yongru Chen, Kai Zhang, Zeliang Zong +4Efficient VLM Inference Large Vision-Language Models
Hikvision Research Institute · East China Normal University · Peking University
ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance
May 24, 2025Duo Li, Zuhao Yang, Xiaoqin Zhang +2Efficient VLM Inference Large Vision-Language Models
CCDS, NTU, Singapore · CCST, ZJUT, China · Terminus AI Lab, UCAS, China
The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gating
Mar 7, 2026Landi He, Xiaoyu Yang, Lijian XuEfficient VLM Inference Visual Token Pruning
Shenzhen University of Advanced Technology

Jun 12, 2026J/K move · Enter open · S save

One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs

Yongru Chen, Kai Zhang, Zeliang Zong, Yuchen Lu+3

Hikvision Research Institute · East China Normal University · Peking University

Efficient VLM Inference Large Vision-Language Models Visual Token Pruning Visual Token Compression

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) have achieved remarkable success across diverse multimodal tasks, yet their practical deployment remains constrained by the computational burden arising from lengthy visual tokens. While visual token pruning has emerged as a promising solution, existing methods suffer from a fundamental limitation: once tokens are pruned at a specific layer, they become inaccessible to all subsequent layers, leading to premature information loss that can compromise model performance. Through empirical studies, we observe that different layers exhibit distinct visual region focus, indicating a varying optimal token subset across layers. Motivated by this insight, we propose Adaptive Layer-wise Visual Token Selection (ALVTS), a novel framework that breaks away from the conventional static token pruning paradigm. ALVTS incorporates a lightweight token selector to identify and route important tokens for further processing, while allowing less important tokens to skip the layer, thus minimizing computational redundancy. These two streams of tokens are seamlessly reintegrated before being fed into subsequent layers, facilitating adaptive compression across the entire model. Grounded in our importance consistency constrained low-rank approximation, the proposed token selection module closely emulates the full attention mechanism, effectively capturing its essential patterns without requiring model retraining. Extensive experiments on LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL validate the effectiveness of our method. With an 89% token compression ratio, ALVTS retains 96.7% of the original model's accuracy, achieving a superior efficiency-accuracy trade-off for LVLM inference.

Abstract

Efficient VLM Inference

Explore similar work

One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs

ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance

The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gating