Large Vision-Language Models (LVLMs) often process hundreds or thousands of visual tokens, resulting in substantial computational and memory overhead during inference. Existing training-free pruning methods typically estimate token importance using language-model attention, visual attention, or feature similarity. However, these methods treat visual tokens with uniform criteria and overlook that tokens may serve different roles during visual encoding. In this work, we investigate the evolution of visual token roles in LVLMs and observe that tokens progressively transition from local detail carriers to globally referenced representatives as encoding proceeds. Based on this observation, we propose BtsPruner, a training-free role-aware visual token pruning framework that preserves complementary token roles under a fixed budget. Specifically, BtsPruner first identifies the token-role transition point based on the evolution of attention concentration. It then discovers globally referenced representatives from later encoding stages and recursively retrieves local detail tokens from earlier stages to recover complementary fine-grained evidence. Extensive experiments across LLaVA-1.5, LLaVA-NeXT, Qwen-VL, InternVL, and CogVLM demonstrate that BtsPruner achieves superior performance retention under aggressive token reduction. On LLaVA-1.5-7B, BtsPruner preserves 91.6% of the dense model's performance using only 32 of 576 visual tokens (5.6%). In the high-resolution LLaVA-NeXT-7B setting, it achieves up to a 5.95x CUDA-latency speedup.
Figures & tables
Figure 1: Qualitative comparison of visual token pruning strategies. Existing methods select tokens according to a unified criterion, while BtsPruner preserves complementary token roles under the same token budget.
Figure 2: Visual token role evolution during LVLM encoding. Left : Attention patterns show a transition from localized interactions to shared representations. Right : Representative local detail and global representative tokens across different encoding stages.
Figure 3: Overview of BtsPruner. BtsPruner first identifies the token role transition during visual encoding, then discovers global representative tokens and recursively recovers local detail tokens through a shared diversity selection module. The selected tokens with complementary roles are combined for efficient LVLM inference.
Method
VQAv2
GQA
VizWiz
SQA IMG
VQA Text
POPE
MME
MMB
MMB CN
MMVet
Avg.
Rel. (%)
Upper Bound (576 tokens)
LLaVA-1.5-7B
78.5
62.0
50.0
66.8
58.2
85.9
1510.7
64.3
58.3
31.1
63.1
100.0%
Retain 128 Tokens ( ↓ 77.8%)
FastV (ECCV24)
73.2
55.8
51.5
68.2
56.9
68.8
1442.3
63.2
56.7
28.0
59.4
94.7%
SparseVLM (ICML25)
76.2
58.4
50.0
65.2
56.8
85.2
1380.4
63.7
58.1
28.3
61.1
96.7%
DivPrune (CVPR25)
75.7
59.3
53.3
68.0
55.5
84.3
1407.7
61.1
54.4
28.7
61.1
96.8%
Table 1: Comparison of visual token pruning methods on LLaVA-1.5-7B across three pruning rates. Avg. denotes the normalized average score across benchmarks, and Rel. denotes the relative performance retention ratio compared with the dense model.
Method
VQAv2
GQA
VQA Text
POPE
MME
Avg.
Rel. (%)
Upper Bound (2880 tokens)
LLaVA-1.6-7B
81.3
62.4
60.2
86.7
1511.8
73.2
100.0%
Retain 640 Tokens ( ↓ 77.8%)
FastV
78.6
60.0
59.0
72.9
1467.6
68.8
94.4%
SparseVLM
77.4
59.8
57.0
81.0
1457.5
69.6
95.1%
DivPrune
77.1
59.8
54.5
82.4
1444.2
69.2
94.3%
Table 2: Performance comparison of visual token pruning methods on LLaVA-1.6-7B, i.e., LLaVA-NeXT-7B, across three pruning rates. Avg. denotes the normalized average score across benchmarks, and Rel. denotes the relative performance retention ratio compared with the dense model.
Method
VQAv2
GQA
VizWiz
SQA
VQA Text
Avg.
Rel. (%)
Qwen-VL-7B (128/256, ↓ 50%)
Qwen-VL
78.8
59.3
35.2
67.1
63.8
60.8
100.0%
FastV
73.3
55.2
35.2
67.3
59.7
58.1
96.0%
DivPrune
77.8
57.8
33.0
67.7
56.6
58.6
95.9%
VisPruner
77.1
57.6
30.9
67.7
55.4
57.7
94.1%
Ours
78.1
58.4
33.2
68.4
61.2
59.9
98.0%
Table 3: Performance across LVLM architectures. Avg. denotes the normalized average score across benchmarks, and Rel. denotes the relative performance retention ratio compared with the dense model.
Global Rep.
Local Detail
TRIM
Diverse
Avg.
Random
Random
×
×
48.5
Our
Random
×
×
54.7
Our
×
×
55.3
√
×
55.6
×
√
56.1
√
√
56.3
Table 4: Ablation study of BtsPruner components on TextVQA. Global Rep. and Local Detail denote global representative discovery and local detail recovery, respectively.
# Token
r
VisPruner
BtsPruner
0.25
56.7
57.7
128
0.50
57.2
57.7
0.75
56.8
57.3
0.25
55.7
56.3
64
0.50
55.8
56.3
0.75
56.0
56.2
Table 5: Influence of the global-representative token ratio r on TextVQA performance.
Method
# Token
FLOPs ↓ (T)
Cache ↓ (MB)
GPU Mem. ↓ (GB)
CUDA Lat. ↓ (ms)
LLaVA-NeXT
2880
40.8
1471
17.8
821
FastV
640
13.2
370
17.6
332
DivPrune
640
11.4
351
15.2
282
VisPruner
640
11.1
351
15.5
264
Ours
640
11.1
351
15.5
270
FastV
160
7.4
125
17.6
207
Table 6: Efficiency comparison on LLaVA-NeXT-7B using the POPE benchmark. Latency is measured as CUDA inference latency on a single RTX 3090 GPU.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Qualitative comparison of visual token pruning strategies.
Figure 5: Visualization of attention evolution during the encoding process across different image samples.
Large Vision-Language Models (LVLMs) face significant computational inefficiencies caused by the large number of visual tokens. Existing visual token pruning methods mainly focus on either retaining individually important tokens or selecting mutually diverse ones. In this work, we revisit visual token pruning from a coverage perspective and formulate it as a biased attention coverage maximization problem. The key idea is to select a compact token subset whose encoder-side outgoing attention can jointly cover the image while assigning higher coverage priority to more informative regions. From this perspective, we propose ACPruner, a training-free visual token pruning framework for efficient LVLM inference. ACPruner first estimates token importance by combining intra-modal saliency and inter-modal relevance, then derives token-wise coverage from attention patterns within the vision encoder, and finally performs greedy selection to maximize the proposed coverage objective. Extensive experiments across multiple LVLM backbones, including LLaVA-1.5-7B/13B, LLaVA-NeXT-7B/13B, Qwen2.5-VL-7B, and LLaVA-OneVision-7B, show that ACPruner consistently achieves strong performance retention while delivering substantial end-to-end inference speedups.
Xu Li, Yuxuan Liang, Yi Zheng +6
College of Computer Science and Artificial Intelligence Fudan University, Shanghai, China
Vision-language models (VLMs) process an image as a sequence of visual tokens, which creates a substantial computational bottleneck during inference. Recent visual token pruning methods address this issue by removing seemingly redundant tokens, yet it remains unclear how these pruning decisions relate to the functional roles of visual tokens. In this work, we analyze visual token pruning through the lens of token roles identified by EmbedLens. We first show that representative pruning methods exhibit distinct token-role biases, but these biases do not directly correlate with downstream performance. To better understand this behavior, we refine the token-role assignment procedure and evaluate role-protected pruning variants. Our results show that preserving non-alive tokens can sometimes maintain or improve performance, suggesting that tokens with weak direct semantic alignment may still affect model behavior under pruning. Our code is publicly available at https://github.com/jaykim9870/Not_All_Redundant_Tokens_Are_Alike.
Vision-Language Models (VLMs) have recently demonstrated remarkable capabilities in visual understanding and reasoning, but they also impose significant computational burdens due to long visual sequence inputs. Recent works address this issue by pruning unimportant visual tokens, achieving substantial computational reduction while maintaining model performance. The core of token pruning lies in determining token importance, with current approaches primarily relying on attention scores from vision encoders or Large Language Models (LLMs). In this paper, we analyze the effectiveness of attention mechanisms in both vision encoders and LLMs. We find that vision encoders suffer from attention sink, leading to poor focus on informative foreground regions, while in LLMs, although prior studies have identified attention bias toward token positions, text-to-vision attention demonstrates resistance to this bias and enables effective pruning guidance in middle layers. Based on these observations, we propose LearnPruner, a two-stage token pruning framework that first removes redundant vision tokens via a learnable pruning module after the vision encoder, then retains only task-relevant tokens in the LLM's middle layer. Experimental results show that our LearnPruner can preserve approximately 95% of the original performance while using only 5.5% of vision tokens, and achieve 3.2× inference acceleration, demonstrating a superior accuracy-efficiency trade-off.