Large vision-language models (LVLMs) have recently been extended to handle multiple input images, but this extension dramatically increases the visual token budget, making visual-token pruning not merely an efficiency option but a practical necessity. Existing visual-token pruning methods, primarily designed for the single-image setting, rely on attention-derived saliency signals to estimate per-token importance. However, this attention-centric design transfers poorly to the multi-image regime: it causes substantial performance degradation on multi-image benchmarks, suffers from strong positional bias, and loses discriminative attention signals as the number of input images increases. To test whether attention-selected tokens are truly important for multi-image reasoning, we conduct two complementary causal probes Leave-One-In and Leave-One-Deletion, which measure token importance by restoring individual tokens to a zeroed visual sequence or removing them from the full sequence. Attention-based pruners provide only marginal gains over random selection, showing that attention is a weak proxy for visual-token importance in multi-image reasoning. Motivated by this gap, we propose Relation and Information Aware Pruning (RI-Prune), a training-free, attention-free pre-decoder selector for multi-image visual token pruning. RI-Prune scores each visual token using a z-normalized combination of three signals computed from projector outputs: query relevance, cross-image correspondence with compact per-image prototypes, and intra image representativeness. By integrating these signals, RI-Prune performs balanced per-image Top-K selection to preserve information across input images. Across multiple multi-image benchmarks and three LVLM backbones, RI-Prune achieves a stronger trade-off between token reduction and task performance than prior pruning methods, particularly under aggressive token budgets.
Figures & tables
Figure 1. Failure modes of attention-based visual token pruning in multi-image LVLMs. (a) Single-image pruning methods retain high performance on single-image benchmarks but degrade sharply on multi-image benchmarks, showing poor transfer to multi-image reasoning. (b) Query attention becomes unreliable across images: it exhibits positional bias and becomes increasingly non-discriminative as the number of input images grows. (c) CLS-based saliency selects tokens with weaker query alignment than query-driven attention, suggesting that CLS attention is less effective at preserving question-relevant visual evidence.
Figure 2. Method-level information gain over random pruning when retaining 4.6% of visual tokens. Bars report the per-sample average gains ΔILOI and ΔILOD over uniformly random pruning under the same token budget. RI-Prune achieves the largest gains across both benchmarks and metrics, indicating that it retains more answer-relevant visual tokens than attention-based pruning strategies.
Figure 3. Overview of the proposed visual token selection framework. Query relevance ( r ), cross-image correspondence ( c ), and intra-image representativeness ( u ) are computed, normalized, and aggregated to select the Top- K vision tokens for the LLM.
Method
Mantis-Eval
MIBench
MIRB
MUIRBench
Avg.
Vanilla
73.3 [-0.18ex]100.0%
54.6 [-0.18ex]100.0%
63.6 [-0.18ex]100.0%
60.9 [-0.18ex]100.0%
63.1 [-0.18ex]100.0%
Retain 45.3% Tokens
FastV (ECCV’24)
71.9 [-0.18ex]98.1%
54.0 [-0.18ex]98.9%
56.0 [-0.18ex]88.1%
51.5 [-0.18ex]84.6%
58.3 [-0.18ex]92.5%
VisionZip (CVPR’25)
71.9 [-0.18ex]98.1%
52.1 [-0.18ex]95.4%
49.5 [-0.18ex]77.8%
55.9 [-0.18ex]91.8%
57.3 [-0.18ex]90.9%
VTW (AAAI’25)
55.8 [-0.18ex]76.1%
45.6 [-0.18ex]83.5%
42.1 [-0.18ex]66.3%
44.4 [-0.18ex]72.9%
47.0 [-0.18ex]74.4%
DivPrune (CVPR’25)
71.9 [-0.18ex]98.1%
54.4 [-0.18ex]99.6%
60.0 [-0.18ex] 94.3%
59.5 [-0.18ex]97.7%
61.4 [-0.18ex]97.4%
Table 1. Performance comparison across multi-image benchmarks under different pruning ratios. Each cell reports accuracy on the first line and relative performance on the second line.
Method
Mantis-Eval
MIBench
MIRB
MUIRBench
Avg.
Vanilla
61.29 [-0.18ex]100.0%
51.37 [-0.18ex]100.0%
52.27 [-0.18ex]100.0%
36.10 [-0.18ex]100.0%
50.3 [-0.18ex]100.0%
Token Budget 5000
FastV (ECCV’24)
43.9 [-0.18ex]71.6%
42.67 [-0.18ex]83.1%
35.7 [-0.18ex]68.3%
29.6 [-0.18ex]82.0%
38.0 [-0.18ex]75.5%
VisionZip (CVPR’25)
59.0 [-0.18ex]96.3%
49.87 [-0.18ex] 97.1%
50.0 [-0.18ex] 95.7%
34.9 [-0.18ex]96.7%
48.4 [-0.18ex] 96.4%
VTW (AAAI’25)
33.6 [-0.18ex]54.8%
38.00 [-0.18ex]74.0%
39.9 [-0.18ex]76.3%
26.1 [-0.18ex]72.3%
34.4 [-0.18ex]68.5%
DivPrune (CVPR’25)
60.4 [-0.18ex] 98.5%
48.57 [-0.18ex] 94.5%
48.8 [-0.18ex]93.4%
35.7 [-0.18ex] 98.9%
48.4 [-0.18ex]96.2%
Table 2. Comparison with SOTA methods on LLaVA-OneVision-7B under different token budgets. Each cell reports accuracy on the first line and relative performance on the second line. Best results are boldfaced, and second-best results are underlined within each token budget.
Method
Mantis-Eval
MIBench
MIRB
MUIRBench
Avg.
Vanilla
78.7 [-0.18ex]100.0%
60.85 [-0.18ex]100.0%
68.75 [-0.18ex]100.0%
64.4 [-0.18ex]100.0%
68.2 [-0.18ex]100.0%
Retain 23.5% Tokens
FastV (ECCV’24)
69.4 [-0.18ex]88.2%
55.1 [-0.18ex]90.6%
59.3 [-0.18ex]86.3%
60.3 [-0.18ex]93.6%
61.0 [-0.18ex]89.5%
VisionZip (CVPR’25)
74.6 [-0.18ex]94.8%
53.8 [-0.18ex]88.4%
62.4 [-0.18ex]90.8%
60.9 [-0.18ex]94.6%
62.9 [-0.18ex]92.3%
VTW (AAAI’25)
63.7 [-0.18ex]80.9%
48.0 [-0.18ex]78.9%
40.1 [-0.18ex]58.3%
58.5 [-0.18ex]90.8%
52.6 [-0.18ex]77.1%
DivPrune (CVPR’25)
64.7 [-0.18ex]82.2%
51.2 [-0.18ex]84.1%
60.1 [-0.18ex]87.4%
61.0 [-0.18ex]94.7%
59.3 [-0.18ex]86.9%
Table 3. Comparison with SOTA methods on Qwen3-VL under different retained token ratios. Each cell reports accuracy on the first line and relative performance on the second line. Best results are boldfaced, and second-best results are underlined within each retained token ratio.
Variant
λr
λc
λu
r
c
u
Acc.
Δ vs Full
Random
0
0
0
40.1
−10.6
r only
1
0
0
✓
44.6
−6.1
c only
0
1
0
✓
43.1
−7.6
u only
0
0
1
✓
42.4
−8.3
r+c
1
1
0
✓
✓
48.6
−2.1
r+u
1
0
1
✓
✓
41.8
−8.9
Table 4. Ablation study of each component and its corresponding weighting coefficient in RI-Prune. Best result is boldfaced.
Method
Retain Ratio
TFLOPs
Latency
Avg. Acc.
Qwen2.5-VL
100.0%
57.8
8.135s
63.1
FastV
4.6%
11.4
7.728s
44.9
DivPrune
4.6%
8.8
8.074s
48.4
SCOPE
4.6%
8.9
7.953s
48.7
VisionZip
4.6%
8.4
5.217s
46.8
VTW
4.6%
8.5
5.125s
22.9
Table 5. Efficiency and accuracy comparison under different retained token ratios. Results are reported on Qwen2.5-VL.
Selection Mode
Accuracy
Selection Overhead (s)
TFLOPs
Global Top- K
48.8
0.909
7.9
Global + min quota
50.3
0.922
7.9
Per-image Top- K
50.7
0.920
7.9
Table 6. Comparison of token selection modes on Qwen2.5-VL at the 4.6% retained-token setting. Selection overhead is measured in seconds, excluding decoder forward computation.
P
2
4
8
16
64
128
Accuracy
45.6
46.9
47.7
50.7
51.0
48.7
Compute (s)
0.0034
0.0035
0.0038
0.0044
0.0082
0.0116
Table 7. Performance comparison according to the number of prototypes P on Qwen2.5-VL at a 4.6% retained token budget.
Benchmark
Vanilla
FastV
DivPrune
SCOPE
VisionZip
VTW
RI-Prune
Video-MME
63.0
49.3
52.2
47.8
44.9
23.2
52.9
TempCompass
67.0
59.8
63.0
59.0
55.0
33.0
61.0
Table 8. Accuracy comparison on video benchmarks. Acc. denotes accuracy (%). Among pruning methods, the best results are boldfaced and the second-best results are underlined.
Figure 4. Qualitative visualization of visual token pruning on MUIRBench ( Wang et al., 2025 ) . Bright regions indicate retained visual tokens. Colored boxes denote question-relevant regions: green indicates successful preservation, red indicates pruning of critical information, and orange indicates ambiguous preservation that ultimately leads to an incorrect prediction. RI-Prune preserves more relevant visual evidence under aggressive token reduction.
With the growing demand for processing multiple image sequences in real-world applications, various visual token pruning methods have emerged to mitigate the computational and context length constraints faced by Large Vision Language Models (LVLMs). However, most existing pruning approaches rely on static strategies that struggle to adapt across different architectural LVLMs and multi-image scenarios, and are additionally constrained by their dependence on attention computations that are incompatible with efficient techniques like FlashAttention. To address these limitations, we propose a training-free, Adaptive Visual Token Pruning (AVTP) framework, applicable to diverse LVLM architectures. We strategically determine pruning layers based on empirical analysis of visual attention distributions across various LVLMs, and implement adaptive pruning ratios in multi-image contexts where images of higher importance retain proportionally more tokens. We conduct extensive experiments across different LVLMs to demonstrate the effectiveness and robustness of AVTP. Specifically, Qwen3VL-8B achieves 2 times inference speedup while maintaining 96.1% of its original accuracy on multiple multi-image benchmarks, InternVL3.5-8B retains 94.1% accuracy, and LLaVA-OV-7B even exceeds its original baseline performance. Our code is available at this link.
Rongyang Zhang, Chengqiang Lu, Cong Li +9
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · Xiaohongshu Inc.
Vision-Language Models (VLMs) have recently demonstrated remarkable capabilities in visual understanding and reasoning, but they also impose significant computational burdens due to long visual sequence inputs. Recent works address this issue by pruning unimportant visual tokens, achieving substantial computational reduction while maintaining model performance. The core of token pruning lies in determining token importance, with current approaches primarily relying on attention scores from vision encoders or Large Language Models (LLMs). In this paper, we analyze the effectiveness of attention mechanisms in both vision encoders and LLMs. We find that vision encoders suffer from attention sink, leading to poor focus on informative foreground regions, while in LLMs, although prior studies have identified attention bias toward token positions, text-to-vision attention demonstrates resistance to this bias and enables effective pruning guidance in middle layers. Based on these observations, we propose LearnPruner, a two-stage token pruning framework that first removes redundant vision tokens via a learnable pruning module after the vision encoder, then retains only task-relevant tokens in the LLM's middle layer. Experimental results show that our LearnPruner can preserve approximately 95% of the original performance while using only 5.5% of vision tokens, and achieve 3.2× inference acceleration, demonstrating a superior accuracy-efficiency trade-off.
Visual token pruning has been widely studied as a practical approach to reducing the computational cost of large vision-language models. However, it struggles to preserve essential visual information, which can lead to substantial performance degradation. In particular, image-based token selection can overlook task-relevant details, while text-guided token selection may fail to capture the text--visual relationships needed for complex reasoning. We find that applying textual guidance too early can limit its ability to identify answer-relevant visual regions, whereas text-to-visual attention becomes more informative at intermediate decoder depths. This finding motivates our training-free method, which separates early vision-guided pruning from deferred text-guided reselection. We first prune visual tokens using vision-encoder attention, retain additional candidates until the decoder midpoint, and then use text-to-visual attention to determine the final visual-token set. Across eight benchmarks and three models, our method outperforms the best-performing baselines by an average of 11.10 and 16.84 percentage points in performance recovery at 80% and 90% pruning, respectively, with comparable or lower LLM-prefill latency than most baselines. The source code is publicly available at https://github.com/kmc3661/DeFT
Minchan Kang, Kyeonghye Park, Seoyoung Cho +2
Korea Advanced Institute of Science and Technology (KAIST) · Hanbat National University