RI-Prune: Relation and Information Aware Visual Token Pruning for Multi-Image Vision-Language Models

Dongjin Lee; Eunju Lee; Jinhee Jang; Yoonji Lee; Youngbin Kim

cs.CVNov 4, 2026

RI-Prune: Relation and Information Aware Visual Token Pruning for Multi-Image Vision-Language Models

Authors: Dongjin Lee, Eunju Lee, Jinhee Jang, Yoonji Lee, Youngbin Kim

Organizations: Chung-Ang University, Seoul, Republic of Korea

Abstract

Large vision-language models (LVLMs) have recently been extended to handle multiple input images, but this extension dramatically increases the visual token budget, making visual-token pruning not merely an efficiency option but a practical necessity. Existing visual-token pruning methods, primarily designed for the single-image setting, rely on attention-derived saliency signals to estimate per-token importance. However, this attention-centric design transfers poorly to the multi-image regime: it causes substantial performance degradation on multi-image benchmarks, suffers from strong positional bias, and loses discriminative attention signals as the number of input images increases. To test whether attention-selected tokens are truly important for multi-image reasoning, we conduct two complementary causal probes Leave-One-In and Leave-One-Deletion, which measure token importance by restoring individual tokens to a zeroed visual sequence or removing them from the full sequence. Attention-based pruners provide only marginal gains over random selection, showing that attention is a weak proxy for visual-token importance in multi-image reasoning. Motivated by this gap, we propose Relation and Information Aware Pruning (RI-Prune), a training-free, attention-free pre-decoder selector for multi-image visual token pruning. RI-Prune scores each visual token using a z-normalized combination of three signals computed from projector outputs: query relevance, cross-image correspondence with compact per-image prototypes, and intra image representativeness. By integrating these signals, RI-Prune performs balanced per-image Top-K selection to preserve information across input images. Across multiple multi-image benchmarks and three LVLM backbones, RI-Prune achieves a stronger trade-off between token reduction and task performance than prior pruning methods, particularly under aggressive token budgets.

Figures & tables

Figure 1. Failure modes of attention-based visual token pruning in multi-image LVLMs. (a) Single-image pruning methods retain high performance on single-image benchmarks but degrade sharply on multi-image benchmarks, showing poor transfer to multi-image reasoning. (b) Query attention becomes unreliable across images: it exhibits positional bias and becomes increasingly non-discriminative as the number of input images grows. (c) CLS-based saliency selects tokens with weaker query alignment than query-driven attention, suggesting that CLS attention is less effective at preserving question-relevant visual evidence.

Figure 2. Method-level information gain over random pruning when retaining $4.6\%$ of visual tokens. Bars report the per-sample average gains $\Delta\mathcal{I}_{\mathrm{LOI}}$ and $\Delta\mathcal{I}_{\mathrm{LOD}}$ over uniformly random pruning under the same token budget. RI-Prune achieves the largest gains across both benchmarks and metrics, indicating that it retains more answer-relevant visual tokens than attention-based pruning strategies.

Figure 3. Overview of the proposed visual token selection framework. Query relevance ( $r$ ), cross-image correspondence ( $c$ ), and intra-image representativeness ( $u$ ) are computed, normalized, and aggregated to select the Top- $K$ vision tokens for the LLM.

Method	Mantis-Eval	MIBench	MIRB	MUIRBench	Avg.
Vanilla	73.3 [-0.18ex]100.0%	54.6 [-0.18ex]100.0%	63.6 [-0.18ex]100.0%	60.9 [-0.18ex]100.0%	63.1 [-0.18ex]100.0%
Retain 45.3% Tokens
FastV (ECCV’24)	71.9 [-0.18ex]98.1%	54.0 [-0.18ex]98.9%	56.0 [-0.18ex]88.1%	51.5 [-0.18ex]84.6%	58.3 [-0.18ex]92.5%
VisionZip (CVPR’25)	71.9 [-0.18ex]98.1%	52.1 [-0.18ex]95.4%	49.5 [-0.18ex]77.8%	55.9 [-0.18ex]91.8%	57.3 [-0.18ex]90.9%
VTW (AAAI’25)	55.8 [-0.18ex]76.1%	45.6 [-0.18ex]83.5%	42.1 [-0.18ex]66.3%	44.4 [-0.18ex]72.9%	47.0 [-0.18ex]74.4%
DivPrune (CVPR’25)	71.9 [-0.18ex]98.1%	54.4 [-0.18ex]99.6%	60.0 [-0.18ex] 94.3%	59.5 [-0.18ex]97.7%	61.4 [-0.18ex]97.4%

Table 1. Performance comparison across multi-image benchmarks under different pruning ratios. Each cell reports accuracy on the first line and relative performance on the second line.

Method	Mantis-Eval	MIBench	MIRB	MUIRBench	Avg.
Vanilla	61.29 [-0.18ex]100.0%	51.37 [-0.18ex]100.0%	52.27 [-0.18ex]100.0%	36.10 [-0.18ex]100.0%	50.3 [-0.18ex]100.0%
Token Budget 5000
FastV (ECCV’24)	43.9 [-0.18ex]71.6%	42.67 [-0.18ex]83.1%	35.7 [-0.18ex]68.3%	29.6 [-0.18ex]82.0%	38.0 [-0.18ex]75.5%
VisionZip (CVPR’25)	59.0 [-0.18ex]96.3%	49.87 [-0.18ex] 97.1%	50.0 [-0.18ex] 95.7%	34.9 [-0.18ex]96.7%	48.4 [-0.18ex] 96.4%
VTW (AAAI’25)	33.6 [-0.18ex]54.8%	38.00 [-0.18ex]74.0%	39.9 [-0.18ex]76.3%	26.1 [-0.18ex]72.3%	34.4 [-0.18ex]68.5%
DivPrune (CVPR’25)	60.4 [-0.18ex] 98.5%	48.57 [-0.18ex] 94.5%	48.8 [-0.18ex]93.4%	35.7 [-0.18ex] 98.9%	48.4 [-0.18ex]96.2%

Table 2. Comparison with SOTA methods on LLaVA-OneVision-7B under different token budgets. Each cell reports accuracy on the first line and relative performance on the second line. Best results are boldfaced, and second-best results are underlined within each token budget.

Method	Mantis-Eval	MIBench	MIRB	MUIRBench	Avg.
Vanilla	78.7 [-0.18ex]100.0%	60.85 [-0.18ex]100.0%	68.75 [-0.18ex]100.0%	64.4 [-0.18ex]100.0%	68.2 [-0.18ex]100.0%
Retain 23.5% Tokens
FastV (ECCV’24)	69.4 [-0.18ex]88.2%	55.1 [-0.18ex]90.6%	59.3 [-0.18ex]86.3%	60.3 [-0.18ex]93.6%	61.0 [-0.18ex]89.5%
VisionZip (CVPR’25)	74.6 [-0.18ex]94.8%	53.8 [-0.18ex]88.4%	62.4 [-0.18ex]90.8%	60.9 [-0.18ex]94.6%	62.9 [-0.18ex]92.3%
VTW (AAAI’25)	63.7 [-0.18ex]80.9%	48.0 [-0.18ex]78.9%	40.1 [-0.18ex]58.3%	58.5 [-0.18ex]90.8%	52.6 [-0.18ex]77.1%
DivPrune (CVPR’25)	64.7 [-0.18ex]82.2%	51.2 [-0.18ex]84.1%	60.1 [-0.18ex]87.4%	61.0 [-0.18ex]94.7%	59.3 [-0.18ex]86.9%

Table 3. Comparison with SOTA methods on Qwen3-VL under different retained token ratios. Each cell reports accuracy on the first line and relative performance on the second line. Best results are boldfaced, and second-best results are underlined within each retained token ratio.

Variant	$\boldsymbol{\lambda_{r}}$	$\boldsymbol{\lambda_{c}}$	$\boldsymbol{\lambda_{u}}$	$\boldsymbol{r}$	$\boldsymbol{c}$	$\boldsymbol{u}$	Acc.	$\boldsymbol{\Delta}$ vs Full
Random	0	0	0				40.1	$-10.6$
$r$ only	1	0	0	✓			44.6	$-6.1$
$c$ only	0	1	0		✓		43.1	$-7.6$
$u$ only	0	0	1			✓	42.4	$-8.3$
$r+c$	1	1	0	✓	✓		48.6	$-2.1$
$r+u$	1	0	1	✓		✓	41.8	$-8.9$

Table 4. Ablation study of each component and its corresponding weighting coefficient in RI-Prune. Best result is boldfaced.

Method	Retain Ratio	TFLOPs	Latency	Avg. Acc.
Qwen2.5-VL	100.0%	57.8	8.135s	63.1
FastV	4.6%	11.4	7.728s	44.9
DivPrune	4.6%	8.8	8.074s	48.4
SCOPE	4.6%	8.9	7.953s	48.7
VisionZip	4.6%	8.4	5.217s	46.8
VTW	4.6%	8.5	5.125s	22.9

Table 5. Efficiency and accuracy comparison under different retained token ratios. Results are reported on Qwen2.5-VL.

Selection Mode	Accuracy	Selection Overhead (s)	TFLOPs
Global Top- $K$	48.8	0.909	7.9
Global $+$ min quota	50.3	0.922	7.9
Per-image Top- $K$	50.7	0.920	7.9

Table 6. Comparison of token selection modes on Qwen2.5-VL at the

4.6\%

retained-token setting. Selection overhead is measured in seconds, excluding decoder forward computation.

$P$	2	4	8	16	64	128
Accuracy	45.6	46.9	47.7	50.7	51.0	48.7
Compute (s)	0.0034	0.0035	0.0038	0.0044	0.0082	0.0116

Table 7. Performance comparison according to the number of prototypes

P

on Qwen2.5-VL at a 4.6% retained token budget.

Benchmark	Vanilla	FastV	DivPrune	SCOPE	VisionZip	VTW	RI-Prune
Video-MME	63.0	49.3	52.2	47.8	44.9	23.2	52.9
TempCompass	67.0	59.8	63.0	59.0	55.0	33.0	61.0

Table 8. Accuracy comparison on video benchmarks. Acc. denotes accuracy (%). Among pruning methods, the best results are boldfaced and the second-best results are underlined.

Visual Token Pruning

Visual token pruning reduces the image and video patches processed by transformers to lower inference cost while preserving task-relevant information.

See all 288 papers

Multi-Image Visual Token Pruning in Large Visual Language Models
Aug 27, 2026Rongyang Zhang, Chengqiang Lu, Cong Li +9Efficient VLM Inference Large Vision-Language Models
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · Xiaohongshu Inc.
LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models
Apr 27, 2026Rinyoichi Takezoe, Yaqian Li, Zihao Bo +3Efficient VLM Inference Attention Sinks
Li Auto Inc.
When Text Matters: Design Principles for Visual Token Pruning in Vision-Language Model
Sep 28, 2026Minchan Kang, Kyeonghye Park, Seoyoung Cho +2Efficient VLM Inference Visual Token Pruning
Korea Advanced Institute of Science and Technology (KAIST) · Hanbat National University

Aug 27, 2026J/K move · Enter open · S save

Multi-Image Visual Token Pruning in Large Visual Language Models

Rongyang Zhang, Chengqiang Lu, Cong Li, Hongchao Gu+8

State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · Xiaohongshu Inc.

Efficient VLM Inference Large Vision-Language Models Visual Token Pruning

Open paper AskPDF arXiv

With the growing demand for processing multiple image sequences in real-world applications, various visual token pruning methods have emerged to mitigate the computational and context length constraints faced by Large Vision Language Models (LVLMs). However, most existing pruning approaches rely on static strategies that struggle to adapt across different architectural LVLMs and multi-image scenarios, and are additionally constrained by their dependence on attention computations that are incompatible with efficient techniques like FlashAttention. To address these limitations, we propose a training-free, Adaptive Visual Token Pruning (AVTP) framework, applicable to diverse LVLM architectures. We strategically determine pruning layers based on empirical analysis of visual attention distributions across various LVLMs, and implement adaptive pruning ratios in multi-image contexts where images of higher importance retain proportionally more tokens. We conduct extensive experiments across different LVLMs to demonstrate the effectiveness and robustness of AVTP. Specifically, Qwen3VL-8B achieves 2 times inference speedup while maintaining 96.1% of its original accuracy on multiple multi-image benchmarks, InternVL3.5-8B retains 94.1% accuracy, and LLaVA-OV-7B even exceeds its original baseline performance. Our code is available at this link.

Abstract

Visual Token Pruning

Explore similar work

Multi-Image Visual Token Pruning in Large Visual Language Models

LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models

When Text Matters: Design Principles for Visual Token Pruning in Vision-Language Model