ReDiPrune: Relevance-Diversity Pre-Projection Token Pruning for Efficient Multimodal LLMs

An Yu; Ting Yu Tsai; Zhenfei Zhang; Weiheng Lu; Felix X. -F. Ye; Ming-Ching Chang

cs.CVMar 25, 2026

ReDiPrune: Relevance-Diversity Pre-Projection Token Pruning for Efficient Multimodal LLMs

Authors: An Yu, Ting Yu Tsai, Zhenfei Zhang, Weiheng Lu, Felix X. -F. Ye, Ming-Ching Chang

Organizations: University at Albany, SUNY · Peking University

Abstract

Recent multimodal large language models are computationally expensive because Transformers must process a large number of visual tokens. We present ReDiPrune, a training-free token pruning method applied before the vision-language projector, where visual features remain rich and discriminative. Unlike post-projection pruning methods that operate on compressed representations, ReDiPrune selects informative tokens directly from vision encoder outputs, preserving fine-grained spatial and semantic cues. Each token is scored by a lightweight rule that jointly consider text-conditioned relevance and max-min diversity, ensuring the selected tokens are both query-relevant and non-redundant. ReDiPrune is fully plug-and-play, requiring no retraining or architectural modifications, and can be seamlessly inserted between the encoder and projector. Across four video and five image benchmarks, it consistently improves the accuracy-efficiency trade-off. For example, on EgoSchema with LLaVA-NeXT-Video-7B, retaining only 15% of visual tokens yields a +2.0% absolute accuracy gain while reducing computation by more than $6\times$ in TFLOPs. Code is available at https://github.com/UA-CVML/ReDiPrune.

Figures & tables

Figure 1: Comparison of pruning strategies in MLLMs: (a) Post-projection pruning selects diverse tokens but ignores text and loses fine details. (b) Text-guided post-projection pruning improves query relevance but discards original visual cues. (c) ReDiPrune prunes tokens before projection using weighted query embeddings, preserving details while balancing accuracy and efficiency.

Figure 2: Overview of ReDiPrune: Given a prompt $p$ , we build a normalized weighted query vector $\hat{q}$ ( $\lx@sectionsign$ 3.2 ). For each frame $I^{(n)}$ , the vision encoder produces tokens $E_{V}^{(n)}$ . ReDiPrune operates before the projector $P(\cdot)$ in the visual feature space, computing text relevance $s_{i}$ and cosine dissimilarity $D_{ij}$ ( $\lx@sectionsign$ 3.3 ). With a per-frame budget $k_{n}$ , we greedily select $k_{n}$ tokens by optimizing a relevance-diversity score $\min_{u\in S}D_{ui}+\alpha s_{i}$ ( $\lx@sectionsign$ 3.4 ), where $u$ indexes tokens already selected in $S$ . The retained tokens $\tilde{E}_{V}^{(n)}$ are projected as $Z_{V}^{(n)}$ and concatenated with text embeddings for decoding by the LLM $f_{\theta}$ , reducing redundancy while preserving query-relevant information.

Figure 3: Qualitative examples from the TGIF dataset [ 6 ] using Video-LLaVA-7B [ 9 ] . For each question, we show the ground-truth (GT) answer and responses from the original model, DivPrune [ 1 ] , CDPruner [ 28 ] , and ReDiPrune. ReDiPrune accurately captures action cues, demonstrating stronger semantic grounding and temporal understanding compared with competing pruning methods.

Figure 4: Sharper, query-aligned attention with ReDiPrune. Frame-wise attention visualization and distribution for different pruning methods on a NextQA [ 23 ] sample. (Left) Each method (Original, DivPrune [ 1 ] , CDPruner [ 28 ] , and ReDiPrune) processes the same query; eight frames (f0-f7) and their attention scores (in $\times 10^{-3}$ ) are shown in descending order of importance. (Right) Histogram of attention across frames. ReDiPrune produces sharper, query-focused attention on the most relevant frames and a more concentrated distribution, indicating stronger alignment with query.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 472 papers

S $^2$ Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models
Sep 1, 2026Yuanyuan Jia, Shunpu Tang, Qianqian YangEfficient VLM Inference Multimodal Large Language Models
College of Information Science and Electronic Engineering, Zhejiang University Hangzhou, China
SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models
Jul 28, 2026Yuchen Wang, Qihui Zhu, Yang Liu +2Efficient Multimodal Inference Multimodal Large Language Models
MoE Key Lab of BIPC, NEL-BITA, University of Science and Technology of China, Hefei, China · ChangXin Memory Technologies, Hefei, China · APKL of BIIP, IAI, Hefei Comprehensive National Science Center, Hefei, China
SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models
Sep 1, 2026Shiyu Li, Zi-Yuan Hu, Shijia Huang +3Efficient VLM Inference Multimodal Large Language Models
1The Chinese University of Hong Kong · 2Weitu AI · 3Peking University

Sep 1, 2026J/K move · Enter open · S save

S $^2$ Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models

Yuanyuan Jia, Shunpu Tang, Qianqian Yang

College of Information Science and Electronic Engineering, Zhejiang University Hangzhou, China

Efficient VLM Inference Multimodal Large Language Models Visual Token Pruning LVLM Pruning

Open paper AskPDF arXiv

Visual token pruning reduces the inference overhead of multimodal large language models (MLLMs) by retaining only a subset of visual tokens. Existing methods usually select tokens based on importance or redundancy. However, we observe that these criteria produce stable spatial biases across inputs and do not always outperform simple Uniform Grid sampling, highlighting the value of broad spatial coverage. Motivated by this, we propose S

^2

Prune, a training-free pruning method that preserves spatial coverage while adapting token density to local image structure. We first divide the image into regions and assign at least one token to each region to preserve coverage. The remaining token budget is then distributed according to Laplacian variation, giving more tokens to regions with richer structure. We then use Early Representation Change (ERC), computed from the first decoder block, to select representative tokens within each region. We evaluate S

^2

Prune across diverse settings and two MLLM architectures. On Qwen2.5-VL-7B-Instruct, it achieves the highest average accuracy among the evaluated training-free pruning methods. With only 32 of the original 576 visual tokens, it still retains 79.3% of the full-model performance. Code is available at https://github.com/yuanyuanjia71-spec/S2Prune.

Abstract

Efficient VLM Inference

Explore similar work

S2^22Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models

SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models

SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models

S $^2$ Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models