Organizations: Morgan Stanley · Clemson University · Arizona State University · Washington University in St. Louis · Texas A&M University · Rice University · Florida State University
Multi-modal large language models (MLLMs) achieve strong visual-language reasoning but suffer from high inference cost due to redundant visual tokens. Recent work explores visual token pruning to accelerate inference, while existing pruning methods overlook the underlying distributional structure of visual representations. We propose OTPrune, a training-free framework that formulates pruning as distribution alignment via optimal transport (OT). By minimizing the 2-Wasserstein distance between the full and pruned token distributions, OTPrune preserves both local diversity and global representativeness while reducing inference cost. Moreover, we derive a tractable submodular objective that enables efficient optimization, and theoretically prove its monotonicity and submodularity, providing a principled foundation for stable and efficient pruning. We further provide a comprehensive analysis that explains how distributional alignment contributes to stable and semantically faithful pruning. Comprehensive experiments on wider benchmarks demonstrate that OTPrune achieves superior performance-efficiency tradeoffs compared to state-of-the-art methods. The code is available at https://github.com/xiwenc1/OTPrune.
Figures & tables
Figure 1: Overview of the proposed OTPrune framework. Given visual and textual inputs, a vision encoder with a projector produces image tokens that are fed into a large language model together with text tokens and reasoning prompts. Existing methods, such as DivPrune , select tokens by maximizing diversity, which may overlook global representativeness. In contrast, OTPrune formulates pruning as distribution alignment by minimizing the 2-Wasserstein (optimal transport) distance between the full and pruned token distributions, thereby preserving both local diversity and global structure for efficient and semantically faithful multimodal reasoning.
Figure 2: Correlation between OT distance and downstream performance. We evaluate several manually designed selection strategies ( First-K , Last-K , Uniform , and Random ) along with the SOTA method DivPrune [ 3 ] . For each strategy, we compute the OT distance between the selected subset and the full token set and measure downstream performance across 11 multimodal benchmarks. Both OT distance (lower distance ⇒ lower rank) and task performance (higher score ⇒ lower rank) are ranked to compute Spearman’s correlation. A strong positive correlation confirms that smaller OT distance corresponds to higher performance.
Method
TFLOP
COCO
Flickr
GQA
MMB
MME
MMMU
Nocaps
OKVQA
POPE
SQA
SEEDB
Avg.
One-shot
(ratio %)
CIDEr
CIDEr
EM
Acc
P-score
Acc
CIDEr
EM
F1
EM
Acc
Rank ↓
Rank ↓
LLaVA 1.5-7B
Original
3.228 (100.00)
1.10
0.75
61.96
64.09
1506
36.44
1.06
53.39
85.84
69.41
66.17
–
–
Category: One-shot Pruning Methods
VTW [ 30 ]
0.603 (18.46)
0.05
0.03
38.94
21.31
681
32.60
0.03
18.64
25.35
65.29
36.13
8.23
3.41
FastV [ 8 ]
0.514 (15.69)
0.06
0.03
38.73
20.62
696
32.00
0.04
18.32
32.84
65.15
35.69
8.41
3.59
DivPrune
0.512 (15.63)
0.96
0.62
56.85
59.19
1328
35.89
0.92
46.98
86.02
68.27
59.47
3.23
1.64
Table 1: Comparison of OTPrune with diverse baselines across 11 multimodal benchmarks. DivPrune ∗ aligns its token retention rate with PruMerge. For fair evaluation, all one-shot methods strictly retain 9.8% of visual tokens, using TFLOPs to indicate computational efficiency. The Avg. Rank ↓ assesses all evaluated methods, while the One-shot Rank ↓ strictly compares one-shot techniques. Our OTPrune consistently delivers the best performance. ( ∙ : requires finetuning; △ : requires calibration).
Configuration
Performance
m
d
k
Method
Win Rate
Optimality Ratio
20
10
5
DivPrune
60.14
74.53
OTPrune
94.56
87.81
30
20
10
DivPrune
84.62
76.48
OTPrune
99.49
93.98
100
50
30
DivPrune
88.77
92.20
Table 2: Comparison between OTPrune and DivPrune on synthetic datasets with varying numbers of samples ( m ), feature dimensions ( d ), and selected subset sizes ( k ). The first two configurations are evaluated exhaustively over all (km) combinations, while the last three configurations use Monte Carlo estimation with 10 million different subsets to approximate the full combinatorial space.
Comparison
p -value
OTPrune vs DivPrune
0.028
Table 5
Figure 3: Comparison with diversity-based methods using LLaVA 1.5-13B. We evaluate OTPrune , DivPrune [ 3 ] , DPP [ 22 ] , and Random sampling across 11 multimodal benchmarks under different pruning ratios {0.05,0.098,0.15,0.2} . We report both absolute performance and relative performance , defined as the ratio between pruned and original model performance ( pruned/original ). We also compute the OT distance between each subset and the full token set and visualize the relative distance normalized by OTPrune, i.e., f(Cmethod)/f(COTPrune) .
Figure 4: Relative OT distance comparison across pruning methods. We evaluate OTPrune , DivPrune [ 3 ] , DPP [ 22 ] , and Random sampling using LLaVA 1.5-13B across 11 multimodal benchmarks under different token ratios {0.05,0.098,0.15,0.2} . Each bar shows the relative OT distance normalized by OTPrune, i.e., f(Cmethod)/f(COTPrune) , where lower values indicate closer alignment with the original token distribution. OTPrune consistently achieves the smallest OT distance across all ratios, demonstrating superior distributional fidelity. The last panel reports the overall average across all 11 datasets.
Figure 5: Sensitivity analysis of the balancing coefficient γ .
Visual token pruning reduces the inference overhead of multimodal large language models (MLLMs) by retaining only a subset of visual tokens. Existing methods usually select tokens based on importance or redundancy. However, we observe that these criteria produce stable spatial biases across inputs and do not always outperform simple Uniform Grid sampling, highlighting the value of broad spatial coverage. Motivated by this, we propose S2Prune, a training-free pruning method that preserves spatial coverage while adapting token density to local image structure. We first divide the image into regions and assign at least one token to each region to preserve coverage. The remaining token budget is then distributed according to Laplacian variation, giving more tokens to regions with richer structure. We then use Early Representation Change (ERC), computed from the first decoder block, to select representative tokens within each region. We evaluate S2Prune across diverse settings and two MLLM architectures. On Qwen2.5-VL-7B-Instruct, it achieves the highest average accuracy among the evaluated training-free pruning methods. With only 32 of the original 576 visual tokens, it still retains 79.3% of the full-model performance. Code is available at https://github.com/yuanyuanjia71-spec/S2Prune.
Yuanyuan Jia, Shunpu Tang, Qianqian Yang
College of Information Science and Electronic Engineering, Zhejiang University Hangzhou, China
Despite their strong multimodal understanding ability, multimodal large language models (MLLMs) incur substantial computational overhead when processing long visual token sequences. To reduce inference costs, recent studies have explored visual token pruning through vision-centric or text-guided strategies. However, these methods often overlook high-norm outlier tokens, i.e., tokens with abnormally large feature norms, leading to suboptimal pruning decisions. In this work, we show that such high-norm outlier tokens are highly redundant in both feature and spatial dimensions, yet are often mistakenly preserved as informative cues by existing methods. Motivated by this observation, we propose SinkPruner, a training-free visual token pruning framework for efficient MLLM inference. SinkPruner follows a coarse-to-fine design with two key modules: a visual sanitizer that filters high-norm redundancies and alleviates attention sink and attention dispersion, and a text-guided pruner that further retains tokens semantically aligned with the text query. Extensive experiments on twelve image-language and four video-language benchmarks demonstrate the effectiveness, efficiency, and generalizability of our framework. Notably, SinkPruner preserves 96.5% (91.8%) of the original performance of LLaVA-1.5 (Qwen2.5-VL) under an 89% token reduction. Experiments further indicate that our visual sanitizer exhibits promising transferability in enhancing the performance of existing pruning methods. Our code is available at https://github.com/LaVi-Lab/SinkPruner.
Shiyu Li, Zi-Yuan Hu, Shijia Huang +3
1The Chinese University of Hong Kong · 2Weitu AI · 3Peking University
Visual token pruning, which aims to compress and prune redundant visual tokens, plays a critical role in efficient inference with large vision-language models (LVLMs). However, existing methods fail to disentangle intra-modal visual redundancy from cross-modal redundancy between vision and language. We show that visual token diversity and task-specific token relevance are two crucial yet orthogonal factors that complement each other in conveying useful information and should therefore be treated separately for more effective visual token pruning. Building upon this insight, we design TODRE, a two-stage and training-free framework that incorporates Token Diversity and task RElevance for effective token compression and efficient LVLM inference. Instead of pruning redundant tokens, we introduce a greedy max-sum diversification algorithm that selects and retains a subset of diverse and representative visual tokens after the vision encoder. On top of that, ToDRE leverages an ``information migration'' mechanism to eliminate task-irrelevant visual tokens within certain decoder layers of the large language model (LLM), further improving token pruning and LVLM inference. Extensive experiments show that ToDRE prunes 90% of visual tokens after the vision encoder as well as all visual tokens in certain LLM decoder layers, leading to a 2.6x speed-up in total inference time while maintaining 95.0% model performance plus excellent model compatibility. The code is available at: this https URL.
Duo Li, Zuhao Yang, Xiaoqin Zhang +2
CCDS, NTU, Singapore · CCST, ZJUT, China · Terminus AI Lab, UCAS, China