EvoPrune: Early-Stage Visual Token Pruning for Efficient MLLMs

Yuhao Chen; Bin Shan; Xin Ye; Cheng Chen

cs.CVMar 4, 2026

EvoPrune: Early-Stage Visual Token Pruning for Efficient MLLMs

Authors: Yuhao Chen, Bin Shan, Xin Ye, Cheng Chen

Organizations: ByteDance, Beijing, China

Abstract

Multimodal Large Language Models (MLLMs) have shown strong performance in vision-language tasks, but their inference efficiency is severely limited by the exponential growth of visual tokens in complex scenarios such as high-resolution images and videos. Existing visual token pruning methods mainly operate after visual encoding, overlooking the substantial computational cost incurred during the encoding stage. To address this issue, we propose EvoPrune, an early-stage visual token pruning method for MLLMs that performs pruning directly during visual encoding. Specifically, EvoPrune employs a layer-wise pruning strategy guided by token similarity, diversity, and attention-based importance to retain the most informative visual tokens at selected encoding layers. Extensive experiments on image and video benchmarks validate the effectiveness of EvoPrune. In particular, on the VideoMME dataset, EvoPrune achieves 2 $\times$ inference speedup with less than 1% performance degradation, demonstrating its potential for latency-sensitive MLLM deployment.

Figures & tables

Figure 1: Inference time of different components in MLLMs with and without visual Pruning under different input scales.

Figure 2: Overview of the EvoPrune framework. Token merging operations are applied at selected visual encoder layers. A composite score matrix integrating semantic similarity, diversity, and attention-based importance, guides token pair selection.

Method	Benchmarks $\uparrow$					Relative Latency $\downarrow$			Avg. $\uparrow$
Metric	VQA V2	MME	MMB EN	MMB CN	MMVet	pre-LLM	LLM	Overall	Acc.	Rel.
Upper Bound, All 576 Tokens (100%)
Native	78.5	1508	61.8	58.1	32.2	1.00	1.00	1.00	76.5	100.0%
Retain 128 Tokens ( $\downarrow$ 77.8%)
FasterVLM	75.5	1353	58.3	54.8	33.1	1.15	0.77	0.85	72.3	94.6%
VisPruner	75.8	1405	58.8	55.2	32.8	1.32	0.77	0.90	73.2	95.7%

Table 1: Comparison of state-of-the-art methods on image-based benchmarks. Relative Latency represents the normalized time breakdown of the pre-LLM, LLM, and Overall stages with respect to the Native(Unpruned). Here, Acc. denotes the average performance across all benchmarks, while Rel. indicates the mean relative accuracy (%) compared to the native baseline.

Method	MVBench $\uparrow$	LongVideoBench $\uparrow$			Video-MME $\uparrow$				Relative Latency $\downarrow$			Avg. $\uparrow$
Metric	test	val	perception	relation	w/o sub	short	medium	long	pre-LLM	LLM	Overall	Acc.	Rel.
Upper Bound, All 64 $\times$ 169 Tokens (100%)
Native	58.0	56.0	64.0	48.9	59.7	71.3	58.0	49.7	1.00	1.00	1.00	57.9	100.0%
Retain 64 $\times$ 64 Tokens ( $\downarrow$ 62.1%)
FasterVLM	58.3	54.9	61.6	49.0	59.4	70.1	57.4	50.6	1.01	0.45	0.77	57.5	99.4%
VisPruner	58.3	54.9	61.6	49.0	59.4	70.1	57.4	50.6	1.01	0.45	0.77	57.5	99.4%

Table 2: Comparison of state-of-the-art methods on video-based benchmarks. Relative Latency represents the normalized time breakdown of the pre-LLM, LLM, and Overall stages with respect to the Native(Unpruned). Here, Acc. denotes the average performance across all benchmarks, while Rel. indicates the mean relative accuracy (%) compared to the native baseline.

Figure 3: Breakdown of Time-To-First-Token (TTFT, 1 unit = 10 ms). The Visual Encoder , Other , and LLM Backbone correspond to the visual encoder, intermediate processing modules (e.g., pooling), and the language model backbone, respectively.

Method	MVBench	LongVideoBench	VideoMME	Avg.	Rel.
Native	58.0	56.0	59.7	57.9	100.0%
EvoPrune (w/o Attn&Div)	54.6	54.3	59.0	55.9	96.5%
EvoPrune (w/o Attn)	55.5	54.9	59.7	56.7	97.9%
EvoPrune (w/o Div)	56.2	55.2	60.4	57.3	98.9%
EvoPrune	56.9	55.5	60.6	57.7	99.7%

Table 3: Ablation study on score factors. We evaluate the contribution of each component—Attention Preservation and Diversity Penalty—by disabling them individually. Results are reported on MVBench, LongVideoBench, and VideoMME. Avg. denotes the mean score across benchmarks, and Rel. is the accuracy relative to the unpruned model.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Performance comparison of different layer-wise merging strategies across video benchmarks We present the Accuracy (top row) and Encoding Time (bottom row) as functions of the retained token budget

B\in\{16,32,64\}

. Each curve represents a specific layer-wise allocation pattern. Specifically, we set the window size

N=13

for First and Last (out of 26 total layers), and the growth/decay rate

\alpha=1

for Increasing and Decreasing .

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs
Jun 30, 2026Yuhao Wang, Mu Qiao, Haiwen Diao +5Efficient VLM Inference Multimodal Large Language Models
School of Future Technology, Dalian University of Technology, Dalian 116024, China · S-Lab, Nanyang Technological University, Singapore 639798 · OPPO Research Institute, Shenzhen 518000, China +1
Structured Redundancy Modeling for Efficient Visual Token Pruning in High-Resolution MLLMs
Jul 25, 2026Jouwon Song, Woohyeong Kim, Kyeongbo KongEfficient VLM Inference Large Vision-Language Models
LG Electronics, Seoul, Republic of Korea · Pusan National University, Busan, Republic of Korea
StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Models
Sep 12, 2026Hansen Zhang, Landi He, Mingde Yao +1Efficient VLM Inference Visual Token Pruning
Shenzhen University of Advanced Technology, Shenzhen, China · School of Computer Science, Nanjing University, Nanjing, China · CUHK MMLab, CPII under InnoHK, Hong Kong, China

Jun 30, 2026J/K move · Enter open · S save

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs

Yuhao Wang, Mu Qiao, Haiwen Diao, Yunzhi Zhuge+4

School of Future Technology, Dalian University of Technology, Dalian 116024, China · S-Lab, Nanyang Technological University, Singapore 639798 · OPPO Research Institute, Shenzhen 518000, China+1

Efficient VLM Inference Multimodal Large Language Models Multimodal Token Compression Visual Token Pruning

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) incur prohibitive inference costs due to long visual token sequences. Training-free visual token reduction provides an efficient solution. However, existing methods distort attention distributions, giving rise to a phenomenon we term Attention Logit Collapse. To address this issue, we propose ERA, an Entropy-guided visual token pruning framework with Rectified Attention for efficient MLLMs. Specifically, ERA comprises three crucial components: Dual-view Entropy Pruning (DEP), Bias-aware Token Recycling (BTR), and Logit-preserving Attention Rectification (LAR). First, DEP identifies representative anchor tokens by jointly modeling visual diversity and head-wise saliency. BTR then recycles pruned tokens into their corresponding anchors while estimating a cluster-level logit bias. Building upon this, LAR injects the estimated bias into attention logits, effectively rectifying the collapse induced by token reduction. Together, these components preserve visual evidence even under aggressive compression, enabling robust performance across single-image, multi-image, and video settings on a wide range of MLLMs. Beyond delivering practical acceleration, ERA establishes logit-preserving visual token pruning as a principled framework for efficient MLLMs, unifying theoretical foundation, algorithmic design, and practical deployment. The code is at https://github.com/924973292/ERA.

Abstract

Efficient VLM Inference

Explore similar work

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs

Structured Redundancy Modeling for Efficient Visual Token Pruning in High-Resolution MLLMs

StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Models