E2S-Pruner: Progressive Two-Stage Evidence Fusion for Visual Token Pruning in Vision-Language Models

Taoyu Qian; Qi Wang; Daqian Shi; Yuanhao Jiang; Shang Gao; Hualong Yu

cs.CVAug 24, 2026

E2S-Pruner: Progressive Two-Stage Evidence Fusion for Visual Token Pruning in Vision-Language Models

Authors: Taoyu Qian, Qi Wang, Daqian Shi, Yuanhao Jiang, Shang Gao, Hualong Yu

Organizations: School of Computer Science and Engineering, Jiangsu University of Science and Technology · Digital Environment Research Institute (DERI), Queen Mary University of London · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · National Institute of Education, Nanyang Technological University

Abstract

Vision-language models typically encode an image into hundreds of visual tokens, incurring substantial inference latency and GPU memory overhead. Existing pruning methods largely rely on attention scores and directly aggregate outputs across attention heads and network layers, making it difficult to characterize evidential uncertainty and conflict. We propose E2S-Pruner, a progressive two-stage evidence-fusion framework for visual token pruning that requires no auxiliary model, trainable parameters, or fine-tuning. In the first stage, E2S-Pruner treats each attention head as an independent evidence source, estimates its reliability from evidence clarity and inter-head consistency, and represents each visual token using three states: important, unimportant, and uncertain. In the second stage, Dempster--Shafer evidence theory is used to quantify inter-layer conflict and fuse complementary evidence from multiple network layers. We further introduce a spatial novelty constraint that promotes coverage of distinct image regions and prevents the retained tokens from concentrating in a few locally salient areas. On LLaVA-1.5-7B, E2S-Pruner retains 98.0%, 96.8%, and 90.6% of the aggregate performance when the average numbers of retained visual tokens are 192, 128, and 64, respectively, while improving throughput by 1.96x and 2.09x under the 128-token and 64-token settings. Experiments on Qwen2-VL-7B further demonstrate cross-model generalization. Code is available at https://github.com/taoyu-qian/E2S-Pruner.git.

Figures & tables

Figure 1: Motivation and overview of E2S-Pruner. Direct aggregation can suppress tokens supported by complementary evidence across layers, whereas the proposed two-stage fusion explicitly models evidential uncertainty and conflict.

Figure 2: Radar-chart comparison of E2S-Pruner on LLaVA-1.5-7B across the evaluated benchmarks, showing the balance between task performance and aggressive visual-token reduction.

m_{l,n}(I)=\rho_{l,n}\bar{p}_{l,n},\quad m_{l,n}(U)=\rho_{l,n}(1-\bar{p}_{l,n}),\quad m_{l,n}(\Theta)=1-\rho_{l,n}

Algorithm 1 E2S-Pruner

Figure 3: Reliability-aware fusion of multi-head evidence. Clear and consistent head responses receive high reliability, whereas ambiguous or conflicting responses contribute greater uncertainty.

Figure 4: Spatial novelty regularization. The visual-token grid is partitioned into spatial cells, and the highest-priority candidate in each occupied cell receives a soft bonus before global Top- $K$ selection.

Figure 5: Qualitative visualization of final-layer visual-token retention under average budgets of 192, 128, and 64 tokens across representative TextVQA, ScienceQA, and MMBench examples. Green markers denote retained tokens, whereas red points indicate pruned patch locations.

Method	GQA	SQA	TextVQA	POPE	MME	MMBench	Avg
Vanilla 576 Tokens
LLaVA-1.5-7B	61.9	69.5	58.2	85.9	1862	64.6	100.0%
Retain 192 Tokens ( $\downarrow 66.7\%$ )
ToME (ICLR’23)	54.3	65.2	52.1	72.4	1563	60.5	88.8%
FastV (ECCV’24)	52.7	67.3	52.5	64.8	1612	61.2	88.2%
HiRED (AAAI’25)	58.7	68.4	47.4	82.8	1737	62.8	93.6%

Table 1: Performance comparison of visual-token pruning methods on LLaVA-1.5-7B, with average scores normalized to the dense baseline.

Figure 6: Qualitative comparison of visual-token retention patterns and model outputs for representative test examples, illustrating how different pruning methods preserve task-relevant image regions.

Variant	SQA	TextVQA	MME	MMBench	Avg.
LLaVA-1.5-7B	69.5	58.2	1862	64.6	100.0%
A1: Vanilla Attention Top- $K$	64.9	55.8	1692	59.2	92.9%
A2: w/o Head Reliability	68.0	55.8	1769	61.1	95.8%
A3: w/o Layer-wise Fusion	68.9	55.8	1763	61.4	96.2%
A4: w/o Conflict Protection	68.9	55.9	1781	62.3	96.8%
A5: w/o Spatial Novelty	68.9	55.9	1774	61.2	96.3%

Table 2: Ablation study of the proposed E2S components.

Schedule	SQA	TextVQA	MME	MMBench	Avg	Throughput
$576\rightarrow 122\rightarrow 48\rightarrow 42$	69.9	56.0	1779	63.5	97.7%	10.37
$576\rightarrow 132\rightarrow 40\rightarrow 32$	69.9	55.9	1783	63.8	97.8%	10.57
$576\rightarrow 142\rightarrow 32\rightarrow 22$	70.1	55.6	1784	63.7	97.7%	10.70
$576\rightarrow 147\rightarrow 38\rightarrow 12$	69.9	55.6	1782	63.9	97.7%	10.41
$576\rightarrow 152\rightarrow 44\rightarrow 2$	69.9	55.8	1785	63.7	97.7%	10.71

Table 3: Ablation study of different visual-token retention schedules.

Method	LLM Generation $\downarrow$ Latency (s)	Total Latency $\downarrow$ (s)	GPU Peak $\downarrow$ Memory (MB)	Throughput $\uparrow$ (item/s)	Accuracy $\uparrow$
LLaVA-1.5-7B	676.11	820.15	15566	5.28	64.6
Avg. Retention 128 Tokens ( $\downarrow 77.8\%$ )
FastV (ECCV’24)	334.87 ( $\downarrow 50.5\%$ )	450.09 ( $\downarrow 45.1\%$ )	15558 ( $\downarrow 0.1\%$ )	9.73 ( $\uparrow 1.84\times$ )	56.1 (86.8%)
SparseVLM (ICML’25)	322.43 ( $\downarrow 52.3\%$ )	435.72 ( $\downarrow 46.9\%$ )	19229 ( $\uparrow 23.5\%$ )	10.05 ( $\uparrow 1.90\times$ )	60.0 (92.9%)
V2Drop (CVPR’26)	410.02 ( $\downarrow 39.4\%$ )	605.69 ( $\downarrow 26.1\%$ )	15046 ( $\downarrow 3.3\%$ )	7.23 ( $\uparrow 1.37\times$ )	61.8 (95.7%)
E2S-Pruner (Ours)	312.48 ( $\downarrow 53.8\%$ )	422.28 ( $\downarrow 48.5\%$ )	18847 ( $\uparrow 21.1\%$ )	10.37 ( $\uparrow 1.96\times$ )	63.5 (98.3%)

Table 4: Efficiency and performance comparison on LLaVA-1.5-7B, where percentages denote changes or performance retention relative to the dense baseline.

Figure 7: Performance–throughput trade-off on LLaVA-1.5-7B under different average visual-token budgets, showing the practical acceleration achieved as the retained sequence becomes shorter.

Method	SQA	POPE	MME	AI2D	Avg.
Qwen2-VL-7B	84.7	86.1	2317	80.5	100.0%
Token Reduction ( $\downarrow 66.7\%$ )
FastV (ECCV’24)	80.0	82.1	2130	76.1	94.1%
DART (EMNNLP’25)	81.4	83.9	2245	78.0	96.8%
V2Drop (CVPR’26)	81.6	87.2	2224	78.0	97.6%
E2S-Pruner (Ours)	83.8	86.8	2228	80.2	98.9%

Table 5: Comparison on Qwen2-VL-7B under different token reduction ratios.

Visual Token Pruning

Visual token pruning reduces the image and video patches processed by transformers to lower inference cost while preserving task-relevant information.

See all 278 papers

LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models
Apr 27, 2026Rinyoichi Takezoe, Yaqian Li, Zihao Bo +3Efficient VLM Inference Attention Sinks
Li Auto Inc.
LearnPrunerV2: Learnable Progressive Token Pruning for Vision-Language Models
Oct 27, 2026Rinyoichi Takezoe, Yaqian Li, Weiren Wang +6Visual Token Pruning LVLM Pruning
Li Auto Inc. Beijing, China · Li Auto Inc. Shanghai, China · Li Auto Inc. Hangzhou, China
Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Models
Mar 6, 2026Jialuo He, Huangxun ChenEfficient VLM Inference Visual Token Pruning
The Hong Kong University of Science and Technology (Guangzhou)

Apr 27, 2026J/K move · Enter open · S save

LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models

Rinyoichi Takezoe, Yaqian Li, Zihao Bo, Anzhou Hou+2

Li Auto Inc.

Efficient VLM Inference Attention Sinks Visual Token Pruning

Open paper AskPDF arXiv

Vision-Language Models (VLMs) have recently demonstrated remarkable capabilities in visual understanding and reasoning, but they also impose significant computational burdens due to long visual sequence inputs. Recent works address this issue by pruning unimportant visual tokens, achieving substantial computational reduction while maintaining model performance. The core of token pruning lies in determining token importance, with current approaches primarily relying on attention scores from vision encoders or Large Language Models (LLMs). In this paper, we analyze the effectiveness of attention mechanisms in both vision encoders and LLMs. We find that vision encoders suffer from attention sink, leading to poor focus on informative foreground regions, while in LLMs, although prior studies have identified attention bias toward token positions, text-to-vision attention demonstrates resistance to this bias and enables effective pruning guidance in middle layers. Based on these observations, we propose LearnPruner, a two-stage token pruning framework that first removes redundant vision tokens via a learnable pruning module after the vision encoder, then retains only task-relevant tokens in the LLM's middle layer. Experimental results show that our LearnPruner can preserve approximately 95% of the original performance while using only 5.5% of vision tokens, and achieve 3.2

\times

inference acceleration, demonstrating a superior accuracy-efficiency trade-off.

Abstract

Visual Token Pruning

Explore similar work

LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models

LearnPrunerV2: Learnable Progressive Token Pruning for Vision-Language Models

Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Models