LearnPrunerV2: Learnable Progressive Token Pruning for Vision-Language Models

Rinyoichi Takezoe; Yaqian Li; Weiren Wang; Zihao Bo; Huihong Shi; Anzhou Hou; Mo Guang; Jiale Yan

cs.CVOct 27, 2026

LearnPrunerV2: Learnable Progressive Token Pruning for Vision-Language Models

Authors: Rinyoichi Takezoe, Yaqian Li, Weiren Wang, Zihao Bo, Huihong Shi, Anzhou Hou, Mo Guang, Jiale Yan,

Organizations: Li Auto Inc. Beijing, China · Li Auto Inc. Shanghai, China · Li Auto Inc. Hangzhou, China

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable multimodal capabilities but suffer from prohibitive computational overhead due to the massive number of visual tokens. While token pruning is a promising solution, existing methods still suffer from two major drawbacks: 1) unreliable pruning criteria that fail to accurately localize task-specific regions due to inherent attention biases or query-agnostic designs, and 2) suboptimal pruning strategies that employ rigid, non-adaptive schedules, overlooking the dynamic evolution of token redundancy across different model depths. To address these challenges, we propose LearnPrunerV2, which extends the previous two-stage heuristic framework (LearnPruner) into a fully learnable architecture. Specifically, we integrate lightweight learnable modules into every layer of the VLM backbone, which estimate token importance by fusing intrinsic visual saliency with extrinsic query relevance captured via a shadow token. By uncovering a consistent staircase redundancy pattern across VLM layers, we further introduce Budget-Deterministic Pruning (BDP) to establish a hardware-friendly schedule with fixed per-layer budgets. Extensive experimental results across various benchmarks and VLMs demonstrate that LearnPrunerV2 consistently outperform previous state-of-the-art methods. Notably, LearnPrunerV2 preserves 99.8% of the original performance when pruning 77.8% of visual tokens on LLaVA-v1.5, achieving a superior accuracy-efficiency trade-off.

Figures & tables

Figure 1. Visualization of cross-modal relevance scores in LLaVA-v1.5. ’Similarity’ denotes the similarity scores between CLIP multimodal features, while ’Layer 8/16 Attn’ represents the attention distribution of the last token within the LLM. Notably, CLIP features primarily capture isolated word-level semantics (e.g., the ’girl’); in contrast, intermediate LLM layers resolve global contextual semantics to accurately localize the intended target based on spatial relations. Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.

Figure 2. Overview of LearnPrunerV2. (a) Framework illustration, a learnable pruning module (LPM) is inserted into each decoder layer to evaluate token importance, and a learnable shadow token is appended after the user input to aggregate semantic context and capture cross-modal relevance. (b) Internal structure of the LPM. The visual hidden state is concatenated with the query-aware feature derived from the shadow token to predict the final importance score. Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.

Figure 3. Statistics of learned pruning trajectories. We visualize the per-layer token retention ratio obtained via threshold-based pruning on LLaVA-v1.5 and Qwen2.5-VL. The solid lines denote the mean values, while the shaded regions represent the variance across 500 randomly sampled image-text pairs from LLaVA-mix665k. Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.

Method	GQA	SQA	VQA T	POPE	MME	VQA v2	MMB	MMB CN	MMVet	Avg.
Upper Bound, 576 Tokens (100%)
Vanilla	61.9	69.5	58.2	85.9	1862	78.5	64.7	58.3	31.1	100.0%
Retain Averaged 128 Tokens ( $\downarrow$ 77.8% )
FastV	49.6	60.2	50.6	59.6	1490	61.8	56.1	51.4	28.1	83.0%
SparseVLM	56.0	67.1	54.9	80.5	1696	73.8	60.0	51.1	29.7	92.9%
VisPruner	58.2	69.1	57.0	84.6	1794	75.8	62.7	57.3	33.7	98.5%

Table 1. Performance comparisons of different pruning methods on LLaVA-v1.5-7B. Avg. denotes the performance relative to the vanilla model, averaged across all benchmarks. Bold and underlined values denote the best and second-best results.

Method	GQA	SQA	VQA T	POPE	MME	VQA v2	MMB	MMB CN	MMVet	Avg.
Upper Bound, 2,880 Tokens (100%)
Vanilla	64.2	70.2	61.3	86.3	1842	81.2	67.9	60.5	40.0	100.0%
Retain Averaged 640 Tokens ( $\downarrow$ 77.8% )
FastV	62.0	69.1	60.0	79.5	1807	79.5	65.8	53.5	33.3	94.4%
SparseVLM	60.3	67.7	57.8	85.3	1772	77.1	65.7	58.6	36.1	95.4%
DART	61.3	68.2	59.5	85.0	1741	78.3	64.9	57.1	36.9	95.7%

Table 2. Performance comparisons of different pruning methods on LLaVA-NeXT-7B.

Method	GQA	SQA	VQA T	MME	MMB	Avg.
Upper Bound, 1280 Tokens (100%)
Vanilla	60.9	75.9	83.3	2314	83.8	100.0%
Retain Averaged 426 Tokens ( $\downarrow$ 66.7% )
FastV	58.4	79.1	81.5	2330	81.4	99.2%
LearnPruner	59.3	77.4	81.1	2339	83.2	99.4%
LearnPrunerV2	60.0	81.4	80.7	2355	83.4	100.8%

Table 3. Performance comparisons of different pruning methods on Qwen2.5-VL-7B.

Table 4. Ablation study on different configurations of the learnable pruning module (LPM).

Table 8

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Method	TGIF-QA		MSVD-QA		MSRVTT-QA
Method	Acc.	Score	Acc.	Score	Acc.	Score
Upper Bound, 2,048 Tokens (100%)
Vanilla	18.1	1.27	64.1	3.41	56.1	2.97
Retain Averaged 456 Tokens ( $\downarrow$ 77.8% )
FastV	16.4	1.17	60.3	3.24	53.0	2.84
LearnPrunerV2	17.8	1.21	64.9	3.39	55.0	2.89

Appendix

Table 7. Video understanding performance comparisons of different pruning methods on Video-LLaVA-7B. Performance is evaluated on the first 1,000 samples of each benchmark. We employ GPT-4.1 to assist in scoring open-ended responses.

Figure 4. Performance comparisons of LearnPrunerV2 models trained with different proportion of the training dataset. Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.

Figure 5. Visualization of the pruning results. Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.

Figure 6. Visualization of the pruning results. Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.

Figure 7. Visualization of the score heatmap. Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.

Figure 8. Visualization of the score heatmap. Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.

Visual Token Pruning

Visual token pruning reduces the image and video patches processed by transformers to lower inference cost while preserving task-relevant information.

See all 276 papers

LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models
Apr 27, 2026Rinyoichi Takezoe, Yaqian Li, Zihao Bo +3Efficient VLM Inference Attention Sinks
Li Auto Inc.
Beyond Surrogate Gradients: Fully Differentiable Token Pruning for Vision-Language Models
May 27, 2026Landi He, Mingde Yao, Shawn Young +1Vision-Language Models Efficient VLM Inference
Shenzhen University of Advanced Technology · CUHK MMLab, CPII under InnoHK
VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Models
Apr 16, 2026Huawei Ji, Yuanhao Sun, Yuan Jin +4Vision-Language Models Efficient VLM Inference
Shanghai Jiao Tong University, Shanghai, China · University of Edinburgh, Edinburgh, UK

Apr 27, 2026J/K move · Enter open · S save

LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models

Rinyoichi Takezoe, Yaqian Li, Zihao Bo, Anzhou Hou+2

Li Auto Inc.

Efficient VLM Inference Attention Sinks Visual Token Pruning

Open paper AskPDF arXiv

Vision-Language Models (VLMs) have recently demonstrated remarkable capabilities in visual understanding and reasoning, but they also impose significant computational burdens due to long visual sequence inputs. Recent works address this issue by pruning unimportant visual tokens, achieving substantial computational reduction while maintaining model performance. The core of token pruning lies in determining token importance, with current approaches primarily relying on attention scores from vision encoders or Large Language Models (LLMs). In this paper, we analyze the effectiveness of attention mechanisms in both vision encoders and LLMs. We find that vision encoders suffer from attention sink, leading to poor focus on informative foreground regions, while in LLMs, although prior studies have identified attention bias toward token positions, text-to-vision attention demonstrates resistance to this bias and enables effective pruning guidance in middle layers. Based on these observations, we propose LearnPruner, a two-stage token pruning framework that first removes redundant vision tokens via a learnable pruning module after the vision encoder, then retains only task-relevant tokens in the LLM's middle layer. Experimental results show that our LearnPruner can preserve approximately 95% of the original performance while using only 5.5% of vision tokens, and achieve 3.2

\times

inference acceleration, demonstrating a superior accuracy-efficiency trade-off.

Abstract

Visual Token Pruning

Explore similar work

LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models

Beyond Surrogate Gradients: Fully Differentiable Token Pruning for Vision-Language Models

VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Models