Vision-Language Models (VLMs) have demonstrated remarkable multimodal capabilities but suffer from prohibitive computational overhead due to the massive number of visual tokens. While token pruning is a promising solution, existing methods still suffer from two major drawbacks: 1) unreliable pruning criteria that fail to accurately localize task-specific regions due to inherent attention biases or query-agnostic designs, and 2) suboptimal pruning strategies that employ rigid, non-adaptive schedules, overlooking the dynamic evolution of token redundancy across different model depths. To address these challenges, we propose LearnPrunerV2, which extends the previous two-stage heuristic framework (LearnPruner) into a fully learnable architecture. Specifically, we integrate lightweight learnable modules into every layer of the VLM backbone, which estimate token importance by fusing intrinsic visual saliency with extrinsic query relevance captured via a shadow token. By uncovering a consistent staircase redundancy pattern across VLM layers, we further introduce Budget-Deterministic Pruning (BDP) to establish a hardware-friendly schedule with fixed per-layer budgets. Extensive experimental results across various benchmarks and VLMs demonstrate that LearnPrunerV2 consistently outperform previous state-of-the-art methods. Notably, LearnPrunerV2 preserves 99.8% of the original performance when pruning 77.8% of visual tokens on LLaVA-v1.5, achieving a superior accuracy-efficiency trade-off.
Figures & tables
Figure 1. Visualization of cross-modal relevance scores in LLaVA-v1.5. ’Similarity’ denotes the similarity scores between CLIP multimodal features, while ’Layer 8/16 Attn’ represents the attention distribution of the last token within the LLM. Notably, CLIP features primarily capture isolated word-level semantics (e.g., the ’girl’); in contrast, intermediate LLM layers resolve global contextual semantics to accurately localize the intended target based on spatial relations. Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.
Figure 2. Overview of LearnPrunerV2. (a) Framework illustration, a learnable pruning module (LPM) is inserted into each decoder layer to evaluate token importance, and a learnable shadow token is appended after the user input to aggregate semantic context and capture cross-modal relevance. (b) Internal structure of the LPM. The visual hidden state is concatenated with the query-aware feature derived from the shadow token to predict the final importance score. Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.
Figure 3. Statistics of learned pruning trajectories. We visualize the per-layer token retention ratio obtained via threshold-based pruning on LLaVA-v1.5 and Qwen2.5-VL. The solid lines denote the mean values, while the shaded regions represent the variance across 500 randomly sampled image-text pairs from LLaVA-mix665k. Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.
Method
GQA
SQA
VQA T
POPE
MME
VQA v2
MMB
MMB CN
MMVet
Avg.
Upper Bound, 576 Tokens (100%)
Vanilla
61.9
69.5
58.2
85.9
1862
78.5
64.7
58.3
31.1
100.0%
Retain Averaged 128 Tokens ( ↓ 77.8% )
FastV
49.6
60.2
50.6
59.6
1490
61.8
56.1
51.4
28.1
83.0%
SparseVLM
56.0
67.1
54.9
80.5
1696
73.8
60.0
51.1
29.7
92.9%
VisPruner
58.2
69.1
57.0
84.6
1794
75.8
62.7
57.3
33.7
98.5%
Table 1. Performance comparisons of different pruning methods on LLaVA-v1.5-7B. Avg. denotes the performance relative to the vanilla model, averaged across all benchmarks. Bold and underlined values denote the best and second-best results.
Method
GQA
SQA
VQA T
POPE
MME
VQA v2
MMB
MMB CN
MMVet
Avg.
Upper Bound, 2,880 Tokens (100%)
Vanilla
64.2
70.2
61.3
86.3
1842
81.2
67.9
60.5
40.0
100.0%
Retain Averaged 640 Tokens ( ↓ 77.8% )
FastV
62.0
69.1
60.0
79.5
1807
79.5
65.8
53.5
33.3
94.4%
SparseVLM
60.3
67.7
57.8
85.3
1772
77.1
65.7
58.6
36.1
95.4%
DART
61.3
68.2
59.5
85.0
1741
78.3
64.9
57.1
36.9
95.7%
Table 2. Performance comparisons of different pruning methods on LLaVA-NeXT-7B.
Method
GQA
SQA
VQA T
MME
MMB
Avg.
Upper Bound, 1280 Tokens (100%)
Vanilla
60.9
75.9
83.3
2314
83.8
100.0%
Retain Averaged 426 Tokens ( ↓ 66.7% )
FastV
58.4
79.1
81.5
2330
81.4
99.2%
LearnPruner
59.3
77.4
81.1
2339
83.2
99.4%
LearnPrunerV2
60.0
81.4
80.7
2355
83.4
100.8%
Table 3. Performance comparisons of different pruning methods on Qwen2.5-VL-7B.
Table 4. Ablation study on different configurations of the learnable pruning module (LPM).
Table 8
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Method
TGIF-QA
MSVD-QA
MSRVTT-QA
Acc.
Score
Acc.
Score
Acc.
Score
Upper Bound, 2,048 Tokens (100%)
Vanilla
18.1
1.27
64.1
3.41
56.1
2.97
Retain Averaged 456 Tokens ( ↓ 77.8% )
FastV
16.4
1.17
60.3
3.24
53.0
2.84
LearnPrunerV2
17.8
1.21
64.9
3.39
55.0
2.89
Appendix
Table 7. Video understanding performance comparisons of different pruning methods on Video-LLaVA-7B. Performance is evaluated on the first 1,000 samples of each benchmark. We employ GPT-4.1 to assist in scoring open-ended responses.
Figure 4. Performance comparisons of LearnPrunerV2 models trained with different proportion of the training dataset. Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.
Figure 5. Visualization of the pruning results. Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.
Figure 6. Visualization of the pruning results. Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.
Figure 7. Visualization of the score heatmap. Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.
Figure 8. Visualization of the score heatmap. Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing to bat.
Vision-Language Models (VLMs) have recently demonstrated remarkable capabilities in visual understanding and reasoning, but they also impose significant computational burdens due to long visual sequence inputs. Recent works address this issue by pruning unimportant visual tokens, achieving substantial computational reduction while maintaining model performance. The core of token pruning lies in determining token importance, with current approaches primarily relying on attention scores from vision encoders or Large Language Models (LLMs). In this paper, we analyze the effectiveness of attention mechanisms in both vision encoders and LLMs. We find that vision encoders suffer from attention sink, leading to poor focus on informative foreground regions, while in LLMs, although prior studies have identified attention bias toward token positions, text-to-vision attention demonstrates resistance to this bias and enables effective pruning guidance in middle layers. Based on these observations, we propose LearnPruner, a two-stage token pruning framework that first removes redundant vision tokens via a learnable pruning module after the vision encoder, then retains only task-relevant tokens in the LLM's middle layer. Experimental results show that our LearnPruner can preserve approximately 95% of the original performance while using only 5.5% of vision tokens, and achieve 3.2× inference acceleration, demonstrating a superior accuracy-efficiency trade-off.
Visual token pruning reduces the computational cost of Vision-Language Models (VLMs) by removing redundant visual tokens. Existing methods typically rely on Gumbel-Softmax to approximate discrete selection during training. However, the optimization is driven by surrogate gradients rather than the true selection process, leading to unreliable learning of token importance. In this paper, we propose DiffPrune, which reformulates pruning as continuous control of token information instead of discrete selection learning. Specifically, we introduce an Information Throttler that modulates each token using variance-preserving noise conditioned on importance scores, where higher scores induce less information suppression during training. This design directly operates on token representations, naturally providing a fully differentiable optimization path for learning token importance. At inference, tokens are removed via hard thresholding on the learned scores. Across ten VLM benchmarks, DiffPrune retains 96.5% of full-model accuracy while accelerating LLM prefill by 2.85x, with only 0.69 ms of inference overhead.
Landi He, Mingde Yao, Shawn Young +1
Shenzhen University of Advanced Technology · CUHK MMLab, CPII under InnoHK
Visual token pruning methods effectively mitigate the quadratic computational growth caused by processing high-resolution images and video frames in vision-language models (VLMs). However, existing approaches rely on predefined pruning configurations without determining whether they achieve computation-performance optimality. In this work, we introduce , a novel framework that formulates visual token pruning as a Pareto configuration optimization problem to automatically identify optimal configurations. Our approach employs continuous relaxation and straight-through estimators to enable gradient-based search, solved via the Augmented Lagrangian method. Extensive experiments across 8 visual benchmarks demonstrate that effectively approximates the empirical Pareto frontier obtained through grid search and generalizes well across various pruning methods and VLM architectures. Furthermore, through learnable kernel functions, we investigate layer-wise pruning patterns and reveal that multi-step progressive pruning captures VLMs' hierarchical compression structure, achieving superior accuracy-efficiency trade-offs compared to single-layer approaches.
Huawei Ji, Yuanhao Sun, Yuan Jin +4
Shanghai Jiao Tong University, Shanghai, China · University of Edinburgh, Edinburgh, UK