Visual token reduction is critical for accelerating Vision-Language Models (VLMs), since visual inputs are represented as token sequences that introduce substantial computational overhead in the LLM backbone. However, most pruning pipelines treat efficiency primarily as a token selection problem and retain a fixed visual token budget across inputs, overlooking the substantial variation in image information density. We propose E-AdaPrune, an energy driven adaptive pruning framework that determines an image specific token budget from the singular value spectrum of the visual feature matrix and passes this budget to existing token selectors. By preserving a certain proportion of spectral energy, our method allocates more tokens to information dense scenes while assigning fewer tokens to redundant scenes, without introducing additional learnable parameters. We evaluate E-AdaPrune across four VLM backbones, three token selectors, and nine benchmarks under matched average token budgets. Results show that E-AdaPrune removes a substantial amount of redundant computation from simple cases and converts the saved budget into larger gains on information rich cases. Notably, on SQAI with Qwen2.5-VL-3B, E-AdaPrune uses 35.8% fewer tokens for simple cases with only a 0.52% relative performance decrease. The saved budget is redirected to hard cases, which receive 52.5% more tokens and achieve a 1.94% relative performance improvement.
Figures & tables
Figure 1: Different images contain different amounts of visual information, indicating that a static token budget may either discard critical details or retain unnecessary redundancy. All examples are from TextVQA ( Singh et al. 2019 ) .
Figure 2: Comparison of static and adaptive pruning. E-AdaPrune determines a content-aware budget k∗ via an image-specific energy criterion, optimizing token retention for varying information densities. V and R denote token importance scores and rankings.
Figure 3: Representative examples illustrating the benefits of controlling the visual token budget from an energy perspective. The hard case contains substantially richer information than the simple case. (a) Under a fixed token budget, FastV uniformly retains 159 tokens for both cases. This budget is redundant for the simple case but insufficient to preserve critical details in the hard case, resulting in an incorrect VQA result. (b) With E-AdaPrune, FastV is assigned an image-specific budget based on the corresponding energy spectrum. For the hard case, the budget is increased to 259 tokens, enabling the model to preserve critical information (e.g., labels on bottles and signs) and produce the correct answer. For the simple case, fewer tokens are allocated to the uninformative background while the answer remains correct. Green ellipses highlight regions exhibiting visible differences in token allocation, while green and red texts denote correct and incorrect answers, respectively.
Figure 4: Energy ratio τ v.s. token count k as an operating point for task performance and token efficiency trade-off. (a)(c) We vary the energy ratio τ from 97.0% to 100% for FastV+E-AdaPrune and evaluate its performance on POPE/TextVQA benchmarks. (b)(d) For each setting in (a)(c), we examine the token count k . Compared with token count k , energy ratio τ provides a smoother operating point.
Figure 5: Matched budget results with FastV on (a) Qwen2.5-VL-3B at τ=95.0% , (b) LLaVA-1.5-13B at τ=99.0% , (c) LLaVA-NeXT-8B at τ=99.0% , and (d) LLaVA-1.5-7B at τ=99.0% . Bars show retained visual tokens, and lines show subset accuracy. Simple and hard denote samples assigned fewer and more tokens than the dataset mean, respectively.
Figure 6: Matched budget results on LLaVA-1.5-7B at τ=99.0% with (a) VisionZip, (b) PDrop, and (c) FastV. Bars show retained visual tokens, and lines show subset accuracy.
Method
MMVet
MMB
MMB CN
Avg.
Base
31.1
64.3
58.3
100.0%
τ =99.8%
298
261
261
273
FastV
28.9(92.9%)
64.0(99.5%)
58.3(100.0%)
97.5%
FastV+E
30.0(96.5%)
64.7(100.6%)
58.7(100.7%)
99.3%
PDrop
31.9(102.6%)
65.0(101.1%)
58.5(100.3%)
101.3%
PDrop+E
33.5(107.7%)
65.2(101.4%)
59.1(101.4%)
103.5%
Table 1: Overall LLaVA-1.5-7B results at τ=99.8% on MMVet, MMB, and MMB CN . "+E" denotes energy based adaptive budgeting. Avg. is computed over the three reported relative scores.
Method
Metric
Compression Ratio
0%
55%
78%
VisionZip (Fixed Token)
MMB
64.3
64.0
62.2
Time (s)
933
722
628
MMB
64.3
64.1
62.5
VisionZip+E-AdaPrune
Time (s)
933
843
781
Table 2: Efficiency and Performance with Conventional SVD.
Method
Dim( t )
Avg. Token
Time( s )
Latency( ms )
MMB
Base
-
576
895
-
64.3
VisionZip
-
130
628
-
62.2
basic SVD
1024
130
781
35
62.5
rSVD ( q=2 )
500
130
701
17
62.5
rSVD ( q=1 )
500
130
687
13
62.5
rSVD ( q=0 )
500
131
695
15
62.4
Table 3: Efficiency improvement with rSVD. Latency denotes the extra per-image overhead relative to VisionZip.
Metric
Schedule
B1
B4
B8
B16
TTFT (ms/request)
FIFO
170.1
164.5
170.7
178.1
Random
164.8
171.1
178.5
Bucketed
159.3
164.7
170.3
Pad(%)
FIFO
0
18.0
28.4
40.2
Random
18.6
28.9
40.8
Bucketed
11.8
20.8
30.8
Table 4: Efficiency under Batch Serving.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Matched budget results on (a)Qwen2.5-VL-3B, τ =96%, (b)LLaVA-1.5-13B, τ =99.8%, and (c)LLaVA-NEXT-8B, τ =99.8%.
Figure 8: Matched budget results on LLaVA-1.5-7B with (a) VisionZip, (b) PDrop, and (c) FastV, all using τ=99.8% .
Vision-language models (VLMs) rely on long visual token sequences for visual understanding, making the prefill stage expensive in both computation and memory. Most existing pruning methods follow an absolute-ranking paradigm, assigning importance scores to visual tokens and retaining a fixed top-K subset. In this work, we argue that this paradigm is fundamentally brittle: attention sinks distort token importance rankings, while image redundancy and query-dependent visual evidence make fixed token budgets unreliable across inputs. We propose OccamToken, a training-free framework that replaces absolute token ranking with register-anchored relative evidence testing. Instead of asking which tokens are globally important, OccamToken evaluates whether a visual token provides information beyond a register-based reference. Our key insight is that register tokens naturally absorb low-information attention patterns, making them a stable reference for identifying genuinely informative visual evidence. Based on this principle, OccamToken performs both image-adaptive redundancy pruning and query-adaptive relevance pruning through dynamic thresholds derived from register attention. Across LLaVA-NeXT, LLaVA-v1.5, and Qwen3-VL, OccamToken consistently improves the accuracy-efficiency trade-off without additional training. Notably, on LLaVA-NeXT, it reduces 2,880 visual tokens to approximately 40 while preserving over 93% of the original accuracy, enabling stable visual token compression even in the extreme 1.4% retention regime.
Vision-Language Models (VLMs) have recently demonstrated remarkable capabilities in visual understanding and reasoning, but they also impose significant computational burdens due to long visual sequence inputs. Recent works address this issue by pruning unimportant visual tokens, achieving substantial computational reduction while maintaining model performance. The core of token pruning lies in determining token importance, with current approaches primarily relying on attention scores from vision encoders or Large Language Models (LLMs). In this paper, we analyze the effectiveness of attention mechanisms in both vision encoders and LLMs. We find that vision encoders suffer from attention sink, leading to poor focus on informative foreground regions, while in LLMs, although prior studies have identified attention bias toward token positions, text-to-vision attention demonstrates resistance to this bias and enables effective pruning guidance in middle layers. Based on these observations, we propose LearnPruner, a two-stage token pruning framework that first removes redundant vision tokens via a learnable pruning module after the vision encoder, then retains only task-relevant tokens in the LLM's middle layer. Experimental results show that our LearnPruner can preserve approximately 95% of the original performance while using only 5.5% of vision tokens, and achieve 3.2× inference acceleration, demonstrating a superior accuracy-efficiency trade-off.
Visual token pruning reduces the computational cost of Vision-Language Models (VLMs) by removing redundant visual tokens. The key is to learn a score that measures whether a token is useful. Existing methods typically rely on Gumbel-Softmax to approximate discrete selection during training. Such selectors make the score depend on the behavior of a relaxed pruning operator, not directly on the consequence of information loss. In this paper, we propose DiffPrune, which gives token scores a direct meaning. During training, DiffPrune keeps all tokens and weakens each token's information according to its score. If weakening a token hurts the task, the scorer is pushed to protect it; if not, the token can receive a lower score. Because the loss is differentiated through this actual information-throttling path, the scorer avoids the unstable surrogate path of relaxed token selection. DiffPrune implements this idea with an Information Throttler, which injects variance-preserving noise into visual tokens, where high-score tokens remain close to their original representations, while low-score tokens carry less original information. At inference, the throttler is removed, and hard top-K pruning is applied using the learned scores. Across ten VLM benchmarks, DiffPrune retains 96.5% of full-model accuracy while accelerating LLM prefill by 2.85x, with only 0.69 ms inference overhead. Code will be publicly available.
Landi He, Mingde Yao, Shawn Young +1
Shenzhen University of Advanced Technology · CUHK MMLab, CPII under InnoHK