Visual tokens dominate inference cost in vision-language models (VLMs), yet many carry redundant information. Existing pruning methods alleviate this but typically rely on attention magnitude or similarity scores. We reformulate visual token pruning as capacity constrained communication: given a fixed budget K, the model must allocate limited bandwidth to maximally preserve visual information. We propose AutoSelect, which attaches a lightweight Scorer and Denoiser to a frozen VLM and trains with only the standard next token prediction loss, without auxiliary objectives or extra annotations. During training, a variance preserving noise gate modulates each token's information flow according to its predicted importance so that gradients propagate through all tokens; a diagonal attention Denoiser then recovers the perturbed representations. At inference, only the Scorer and a hard top-K selection remain, adding negligible latency. On ten VLM benchmarks, AutoSelect retains 96.5% of full model accuracy while accelerating LLM prefill by 2.85x with only 0.69 ms overhead, and transfers to different VLM backbones without architecture-specific tuning. Code is available at https://github.com/MedHK23/AutoSelect.
Figures & tables
Figure 1: Two views of visual token pruning. (Top) Hard pruning directly discards a subset of tokens. (Bottom) Our capacity-constrained formulation retains all tokens but limits the total information throughput to the same budget through a bandwidth-limited channel.
Figure 2: Overview of the AutoSelect framework. Visual tokens from the frozen Image Encoder pass through a learnable Scorer that assigns per-token importance scores. These scores are polarized by the differentiable Soft Top- K operator under a fixed bandwidth budget K . During training (lower path), a VP Noise Gate injects variance-preserving noise into each token in inverse proportion to its score; the Denoiser then maps the perturbed sequence back toward the LLM’s expected input space. At inference (upper path), the Denoiser and noise injection are discarded: Hard Top- K retains the K highest-scoring tokens with their original position indices. All base VLM parameters, including the Image Encoder, modality projector, and LLM, remain frozen.
Method
Loc.
GQA
MMB
MMB CN
MME
POPE
SQA
VQA v2
VQA Text
SEED
VizWiz
Avg.
Upper Bound, 576 Tokens (100%)
Vanilla
-
61.9
64.7
58.1
1862
85.9
69.5
78.5
58.2
60.5
54.3
100%
Retain 192 Tokens (66.7% pruning)
ToMe (ICLR’23)
Pre
54.3
60.5
-
1563
72.4
65.2
68.0
52.1
-
-
88.5%
FastV (ECCV’24)
L2–L32
52.7
61.2
53.5
1612
64.8
67.3
67.1
52.5
57.1
50.8
89.4%
SparseVLM (ICML’25)
L1–L32
57.6
62.5
58.6
1721
83.6
69.1
75.6
56.1
55.8
50.5
95.8%
Table 1: Results on LLaVA-1.5-7B under different token budgets. “Loc.” denotes pruning location (Pre = before LLM; L a –L b = from LLM layer a to b ). Upper bound uses all 576 visual tokens; remaining blocks report 192/128/64 retained tokens (66.7%/77.8%/88.9% pruning). “Avg.” is the mean ratio of each benchmark score to its upper bound. Formatting: within each Retain- K block, the best Avg. is colored red and the second best is colored blue ; our method is highlighted in blue .
Method
Loc.
GQA
MMB
MMB CN
MME
POPE
SQA
VQA v2
VQA Text
Avg.
Upper Bound, 2880 Tokens (100%)
Vanilla
-
64.2
67.4
60.6
1851
86.5
70.1
80.8
64.9
100%
Retain 320 Tokens (88.9% pruning)
FastV (ECCV’24)
L2–L32
55.9
61.6
51.9
1661
71.7
62.8
71.9
55.7
87.6%
PDrop (CVPR’25)
L8–L24
56.4
63.4
56.2
1663
77.6
67.5
73.5
54.4
90.7%
DART (EMNLP’25)
L2
61.7
65.3
58.2
1710
84.1
68.4
79.1
58.7
95.6%
Table 2: Results on LLaVA-NEXT-7B. “Loc.” denotes pruning location (Pre = before LLM; L a –L b = from LLM layer a to b ). “Avg.” is the mean ratio of each benchmark score to its upper bound. Formatting: the best Avg. is colored red and the second best is colored blue ; our method is highlighted in blue .
Method
MMB
MME
POPE
SQA
VQA Text
Avg.
Upper Bound (100%)
Vanilla
82.8
2304
86.1
84.7
84.8
100%
Token Pruning Rate = 66.7%
FastV (ECCV’24)
75.7
2072
82.2
78.5
77.9
92.3%
HoloV (NeurIPS’25)
78.3
2093
85.0
79.8
78.9
94.3 %
AutoSelect (Ours)
81.7
2279
84.9
86.4
79.0
98.3 %
Table 3: Results on Qwen2.5-VL-7B. “Avg.” is the mean ratio of each benchmark score to its upper bound. Formatting: the best Avg. is colored red and the second best is colored blue ; our method is highlighted in blue .
Figure 6
Method
Loc.
K
FLOPs (T)
Vision Enc. (ms)
Pruning Module (ms)
LLM Prefill (ms)
Prefill Total (ms)
Full (LLaVA)
—
576
8.89
29.71
0.00
118.66
149.51
PruneSID
Pre
64
2.13
31.80
43.39
39.74
115.84
PyramidDrop
L8–L24
64
4.89
29.54
5.02
70.08
105.90
HoloV
Pre
64
2.15
31.96
2.77
41.48
77.39
AutoSelect (Ours)
Pre
64
2.13
29.41
0.69
41.61
72.73
Table 4: Efficiency and prefill-stage breakdown on LLaVA-1.5-7B ( K=64 ). “Loc.” denotes pruning location, consistent with previous tables. Prefill Total is time-to-first-token (TTFT). All latencies are in milliseconds, averaged over 30 forward passes.
Configuration
Modification
K=64
K=128
K=192
Base (VP noise + diagonal) †
–
95.5 %
97.3 %
98.3 %
Global attention
diagonal → global
92.7% (-2.8)
95.0% (-2.3)
95.9% (-2.4)
Scale gating
VP noise → scale
93.2% (-2.3)
94.8% (-2.5)
96.5% (-1.8)
Table 5: Ablation studies on LLaVA-1.5-7B. Each row reports the average performance retention (%) over GQA, MMB, POPE, and MME across three token budgets. “Base” denotes the full AutoSelect configuration (VP noise + diagonal attention).
Figure 5: Validation of VP-noise gating as a differentiable proxy for hard Top- K pruning. Both strategies are applied after patch embedding and before the vision encoder, matching the insertion point used by AutoSelect. Both use identical random score maps to control for scoring quality. Quantitatively (a) and qualitatively (b), the two mechanisms produce equivalent information restriction across all budget levels, justifying VP-noise gating as a continuous, gradient-friendly training surrogate.
Visual token pruning reduces the computational cost of Vision-Language Models (VLMs) by removing redundant visual tokens. The key is to learn a score that measures whether a token is useful. Existing methods typically rely on Gumbel-Softmax to approximate discrete selection during training. Such selectors make the score depend on the behavior of a relaxed pruning operator, not directly on the consequence of information loss. In this paper, we propose DiffPrune, which gives token scores a direct meaning. During training, DiffPrune keeps all tokens and weakens each token's information according to its score. If weakening a token hurts the task, the scorer is pushed to protect it; if not, the token can receive a lower score. Because the loss is differentiated through this actual information-throttling path, the scorer avoids the unstable surrogate path of relaxed token selection. DiffPrune implements this idea with an Information Throttler, which injects variance-preserving noise into visual tokens, where high-score tokens remain close to their original representations, while low-score tokens carry less original information. At inference, the throttler is removed, and hard top-K pruning is applied using the learned scores. Across ten VLM benchmarks, DiffPrune retains 96.5% of full-model accuracy while accelerating LLM prefill by 2.85x, with only 0.69 ms inference overhead. Code will be publicly available.
Landi He, Mingde Yao, Shawn Young +1
Shenzhen University of Advanced Technology · CUHK MMLab, CPII under InnoHK
Recent advances have explored visual token pruning to accelerate the inference of large vision-language models (LVLMs). However, existing methods often struggle to balance token importance and diversity: importance-based methods tend to retain redundant tokens, whereas diversity-based methods may overlook informative ones. This trade-off becomes especially problematic under high reduction ratios, where preserving only a small subset of visual tokens is critical. To address this issue, we propose ID-Selection, a simple yet effective token selection strategy for efficient LVLM inference. The key idea is to couple importance estimation with diversity-aware iterative selection: each token is first assigned an importance score, after which high-scoring tokens are selected one by one while the scores of similar tokens are progressively suppressed. In this way, ID-Selection preserves informative tokens while reducing redundancy in a unified selection process. Extensive experiments across 5 LVLM backbones and 16 main benchmarks demonstrate that ID-Selection consistently achieves superior performance and efficiency, especially under extreme pruning ratios. For example, on LLaVA-1.5-7B, ID-Selection prunes 97.2% of visual tokens, retaining only 16 tokens, while reducing inference FLOPs by over 97% and preserving 91.8% of the original performance, all without additional training.
Vision-language models (VLMs) rely on long visual token sequences for visual understanding, making the prefill stage expensive in both computation and memory. Most existing pruning methods follow an absolute-ranking paradigm, assigning importance scores to visual tokens and retaining a fixed top-K subset. In this work, we argue that this paradigm is fundamentally brittle: attention sinks distort token importance rankings, while image redundancy and query-dependent visual evidence make fixed token budgets unreliable across inputs. We propose OccamToken, a training-free framework that replaces absolute token ranking with register-anchored relative evidence testing. Instead of asking which tokens are globally important, OccamToken evaluates whether a visual token provides information beyond a register-based reference. Our key insight is that register tokens naturally absorb low-information attention patterns, making them a stable reference for identifying genuinely informative visual evidence. Based on this principle, OccamToken performs both image-adaptive redundancy pruning and query-adaptive relevance pruning through dynamic thresholds derived from register attention. Across LLaVA-NeXT, LLaVA-v1.5, and Qwen3-VL, OccamToken consistently improves the accuracy-efficiency trade-off without additional training. Notably, on LLaVA-NeXT, it reduces 2,880 visual tokens to approximately 40 while preserving over 93% of the original accuracy, enabling stable visual token compression even in the extreme 1.4% retention regime.