Visual token pruning aims to reduce inference cost while preserving the capabilities of the original VLM. Yet existing evaluations rely on aggregate task accuracy and retained token count as proxies for pruning quality and actual efficiency. We reveal two limitations: 1) Aggregate accuracy can obscure whether a pruning method truly identifies critical visual tokens; many examples remain solvable under arbitrary token removal and models can exploit textual shortcuts when visual evidence is degraded. 2) Retained token count is an unreliable proxy for efficiency, as identical token budgets can incur different end-to-end costs depending on where and how pruning is performed. Consequently, comparisons based on aggregate accuracy and retained token counts fail to recover the global performance-efficiency Pareto frontier across methods. We introduce PAA-Bench, a plug-and-play evaluation framework for existing benchmarks, to evaluate whether pruning succeeds through meaningful token selection and whether this success translates into real speedup. To determine when token choice actually matters, we propose Random-Pass@k to measure how often an example remains correct under random pruning, enabling comprehensive evaluation across different levels of selection difficulty. To compare efficiency fairly, we introduce Preservation-Constrained Speedup (PCS) to report the maximum end-to-end speedup each method achieves under a shared preservation target. Experiments across 8 methods, 3 VLM backbones, and 9 benchmarks show that stronger token compression does not necessarily yield greater speedup, method differences widen on examples where token selection matters, and efficiency rankings change with the preservation target.
Figures & tables
Figure 1: Pitfalls in visual-token pruning evaluation. (a) Corrections can arise from distractor removal or text priors and mask losses of originally correct predictions. (b) Under the budget of 64 tokens, easy examples dominate MME, MMBench-EN and POPE under Random-Pass@10. (c) At 95% correctness preservation, fewer retained tokens do not guarantee greater end-to-end speedup.
Figure 2: Diagnostics of token-pruning evaluation. (a) Correction gains can mask preservation losses : correct-to-wrong and wrong-to-correct rates for LLaVA-1.5 at 128 tokens (left) and Qwen3-VL-8B at 20% token retention (right). (b) Stable corrections overlap with text-only behavior : counts of Stable, Text-only corrections, and their intersection across budgets on LLaVA-1.5. (c) Method gaps widen as the Easy-example proportion decreases : overall accuracy of Random, CDPruner, and SparseVLM across mixtures with 0–100% Easy examples. Details in Appendix G.1 .
Method
GQA
VizWiz
SQA I
VQA T
POPE
MME
VQA v2
MMB
MMB CN
S-Avg
W-Avg
Rel.
Retain 128 visual tokens
Random
56.7
56.3
55.2
55.1
57.8
56.5
57.3
55.3
54.3
56.1
85.6
90.2
CDPruner
73.8
74.4
71.7
86.7
68.9
73.7
80.0
69.2
70.1
74.3
92.1
98.3
DivPrune
68.8
72.7
70.1
83.3
68.7
69.5
74.9
66.8
66.2
71.2
91.2
97.4
FastV
63.3
81.1
85.2
89.7
73.1
83.2
75.4
77.2
82.1
78.9
90.9
93.5
DART
72.8
83.8
85.0
86.1
79.7
81.3
80.9
79.6
83.3
81.4
93.0
96.6
Table 1: Correctness preservation on LLaVA-1.5 (%). Dataset scores average the three difficulty strata. S-Avg and W-Avg average stratum-weighted and sample-weighted scores across nine datasets. Rel. averages pruned-to-dense score ratios over the same nine datasets. Bold/underline: best/second-best per budget. Full results appear in Appendix D .
Figure 3: Method differences widen with selection difficulty. (a) LLaVA-1.5, 64 tokens; (b) Qwen3-VL, 30% retention; (c) LLaVA-1.5, 32 tokens; (d) LLaVA-NeXT, 160 tokens. Random reports the mean of 10 trials. Implementation details are in Appendix C.3 .
Method
GQA
VizWiz
SQA I
VQA T
POPE
MME
MMB
MMB CN
S-Avg
W-Avg
Rel.
Retain 640 visual tokens
Random
57.2
56.0
56.9
55.3
59.1
58.0
56.3
56.3
56.9
89.6
93.6
DART
83.2
86.8
86.0
90.3
89.1
86.3
80.7
87.6
86.3
96.1
99.1
DivPrune
62.5
69.8
68.7
75.1
75.0
75.1
67.9
58.9
69.1
91.7
99.0
VisionZip
64.7
75.1
63.8
86.0
73.5
72.5
73.2
66.3
71.9
92.7
100.1
Retain 320 visual tokens
Table 2: Correctness preservation on LLaVA-NeXT (%). Selected methods at 640 and 320 retained visual tokens. Dataset scores average the three difficulty strata; S-Avg and W-Avg average stratum-weighted and sample-weighted scores across eight datasets. Bold/underline: best/second-best among the shown methods per budget. Full results appear in Appendix D .
Figure 4: Fewer tokens do not guarantee faster inference. Bars show end-to-end speedup at nominal 95% and 90% dense-correct preservation targets; numbers below report retained visual tokens. Dashed lines mark the dense baseline ( 1× ). Y-axis ranges vary.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark / table label
Main capability examined
Model coverage
GQA
Objects, attributes, relations, and compositional reasoning
Primary three
VizWiz
Real-world visual questions from blind users
All five
ScienceQA-IMG / SQA I
Science questions with image context
All five
TextVQA / VQA T
Reading and reasoning about text in images
All five
POPE
Object existence and object hallucination
All five
MME
Perceptual and cognitive judgments
All five
Appendix
Table 3: Benchmark variants in the complete token-selection tables. “Primary three” denotes LLaVA-1.5-7B, LLaVA-NeXT-7B, and Qwen3-VL-8B; “All five” additionally includes Qwen3-VL-2B and LLaVA-1.5-13B. Coverage here refers to the reported token-selection results, not every PCS or diagnostic experiment.
Model
Budgets
Benchmark variants
Methods + Random
LLaVA-1.5-7B
128, 64, 32 tokens
9
8+1
LLaVA-NeXT-7B
640, 320, 160 tokens
8
7+1
Qwen3-VL-8B
10%, 20% retention
8
5+1
Qwen3-VL-2B
20%, 10% retention
7
5+1
LLaVA-1.5-13B
128, 64, 32 tokens
7
5+1
Appendix
Table 4: Budgets and coverage of the complete fixed-budget selection tables. Qwen3-VL uses retention fractions rather than a common absolute token count.
Method
GQA
VizWiz
SQA I
VQA T
POPE
MME
VQA v2
MMB
MMB CN
S-Avg
W-Avg
Retain 128 visual tokens
Random
56.71
56.33
55.22
55.11
57.79
56.48
57.34
55.28
54.34
56.07
85.61
CDPruner
73.75
74.38
71.68
86.74
68.89
73.69
80.01
69.16
70.09
74.27
92.10
DivPrune
68.80
72.74
70.11
83.31
68.68
69.48
74.86
66.80
66.18
71.22
91.21
FastV
63.29
81.11
85.23
89.65
73.10
83.15
75.44
77.17
82.13
78.92
90.94
VisionZip
65.27
78.96
76.82
88.60
68.41
73.85
76.68
65.70
76.21
74.50
91.99
Appendix
Table 5: LLaVA-1.5 results on nine benchmarks with 128, 64, and 32 retained visual tokens. S-Avg and W-Avg are equal-weight averages across datasets of the three-stratum and sample-weighted scores, respectively. Bold and underlined values indicate the best and second-best results, respectively, within each column and token budget.
Method
GQA
VizWiz
SQA I
VQA T
POPE
MME
MMB
MMB CN
S-Avg
W-Avg
Retain 640 visual tokens
Random
57.16
55.96
56.88
55.29
59.06
57.99
56.31
56.29
56.87
89.57
CDPruner
65.81
73.74
67.49
82.59
77.47
76.54
80.35
69.28
74.16
92.87
DivPrune
62.49
69.79
68.69
75.07
74.96
75.06
67.93
58.94
69.12
91.71
FastV
79.78
84.88
84.40
91.48
79.95
85.02
80.84
87.45
84.22
95.76
VisionZip
64.67
75.05
63.82
86.04
73.53
72.45
73.22
66.29
71.88
92.73
Appendix
Table 6: Complete LLaVA-NeXT results with 640, 320, and 160 retained visual tokens. S-Avg denotes the three-stratum aggregate and W-Avg the sample-weighted aggregate. Bold and underlined values indicate the best and second-best results, respectively, within each column and token budget.
Method
GQA
VizWiz
SQA I
VQA T
POPE
MME
MMB
MMB CN
S-Avg
W-Avg
Retain 10% of visual tokens
Random
56.37
55.67
55.08
54.32
56.60
56.47
56.44
56.58
55.94
79.14
CDPruner
60.35
59.65
59.31
69.35
59.79
60.20
64.31
62.14
61.89
81.62
DivPrune
65.50
60.55
56.87
70.79
78.22
63.69
61.02
57.37
64.25
82.19
FastV
54.60
50.03
51.83
52.53
83.59
60.41
55.16
53.64
57.72
72.36
VisionZip
58.23
69.53
62.52
61.23
85.37
60.37
57.09
54.28
63.58
80.08
Appendix
Table 7: Qwen3-VL results at 10% and 20% visual-token retention. S-Avg denotes the three-stratum aggregate and W-Avg the sample-weighted aggregate. Bold and underlined values indicate the best and second-best results, respectively, within each column and retention budget.
Method
VizWiz
SQA I
TextVQA
POPE
MME
MMB-EN
MMB-CN
S-Avg
W-Avg
Retain 20% of visual tokens
Random
56.64
56.45
55.16
54.72
56.51
56.45
57.24
56.17
86.27
CDPruner
67.43
74.81
82.71
100.00
70.51
73.08
68.56
76.73
91.53
DivPrune
68.98
71.67
83.34
66.67
69.58
75.73
66.73
71.82
92.34
FastV
67.75
49.57
72.99
44.44
61.38
53.32
53.12
57.51
83.75
VisionZip
65.75
61.91
57.46
77.78
56.27
57.23
56.70
61.87
85.71
Appendix
Table 8: Token-selection results for Qwen3-VL-2B on seven benchmarks. Dataset columns report three-stratum preservation scores; S-Avg and W-Avg follow Section A.3 . All values are percentages. Bold and underlined values mark the best and second-best scores within each column and budget, respectively.
Method
VizWiz
SQA I
TextVQA
POPE
MME
MMB-EN
MMB-CN
S-Avg
W-Avg
Retain 128 visual tokens
Random
56.44
58.24
54.60
57.23
58.04
56.39
55.00
56.56
86.34
CDPruner
73.72
73.08
85.44
66.80
69.79
71.35
67.20
72.48
92.44
DivPrune
74.54
68.76
81.40
64.78
66.65
67.71
63.49
69.62
91.88
FastV
82.73
88.90
86.38
79.88
89.36
84.87
78.63
84.39
93.95
VisionZip
78.82
78.40
88.11
68.05
70.17
69.59
67.77
74.42
93.07
Appendix
Table 9: Token-selection results for LLaVA-1.5-13B on seven benchmarks. Dataset columns report three-stratum preservation scores; S-Avg and W-Avg follow Section A.3 . All values are percentages. Bold and underlined values mark the best and second-best scores within each column and budget, respectively.
Method
MMBench-EN
TextVQA
POPE
GQA
Speedup
Tokens
Speedup
Tokens
Speedup
Tokens
Speedup
Tokens
95% correctness-preservation target
Dense
1.00 ×
576
1.00 ×
576
1.00 ×
576
1.00 ×
576
Random
1.41 ×
266
1.08 ×
461
1.71 ×
177
1.19 ×
426
VisionZip
1.56 ×
183
1.32 ×
230
1.70 ×
145
1.18 ×
392
FastV
1.71 ×
115
1.41 ×
142
1.33 ×
270
1.23 ×
350
Appendix
Table 10: End-to-end speedup and retained visual tokens for LLaVA-1.5-7B . Bold marks the highest reported speedup within each dataset and preservation target, including ties and Random. Dashes denote unreported results.
Method
MMBench-EN
TextVQA
POPE
GQA
Speedup
Tokens
Speedup
Tokens
Speedup
Tokens
Speedup
Tokens
95% correctness-preservation target
Dense
1.00 ×
1927
1.00 ×
2310
1.00 ×
2250
1.00 ×
2239
Random
2.07 ×
617
1.12 ×
1951
2.43 ×
571
1.35 ×
1482
VisionZip
1.94 ×
626
1.07 ×
2022
2.47 ×
446
—
FastV
2.24 ×
385
1.84 ×
577
2.29 ×
474
1.73 ×
896
Appendix
Table 11: End-to-end speedup and retained visual tokens for LLaVA-NeXT-7B . Bold marks the highest reported speedup within each dataset and preservation target, including ties and Random. Dashes denote unreported results.
Method
MMBench-EN
TextVQA
POPE
GQA
Speedup
Tokens
Speedup
Tokens
Speedup
Tokens
Speedup
Tokens
95% correctness-preservation target
Dense
1.00 ×
155
1.00 ×
750
1.00 ×
269
1.00 ×
268
Random
1.12 ×
45
1.11 ×
489
1.26 ×
77
1.17 ×
186
VisionZip
1.02 ×
45
0.95 ×
289
1.10 ×
56
0.97 ×
141
FastV
1.03 ×
65
1.15 ×
423
1.22 ×
99
1.16 ×
179
Appendix
Table 12: End-to-end speedup and retained visual tokens for Qwen3-VL-8B . Bold marks the highest reported speedup within each dataset and preservation target, including ties and Random. Dashes denote unreported results.
Case
1
2
3
4
5
6
7
8
9
10
LLaVA-1.5-7B / VQAv2 / 32 tokens
Easy
1
1
1
1
1
1
1
0
1
1
Medium
Food
Ice cream
Ice cream
Ice cream
Ice cream
Food
Ice cream
Ice cream
Ice cream
Coffee
Hard
White
White
White
White
Orange
Blue
Brown
White
Blue
White
LLaVA-NeXT-7B / GQA / 320 tokens
Easy
Sink
Sink
Pipe
Sink
Sink
Sink
Sink
Sink
Sink
Sink
Appendix
Table 13: Complete random-trial answers for the nine displayed cases. Columns index trials 1–10. Bold answers are correct under the saved scorer; plain answers are incorrect. Cases appear in the same order as the figure rows. Each dense prediction is correct.
Figure 5: Random pruning on LLaVA-1.5-7B at 32/576 tokens. Each VQAv2 row pairs the unpruned model input with one successful and one unsuccessful saved mask at the same token budget. Blue and orange answer labels also indicate in text whether the prediction is correct. Clear patches are retained, while faded patches are removed. Trial numbers are one-based; all ten outcomes are reported in Table 13 .
Figure 6: Random pruning on LLaVA-NeXT-7B at 320 tokens. The GQA examples show separate local and global views for the dense input and for both selected masks. G, L, and N denote the numbers of global, local, and image-newline tokens, respectively; together they sum to the corresponding token budget. Newline tokens do not correspond to spatial image patches. Clear positions are retained, while faded positions are removed.
Figure 7: Random pruning on Qwen3-VL-8B at a nominal 20% token budget. The GQA examples pair the original image with saved overlays from one successful and one unsuccessful token-selection trial. The actual budgets are 60/300, 38/192, and 60/300 tokens; the middle case uses ⌊0.2×192⌋=38 . Clear positions are retained, while faded positions are removed. The original image is included only as a spatial reference and should not be read as a visualization of internal model features.
Vision-language models (VLMs) rely on long visual token sequences for visual understanding, making the prefill stage expensive in both computation and memory. Most existing pruning methods follow an absolute-ranking paradigm, assigning importance scores to visual tokens and retaining a fixed top-K subset. In this work, we argue that this paradigm is fundamentally brittle: attention sinks distort token importance rankings, while image redundancy and query-dependent visual evidence make fixed token budgets unreliable across inputs. We propose OccamToken, a training-free framework that replaces absolute token ranking with register-anchored relative evidence testing. Instead of asking which tokens are globally important, OccamToken evaluates whether a visual token provides information beyond a register-based reference. Our key insight is that register tokens naturally absorb low-information attention patterns, making them a stable reference for identifying genuinely informative visual evidence. Based on this principle, OccamToken performs both image-adaptive redundancy pruning and query-adaptive relevance pruning through dynamic thresholds derived from register attention. Across LLaVA-NeXT, LLaVA-v1.5, and Qwen3-VL, OccamToken consistently improves the accuracy-efficiency trade-off without additional training. Notably, on LLaVA-NeXT, it reduces 2,880 visual tokens to approximately 40 while preserving over 93% of the original accuracy, enabling stable visual token compression even in the extreme 1.4% retention regime.
Visual token pruning is a promising approach to reducing the inference cost of large vision-language models (LVLMs), yet aggressive token reduction often causes substantial performance degradation. We identify three key factors behind this degradation: text-guided selection bias, information loss from discarded tokens, and positional distortion caused by sequence compaction. Based on these observations, we propose VPRune, a training-free pre-LLM pruning framework consisting of visual-only diversity selection, similarity-guided token recycling, and position-preserving restoration. Experiments on FastVLM-1.5B across multiple vision-language benchmarks demonstrate that VPRune achieves a favorable accuracy--compression trade-off, with particularly pronounced advantages under aggressive compression. Furthermore, evaluations on edge-device show that VPRune effectively reduces end-to-end inference latency while maintaining superior task performance, demonstrating its practicality for resource-constrained LVLM deployment.
Guangchuan Lv, Dianxing Shi, Dingjie Fu
Northeastern University · Beihang University · Huazhong University of Science and Technology
Vision-Language Models (VLMs) have exhibited impressive performance across diverse visual scenarios. However, this success comes at the cost of explosive growth in visual tokens, which imposes substantial memory and computational overhead during inference, ultimately increasing latency. To improve VLM inference efficiency, a typical class of visual token pruning methods estimates token importance by aggregating attention scores across all heads in the pruning layer of the Large Language Model (LLM) backbone and prunes tokens based on aggregated scores. However, in this paper, we reveal a compelling phenomenon: the capability to pinpoint critical visual tokens is concentrated within a small fraction of heads. Aggregation exclusively on these heads can improve task performance. Inspired by this observation, we propose ProViP, a training-free progressive visual token pruning framework. ProViP first removes redundant visual tokens based on the embedding similarity of input tokens before reasoning of the LLM backbone, and then further prunes tokens during reasoning via head-aware pruning. Experiments demonstrate that ProViP delivers outstanding task performance and inference efficiency. For instance, when applied to LLaVA-1.5-7B, ProViP retains 95.9% of the original performance and achieves 1.62x inference speedup under an 88.9% pruning ratio.
Chaofang Ma, Lin Jiang, Carol Jingyi Li +4
The Hong Kong University of Science and Technology · Northeastern University · The Hong Kong University of Science and Technology (GZ)