Are We Evaluating VLM Token Pruning Right? Looking Beyond Accuracy and Token Count

Geng Li; Guohao Chen; Shilin Shan; Jianfei Yang

cs.AIOct 17, 2026

Are We Evaluating VLM Token Pruning Right? Looking Beyond Accuracy and Token Count

Authors: Geng Li, Guohao Chen, Shilin Shan, Jianfei Yang

Organizations: Nanyang Technological University

Abstract

Visual token pruning aims to reduce inference cost while preserving the capabilities of the original VLM. Yet existing evaluations rely on aggregate task accuracy and retained token count as proxies for pruning quality and actual efficiency. We reveal two limitations: 1) Aggregate accuracy can obscure whether a pruning method truly identifies critical visual tokens; many examples remain solvable under arbitrary token removal and models can exploit textual shortcuts when visual evidence is degraded. 2) Retained token count is an unreliable proxy for efficiency, as identical token budgets can incur different end-to-end costs depending on where and how pruning is performed. Consequently, comparisons based on aggregate accuracy and retained token counts fail to recover the global performance-efficiency Pareto frontier across methods. We introduce PAA-Bench, a plug-and-play evaluation framework for existing benchmarks, to evaluate whether pruning succeeds through meaningful token selection and whether this success translates into real speedup. To determine when token choice actually matters, we propose Random-Pass@k to measure how often an example remains correct under random pruning, enabling comprehensive evaluation across different levels of selection difficulty. To compare efficiency fairly, we introduce Preservation-Constrained Speedup (PCS) to report the maximum end-to-end speedup each method achieves under a shared preservation target. Experiments across 8 methods, 3 VLM backbones, and 9 benchmarks show that stronger token compression does not necessarily yield greater speedup, method differences widen on examples where token selection matters, and efficiency rankings change with the preservation target.

Figures & tables

Figure 1: Pitfalls in visual-token pruning evaluation. (a) Corrections can arise from distractor removal or text priors and mask losses of originally correct predictions. (b) Under the budget of 64 tokens, easy examples dominate MME, MMBench-EN and POPE under Random-Pass@10. (c) At 95% correctness preservation, fewer retained tokens do not guarantee greater end-to-end speedup.

Figure 2: Diagnostics of token-pruning evaluation. (a) Correction gains can mask preservation losses : correct-to-wrong and wrong-to-correct rates for LLaVA-1.5 at 128 tokens (left) and Qwen3-VL-8B at 20% token retention (right). (b) Stable corrections overlap with text-only behavior : counts of Stable, Text-only corrections, and their intersection across budgets on LLaVA-1.5. (c) Method gaps widen as the Easy-example proportion decreases : overall accuracy of Random, CDPruner, and SparseVLM across mixtures with 0–100% Easy examples. Details in Appendix G.1 .

Method	GQA	VizWiz	SQA I	VQA T	POPE	MME	VQA v2	MMB	MMB CN	S-Avg	W-Avg	Rel.
Retain 128 visual tokens
Random	56.7	56.3	55.2	55.1	57.8	56.5	57.3	55.3	54.3	56.1	85.6	90.2
CDPruner	73.8	74.4	71.7	86.7	68.9	73.7	80.0	69.2	70.1	74.3	92.1	98.3
DivPrune	68.8	72.7	70.1	83.3	68.7	69.5	74.9	66.8	66.2	71.2	91.2	97.4
FastV	63.3	81.1	85.2	89.7	73.1	83.2	75.4	77.2	82.1	78.9	90.9	93.5
DART	72.8	83.8	85.0	86.1	79.7	81.3	80.9	79.6	83.3	81.4	93.0	96.6

Table 1: Correctness preservation on LLaVA-1.5 (%). Dataset scores average the three difficulty strata. S-Avg and W-Avg average stratum-weighted and sample-weighted scores across nine datasets. Rel. averages pruned-to-dense score ratios over the same nine datasets. Bold/underline: best/second-best per budget. Full results appear in Appendix D .

Figure 3: Method differences widen with selection difficulty. (a) LLaVA-1.5, 64 tokens; (b) Qwen3-VL, 30% retention; (c) LLaVA-1.5, 32 tokens; (d) LLaVA-NeXT, 160 tokens. Random reports the mean of 10 trials. Implementation details are in Appendix C.3 .

Method	GQA	VizWiz	SQA I	VQA T	POPE	MME	MMB	MMB CN	S-Avg	W-Avg	Rel.
Retain 640 visual tokens
Random	57.2	56.0	56.9	55.3	59.1	58.0	56.3	56.3	56.9	89.6	93.6
DART	83.2	86.8	86.0	90.3	89.1	86.3	80.7	87.6	86.3	96.1	99.1
DivPrune	62.5	69.8	68.7	75.1	75.0	75.1	67.9	58.9	69.1	91.7	99.0
VisionZip	64.7	75.1	63.8	86.0	73.5	72.5	73.2	66.3	71.9	92.7	100.1
Retain 320 visual tokens

Table 2: Correctness preservation on LLaVA-NeXT (%). Selected methods at 640 and 320 retained visual tokens. Dataset scores average the three difficulty strata; S-Avg and W-Avg average stratum-weighted and sample-weighted scores across eight datasets. Bold/underline: best/second-best among the shown methods per budget. Full results appear in Appendix D .

Figure 4: Fewer tokens do not guarantee faster inference. Bars show end-to-end speedup at nominal 95% and 90% dense-correct preservation targets; numbers below report retained visual tokens. Dashed lines mark the dense baseline (

1\times

). Y-axis ranges vary.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark / table label	Main capability examined	Model coverage
GQA	Objects, attributes, relations, and compositional reasoning	Primary three
VizWiz	Real-world visual questions from blind users	All five
ScienceQA-IMG / SQA I	Science questions with image context	All five
TextVQA / VQA T	Reading and reasoning about text in images	All five
POPE	Object existence and object hallucination	All five
MME	Perceptual and cognitive judgments	All five

Appendix

Table 3: Benchmark variants in the complete token-selection tables. “Primary three” denotes LLaVA-1.5-7B, LLaVA-NeXT-7B, and Qwen3-VL-8B; “All five” additionally includes Qwen3-VL-2B and LLaVA-1.5-13B. Coverage here refers to the reported token-selection results, not every PCS or diagnostic experiment.

Model	Budgets	Benchmark variants	Methods + Random
LLaVA-1.5-7B	128, 64, 32 tokens	9	$8+1$
LLaVA-NeXT-7B	640, 320, 160 tokens	8	$7+1$
Qwen3-VL-8B	10%, 20% retention	8	$5+1$
Qwen3-VL-2B	20%, 10% retention	7	$5+1$
LLaVA-1.5-13B	128, 64, 32 tokens	7	$5+1$

Appendix

Table 4: Budgets and coverage of the complete fixed-budget selection tables. Qwen3-VL uses retention fractions rather than a common absolute token count.

Method	GQA	VizWiz	SQA I	VQA T	POPE	MME	VQA v2	MMB	MMB CN	S-Avg	W-Avg
Retain 128 visual tokens
Random	56.71	56.33	55.22	55.11	57.79	56.48	57.34	55.28	54.34	56.07	85.61
CDPruner	73.75	74.38	71.68	86.74	68.89	73.69	80.01	69.16	70.09	74.27	92.10
DivPrune	68.80	72.74	70.11	83.31	68.68	69.48	74.86	66.80	66.18	71.22	91.21
FastV	63.29	81.11	85.23	89.65	73.10	83.15	75.44	77.17	82.13	78.92	90.94
VisionZip	65.27	78.96	76.82	88.60	68.41	73.85	76.68	65.70	76.21	74.50	91.99

Appendix

Table 5: LLaVA-1.5 results on nine benchmarks with 128, 64, and 32 retained visual tokens. S-Avg and W-Avg are equal-weight averages across datasets of the three-stratum and sample-weighted scores, respectively. Bold and underlined values indicate the best and second-best results, respectively, within each column and token budget.

Method	GQA	VizWiz	SQA I	VQA T	POPE	MME	MMB	MMB CN	S-Avg	W-Avg
Retain 640 visual tokens
Random	57.16	55.96	56.88	55.29	59.06	57.99	56.31	56.29	56.87	89.57
CDPruner	65.81	73.74	67.49	82.59	77.47	76.54	80.35	69.28	74.16	92.87
DivPrune	62.49	69.79	68.69	75.07	74.96	75.06	67.93	58.94	69.12	91.71
FastV	79.78	84.88	84.40	91.48	79.95	85.02	80.84	87.45	84.22	95.76
VisionZip	64.67	75.05	63.82	86.04	73.53	72.45	73.22	66.29	71.88	92.73

Appendix

Table 6: Complete LLaVA-NeXT results with 640, 320, and 160 retained visual tokens. S-Avg denotes the three-stratum aggregate and W-Avg the sample-weighted aggregate. Bold and underlined values indicate the best and second-best results, respectively, within each column and token budget.

Method	GQA	VizWiz	SQA I	VQA T	POPE	MME	MMB	MMB CN	S-Avg	W-Avg
Retain 10% of visual tokens
Random	56.37	55.67	55.08	54.32	56.60	56.47	56.44	56.58	55.94	79.14
CDPruner	60.35	59.65	59.31	69.35	59.79	60.20	64.31	62.14	61.89	81.62
DivPrune	65.50	60.55	56.87	70.79	78.22	63.69	61.02	57.37	64.25	82.19
FastV	54.60	50.03	51.83	52.53	83.59	60.41	55.16	53.64	57.72	72.36
VisionZip	58.23	69.53	62.52	61.23	85.37	60.37	57.09	54.28	63.58	80.08

Appendix

Table 7: Qwen3-VL results at 10% and 20% visual-token retention. S-Avg denotes the three-stratum aggregate and W-Avg the sample-weighted aggregate. Bold and underlined values indicate the best and second-best results, respectively, within each column and retention budget.

Retain 20% of visual tokens
Method	VizWiz	SQA I	TextVQA	POPE	MME	MMB-EN	MMB-CN	S-Avg	W-Avg
Random	56.64	56.45	55.16	54.72	56.51	56.45	57.24	56.17	86.27
CDPruner	67.43	74.81	82.71	100.00	70.51	73.08	68.56	76.73	91.53
DivPrune	68.98	71.67	83.34	66.67	69.58	75.73	66.73	71.82	92.34
FastV	67.75	49.57	72.99	44.44	61.38	53.32	53.12	57.51	83.75
VisionZip	65.75	61.91	57.46	77.78	56.27	57.23	56.70	61.87	85.71

Appendix

Table 8: Token-selection results for Qwen3-VL-2B on seven benchmarks. Dataset columns report three-stratum preservation scores; S-Avg and W-Avg follow Section A.3 . All values are percentages. Bold and underlined values mark the best and second-best scores within each column and budget, respectively.

Retain 128 visual tokens
Method	VizWiz	SQA I	TextVQA	POPE	MME	MMB-EN	MMB-CN	S-Avg	W-Avg
Random	56.44	58.24	54.60	57.23	58.04	56.39	55.00	56.56	86.34
CDPruner	73.72	73.08	85.44	66.80	69.79	71.35	67.20	72.48	92.44
DivPrune	74.54	68.76	81.40	64.78	66.65	67.71	63.49	69.62	91.88
FastV	82.73	88.90	86.38	79.88	89.36	84.87	78.63	84.39	93.95
VisionZip	78.82	78.40	88.11	68.05	70.17	69.59	67.77	74.42	93.07

Appendix

Table 9: Token-selection results for LLaVA-1.5-13B on seven benchmarks. Dataset columns report three-stratum preservation scores; S-Avg and W-Avg follow Section A.3 . All values are percentages. Bold and underlined values mark the best and second-best scores within each column and budget, respectively.

95% correctness-preservation target
Method	MMBench-EN		TextVQA		POPE		GQA
	Speedup	Tokens	Speedup	Tokens	Speedup	Tokens	Speedup	Tokens
Dense	1.00 $\times$	576	1.00 $\times$	576	1.00 $\times$	576	1.00 $\times$	576
Random	1.41 $\times$	266	1.08 $\times$	461	1.71 $\times$	177	1.19 $\times$	426
VisionZip	1.56 $\times$	183	1.32 $\times$	230	1.70 $\times$	145	1.18 $\times$	392
FastV	1.71 $\times$	115	1.41 $\times$	142	1.33 $\times$	270	1.23 $\times$	350

Appendix

Table 10: End-to-end speedup and retained visual tokens for LLaVA-1.5-7B . Bold marks the highest reported speedup within each dataset and preservation target, including ties and Random. Dashes denote unreported results.

Method	MMBench-EN		TextVQA		POPE		GQA
	Speedup	Tokens	Speedup	Tokens	Speedup	Tokens	Speedup	Tokens
95% correctness-preservation target
Dense	1.00 $\times$	1927	1.00 $\times$	2310	1.00 $\times$	2250	1.00 $\times$	2239
Random	2.07 $\times$	617	1.12 $\times$	1951	2.43 $\times$	571	1.35 $\times$	1482
VisionZip	1.94 $\times$	626	1.07 $\times$	2022	2.47 $\times$	446	—
FastV	2.24 $\times$	385	1.84 $\times$	577	2.29 $\times$	474	1.73 $\times$	896

Appendix

Table 11: End-to-end speedup and retained visual tokens for LLaVA-NeXT-7B . Bold marks the highest reported speedup within each dataset and preservation target, including ties and Random. Dashes denote unreported results.

95% correctness-preservation target
Method	MMBench-EN		TextVQA		POPE		GQA
	Speedup	Tokens	Speedup	Tokens	Speedup	Tokens	Speedup	Tokens
Dense	1.00 $\times$	155	1.00 $\times$	750	1.00 $\times$	269	1.00 $\times$	268
Random	1.12 $\times$	45	1.11 $\times$	489	1.26 $\times$	77	1.17 $\times$	186
VisionZip	1.02 $\times$	45	0.95 $\times$	289	1.10 $\times$	56	0.97 $\times$	141
FastV	1.03 $\times$	65	1.15 $\times$	423	1.22 $\times$	99	1.16 $\times$	179

Appendix

Table 12: End-to-end speedup and retained visual tokens for Qwen3-VL-8B . Bold marks the highest reported speedup within each dataset and preservation target, including ties and Random. Dashes denote unreported results.

LLaVA-1.5-7B / VQAv2 / 32 tokens
Case	1	2	3	4	5	6	7	8	9	10
Easy	1	1	1	1	1	1	1	0	1	1
Medium	Food	Ice cream	Ice cream	Ice cream	Ice cream	Food	Ice cream	Ice cream	Ice cream	Coffee
Hard	White	White	White	White	Orange	Blue	Brown	White	Blue	White
LLaVA-NeXT-7B / GQA / 320 tokens
Easy	Sink	Sink	Pipe	Sink	Sink	Sink	Sink	Sink	Sink	Sink

Appendix

Table 13: Complete random-trial answers for the nine displayed cases. Columns index trials 1–10. Bold answers are correct under the saved scorer; plain answers are incorrect. Cases appear in the same order as the figure rows. Each dense prediction is correct.

Figure 5: Random pruning on LLaVA-1.5-7B at 32/576 tokens. Each VQAv2 row pairs the unpruned model input with one successful and one unsuccessful saved mask at the same token budget. Blue and orange answer labels also indicate in text whether the prediction is correct. Clear patches are retained, while faded patches are removed. Trial numbers are one-based; all ten outcomes are reported in Table 13 .

Figure 6: Random pruning on LLaVA-NeXT-7B at 320 tokens. The GQA examples show separate local and global views for the dense input and for both selected masks. G, L, and N denote the numbers of global, local, and image-newline tokens, respectively; together they sum to the corresponding token budget. Newline tokens do not correspond to spatial image patches. Clear positions are retained, while faded positions are removed.

Figure 7: Random pruning on Qwen3-VL-8B at a nominal 20% token budget. The GQA examples pair the original image with saved overlays from one successful and one unsuccessful token-selection trial. The actual budgets are 60/300, 38/192, and 60/300 tokens; the middle case uses $\lfloor 0.2\times 192\rfloor=38$ . Clear positions are retained, while faded positions are removed. The original image is included only as a spatial reference and should not be read as a visualization of internal model features.

Visual Token Pruning

Visual token pruning reduces the image and video patches processed by transformers to lower inference cost while preserving task-relevant information.

See all 269 papers

OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning
May 28, 2026Geng Li, Guohao Chen, Ting Chen +6Efficient VLM Inference Large Vision-Language Models
1Nanyang Technological University (NTU)
VPRune: Efficient Training-free Pre-LLM Visual Token Pruning
Sep 21, 2026Guangchuan Lv, Dianxing Shi, Dingjie FuEfficient VLM Inference Large Vision-Language Models
Northeastern University · Beihang University · Huazhong University of Science and Technology
Not All Attention Heads Contribute to Critical Visual Token Selection: Head-Aware Pruning Matters More
Aug 26, 2026Chaofang Ma, Lin Jiang, Carol Jingyi Li +4Efficient VLM Inference Visual Token Pruning
The Hong Kong University of Science and Technology · Northeastern University · The Hong Kong University of Science and Technology (GZ)

May 28, 2026J/K move · Enter open · S save

OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning

Geng Li, Guohao Chen, Ting Chen, Shilin Shan+5

1Nanyang Technological University (NTU)

Efficient VLM Inference Large Vision-Language Models Visual Token Pruning Visual Token Compression LVLM Pruning

Open paper AskPDF arXiv

Vision-language models (VLMs) rely on long visual token sequences for visual understanding, making the prefill stage expensive in both computation and memory. Most existing pruning methods follow an absolute-ranking paradigm, assigning importance scores to visual tokens and retaining a fixed top-K subset. In this work, we argue that this paradigm is fundamentally brittle: attention sinks distort token importance rankings, while image redundancy and query-dependent visual evidence make fixed token budgets unreliable across inputs. We propose OccamToken, a training-free framework that replaces absolute token ranking with register-anchored relative evidence testing. Instead of asking which tokens are globally important, OccamToken evaluates whether a visual token provides information beyond a register-based reference. Our key insight is that register tokens naturally absorb low-information attention patterns, making them a stable reference for identifying genuinely informative visual evidence. Based on this principle, OccamToken performs both image-adaptive redundancy pruning and query-adaptive relevance pruning through dynamic thresholds derived from register attention. Across LLaVA-NeXT, LLaVA-v1.5, and Qwen3-VL, OccamToken consistently improves the accuracy-efficiency trade-off without additional training. Notably, on LLaVA-NeXT, it reduces 2,880 visual tokens to approximately 40 while preserving over 93% of the original accuracy, enabling stable visual token compression even in the extreme 1.4% retention regime.

Abstract

Visual Token Pruning

Explore similar work

OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning

VPRune: Efficient Training-free Pre-LLM Visual Token Pruning

Not All Attention Heads Contribute to Critical Visual Token Selection: Head-Aware Pruning Matters More