Visual token pruning improves the efficiency of multimodal large language models (MLLMs) by removing redundant visual tokens, yet the reliability of existing selection signals has not been systematically examined. We analyze token selection under semantics-preserving perturbations and across representative pruning criteria, revealing two limitations. First, attention-based importance shifts non-uniformly under perturbations, so magnitude alone cannot distinguish stable evidence from unstable saliency (local instability). Second, attention- and diversity-based methods select substantially different subsets despite comparable performance, showing that useful evidence is not confined to a unique subset (global inconsistency). These findings motivate reliable token-set construction. We instantiate this perspective in CoRePruner, a training-free framework combining instruction-aware diversity with reliability-aware attention. Across image benchmarks, multiple MLLM architectures, and token budgets, CoRePruner improves performance, especially under aggressive pruning, and remains competitive on video benchmarks. Our results establish reliability as a central consideration in visual token selection.
Figures & tables
Figure 1: Limitations of existing criteria and CoRePruner. (1) Local instability : attention shifts across semantically equivalent inputs. (2) Global inconsistency : attention and diversity select different tokens yet perform similarly. CoRePruner combines stable attention with instruction-aligned coverage to construct reliable token sets.
Figure 2Figure 3
Figure 5: Method overview of CoRePruner. Stage I selects task-relevant diverse tokens using DPP with an instruction-modulated kernel. Stage II selects reliability-aware attention anchors from the remaining tokens. The final token set is S∗=S∪A .
Method
VQA v2
GQA
VQA Text
MME
MMB EN
MMB CN
MMVet
SQA Img
POPE
Avg.
Rel.
Upper Bound, All 576 Tokens (100%)
LLaVA-1.5-7B
78.5
61.9
58.2
1506.5
64.7
58.1
31.1
69.5
85.9
64.8
100%
Retain 128 Tokens ( ↓ 77.8%)
FastV (ECCV24)
61.8
49.6
50.6
1368.9
56.1
51.4
28.1
60.2
59.6
54.0
84.2%
SparseVLM (ICML25)
73.8
56.0
54.9
1399.3
60.0
51.1
30.0
67.1
80.5
60.4
93.2%
PDrop (CVPR25)
74.3
57.1
56.6
1444.1
61.6
55.3
27.6
68.4
77.5
61.2
94.2%
Table 2: Performance comparison on LLaVA-1.5-7B under strict visual token budgets (128, 64, 32). CoRePruner, CoRePruner † , and CoRePruner ‡ use cosine, RBF, and polynomial diversity kernels, respectively, demonstrating the plug-and-play nature. Avg. is the mean over 9 benchmarks, and Rel. is the average performance retention. CoRePruner achieves the best overall performance, with larger gains under aggressive pruning.
Method
VQA Text
MME
MMB EN
MMB CN
Avg.
Rel.
Upper Bound, All 1296 Tokens (100%)
Qwen2.5-VL-7B
84.8
2304
82.8
83.2
91.5
100.0%
Retain 512 Tokens ( ↓ 60.5%)
FastV (ECCV24)
84.1
2317
82.0
81.1
90.8
99.1%
DivPrune (CVPR25)
81.8
2279
81.6
82.1
89.9
98.2%
CDPruner (NeurIPS25)
82.1
2323
82.2
81.4
90.5
98.7%
Table 3: Performance comparison on Qwen2.5-VL-7B under constrained token budgets. Consistent improvements show that CoRePruner generalizes to architectures with different visual-token generation mechanisms, with margins widening as the budget tightens. Notation follows Table 2 .
Method
MVBench
VideoMME
test
w/o sub
short
medium
long
Upper Bound, All 6,272 Tokens (100%)
Qwen2.5-VL-7B
60.7
61.3
70.8
61.2
51.9
Retain 1568 Tokens ( ↓ 75%)
VisPruner (ICCV25)
59.8
58.7
68.0
57.6
50.0
CDPruner (NeurIPS25)
58.5
58.9
67.7
59.3
49.7
Table 4: Video-understanding performance on MVBench and VideoMME with Qwen2.5-VL-7B. CoRePruner matches or exceeds the strongest baseline on most metrics, with clearer gains under 85% and 90% token reduction.
Figure 6: Effects of token allocation and perturbation count. Left: Accuracy generally peaks near a Stage II ratio of 0.3 across token budgets. Right: Performance versus the number of perturbed images K used to estimate attention reliability, with fluctuations when K≥4 .
Figure 7: Feature-space coverage of retained tokens, measured by the RMS-std of their features, across Stage I ratios on GQA (Left) and TextVQA (Right) with LLaVA-1.5-7B. Coverage peaks at intermediate ratios, supporting complementary token selection.
Method
Prefill
Decode
KV
GPU
Time
Time
Cache
Mem.
LLaVA-NeXT-7B
254.7
31.0
1465.3
17.1
VisPruner
35.4
20.0
185.3
15.2
CDPruner
35.4
20.0
185.3
14.4
CoRePruner
35.7
20.0
185.3
16.6
Table 5: Efficiency analysis on LLaVA-NeXT-7B with 320 tokens. CoRePruner matches prior pruning methods in prefill and decode latency and KV cache size, with additional GPU memory overhead. Prefill is reported per input and decode per generated token (ms); KV cache in MB, and memory in GB. Prefill and decode exclude vision encoding and token selection.
Benchmark
Stage I
Stage II
w/o
Ours
-only
-only
Reliability
Retain 64 Tokens ( ↓ 88.9%)
VQA Text
55.1
55.9
56.1
56.4
GQA
58.7
56.0
58.6
59.0
MME
1410.2
1437.6
1430.0
1438.2
MMB EN
61.2
61.3
61.3
61.4
Table 6: Ablation analysis on LLaVA-1.5-7B. Stage I-only and Stage II-only use a single stage; w/o Reliability replaces the Stage II score with raw attention. Both stages contribute, and removing the stability weighting lowers accuracy on every metric at both budgets.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
# Tokens
Groups
Attention Score
Uncertainty
Mean
Std.
Mean
Std.
128
U↑
0.0029
0.0105
0.0028
0.0120
U↓
0.0024
0.0072
0.0010
0.0067
256
U↑
0.0014
0.0068
0.0018
0.0106
U↓
0.0012
0.0047
0.0005
0.0044
320
U↑
0.0011
0.0058
0.0014
0.0091
Appendix
Table 7: Attention and uncertainty statistics of the high-uncertainty ( U↑ ) and low-uncertainty ( U↓ ) token groups used in Figure 3 (a). For a fair comparison, the two groups are constructed to have closely matched attention-score distributions, including similar mean attention magnitudes, allowing the effect of uncertainty to be examined while controlling for raw attention strength.
Figure 8: Ablation of the Stage-II token allocation ratio on MMBench. Performance peaks at 0.5, indicating that reasoning-centric benchmarks benefit from a larger allocation to reliability-aware tokens.
Package
Version
Primary use
Python
3.10.19
Runtime environment
PyTorch
2.1.2
Model inference
torchvision
0.16.2
Image transformations
Transformers
4.37.2
Model loading and generation
NumPy
1.22.3
Perturbation and numerical operations
Pillow
10.4.0
Image preprocessing
Appendix
Table 8: Core software packages used for model inference and evaluation.
Artifact
Checkpoint / Scope
License / Usage terms
Model checkpoints
LLaVA-1.5-7B
liuhaotian/llava-v1.5-7b
Llama 2 Community License Agreement
LLaVA-NeXT-7B
liuhaotian/llava-v1.6-vicuna-7b
Llama 2 Community License Agreement
Qwen2.5-VL-7B
Qwen/Qwen2.5-VL-7B-Instruct
Apache License 2.0
Image-understanding benchmarks
VQAv2
Annotations and COCO images
CC BY 4.0 for VQA annotations; COCO image licenses apply
Appendix
Table 9: Model checkpoints and applicable licenses and usage terms for the models and evaluation benchmarks used in our experiments.
Benchmark
Evaluation Split
# Eval Samples
VQAv2
test-dev2015
107,394
GQA
test-dev (balanced)
12,578
TextVQA
val
5,734
MME
test
2,374
MMBench-EN
dev
1,164
MMBench-CN
dev
1,164
Appendix
Table 10: Evaluation benchmarks, evaluation splits, and the number of evaluation samples used in our experiments.
Method
VQA v2
GQA
VQA Text
MME
SQA Img
POPE
Avg.
Rel.
Upper Bound, All 2880 Tokens (100%)
LLaVA-NeXT-7B
81.3
62.5
60.3
1511.8
67.5
86.8
72.3
100%
Retain 640 Tokens ( ↓ 77.8%)
FastV (ECCV24)
77.0
58.9
58.1
1412.6
67.4
79.5
68.6
95.0%
SparseVLM (ICML25)
77.1
60.3
57.8
1456.8
67.6
85.3
70.2
97.0%
PDrop (CVPR25)
79.1
60.0
57.8
1475.9
66.7
83.8
70.2
97.0%
Appendix
Table 11: Performance comparison on LLaVA-NeXT-7B under identical token budgets. CoRePruner consistently improves performance, demonstrating that our method generalizes beyond the LLaVA-1.5 backbone. Notation follows Table 2 .
Table 12: Performance (%) on GQA after removing tokens selected by attention vs. random pruning. (a) On clean images, removing the top- k attended tokens ( Tk ) causes a larger accuracy drop than removing randomly selected tokens. (b) On perturbed images, we remove Tk identified from clean inputs. Removing Tk still results in a substantially larger performance drop than random pruning.
Retained Tokens
Ratio
Recall@ k
448 ( ↓ 22.2%)
3.13
0.91
320 ( ↓ 44.4%)
1.99
0.78
192 ( ↓ 66.7%)
1.78
0.82
64 ( ↓ 88.9%)
2.52
0.70
Appendix
Table 13: Quantitative analysis of token prominence under perturbations. The high Ratio (up to 3.13) and Recall@ k (up to 0.91) show that tokens identified as important under clean inputs remain prominent and are largely preserved within the perturbed top- k set.
Figure 9: Comparison of token selection for the same image under different instructions. Instruction-aware diversity-based token selection (yellow) shifts toward instructions-relevant regions, while attention-based token selection (blue) focuses on visually salient areas. Overlap is shown in pink .
Figure 10: High-overlap cases between diversity-based and attention-based token selections. When questions target prominent foreground objects, both strategies converge on the queried instance, leading to relatively high IoU. Notation follows Figure 9 .
Figure 11: Low-overlap cases between diversity-based and attention-based token selections. For questions targeting background or peripheral cues, attention-based selection focuses on salient regions while instruction-aware diversity captures subtle task-relevant evidence. Notation follows Figure 9 .
Method
VQA v2
GQA
VQA Text
MME
MMB EN
MMB CN
MMVet
SQA Img
POPE
Avg.
Retain 128 Tokens ( ↓ 77.8%)
DivPrune
76.0
59.4
55.9
1405.1
61.5
54.8
30.6
68.6
87.0
62.7
VisPruner
75.8
58.2
57.0
1461.4
62.7
57.3
33.7
69.1
84.6
63.5
CDPruner
76.6
59.9
56.2
1431.4
63.1
55.0
32.8
69.0
87.7
63.5
CoRePruner
77.0±0.02
60.2±0.05
57.2±0.08
1457.6±1.04
63.2±0.10
56.5±0.16
33.3±0.36
68.9±0.16
87.6±0.03
64.1
CoRePruner †
76.9±0.02
60.3±0.06
57.3±0.05
1449.8±1.64
62.9±0.09
56.9±0.26
32.7±0.25
68.6±0.10
87.5±0.09
64.0
Appendix
Table 14: Performance of LLaVA-1.5-7B under 128- and 64-token budgets. CoRePruner results are reported as mean ± standard deviation over five seeds. Notation follows Table 2 .
Benchmark
Stage I-only
Stage II-only
w/o ρ
ρ=exp(−u)
Ours ( ρ=1+u1 )
Retain 128 Tokens ( ↓ 77.8%)
VQA Text
56.2±0.02
57.0±0.08
57.0±0.05
57.2±0.08
57.2±0.12
GQA
59.7±0.02
58.7±0.07
59.9±0.04
60.2±0.05
60.2±0.03
MME
1419.4±0.87
1415.6±4.68
1449.7±2.37
1457.6±1.04
1455.8±1.80
MMB EN
62.7±0.19
62.0±0.35
62.5±0.25
63.2±0.10
63.5±0.12
MMVet
30.6±0.78
31.5±0.26
32.4±0.19
33.3±0.36
33.2±0.39
Appendix
Table 15: Stage-wise and ρ -form ablation on LLaVA-1.5-7B under 128- and 64-token budgets. Results are reported as mean ± standard deviation over five seeds.
Multimodal large language models (MLLMs) have achieved strong multimodal reasoning capabilities, but their efficiency is limited by the large number of visual tokens, which introduces substantial computational overhead. Visual token pruning offers a natural solution, yet existing methods are imperfect: attention-based criteria tend to retain redundant tokens, while diversity-based criteria are often agnostic to user instructions. Even methods that combine multiple criteria still lack a principled formulation of the intrinsic objective of token pruning. In this paper, we revisit visual token pruning from a first-principles perspective and formulate it as constructing Token Optimal Preservation Sets. Through a top-down information-theoretic analysis, we identify three fundamental principles for effective token selection: Task Relevance, Information Coverage, and Semantic Diversity. Based on these principles, we propose TOPS, a training-free and model-agnostic pruning module that can be applied to various MLLMs. Extensive experiments on 7 MLLM backbones and 14 benchmarks demonstrate that TOPS outperforms prior methods under diverse pruning settings. Notably, on LLaVA-NeXT, TOPS removes 77.8% of visual tokens while preserving 100.0% and 100.6% performance on its 7B and 13B models, respectively, suggesting that pruning redundant visual tokens can sometimes mitigate hallucination and inspire future lightweight MLLM design. Our code is available at https://github.com/MAX-7237/TOPS.
Tinghao Wang, Yichen Guo, Rui Huang +9
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · University of Electronic Science and Technology of China · Nanyang Technological University +1
Visual token pruning speeds up multimodal large language models (MLLMs) by keeping a small subset of the visual tokens, and pruning methods are compared by the accuracy they retain. We study what pruning does to the confidence of these models, across common selectors, several MLLMs, and different output formats. Pruning errors concentrate on questions whose evidence the selector removed, and confidence does not register the removal. When the queried object loses all of its tokens, accuracy on these questions drops from 59% to 17%, while confidence stays at the unpruned level. Returning a few object tokens to the kept set recovers most of the lost accuracy. Selectors that keep the most attended tokens remove such evidence most often and produce confident errors, which temperature scaling cannot re-rank. Selectors that avoid keeping redundant tokens stay close to the calibration of the unpruned model. We then use the confidence of the pruned model to set a per-question token budget. The model answers with few tokens first and again with all tokens when its confidence is low. Conformal risk control sets the threshold to bound the expected deviation from the unpruned model. With coverage-based selection, this cascade needs about a third of the prefill tokens of the unpruned model, while with FastV it needs more than the unpruned model. The savings come mainly from how well the confidence ranks the answers that differ from the unpruned ones.
Kaizhen Tan, Yang Feng, Heqing Du +3
Carnegie Mellon University, Pittsburgh, PA, USA · Columbia University, New York, NY, USA
Training-free visual-token pruning often uses token importance, redundancy, or related selection criteria as proxies for safe removal. We show that these signals alone do not fully characterize removability, which is conditioned on both representation depth and the surrounding deletion set. Controlled interventions demonstrate that removing the same tokens at different depths produces substantially different downstream perturbations, while changing only the deletion context at a fixed depth alters candidate marginals and pruning-boundary decisions. These findings show that token importance alone cannot determine when a token is safely removable or how its removability changes under joint deletion. Motivated by this perspective, we propose CoRePrune, a training-free two-stage framework. Progressive Perturbation-Aware Visual Pruning refreshes deletion effects as visual representations evolve, while Set-Conditioned Refinement reevaluates candidate rescue benefits under the current deletion set after visual--text interaction. Across five multimodal large language model backbones covering standard images, high-resolution inputs, and video, CoRePrune preserves performance under aggressive token budgets. On Qwen3.5, with a final budget of 128 visual tokens, it retains 90.3% of dense-model performance while reducing aggregate prefill time by 51.0%.