How Reliable Is Visual Token Selection? Rethinking Pruning in Multimodal LLMs
Organizations: Korea Advanced Institute of Science and Technology (KAIST)
Abstract
Visual token pruning improves the efficiency of multimodal large language models (MLLMs) by removing redundant visual tokens, yet the reliability of existing selection signals has not been systematically examined. We analyze token selection under semantics-preserving perturbations and across representative pruning criteria, revealing two limitations. First, attention-based importance shifts non-uniformly under perturbations, so magnitude alone cannot distinguish stable evidence from unstable saliency (local instability). Second, attention- and diversity-based methods select substantially different subsets despite comparable performance, showing that useful evidence is not confined to a unique subset (global inconsistency). These findings motivate reliable token-set construction. We instantiate this perspective in CoRePruner, a training-free framework combining instruction-aware diversity with reliability-aware attention. Across image benchmarks, multiple MLLM architectures, and token budgets, CoRePruner improves performance, especially under aggressive pruning, and remains competitive on video benchmarks. Our results establish reliability as a central consideration in visual token selection.
Figures & tables
| Method | VQA v2 | GQA | VQA Text | MME | MMB EN | MMB CN | MMVet | SQA Img | POPE | Avg. | Rel. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Upper Bound, All 576 Tokens (100%) | |||||||||||
| LLaVA-1.5-7B | 78.5 | 61.9 | 58.2 | 1506.5 | 64.7 | 58.1 | 31.1 | 69.5 | 85.9 | 64.8 | 100% |
| Retain 128 Tokens ( 77.8%) | |||||||||||
| FastV | 61.8 | 49.6 | 50.6 | 1368.9 | 56.1 | 51.4 | 28.1 | 60.2 | 59.6 | 54.0 | 84.2% |
| SparseVLM | 73.8 | 56.0 | 54.9 | 1399.3 | 60.0 | 51.1 | 30.0 | 67.1 | 80.5 | 60.4 | 93.2% |
| PDrop | 74.3 | 57.1 | 56.6 | 1444.1 | 61.6 | 55.3 | 27.6 | 68.4 | 77.5 | 61.2 | 94.2% |
| Method | VQA Text | MME | MMB EN | MMB CN | Avg. | Rel. |
| Upper Bound, All 1296 Tokens (100%) | ||||||
| Qwen2.5-VL-7B | 84.8 | 2304 | 82.8 | 83.2 | 91.5 | 100.0% |
| Retain 512 Tokens ( 60.5%) | ||||||
| FastV | 84.1 | 2317 | 82.0 | 81.1 | 90.8 | 99.1% |
| DivPrune | 81.8 | 2279 | 81.6 | 82.1 | 89.9 | 98.2% |
| CDPruner | 82.1 | 2323 | 82.2 | 81.4 | 90.5 | 98.7% |
| Method | MVBench | VideoMME | |||
|---|---|---|---|---|---|
| test | w/o sub | short | medium | long | |
| Upper Bound, All 6,272 Tokens (100%) | |||||
| Qwen2.5-VL-7B | 60.7 | 61.3 | 70.8 | 61.2 | 51.9 |
| Retain 1568 Tokens ( 75%) | |||||
| VisPruner | 59.8 | 58.7 | 68.0 | 57.6 | 50.0 |
| CDPruner | 58.5 | 58.9 | 67.7 | 59.3 | 49.7 |
| Method | Prefill | Decode | KV | GPU |
|---|---|---|---|---|
| Time | Time | Cache | Mem. | |
| LLaVA-NeXT-7B | 254.7 | 31.0 | 1465.3 | 17.1 |
| VisPruner | 35.4 | 20.0 | 185.3 | 15.2 |
| CDPruner | 35.4 | 20.0 | 185.3 | 14.4 |
| CoRePruner | 35.7 | 20.0 | 185.3 | 16.6 |
| Benchmark | Stage I | Stage II | w/o | Ours |
|---|---|---|---|---|
| -only | -only | Reliability | ||
| Retain 64 Tokens ( 88.9%) | ||||
| VQA Text | 55.1 | 55.9 | 56.1 | 56.4 |
| GQA | 58.7 | 56.0 | 58.6 | 59.0 |
| MME | 1410.2 | 1437.6 | 1430.0 | 1438.2 |
| MMB EN | 61.2 | 61.3 | 61.3 | 61.4 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| # Tokens | Groups | Attention Score | Uncertainty | ||
|---|---|---|---|---|---|
| Mean | Std. | Mean | Std. | ||
| 128 | 0.0029 | 0.0105 | 0.0028 | 0.0120 | |
| 0.0024 | 0.0072 | 0.0010 | 0.0067 | ||
| 256 | 0.0014 | 0.0068 | 0.0018 | 0.0106 | |
| 0.0012 | 0.0047 | 0.0005 | 0.0044 | ||
| 320 | 0.0011 | 0.0058 | 0.0014 | 0.0091 | |
| Package | Version | Primary use |
|---|---|---|
| Python | 3.10.19 | Runtime environment |
| PyTorch | 2.1.2 | Model inference |
| torchvision | 0.16.2 | Image transformations |
| Transformers | 4.37.2 | Model loading and generation |
| NumPy | 1.22.3 | Perturbation and numerical operations |
| Pillow | 10.4.0 | Image preprocessing |
| Artifact | Checkpoint / Scope | License / Usage terms |
| Model checkpoints | ||
| LLaVA-1.5-7B | liuhaotian/llava-v1.5-7b | Llama 2 Community License Agreement |
| LLaVA-NeXT-7B | liuhaotian/llava-v1.6-vicuna-7b | Llama 2 Community License Agreement |
| Qwen2.5-VL-7B | Qwen/Qwen2.5-VL-7B-Instruct | Apache License 2.0 |
| Image-understanding benchmarks | ||
| VQAv2 | Annotations and COCO images | CC BY 4.0 for VQA annotations; COCO image licenses apply |
| Benchmark | Evaluation Split | # Eval Samples |
|---|---|---|
| VQAv2 | test-dev2015 | 107,394 |
| GQA | test-dev (balanced) | 12,578 |
| TextVQA | val | 5,734 |
| MME | test | 2,374 |
| MMBench-EN | dev | 1,164 |
| MMBench-CN | dev | 1,164 |
| Method | VQA v2 | GQA | VQA Text | MME | SQA Img | POPE | Avg. | Rel. |
| Upper Bound, All 2880 Tokens (100%) | ||||||||
| LLaVA-NeXT-7B | 81.3 | 62.5 | 60.3 | 1511.8 | 67.5 | 86.8 | 72.3 | 100% |
| Retain 640 Tokens ( 77.8%) | ||||||||
| FastV | 77.0 | 58.9 | 58.1 | 1412.6 | 67.4 | 79.5 | 68.6 | 95.0% |
| SparseVLM | 77.1 | 60.3 | 57.8 | 1456.8 | 67.6 | 85.3 | 70.2 | 97.0% |
| PDrop | 79.1 | 60.0 | 57.8 | 1475.9 | 66.7 | 83.8 | 70.2 | 97.0% |
| Retained Tokens | Ratio | Recall@ |
|---|---|---|
| 448 ( 22.2%) | 3.13 | 0.91 |
| 320 ( 44.4%) | 1.99 | 0.78 |
| 192 ( 66.7%) | 1.78 | 0.82 |
| 64 ( 88.9%) | 2.52 | 0.70 |
| Method | VQA v2 | GQA | VQA Text | MME | MMB EN | MMB CN | MMVet | SQA Img | POPE | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| Retain 128 Tokens ( 77.8%) | ||||||||||
| DivPrune | 76.0 | 59.4 | 55.9 | 1405.1 | 61.5 | 54.8 | 30.6 | 68.6 | 87.0 | 62.7 |
| VisPruner | 75.8 | 58.2 | 57.0 | 1461.4 | 62.7 | 57.3 | 33.7 | 69.1 | 84.6 | 63.5 |
| CDPruner | 76.6 | 59.9 | 56.2 | 1431.4 | 63.1 | 55.0 | 32.8 | 69.0 | 87.7 | 63.5 |
| CoRePruner | 64.1 | |||||||||
| CoRePruner † | 64.0 | |||||||||
| Benchmark | Stage I-only | Stage II-only | w/o | Ours ( ) | |
|---|---|---|---|---|---|
| Retain 128 Tokens ( 77.8%) | |||||
| VQA Text | |||||
| GQA | |||||
| MME | |||||
| MMB EN | |||||
| MMVet | |||||
Visual Token Pruning
Visual token pruning reduces the image and video patches processed by transformers to lower inference cost while preserving task-relevant information.
See all 298 papers