Disagree to Prune: Reallocating Visual Tokens Where a Small VLM Dissents
Organizations: Nanjing University · Shanghai Artificial Intelligence Laboratory · Fudan University · Wuhan University ·
Abstract
Visual token pruning speeds up large vision-language models (VLMs) by keeping only a part of their visual tokens. Most existing methods improve the ranking of the tokens and then answer every question once with the same budget. With strong rankings, pruning hurts only a few questions, and signals computed on the kept tokens see little of what it removed. We find that, on most of these questions, a small VLM given the full image disagrees with the pruned answer, since it still sees the missing evidence. We propose PRUDENT, which first answers with about half the budget and spends the saved tokens on re-answering the disagreeing questions at twice the budget. We show when such re-answering pays and why the small VLM should read the full image. On 18 benchmarks, PRUDENT outperforms eight existing pruning methods at the same average budget of visual tokens. Its lead widens as the budget tightens, and holds across five InternVL sizes. PRUDENT also has the highest average in two more model families. The disagreement of the small VLM flags most pruning failures of all eight methods, and re-answering on it also improves the two strongest with their own rankings. Other checkers give similar gains, and the extra tokens go where the first pass hurts the answer. These results suggest that identifying which pruned answers are trustworthy provides an additional lever for visual token pruning beyond the ranking of tokens. Code and data are available at https://github.com/LiZaiyuan0619/PRUDENT.
Figures & tables
| Text-rich | General | Grounding | |||||||||||||||||
| Method | ChartQA | DocVQA | InfoVQA | TextVQA | OCRB. | WebSRC | GQA | VQAv2 | OK-VQA | MMB | MMB-CN | MME | POPE | SEED | AI2D | SQA-I | RefC-A | RefC-B | Avg. |
| Full tokens | 88.9 | 93.4 | 77.4 | 83.7 | 85.4 | 94.3 | 67.5 | 83.4 | 58.0 | 89.2 | 88.8 | 2466 | 90.8 | 79.2 | 87.8 | 99.4 | 94.6 | 88.5 | 100.0 |
| Retain 2.5% of the visual tokens | |||||||||||||||||||
| ToMe | 19.3 | 14.8 | 29.4 | 24.9 | 5.5 | 24.3 | 55.4 | 62.0 | 28.5 | 70.7 | 70.4 | 1780 | 82.7 | 62.2 | 69.1 | 82.1 | 14.0 | 11.1 | 51.8 |
| VisionZip | 17.7 | 12.9 | 29.0 | 15.4 | 4.8 | 19.7 | 48.2 | 51.7 | 16.5 | 53.2 | 52.7 | 1561 | 73.3 | 56.8 | 68.3 | 78.4 | 12.7 | 8.7 | 44.0 |
| ZOO-Prune | 25.5 | 17.2 | 27.9 | 24.2 | 5.8 | 25.7 | 55.7 | 63.8 | 29.3 | 70.6 | 68.9 | 1812 | 85.0 | 64.9 | 70.0 | 84.6 | 13.6 | 11.6 | 53.0 |
| Text-rich | General | Grounding | |||||||||||||||||
| Method | ChartQA | DocVQA | InfoVQA | TextVQA | OCRB. | WebSRC | GQA | VQAv2 | OK-VQA | MMB | MMB-CN | MME | POPE | SEED | AI2D | SQA-I | RefC-A | RefC-B | Avg. |
| Full tokens | 86.6 | 91.7 | 71.2 | 80.9 | 84.2 | 90.8 | 62.9 | 79.7 | 49.0 | 85.9 | 84.4 | 2384 | 91.0 | 78.1 | 84.1 | 98.4 | 94.3 | 86.5 | 100.0 |
| Retain 2.5% of the visual tokens | |||||||||||||||||||
| ToMe | 16.5 | 10.2 | 24.1 | 18.2 | 4.4 | 18.2 | 46.6 | 54.8 | 21.5 | 59.1 | 56.7 | 1534 | 77.2 | 56.5 | 66.5 | 77.5 | 11.3 | 9.6 | 46.5 |
| VisionZip | 14.0 | 9.5 | 24.6 | 11.5 | 3.0 | 17.4 | 41.6 | 47.0 | 10.2 | 44.2 | 43.6 | 1343 | 68.5 | 52.0 | 66.4 | 74.3 | 9.7 | 8.7 | 40.0 |
| ZOO-Prune | 27.4 | 14.7 | 23.1 | 25.1 | 9.1 | 24.6 | 50.7 | 62.0 | 29.3 | 66.9 | 66.2 | 1728 | 86.0 | 63.9 | 67.6 | 86.1 | 41.6 | 39.6 | 57.2 |
| Method | 8B | 14B | 26B | 38B | 78B |
|---|---|---|---|---|---|
| ToMe | 15.5 | 16.1 | 19.1 | 18.5 | 21.2 |
| VisionZip | 14.9 | 15.5 | 16.5 | 15.5 | 17.1 |
| ZOO-Prune | 18.8 | 17.6 | 20.2 | 18.3 | 19.1 |
| FastV | 15.7 | 21.8 | 17.2 | 22.8 | 19.7 |
| CDPruner | 21.1 | 21.7 | 25.3 | 20.9 | 26.8 |
| MMTok | 17.6 | 16.9 | 19.1 | 20.6 | 20.8 |
| Qwen3-VL-8B, 10% | Qwen3-VL-32B, 10% | Gemma 4 31B, 20% | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | TextVQA | ChartQA | DocVQA | GQA | MMB | Avg. | TextVQA | ChartQA | DocVQA | GQA | MMB | Avg. | TextVQA | ChartQA | DocVQA | GQA | MMB | Avg. |
| Full tokens | 82.1 | 85.2 | 95.5 | 61.5 | 84.6 | 100.0 | 83.5 | 87.3 | 96.2 | 62.5 | 88.1 | 100.0 | 75.8 | 68.0 | 93.0 | 47.3 | 84.1 | 100.0 |
| ToMe | 36.2 | 25.2 | 28.3 | 51.5 | 64.9 | 57.2 | 39.5 | 21.2 | 30.2 | 54.4 | 73.7 | 59.5 | 60.8 | 39.5 | 40.3 | 44.7 | 82.7 | 76.1 |
| VisionZip | 49.1 | 35.1 | 44.1 | 49.5 | 65.6 | 60.3 | 50.5 | 33.1 | 40.1 | 53.2 | 77.1 | 63.1 | 62.6 | 65.4 | 60.0 | 44.4 | 83.6 | 86.6 |
| ZOO-Prune | 44.9 | 37.9 | 45.5 | 54.1 | 68.8 | 65.2 | 56.7 | 35.2 | 55.8 | 56.1 | 77.1 | 69.5 | n/a | n/a | n/a | n/a | n/a | n/a |
| FastV | 35.5 | 23.1 | 29.0 | 44.7 | 57.8 | 48.2 | 60.3 | 22.4 | 33.4 | 46.7 | 71.0 | 58.5 | 49.6 | 36.6 | 23.1 | 40.8 | 77.3 | 66.6 |
| Text-rich | General | |||||
| 2.5% | 5% | 10% | 2.5% | 5% | 10% | |
| PRUDENT (Ours) | 84.8 | 90.6 | 95.2 | 94.9 | 96.8 | 98.1 |
| Change one part | ||||||
| IF-Prune’s ranking in both passes | 7.8 | 5.4 | 2.9 | 0.5 | 0.3 | 0.2 |
| No checker, one pass at | 2.4 | 1.9 | 1.8 | 1.8 | 1.2 | 1.0 |
| The checker never abstains | 1.1 | 1.2 | 1.0 | 0.4 | 0.3 | 0.2 |
Appendix figures & tables52 assets
Supplementary material from the paper’s appendix.
Appendix
| InternVL3-38B | InternVL3-8B | |
| Candidate questions | 771 | 749 |
| Pairs built | 411 | 409 |
| with a bit-identical first-pass input | 14 | 16 |
| Pruned answer unchanged (%) | 90.0 (86.7 to 92.6) | 91.0 (87.8 to 93.4) |
| Full-token answer follows the edit (%) | 74.9 (70.5 to 78.9) | 69.7 (65.1 to 73.9) |
| Valid twins | 198 | 194 |
| Text-rich, InternVL3-38B | Text-rich, InternVL3-8B | General, InternVL3-38B | General, InternVL3-8B | |||||||||
| Budget | Thr. | Ours | Thr. | Ours | Thr. | Ours | Thr. | Ours | ||||
| 1% | 0.45 | 0.43 | 0.44 | 0.42 | 0.43 | 0.44 | 0.17 | 0.29 | 0.27 | 0.15 | 0.25 | 0.28 |
| 1.5% | 0.37 | 0.40 | 0.42 | 0.33 | 0.38 | 0.38 | 0.13 | 0.25 | 0.22 | 0.11 | 0.22 | 0.23 |
| 2% | 0.33 | 0.38 | 0.41 | 0.28 | 0.35 | 0.37 | 0.13 | 0.22 | 0.20 | 0.10 | 0.18 | 0.19 |
| 2.5% | 0.32 | 0.40 | 0.41 | 0.28 | 0.38 | 0.39 | 0.13 | 0.21 | 0.20 | 0.10 | 0.16 | 0.16 |
| 3% | 0.25 | 0.35 | 0.37 | 0.23 | 0.34 | 0.35 | 0.09 | 0.19 | 0.17 | 0.08 | 0.15 | 0.15 |
| Pruned view | Full view (Ours) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Budget | Random | Confidence | Order | Ceiling | Order | Ceiling | Order | Ceiling | Oracle | |
| Text-rich InternVL3-38B | 1% | 4.8 | 1.8 | 0.8 | 0.6 | 0.3 | 0.2 | 0.0 | 0.5 | 7.1 |
| 2.5% | 3.9 | 0.6 | 0.5 | 0.7 | 1.8 | 1.9 | 2.3 | 2.5 | 6.5 | |
| 5% | 2.0 | 0.0 | 1.0 | 1.1 | 2.0 | 2.2 | 2.7 | 3.0 | 5.5 | |
| 10% | 2.0 | 0.2 | 0.8 | 0.8 | 1.5 | 1.6 | 2.2 | 2.3 | 3.8 | |
| 20% | 1.2 | 0.2 | 0.7 | 0.8 | 1.1 | 1.2 | 1.5 | 1.6 | 2.6 | |
| Text-rich, InternVL3-38B | Text-rich, InternVL3-8B | General, InternVL3-38B | General, InternVL3-8B | ||||||||||
| Budget | Splits | Gap | Gap | Gap | Gap | ||||||||
| 1% | 34 | 0.50 | 1.5 | 1.3 | 0.50 | 1.5 | 1.1 | 0.50 | 1.5 | 0.3 | 0.50 | 1.8 | 0.1 |
| 1.5% | 48 | 0.50 | 1.3 | 1.5 | 0.42 | 1.3 | 1.2 | 0.42 | 1.8 | 0.6 | 0.42 | 2.0 | 0.1 |
| 2% | 60 | 0.38 | 1.5 | 1.5 | 0.38 | 1.8 | 1.3 | 0.31 | 2.0 | 0.5 | 0.25 | 2.2 | 0.2 |
| 2.5% | 70 | 0.30 | 1.6 | 1.3 | 0.30 | 1.8 | 0.9 | 0.25 | 2.4 | 0.3 | 0.30 | 2.4 | 0.2 |
| 3% | 78 | 0.29 | 1.7 | 0.7 | 0.25 | 1.7 | 1.0 | 0.25 | 2.5 | 0.5 | 0.25 | 2.6 | 0.2 |
| Text-rich | General | |||||||
|---|---|---|---|---|---|---|---|---|
| Budget | Thr. | Ours | Bound | Thr. | Ours | Bound | ||
| 1% | 0.51 | 0.42 | 0.42 | 0.92 | 0.08 | 0.24 | 0.25 | 0.62 |
| 1.5% | 0.57 | 0.44 | 0.44 | 0.92 | 0.10 | 0.36 | 0.38 | 0.87 |
| 2% | 0.52 | 0.42 | 0.43 | 0.86 | 0.23 | 0.45 | 0.46 | 0.95 |
| 2.5% | 0.54 | 0.45 | 0.46 | 0.81 | 0.34 | 0.49 | 0.49 | 0.92 |
| 3% | 0.42 | 0.38 | 0.40 | 0.79 | 0.50 | 0.50 | 0.51 | 1.00 |
| Retain 2.5% of the visual tokens | Retain 5% of the visual tokens | Retain 10% of the visual tokens | |||||||||||||
| Benchmark | Conf. | Pruned | Ours | Oracle | Conf. | Pruned | Ours | Oracle | Conf. | Pruned | Ours | Oracle | |||
| ChartQA | 0.2 | 1.8 | 3.1 | 4.1 | 9.0 | 1.6 | 3.4 | 4.3 | 5.2 | 8.6 | 0.7 | 2.5 | 3.7 | 4.8 | 6.1 |
| DocVQA | 0.7 | 2.1 | 3.5 | 4.4 | 6.1 | 0.1 | 1.0 | 2.4 | 3.3 | 4.4 | 0.1 | 1.2 | 2.0 | 2.3 | 3.0 |
| InfoVQA | 1.6 | 1.5 | 0.4 | 0.3 | 7.7 | 0.1 | 0.3 | 1.4 | 1.9 | 7.6 | 0.7 | 0.0 | 0.4 | 1.0 | 5.8 |
| TextVQA | 0.5 | 1.7 | 2.7 | 2.8 | 4.4 | 0.4 | 0.8 | 1.3 | 1.4 | 2.9 | 0.1 | 0.4 | 0.8 | 0.6 | 2.0 |
| OCRBench | 2.0 | 1.0 | 0.2 | 0.1 | 5.9 | 2.1 | 0.7 | 0.4 | 0.8 | 4.2 | 1.8 | 0.4 | 0.1 | 1.0 | 1.6 |
| Retain 2.5% of the visual tokens | Retain 5% of the visual tokens | Retain 10% of the visual tokens | |||||||||||||
| Benchmark | Conf. | Pruned | Ours | Oracle | Conf. | Pruned | Ours | Oracle | Conf. | Pruned | Ours | Oracle | |||
| ChartQA | 2.1 | 3.4 | 5.5 | 5.8 | 8.7 | 0.6 | 0.6 | 1.5 | 2.4 | 4.8 | 0.8 | 1.6 | 2.8 | 2.7 | 4.5 |
| DocVQA | 0.8 | 1.4 | 3.3 | 3.6 | 6.6 | 0.0 | 1.3 | 2.5 | 3.5 | 5.3 | 0.1 | 1.3 | 2.3 | 2.9 | 4.1 |
| InfoVQA | 1.0 | 0.1 | 1.1 | 1.3 | 10.2 | 1.6 | 1.3 | 0.1 | 0.2 | 8.1 | 0.7 | 0.2 | 0.7 | 1.4 | 6.5 |
| TextVQA | 0.2 | 1.6 | 3.0 | 3.1 | 4.8 | 0.1 | 1.1 | 1.7 | 1.5 | 3.2 | 0.2 | 0.9 | 1.5 | 1.1 | 2.6 |
| OCRBench | 2.8 | 1.0 | 0.0 | 0.3 | 6.0 | 2.1 | 1.0 | 0.4 | 0.4 | 3.9 | 1.6 | 0.2 | 0.3 | 0.2 | 1.9 |
| Text-rich | General | ||||||||
| View | Checker | Recall | False alarm | Differs | Recall | False alarm | Differs | ||
| First pass keeps 1% of the visual tokens | |||||||||
| Full | InternVL3-1B | 39.0 | 90.1 | 21.6 | 14.9 | 26.0 | 74.4 | 20.6 | 19.7 |
| InternVL3-2B | 30.5 | 91.5 | 13.6 | 10.5 | 19.7 | 78.0 | 14.4 | 14.7 | |
| InternVL3-8B | 25.5 | 92.5 | 10.5 | 8.6 | 14.4 | 80.2 | 9.2 | 10.6 | |
| InternVL3-14B | 23.1 | 93.2 | 9.3 | 7.6 | 13.8 | 80.2 | 8.5 | 10.3 | |
| Text-rich, InternVL3-38B | Text-rich, InternVL3-8B | General, InternVL3-38B | General, InternVL3-8B | Evidence pruned | ||||||
| Budget | Full | Pruned | Full | Pruned | Full | Pruned | Full | Pruned | 38B | 8B |
| 1% | 8.2 | 23.1 | 7.3 | 23.8 | 13.3 | 24.8 | 14.3 | 25.6 | 49.2 | 47.2 |
| 1.5% | 9.4 | 22.7 | 8.8 | 23.6 | 13.6 | 23.0 | 14.6 | 23.5 | 44.2 | 41.5 |
| 2% | 10.1 | 22.1 | 9.4 | 22.5 | 13.9 | 21.7 | 14.8 | 22.4 | 38.9 | 36.8 |
| 2.5% | 10.1 | 22.1 | 9.4 | 22.5 | 13.9 | 21.7 | 14.8 | 22.4 | 38.9 | 36.8 |
| 3% | 11.5 | 21.9 | 10.6 | 21.5 | 14.3 | 20.1 | 15.1 | 20.7 | 34.1 | 31.6 |
| Text-rich | General | |||
|---|---|---|---|---|
| Budget | Full | Pruned | Full | Pruned |
| 1% | 4.6 | 18.9 | 9.9 | 39.1 |
| 1.5% | 5.9 | 19.0 | 10.1 | 39.3 |
| 2% | 6.9 | 18.5 | 10.2 | 38.5 |
| 2.5% | 6.9 | 18.5 | 10.2 | 38.5 |
| 3% | 8.2 | 17.1 | 10.4 | 37.9 |
| Benchmark | Split | Questions | Metric |
| Text-rich | |||
| ChartQA [ 37 ] | test | 2,500 | relaxed accuracy |
| DocVQA [ 38 ] | validation | 5,349 | ANLS |
| InfoVQA [ 39 ] | validation | 2,801 | ANLS |
| TextVQA [ 42 ] | validation | 5,000 | VQA accuracy |
| OCRBench [ 33 ] | test | 1,000 | accuracy |
| Method | ChartQA | DocVQA | InfoVQA | TextVQA | OCRB. | GQA | VQAv2 | OK-VQA | MMB | POPE | Mean |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-VL-8B, retain 50% | |||||||||||
| ToMe | 79.9 | 89.8 | 79.5 | 94.7 | 76.8 | 99.4 | 96.9 | 94.6 | 98.4 | 98.1 | 90.8 |
| VisionZip | 93.5 | 97.8 | 90.8 | 98.9 | 88.2 | 97.2 | 98.0 | 97.3 | 98.7 | 100.0 | 96.0 |
| ZOO-Prune | 91.0 | 95.0 | 83.9 | 91.2 | 92.4 | 98.4 | 98.2 | 91.7 | 98.4 | 102.2 | 94.2 |
| CDPruner | 89.2 | 87.0 | 72.6 | 94.0 | 88.2 | 96.6 | 98.6 | 98.2 | 98.4 | 99.5 | 92.2 |
| MMTok | 93.1 | 95.2 | 85.7 | 96.6 | 86.3 | 98.4 | 98.1 | 98.1 | 98.8 | 100.0 | 95.0 |
| Fail. | Right | Disagr. | AUROC | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | 2.5% | 5% | 10% | 2.5% | 5% | 10% | 2.5% | 5% | 10% | Conf. | Ours |
| ToMe | 38.9 | 36.0 | 28.2 | 84.3 | 83.8 | 82.0 | 95.9 | 95.8 | 95.1 | 0.71 | 0.87 |
| VisionZip | 44.6 | 36.7 | 27.6 | 85.3 | 84.2 | 82.4 | 96.4 | 96.0 | 95.5 | 0.70 | 0.86 |
| ZOO-Prune | 35.4 | 29.8 | 23.3 | 83.2 | 81.9 | 79.9 | 95.6 | 95.2 | 94.3 | 0.74 | 0.87 |
| FastV | 36.6 | 28.6 | 20.4 | 84.2 | 82.5 | 79.9 | 96.0 | 95.4 | 94.7 | 0.72 | 0.89 |
| CDPruner | 34.7 | 29.3 | 23.3 | 83.5 | 82.2 | 80.4 | 95.6 | 95.2 | 94.6 | 0.72 | 0.88 |
| InternVL3-38B | InternVL3-8B | |||||||||||
| 2.5% | 5% | 10% | 2.5% | 5% | 10% | |||||||
| Benchmark | Fail. | Right | Fail. | Right | Fail. | Right | Fail. | Right | Fail. | Right | Fail. | Right |
| Text-rich | ||||||||||||
| ChartQA | 38.3 | 79.7 | 27.4 | 76.4 | 14.4 | 67.2 | 28.7 | 78.3 | 18.4 | 69.8 | 11.0 | 61.5 |
| DocVQA | 15.5 | 74.6 | 10.7 | 67.5 | 6.7 | 64.1 | 18.5 | 79.0 | 14.1 | 75.3 | 9.4 | 72.6 |
| InfoVQA | 26.1 | 61.6 | 20.2 | 56.8 | 14.8 | 52.0 | 28.0 | 69.3 | 21.2 | 66.5 | 16.2 | 61.4 |
| InternVL3-38B | InternVL3-8B | |||||||||||||||
| Benchmark | Rand. | 2B conf. | SGL exit | Two f prun. | 1B dis. | VLM conf. | 2B dis. | Ours | Rand. | 2B conf. | SGL exit | Two f prun. | 1B dis. | VLM conf. | 2B dis. | Ours |
| Text-rich | ||||||||||||||||
| ChartQA | 0.50 | 0.65 | 0.70 | 0.66 | 0.85 | 0.82 | 0.86 | 0.90 | 0.49 | 0.71 | 0.78 | 0.73 | 0.85 | 0.83 | 0.86 | 0.91 |
| DocVQA | 0.49 | 0.56 | 0.68 | 0.77 | 0.84 | 0.73 | 0.88 | 0.91 | 0.50 | 0.55 | 0.67 | 0.74 | 0.85 | 0.71 | 0.88 | 0.91 |
| InfoVQA | 0.51 | 0.57 | 0.63 | 0.68 | 0.69 | 0.70 | 0.77 | 0.81 | 0.51 | 0.54 | 0.60 | 0.66 | 0.69 | 0.64 | 0.76 | 0.79 |
| TextVQA | 0.49 | 0.65 | 0.69 | 0.72 | 0.80 | 0.74 | 0.84 | 0.89 | 0.49 | 0.65 | 0.68 | 0.70 | 0.82 | 0.76 | 0.85 | 0.89 |
| InternVL3-38B | InternVL3-8B | |||||||||||
| vs. IF-Prune | vs. SGL | vs. IF-Prune | vs. SGL | |||||||||
| Benchmark | 2.5% | 5% | 10% | 2.5% | 5% | 10% | 2.5% | 5% | 10% | 2.5% | 5% | 10% |
| Text-rich | ||||||||||||
| ChartQA | ||||||||||||
| DocVQA | ||||||||||||
| InfoVQA | ||||||||||||
| Ours vs. | ToMe | VisionZip | ZOO-Prune | FastV | CDPruner | MMTok | SGL | IF-Prune |
|---|---|---|---|---|---|---|---|---|
| InternVL3-38B (54 settings) | 54 | 54 | 54 | 54 | 54 | 53 | 52 | 48 |
| InternVL3-8B (54 settings) | 54 | 54 | 54 | 54 | 54 | 53 | 51 | 42 |
| Both (108 settings) | 108 | 108 | 108 | 108 | 108 | 106 | 103 | 90 |
| Both, Holm-corrected | 108 | 108 | 108 | 107 | 108 | 106 | 97 | 70 |
| Method | TextVQA | ChartQA | DocVQA | InfoVQA | OCRB. | WebSRC | GQA | MMB | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| InternVL3-38B | |||||||||
| Full tokens | 83.7 | 88.9 | 93.4 | 77.4 | 85.4 | 94.3 | 67.5 | 89.2 | 100.0 |
| Retain 20% of the visual tokens | |||||||||
| ToMe | 64.0 | 47.6 | 44.1 | 39.5 | 44.3 | 54.4 | 65.6 | 85.4 | 66.3 |
| VisionZip | 62.3 | 70.4 | 54.3 | 39.6 | 38.7 | 57.4 | 64.2 | 84.9 | 69.9 |
| ZOO-Prune | 57.1 | 72.0 | 52.3 | 38.3 | 39.6 | 64.7 | 65.3 | 85.5 | 70.3 |
| Text-rich | General | |||||||||
| Method | ChartQA | DocVQA | InfoVQA | TextVQA | OCRB. | WebSRC | GQA | VQAv2 | MMB | Avg. |
| Full tokens | 87.9 | 93.2 | 74.6 | 81.0 | 84.5 | 93.8 | 63.6 | 79.8 | 87.1 | 100.0 |
| Retain 2.5% of the visual tokens | ||||||||||
| ToMe | 16.2 | 11.2 | 26.4 | 17.7 | 4.0 | 19.9 | 49.1 | 57.3 | 59.2 | 36.7 |
| VisionZip | 15.6 | 11.0 | 26.9 | 11.6 | 3.0 | 19.0 | 42.4 | 47.3 | 41.5 | 30.8 |
| ZOO-Prune | 23.9 | 14.2 | 25.8 | 19.7 | 5.3 | 23.3 | 50.6 | 61.0 | 62.0 | 40.0 |
| Text-rich | General | |||||||||
| Method | ChartQA | DocVQA | InfoVQA | TextVQA | OCRB. | WebSRC | GQA | VQAv2 | MMB | Avg. |
| Full tokens | 86.6 | 92.2 | 72.6 | 83.1 | 82.4 | 91.0 | 64.9 | 83.5 | 85.6 | 100.0 |
| Retain 2.5% of the visual tokens | ||||||||||
| ToMe | 17.0 | 15.1 | 28.3 | 23.2 | 8.1 | 26.0 | 49.1 | 59.0 | 60.3 | 39.8 |
| VisionZip | 17.1 | 11.4 | 27.0 | 14.7 | 2.6 | 18.4 | 44.1 | 52.8 | 45.1 | 32.7 |
| ZOO-Prune | 28.2 | 20.3 | 26.6 | 29.4 | 13.1 | 28.3 | 53.0 | 66.1 | 63.3 | 45.4 |
| Text-rich | General | |||||||||
| Method | ChartQA | DocVQA | InfoVQA | TextVQA | OCRB. | WebSRC | GQA | VQAv2 | MMB | Avg. |
| Full tokens | 89.2 | 94.0 | 79.6 | 85.4 | 86.5 | 93.8 | 65.7 | 83.6 | 89.6 | 100.0 |
| Retain 2.5% of the visual tokens | ||||||||||
| ToMe | 22.6 | 16.8 | 31.5 | 28.0 | 10.1 | 28.1 | 54.8 | 65.2 | 72.2 | 44.3 |
| VisionZip | 20.2 | 13.4 | 29.5 | 17.9 | 4.8 | 21.8 | 48.1 | 55.2 | 54.3 | 36.0 |
| ZOO-Prune | 24.6 | 18.3 | 28.7 | 26.8 | 10.0 | 28.6 | 55.0 | 68.0 | 71.0 | 44.5 |
| Text-rich | General | ||||||||||
| Method | ChartQA | DocVQA | InfoVQA | TextVQA | OCRB. | GQA | VQAv2 | OK-VQA | MMB | POPE | Avg. |
| Full tokens | 85.2 | 95.5 | 82.0 | 82.1 | 85.5 | 61.5 | 80.7 | 50.7 | 84.6 | 90.8 | 100.0 |
| Retain 10% of the visual tokens | |||||||||||
| ToMe | 25.2 | 28.3 | 28.8 | 36.2 | 27.7 | 51.5 | 66.0 | 34.8 | 64.9 | 81.9 | 57.2 |
| VisionZip | 35.1 | 44.1 | 32.3 | 49.1 | 19.3 | 49.5 | 63.6 | 34.1 | 65.6 | 81.1 | 60.3 |
| ZOO-Prune | 37.9 | 45.5 | 31.0 | 44.9 | 34.3 | 54.1 | 70.1 | 37.3 | 68.8 | 88.7 | 65.2 |
| Text-rich | General | ||||||||||
| Method | ChartQA | DocVQA | InfoVQA | TextVQA | OCRB. | GQA | VQAv2 | OK-VQA | MMB | POPE | Avg. |
| Full tokens | 87.3 | 96.2 | 86.4 | 83.5 | 86.3 | 62.5 | 82.4 | 51.0 | 88.1 | 91.1 | 100.0 |
| Retain 10% of the visual tokens | |||||||||||
| ToMe | 21.2 | 30.2 | 29.5 | 39.5 | 28.2 | 54.4 | 69.2 | 40.8 | 73.7 | 82.6 | 59.5 |
| VisionZip | 33.1 | 40.1 | 31.7 | 50.5 | 24.9 | 53.2 | 68.9 | 39.3 | 77.1 | 83.8 | 63.1 |
| ZOO-Prune | 35.2 | 55.8 | 34.3 | 56.7 | 38.6 | 56.1 | 72.0 | 43.3 | 77.1 | 86.1 | 69.5 |
| Text-rich | General | ||||||||||
| Method | ChartQA | DocVQA | InfoVQA | TextVQA | OCRB. | GQA | VQAv2 | OK-VQA | MMB | POPE | Avg. |
| Full tokens | 68.0 | 93.0 | 78.4 | 75.8 | 83.9 | 47.3 | 64.5 | 38.0 | 84.1 | 91.0 | 100.0 |
| Retain 20% of the visual tokens | |||||||||||
| ToMe | 39.5 | 40.3 | 25.5 | 60.8 | 50.5 | 44.7 | 62.2 | 37.8 | 82.7 | 88.8 | 76.1 |
| VisionZip | 65.4 | 60.0 | 44.4 | 62.6 | 67.2 | 44.4 | 62.0 | 37.5 | 83.6 | 89.5 | 86.6 |
| ZOO-Prune | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
| IF-Prune, 5% | IF-Prune, 10% | IF-Prune, 20% | SGL, 5% | SGL, 10% | SGL, 20% | |||||||
| Benchmark | Alone | First | Alone | First | Alone | First | Alone | First | Alone | First | Alone | First |
| InternVL3-38B | ||||||||||||
| ChartQA | 63.4 | 63.5 | 76.5 | 76.7 | 84.6 | 86.5 ∗ | 68.3 | 72.2 ∗ | 74.2 | 79.4 ∗ | 81.3 | 84.9 ∗ |
| DocVQA | 82.7 | 84.3 ∗ | 87.1 | 89.7 ∗ | 90.4 | 91.7 ∗ | 85.0 | 87.4 ∗ | 87.9 | 90.3 ∗ | 90.6 | 92.0 ∗ |
| InfoVQA | 59.2 | 58.3 | 65.5 | 66.2 | 70.9 | 71.1 | 60.0 | 60.8 | 64.9 | 65.1 | 69.7 | 70.3 |
| TextVQA | 79.0 | 81.4 ∗ | 81.5 | 82.7 ∗ | 82.5 | 83.2 ∗ | 79.7 | 81.4 ∗ | 81.4 | 82.8 ∗ | 82.7 | 83.2 ∗ |
| Text-rich | General | |||||
| Method | 2.5% | 5% | 10% | 2.5% | 5% | 10% |
| InternVL3-38B | ||||||
| IF-Prune | 75.4 | 83.7 | 91.1 | 91.2 | 94.2 | 96.6 |
| Ours, 1B checker | 84.3 | 89.8 | 94.5 | 93.1 | 95.9 | 97.5 |
| Ours, 2B checker (default) | 84.1 | 90.4 | 94.8 | 93.8 | 96.2 | 97.9 |
| Ours, 8B checker | 84.1 | 90.9 | 95.4 | 94.1 | 96.3 | 98.0 |
| Retain 2.5% | Retain 5% | Retain 10% | |||||||||||
| Benchmark | Full | IF-Prune | 1B | 2B | 8B | IF-Prune | 1B | 2B | 8B | IF-Prune | 1B | 2B | 8B |
| Text-rich | |||||||||||||
| ChartQA | 88.9 | 52.2 | 73.6 | 71.9 | 72.0 | 63.4 | 77.9 | 78.2 | 78.1 | 76.5 | 83.6 | 83.8 | 85.2 |
| DocVQA | 93.4 | 77.2 | 84.8 | 85.5 | 85.5 | 82.7 | 88.0 | 88.4 | 89.4 | 87.1 | 90.6 | 91.1 | 91.4 |
| InfoVQA | 77.4 | 52.5 | 57.9 | 57.6 | 57.6 | 59.2 | 62.2 | 63.9 | 63.9 | 65.5 | 68.1 | 68.2 | 68.4 |
| TextVQA | 83.7 | 75.6 | 80.3 | 80.8 | 81.1 | 79.0 | 82.0 | 82.1 | 82.7 | 81.5 | 82.8 | 82.9 | 83.1 |
| Retain 2.5% | Retain 5% | Retain 10% | ||||||||
| Benchmark | Full | IF-Prune | 1B | 2B | IF-Prune | 1B | 2B | IF-Prune | 1B | 2B |
| Text-rich | ||||||||||
| ChartQA | 86.6 | 60.0 | 75.6 | 76.1 | 70.2 | 79.1 | 78.5 | 77.6 | 82.5 | 83.3 |
| DocVQA | 91.7 | 72.5 | 82.4 | 82.9 | 77.5 | 85.5 | 85.5 | 82.8 | 88.2 | 88.4 |
| InfoVQA | 71.2 | 45.8 | 52.8 | 53.1 | 53.2 | 57.8 | 58.4 | 58.7 | 63.3 | 64.3 |
| TextVQA | 80.9 | 73.3 | 77.7 | 78.4 | 76.5 | 80.0 | 79.9 | 79.0 | 80.9 | 81.2 |
| InternVL3-38B | InternVL3-78B | |||||||
| Method | Acc. (%) | Latency (ms) | Memory (GB) | FLOPs (%) | Acc. (%) | Latency (ms) | Memory (GB) | FLOPs (%) |
| Full tokens | 100.0 | 499 | 75.2 | 100.0 | 100.0 | 895 | 150.7 | 100.0 |
| Full tokens + draft | 100.0 | 475 | 77.1 | 103.2 | 100.0 | 811 | 152.7 | 101.8 |
| ToMe | 42.2 | 336 | 73.2 | 45.2 | 45.2 | 536 | 148.3 | 29.3 |
| VisionZip | 40.9 | 336 | 73.2 | 45.2 | 42.6 | 536 | 148.3 | 29.3 |
| ZOO-Prune | 47.5 | 487 | 73.2 | 66.3 | 51.4 | 814 | 148.3 | 51.8 |
| Text-rich | General | Mean | ||||||||||||||||
| ChartQA | DocVQA | InfoVQA | TextVQA | OCRB. | WebSRC | GQA | VQAv2 | OK-VQA | MMB | MMB-CN | MME | POPE | SEED | AI2D | SQA-I | Text | Gen. | |
| Retain 2.5% of the visual tokens | ||||||||||||||||||
| PRUDENT (Ours) | 80.8 | 91.5 | 74.4 | 96.6 | 77.6 | 83.3 | 97.5 | 93.1 | 81.9 | 95.7 | 94.7 | 96.6 | 99.0 | 96.3 | 90.3 | 93.1 | 84.1 | 93.8 |
| Change one part | ||||||||||||||||||
| IF-Prune’s ranking in both passes | 18.8 | 7.3 | 7.3 | 3.3 | 4.3 | 5.3 | 0.9 | +0.5 | 6.7 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 7.7 | 0.7 |
| No checker, one pass at | 3.5 | 3.7 | +0.5 | 3.2 | 0.2 | 1.8 | 1.2 | 3.8 | +1.4 | 1.4 | 1.5 | 3.7 | 0.4 | 1.2 | 3.8 | 2.4 | 2.0 | 1.8 |
| Text-rich | General | Mean | ||||||||||||||||
| ChartQA | DocVQA | InfoVQA | TextVQA | OCRB. | WebSRC | GQA | VQAv2 | OK-VQA | MMB | MMB-CN | MME | POPE | SEED | AI2D | SQA-I | Text | Gen. | |
| Retain 2.5% of the visual tokens | ||||||||||||||||||
| PRUDENT (Ours) | 87.9 | 90.3 | 74.6 | 96.9 | 80.3 | 83.8 | 99.5 | 96.9 | 94.2 | 95.8 | 94.1 | 96.6 | 99.5 | 97.3 | 91.6 | 94.4 | 85.6 | 96.0 |
| Change one part | ||||||||||||||||||
| IF-Prune’s ranking in both passes | 14.0 | 9.4 | 10.0 | 3.2 | 4.4 | 6.4 | 1.6 | 1.0 | +0.6 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 7.9 | 0.2 |
| No checker, one pass at | 6.4 | 3.6 | 1.5 | 3.8 | 0.0 | 1.5 | 2.4 | 1.8 | 0.5 | 1.8 | 1.1 | 2.6 | 0.5 | 1.7 | 3.5 | 2.9 | 2.8 | 1.9 |
| Text-rich | General | Mean | ||||||||||||||||
| ChartQA | DocVQA | InfoVQA | TextVQA | OCRB. | WebSRC | GQA | VQAv2 | OK-VQA | MMB | MMB-CN | MME | POPE | SEED | AI2D | SQA-I | Text | Gen. | |
| Retain 2.5% of the visual tokens | ||||||||||||||||||
| PRUDENT (Ours) | 80.8 | 91.5 | 74.4 | 96.6 | 77.6 | 83.3 | 97.5 | 93.1 | 81.9 | 95.7 | 94.7 | 96.6 | 99.0 | 96.3 | 90.3 | 93.1 | 84.1 | 93.8 |
| Change when the checker abstains, default probability 0.5 and 10 words | ||||||||||||||||||
| Below probability 0.3 | 1.6 | 0.1 | 0.5 | 0.1 | 0.2 | 0.3 | 0.3 | 0.6 | 3.3 | +0.1 | 0.5 | +0.4 | 0.0 | 0.0 | +0.5 | 0.0 | 0.5 | 0.4 |
| Below probability 0.4 | 0.8 | 0.0 | 0.4 | 0.0 | 0.2 | 0.2 | 0.1 | 0.1 | 1.8 | 0.0 | 0.4 | +0.4 | 0.0 | 0.0 | +0.3 | 0.0 | 0.3 | 0.2 |
| Text-rich | General | Mean | ||||||||||||||||
| ChartQA | DocVQA | InfoVQA | TextVQA | OCRB. | WebSRC | GQA | VQAv2 | OK-VQA | MMB | MMB-CN | MME | POPE | SEED | AI2D | SQA-I | Text | Gen. | |
| Retain 2.5% of the visual tokens | ||||||||||||||||||
| PRUDENT (Ours) | 87.9 | 90.3 | 74.6 | 96.9 | 80.3 | 83.8 | 99.5 | 96.9 | 94.2 | 95.8 | 94.1 | 96.6 | 99.5 | 97.3 | 91.6 | 94.4 | 85.6 | 96.0 |
| Change when the checker abstains, default probability 0.5 and 10 words | ||||||||||||||||||
| Below probability 0.3 | 0.7 | 0.0 | 1.1 | 0.2 | 0.4 | 0.5 | 0.1 | +0.5 | 0.9 | +0.2 | 0.0 | 0.0 | +0.1 | +0.1 | +0.2 | +0.1 | 0.5 | 0.0 |
| Below probability 0.4 | 0.7 | +0.1 | 0.6 | +0.2 | 0.0 | 0.4 | 0.0 | +0.4 | 0.6 | 0.0 | 0.1 | 0.1 | +0.1 | +0.1 | 0.0 | +0.1 | 0.2 | 0.0 |
| First pass 2.5% | First pass 5% | First pass 7.5% | |||||||
| Re-answer | 20% | 30% | 40% | 20% | 30% | 40% | 20% | 30% | 40% |
| InternVL3-38B | |||||||||
| ChartQA | 94.2 | 92.0 | 88.8 | 94.2 | 91.9 | 89.8 | 91.2 | 90.1 | 89.9 |
| DocVQA | 97.4 | 96.9 | 96.0 | 97.5 | 96.7 | 96.2 | 96.6 | 96.3 | 96.1 |
| InfoVQA | 88.7 | 85.9 | 84.4 | 88.2 | 86.6 | 85.4 | 87.5 | 87.1 | 86.6 |
| TextVQA | 98.9 | 99.2 | 98.7 | 99.2 | 99.2 | 99.0 | 98.9 | 98.6 | 98.6 |
| InternVL3-38B | InternVL3-8B | |||||||||||||||||
| 2.5% | 5% | 10% | 2.5% | 5% | 10% | |||||||||||||
| Benchmark | Re. | Hurt | Kept | Re. | Hurt | Kept | Re. | Hurt | Kept | Re. | Hurt | Kept | Re. | Hurt | Kept | Re. | Hurt | Kept |
| Text-rich | ||||||||||||||||||
| ChartQA | 25.6 | 69.7 | 11.7 | 21.3 | 58.5 | 7.6 | 21.3 | 51.8 | 3.4 | 25.6 | 62.7 | 5.4 | 21.3 | 53.1 | 4.5 | 21.3 | 34.5 | 1.8 |
| DocVQA | 25.2 | 71.7 | 6.6 | 21.0 | 59.0 | 3.9 | 21.0 | 43.0 | 1.9 | 25.2 | 73.4 | 9.1 | 21.0 | 61.7 | 5.6 | 21.0 | 48.9 | 2.5 |
| InfoVQA | 23.0 | 63.7 | 26.4 | 19.2 | 53.9 | 20.5 | 19.2 | 52.4 | 14.8 | 23.0 | 61.6 | 29.7 | 19.2 | 55.9 | 24.3 | 19.2 | 49.3 | 15.3 |
| Budget | ToMe | VisionZip | ZOO-Prune | FastV | CDPruner | MMTok | SGL | IF-Prune | Ours |
|---|---|---|---|---|---|---|---|---|---|
| InternVL3-38B | |||||||||
| 2.5% | 43.8 | 51.8 | 42.2 | 44.5 | 44.9 | 39.9 | 20.0 | 16.0 | 11.9 |
| 5% | 40.2 | 41.8 | 35.6 | 35.7 | 38.5 | 32.5 | 13.6 | 11.0 | 8.0 |
| 10% | 31.6 | 31.1 | 27.6 | 25.1 | 31.0 | 25.2 | 8.2 | 7.3 | 5.3 |
| InternVL3-8B | |||||||||
| 2.5% | 49.3 | 56.0 | 38.4 | 46.1 | 39.0 | 42.0 | 17.6 | 12.5 | 8.8 |
| InternVL3-38B | InternVL3-8B | |||||||||||||||||
| 2.5% | 5% | 10% | 2.5% | 5% | 10% | |||||||||||||
| Benchmark | Fix. | Brk. | Rec. | Fix. | Brk. | Rec. | Fix. | Brk. | Rec. | Fix. | Brk. | Rec. | Fix. | Brk. | Rec. | Fix. | Brk. | Rec. |
| Text-rich | ||||||||||||||||||
| ChartQA | 25.0 | 5.3 | 62.3 | 19.7 | 4.9 | 68.1 | 10.9 | 3.6 | 68.3 | 20.4 | 4.2 | 65.7 | 12.2 | 3.9 | 60.3 | 8.2 | 2.6 | 61.5 |
| DocVQA | 10.3 | 3.2 | 63.4 | 7.1 | 2.1 | 61.7 | 5.0 | 1.5 | 67.4 | 12.2 | 2.9 | 61.8 | 9.4 | 2.4 | 61.4 | 6.8 | 1.9 | 64.0 |
| InfoVQA | 14.1 | 9.2 | 43.8 | 11.6 | 7.4 | 48.1 | 8.7 | 6.1 | 45.5 | 15.1 | 8.3 | 44.3 | 12.7 | 7.7 | 49.7 | 11.2 | 5.5 | 53.2 |
Visual Token Pruning
Visual token pruning reduces the image and video patches processed by transformers to lower inference cost while preserving task-relevant information.
See all 298 papers