DiFiT: Distribution-Fidelity-guided Visual Token Pruning for Multimodal Large Language Models
Organizations: School of Artificial Intelligence, Anhui University, Hefei 230601, China · College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China · State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Anhui University, Hefei 230601, China
Abstract
Visual token pruning reduces the inference cost of multimodal large language models, but answer accuracy only partly reflects its effect. Pruning can change the probabilities of alternative response tokens even when greedy answers remain correct and unchanged. We compare full and pruned response distributions under shared reference prefixes. Average predictive drift grows with compression on fixed correctly answered samples. Separate analyses associate larger drift with more confidence-decision disagreements and sampling failures. We propose DiFiT (Distribution-Fidelity-guided Token pruning) to learn discrete selection from reference answers and full-token response distributions. Dual-View Distillation matches probabilities across the vocabulary, while answer supervision supports task quality. A learned scorer and differentiable Top- learn which visual evidence to retain. A multi-budget mask objective supports different retention ratios through one shared ranking over a frozen model. On Qwen2.5-VL-7B, DiFiT reduces sampling-induced negative flips by 30.3% relative to FastV at 10% retention on four-task greedy-correct controls. It preserves 98.74% of baseline performance on MME at 30% retention.
Figures & tables
| Method | DocVQA | TextVQA | OCRBench | ScienceQA | AI2D | MMMU | MME | POPE | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| ANLS | Acc. | Acc. | EM | Acc. | Acc. | Score | F1 | ||
| Qwen2.5-VL-7B | 94.78 | 83.67 | 83.40 | 87.26 | 93.65 | 50.78 | 2342.15 | 86.19 | 100.00% |
| Retain 30% Tokens (70% Compression Ratio) | |||||||||
| FastV (ECCV’2024) [ 6 ] | 84.01 | 80.22 | 68.70 | 83.06 | 86.92 | 49.44 | 2263.58 | 80.47 | 92.78% |
| LLaVA-PruMerge (ICCV’2025) [ 8 ] | 73.95 | 73.71 | 64.80 | 85.22 | 82.77 | 47.67 | 2239.64 | 83.69 | 89.56% |
| VisionZip (CVPR’2025) [ 7 ] | 86.11 | 77.30 | 71.10 | 86.61 | 87.86 | 49.44 | 2276.04 | 84.73 | 94.30% |
| Method | DocVQA | TextVQA | OCRBench | ScienceQA | AI2D | MMMU | MME | POPE | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| ANLS | Acc. | Acc. | EM | Acc. | Acc. | Score | F1 | ||
| Qwen2.5-VL-3B | 92.76 | 77.87 | 78.80 | 80.37 | 90.84 | 46.78 | 2168.46 | 86.94 | 100.00% |
| Retain 30% Tokens (70% Compression Ratio) | |||||||||
| FastV (ECCV’2024) [ 6 ] | 81.66 | 74.02 | 62.90 | 78.63 | 86.40 | 46.00 | 2048.53 | 82.94 | 93.01% |
| LLaVA-PruMerge (ICCV’2025) [ 8 ] | 65.43 | 64.29 | 55.30 | 78.93 | 79.24 | 46.00 | 2051.01 | 84.78 | 87.39% |
| VisionZip (CVPR’2025) [ 7 ] | 81.38 | 67.71 | 63.50 | 79.47 | 84.88 | 45.44 | 2065.69 | 86.00 | 92.36% |
| Method | DocVQA | TextVQA | OCRBench | ScienceQA | AI2D | MMMU | MME | POPE | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| ANLS | Acc. | Acc. | EM | Acc. | Acc. | Score | F1 | ||
| LLaVA-OV-1.5-8B | 97.87 | 79.51 | 83.00 | 98.56 | 94.01 | 56.44 | 2271.32 | 88.46 | 100.00% |
| Retain 30% Tokens (70% Compression Ratio) | |||||||||
| FastV (ECCV’2024) [ 6 ] | 86.51 | 72.27 | 59.60 | 91.22 | 83.52 | 53.67 | 2019.84 | 70.41 | 87.01% |
| DivPrune (CVPR’2025) [ 12 ] | 85.23 | 76.03 | 64.50 | 94.65 | 89.35 | 54.11 | 2104.77 | 88.23 | 92.47% |
| DiFiT † | 94.98 | 77.50 | 74.60 | 97.17 | 91.45 | 55.00 | 2221.99 | 87.00 | 96.74% |
| Method | Acc. | Retention | Peak Mem. | Prefill | Speedup |
|---|---|---|---|---|---|
| (%) | (%) | (GB) | (ms) | ( ) | |
| Baseline | 68.10 | 100.00 | 25.97 | 1413.34 | 1.00 |
| FastV | 65.75 | 96.55 | 24.63 | 851.76 | 1.66 |
| DART | 65.80 | 96.62 | 18.93 | 832.58 | 1.70 |
| DivPrune | 65.85 | 96.69 | 17.55 | 1184.00 | 1.19 |
| DiFiT | 66.38 | 97.47 | 16.45 | 603.90 | 2.34 |
| Sample set | Same answer required | Same samples across budgets | Correctness required | Main use |
|---|---|---|---|---|
| Output-matched | ✓ | ✗ | ✗ | Compare methods at each budget (Table VI (A)). |
| Fixed output-matched | ✓ | ✓ | ✗ | Follow drift as compression increases (Fig. 4 (c,d)). |
| Fixed correct output-matched | ✓ | ✓ | ✓ | Follow drift on consistently correct samples (Table VI (B)). |
| Correct output-matched | ✓ | ✗ | ✓ | Relate drift to behavior (Fig. 6 ; Table X ). |
| Common-correct | ✗ ∗ | ✗ | ✓ | Confidence controls and sampling tests. |
| A. Output-matched set Four tasks; correctness is not required. | ||||||||||||
| 30% retained ( ) | 20% retained ( ) | 10% retained ( ) | ||||||||||
| Method | JS | KL | Top-5 | Top-10 | JS | KL | Top-5 | Top-10 | JS | KL | Top-5 | Top-10 |
| FastV | 0.0195 | 0.1327 | 90.60 | 90.56 | 0.0234 | 0.1460 | 88.99 | 89.06 | 0.0171 | 0.0864 | 87.97 | 87.13 |
| VisionZip | 0.0162 | 0.1033 | 89.95 | 89.67 | 0.0239 | 0.1457 | 87.76 | 87.32 | 0.0254 | 0.1565 | 83.28 | 82.04 |
| DivPrune | 0.0148 | 0.0800 | 88.07 | 87.84 | 0.0202 | 0.1177 | 85.63 | 85.35 | 0.0289 | 0.1512 | 82.42 | 81.51 |
| DiFiT | 0.0092 | 0.0612 | 91.14 | 90.85 | 0.0129 | 0.0782 | 89.21 | 88.92 | 0.0127 | 0.0636 | 86.71 | 85.61 |
| Method | AURC | Cov. Gap | Risk Gap | A/R (%) |
| Retain 20% Tokens | ||||
| FastV | 0.1398 | 13.87 | ||
| VisionZip | 0.1565 | 14.94 | ||
| DivPrune | 0.1660 | 14.54 | ||
| DiFiT | 0.0849 | 8.74 | ||
| Retain 10% Tokens | ||||
| Common-correct | Output-matched | |||
|---|---|---|---|---|
| Method | 20% | 10% | 20% | 10% |
| FastV | 7.59 | 9.39 | 4.48 | 5.41 |
| VisionZip | 9.38 | 15.79 | 5.53 | 7.93 |
| DivPrune | 9.27 | 14.48 | 6.68 | 9.10 |
| DiFiT | 6.42 | 9.15 | 4.29 | 6.40 |
| Method | Sampled Score | Neg. Flip | Majority | Emp. JS |
|---|---|---|---|---|
| Retain 20% Tokens; | ||||
| Full | 99.05 | 1.31 | 99.75 | – |
| FastV | 98.07 | 2.40 | 99.32 | 0.07564 |
| VisionZip | 97.65 | 2.89 | 98.96 | 0.09239 |
| DivPrune | 97.03 | 3.75 | 98.89 | 0.09659 |
| DiFiT | 98.63 | 1.81 | 99.64 | 0.07494 |
| Method | A/R OR [95% CI] | Negative-flip OR [95% CI] |
|---|---|---|
| DiFiT | 1.283 [1.202, 1.368] | 1.335 [1.250, 1.426] |
| FastV | 1.589 [1.478, 1.708] | 1.547 [1.457, 1.642] |
| VisionZip | 1.357 [1.288, 1.430] | 1.380 [1.314, 1.450] |
| DivPrune | 1.373 [1.306, 1.443] | 1.389 [1.327, 1.455] |
| Config | DocVQA | OCRBench | AI2D | MME | POPE | Avg. |
|---|---|---|---|---|---|---|
| ANLS | Acc. | Acc. | Score | F1 | Ret. | |
| : – | 90.17 | 74.40 | 89.51 | 2287.93 | 81.47 | 94.43% |
| : – | 90.61 | 76.10 | 90.41 | 2262.23 | 82.41 | 95.12% |
| 89.65 | 76.80 | 91.03 | 2283.25 | 82.80 | 95.49% | |
| 88.26 | 69.60 | 88.44 | 2233.91 | 83.91 | 92.75% | |
| 90.03 | 73.60 | 90.19 | 2267.36 | 83.22 | 94.58% |
| Data Ratio | Time | DocVQA | OCRBench | MME |
|---|---|---|---|---|
| 10 / 10 / 1 | 40m | 88.09 | 73.40 | 2264.89 |
| 100 / – / – | 50m | 84.52 | 64.70 | 2169.48 |
| 50 / 50 / – | 1h40m | 89.37 | 75.00 | 2291.62 |
| – / 100 / – | 2h50m | 87.65 | 74.70 | 2287.69 |
| 100 / 100 / 10 | 5h30m | 90.17 | 76.10 | 2295.87 |
| Variant | AI2D | DocVQA | OCRBench | TextVQA | Macro Ret. | Macro KL | Macro JS | Top-5 | Top-10 |
|---|---|---|---|---|---|---|---|---|---|
| Hard-teacher distillation | 90.97 | 90.49 | 75.30 | 75.37 | 93.24% | 0.1785 | 0.0263 | 86.15% | 85.67% |
| CE + MBL (w/o DVD) | 89.93 | 89.47 | 73.80 | 71.39 | 91.06% | 0.2014 | 0.0298 | 85.65% | 85.08% |
| w/o cross-budget MBL (single-budget) | 89.64 | 89.13 | 74.10 | 75.55 | 92.22% | 0.1690 | 0.0259 | 87.26% | 86.99% |
| DiFiT (soft DVD + MBL) | 91.39 | 90.01 | 75.60 | 78.18 | 94.16% | 0.1609 | 0.0233 | 87.57% | 87.16% |
Visual Token Pruning
Visual token pruning reduces the image and video patches processed by transformers to lower inference cost while preserving task-relevant information.
See all 295 papers