Organizations: School of Artificial Intelligence, Anhui University, Hefei 230601, China · College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China · State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Anhui University, Hefei 230601, China
Visual token pruning reduces the inference cost of multimodal large language models, but answer accuracy only partly reflects its effect. Pruning can change the probabilities of alternative response tokens even when greedy answers remain correct and unchanged. We compare full and pruned response distributions under shared reference prefixes. Average predictive drift grows with compression on fixed correctly answered samples. Separate analyses associate larger drift with more confidence-decision disagreements and sampling failures. We propose DiFiT (Distribution-Fidelity-guided Token pruning) to learn discrete selection from reference answers and full-token response distributions. Dual-View Distillation matches probabilities across the vocabulary, while answer supervision supports task quality. A learned scorer and differentiable Top-K learn which visual evidence to retain. A multi-budget mask objective supports different retention ratios through one shared ranking over a frozen model. On Qwen2.5-VL-7B, DiFiT reduces sampling-induced negative flips by 30.3% relative to FastV at 10% retention on four-task greedy-correct controls. It preserves 98.74% of baseline performance on MME at 30% retention.
Figures & tables
Fig. 1: Probability drift and behavior changes beyond greedy answers. Left: KL/JS drift on OCRBench common-correct sets at each budget, a broad control. Middle: four-task accept/reject disagreement and selective-risk increase when reusing a full-model threshold set for 90% coverage. Right: negative flips and sampled score on sets all systems answer correctly under greedy decoding ( N=7,141 at 20%; N=4,427 at 10%). Pale-red bands identify DiFiT and are not confidence intervals. Figure 4 and Table X provide stricter output controls and adjusted drift–behavior associations.
Fig. 2: Overview of DiFiT. DVD matches the softly weighted student view to full-token response distributions. DPS scores visual importance and text relevance. MBL aligns soft masks with hard selection across budgets. CE and DVD supervise the 30% student path; MBL supervises masks at 30%, 20%, and 10%. Inference gathers hard Top- K tokens into a shortened sequence for generation.
Method
DocVQA
TextVQA
OCRBench
ScienceQA
AI2D
MMMU
MME
POPE
Avg.
ANLS
Acc.
Acc.
EM
Acc.
Acc.
Score
F1
Qwen2.5-VL-7B
94.78
83.67
83.40
87.26
93.65
50.78
2342.15
86.19
100.00%
Retain 30% Tokens (70% Compression Ratio)
FastV (ECCV’2024) [ 6 ]
84.01
80.22
68.70
83.06
86.92
49.44
2263.58
80.47
92.78%
LLaVA-PruMerge (ICCV’2025) [ 8 ]
73.95
73.71
64.80
85.22
82.77
47.67
2239.64
83.69
89.56%
VisionZip (CVPR’2025) [ 7 ]
86.11
77.30
71.10
86.61
87.86
49.44
2276.04
84.73
94.30%
TABLE I: Image-language results on Qwen2.5-VL-7B. Avg. is the mean of eight task scores divided by their full-token scores (%). Bold and underlined values mark the best and second-best pruning results; † marks training-based methods.
Method
DocVQA
TextVQA
OCRBench
ScienceQA
AI2D
MMMU
MME
POPE
Avg.
ANLS
Acc.
Acc.
EM
Acc.
Acc.
Score
F1
Qwen2.5-VL-3B
92.76
77.87
78.80
80.37
90.84
46.78
2168.46
86.94
100.00%
Retain 30% Tokens (70% Compression Ratio)
FastV (ECCV’2024) [ 6 ]
81.66
74.02
62.90
78.63
86.40
46.00
2048.53
82.94
93.01%
LLaVA-PruMerge (ICCV’2025) [ 8 ]
65.43
64.29
55.30
78.93
79.24
46.00
2051.01
84.78
87.39%
VisionZip (CVPR’2025) [ 7 ]
81.38
67.71
63.50
79.47
84.88
45.44
2065.69
86.00
92.36%
TABLE II: Image-language results on Qwen2.5-VL-3B. Full tokens define 100% retention. Bold and underlined values mark the best and second-best pruning results; † denotes training-based methods that use the same training data.
Method
DocVQA
TextVQA
OCRBench
ScienceQA
AI2D
MMMU
MME
POPE
Avg.
ANLS
Acc.
Acc.
EM
Acc.
Acc.
Score
F1
LLaVA-OV-1.5-8B
97.87
79.51
83.00
98.56
94.01
56.44
2271.32
88.46
100.00%
Retain 30% Tokens (70% Compression Ratio)
FastV (ECCV’2024) [ 6 ]
86.51
72.27
59.60
91.22
83.52
53.67
2019.84
70.41
87.01%
DivPrune (CVPR’2025) [ 12 ]
85.23
76.03
64.50
94.65
89.35
54.11
2104.77
88.23
92.47%
DiFiT †
94.98
77.50
74.60
97.17
91.45
55.00
2221.99
87.00
96.74%
TABLE III: Image-language results on LLaVA-OneVision-1.5-8B. Full tokens define 100% retention. Bold and underlined values mark the best and second-best pruning results; † denotes training-based methods that use the same training data.
Method
Acc.
Retention
Peak Mem.
Prefill
Speedup
(%)
(%)
(GB)
(ms)
( × )
Baseline
68.10
100.00
25.97
1413.34
1.00
FastV
65.75
96.55
24.63
851.76
1.66
DART
65.80
96.62
18.93
832.58
1.70
DivPrune
65.85
96.69
17.55
1184.00
1.19
DiFiT
66.38
97.47
16.45
603.90
2.34
TABLE IV: MVBench performance and efficiency at 20% retention. Qwen2.5-VL-7B is used for all methods in this comparison.
Sample set
Same answer required
Same samples across budgets
Correctness required
Main use
Output-matched
✓
✗
✗
Compare methods at each budget (Table VI (A)).
Fixed output-matched
✓
✓
✗
Follow drift as compression increases (Fig. 4 (c,d)).
Fixed correct output-matched
✓
✓
✓
Follow drift on consistently correct samples (Table VI (B)).
Correct output-matched
✓
✗
✓
Relate drift to behavior (Fig. 6 ; Table X ).
Common-correct
✗ ∗
✗
✓
Confidence controls and sampling tests.
TABLE V: Controlled sample sets used in the predictive-drift and behavior analyses. All sets compare the full model and four pruning methods. “Fixed” means the same samples are used at all token budgets.
A. Output-matched set Four tasks; correctness is not required.
30% retained ( N=9,189 )
20% retained ( N=7,201 )
10% retained ( N=4,383 )
Method
JS ↓
KL ↓
Top-5 ↑
Top-10 ↑
JS ↓
KL ↓
Top-5 ↑
Top-10 ↑
JS ↓
KL ↓
Top-5 ↑
Top-10 ↑
FastV
0.0195
0.1327
90.60
90.56
0.0234
0.1460
88.99
89.06
0.0171
0.0864
87.97
87.13
VisionZip
0.0162
0.1033
89.95
89.67
0.0239
0.1457
87.76
87.32
0.0254
0.1565
83.28
82.04
DivPrune
0.0148
0.0800
88.07
87.84
0.0202
0.1177
85.63
85.35
0.0289
0.1512
82.42
81.51
DiFiT
0.0092
0.0612
91.14
90.85
0.0129
0.0782
89.21
88.92
0.0127
0.0636
86.71
85.61
TABLE VI: Predictive fidelity under strict output controls on Qwen2.5-VL-7B. All metrics are equally weighted task averages evaluated under shared reference prefixes. Lower JS/KL and higher vocabulary overlap indicate greater fidelity.
Figure 9
Method
AURC ↓
∣ Cov. Gap ∣↓
Risk Gap ↓
A/R (%) ↓
Retain 20% Tokens
FastV
0.1398
7.64
+10.28
13.87
VisionZip
0.1565
8.65
+12.91
14.94
DivPrune
0.1660
8.60
+12.40
14.54
DiFiT
0.0849
2.32
+3.70
8.74
Retain 10% Tokens
TABLE VII: Results from reusing a fixed threshold set for 90% full-model coverage, averaged over five splits and four tasks. Coverage gap is the absolute pruned-minus-full difference; risk gap remains signed (pp). A/R is accept/reject disagreement. Bold values mark the best result among pruning methods within each token budget.
Common-correct
Output-matched
Method
20%
10%
20%
10%
FastV
7.59
9.39
4.48
5.41
VisionZip
9.38
15.79
5.53
7.93
DivPrune
9.27
14.48
6.68
9.10
DiFiT
6.42
9.15
4.29
6.40
TABLE VIII: Accept/reject disagreement (%) at 90% target coverage on common-correct and output-matched sets, averaged across five splits and four tasks. Lower values indicate closer agreement with the full model’s acceptance decisions.
Fig. 5: Separate DocVQA results at different token budgets on the full evaluation set for DiFiT. ANLS includes standard-error bars; performance retention uses the full-token ANLS of 94.78% as its denominator. Predictive divergences use the full evaluation set without requiring identical answers. In the top panel, dashed denotes retention and solid denotes ANLS.
Method
Sampled Score ↑
Neg. Flip ↓
Majority ↑
Emp. JS ↓
Retain 20% Tokens; N=7,141
Full
99.05
1.31
99.75
–
FastV
98.07
2.40
99.32
0.07564
VisionZip
97.65
2.89
98.96
0.09239
DivPrune
97.03
3.75
98.89
0.09659
DiFiT
98.63
1.81
99.64
0.07494
TABLE IX: Four-task average stochastic results; N is the total eligible count. Full and all pruning methods must answer correctly under greedy decoding. Empirical JS is against Full; bold marks the best result among the pruning methods.
Fig. 6: Decision disagreements and sampling errors increase with predictive drift on correct, output-matched samples. Five equal-sized groups run from low to high JS within each method, task, and budget. Q1 and Q5 are the lowest- and highest-drift groups. Both rates increase from the lowest- to the highest-drift group for every pruning method.
Method
A/R OR [95% CI]
Negative-flip OR [95% CI]
DiFiT
1.283 [1.202, 1.368]
1.335 [1.250, 1.426]
FastV
1.589 [1.478, 1.708]
1.547 [1.457, 1.642]
VisionZip
1.357 [1.288, 1.430]
1.380 [1.314, 1.450]
DivPrune
1.373 [1.306, 1.443]
1.389 [1.327, 1.455]
TABLE X: Association of JS drift with decision disagreements and sampling errors after adjustment. Entries are odds ratios per one-SD increase in JS, with 95% cluster-robust CIs. Models control full-model confidence, prediction margin, task, and budget. The A/R analysis has 15,436 observations; the sampling analysis has 86,680 trials from 8,668 observations. Samples can recur across budgets.
Config
DocVQA
OCRBench
AI2D
MME
POPE
Avg.
ANLS
Acc.
Acc.
Score
F1
Ret.
λ : 0.1 – 1.0
90.17
74.40
89.51
2287.93
81.47
94.43%
λ : 0.1 – 3.0
90.61
76.10
90.41
2262.23
82.41
95.12%
γ=0
89.65
76.80
91.03
2283.25
82.80
95.49%
γ=1.0
88.26
69.60
88.44
2233.91
83.91
92.75%
τ=1
90.03
73.60
90.19
2267.36
83.22
94.58%
TABLE XI: Hyperparameter sensitivity on Qwen2.5-VL-7B at 20% token retention. Each variant changes one setting from λ:0.1 – 2.0 , γ=0.5 , τ=2 . Avg. Ret. averages the five scores normalized by Full in Table I . The last row uses our scores from that table.
Data Ratio
Time
DocVQA
OCRBench
MME
10 / 10 / 1
40m
88.09
73.40
2264.89
100 / – / –
50m
84.52
64.70
2169.48
50 / 50 / –
1h40m
89.37
75.00
2291.62
– / 100 / –
2h50m
87.65
74.70
2287.69
100 / 100 / 10
5h30m
90.17
76.10
2295.87
TABLE XII: Training-data mixtures at 20% retention from earlier independent checkpoints, distinct from Table I . Ratios denote ChartQA/OCR-VQA/COCO.
Fig. 7: DocVQA performance–efficiency trade-off. The horizontal axis shows prefill speedup relative to the full-token model, and the vertical axis shows performance retention. Bubble sizes and annotations indicate peak GPU memory. Token-selection overhead can increase peak memory despite a smaller retained sequence.
Variant
AI2D
DocVQA
OCRBench
TextVQA
Macro Ret.
Macro KL ↓
Macro JS ↓
Top-5 ↑
Top-10 ↑
Hard-teacher distillation
90.97
90.49
75.30
75.37
93.24%
0.1785
0.0263
86.15%
85.67%
CE + MBL (w/o DVD)
89.93
89.47
73.80
71.39
91.06%
0.2014
0.0298
85.65%
85.08%
w/o cross-budget MBL (single-budget)
89.64
89.13
74.10
75.55
92.22%
0.1690
0.0259
87.26%
86.99%
DiFiT (soft DVD + MBL)
91.39
90.01
75.60
78.18
94.16%
0.1609
0.0233
87.57%
87.16%
TABLE XIII: Comparison of training losses at 20% retention. Task columns use each benchmark’s standard score under greedy decoding. Macro retention averages each task score divided by its full-token reference. KL, JS, and overlap are averages across tasks on the full evaluation sets. All variants use identical data and optimization settings.
Fig. 8: Qualitative token selections for a dense infographic at 10% retention (top) and scene text at 20% (bottom). Columns show the input image and retained regions for FastV, DivPrune, and DiFiT, with generated answers below each selection.
Shenzhen University of Advanced Technology, Shenzhen, China · School of Computer Science, Nanjing University, Nanjing, China · CUHK MMLab, CPII under InnoHK, Hong Kong, China