Organizations: EIT-NLP Lab, Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University · Uppsala University · University of Science and Technology of China
Visual token pruning for multimodal large language models (MLLMs) appears highly effective, with existing methods retaining over 95% of unpruned performance on common VQA benchmarks using a quarter of the visual tokens or fewer. However, our extensive evaluation of ten methods across three MLLM families and five ability groups reveals a critical gap: strong general performance often comes at the cost of severe degradation on fine-grained tasks such as OCR and visual grounding. We trace these failures to two limitations of the prevailing paradigm of retaining the highest-ranked native patches. A single importance ranking overlooks the complementary evidence required by different tasks, while sparse patch selection disrupts both image content and spatial integrity. To address these limitations, we propose Bifocal, a training-free, two-stage pruning method built on a mixed-granularity framework. It allocates separate budgets to query-focused and image-intrinsic evidence, complemented by a pooled coarse representation of the entire image. Before the LLM, Bifocal matches question words to visual regions through a soft conjunction balanced against norm-debiased image saliency, while selecting image-intrinsic evidence to maximize representational coverage. Inside the LLM, it further prunes tokens using density-adjusted text-to-image attention. With roughly a quarter of the original visual tokens, Bifocal is the only evaluated method to retain over 90% of unpruned performance in every ability group across all three models, and ranks first on average in all nine model-budget settings. The code is available at: https://github.com/EIT-NLP/Bifocal
Figures & tables
Figure 1: Performance overview of Bifocal on LLaVA-1.5-7B. (a) General versus grounding performance of pruning methods in our evaluation, under matched budgets. (b, c) Radar charts over 20 evaluations in five ability groups, at an average of 128 visual tokens. Bifocal achieves state-of-the-art overall performance.
Figure 2: The image and text routes under a fixed token budget. image route; text route. Left: tokens selected by each route on two examples; marks tokens selected by both. Right: performance relative to the baseline on 20 benchmarks at 144 of 576 tokens; half splits the budget evenly between the two routes, and random retains tokens at random.
Figure 3: Left: experiments testing positional and content integrity. Every row except ① keeps 144 of the 576 visual tokens; each cell reports RefCOCO testA / testB, with the best pruned result in bold and the second best underlined. Right: the tokens each row retains on one example. ground-truth box; predicted box; retained patch; extra native patch; pooled token, averaging its tile.
Figure 4: Overview of Bifocal , a training-free, two-stage visual token pruning method. Stage 1 runs before the LLM and retains a mixed-granularity representation: fine evidence at native-patch resolution, of two kinds, query-focused evidence selected for the question and image-intrinsic evidence selected from the image itself, and coarse tokens obtained by average pooling. Stage 2 keeps the top- K tokens by text-to-image attention inside the LLM.
Method
General
Reasoning
OCR
Hall.
Avg.
VQA v2
GQA
MMB
VQA OK
SQA-I
AI2D
VQA T
OCR-B
DocVQA
ChartQA
POPE
Baseline
78.5
61.9
64.1
53.5
69.4
55.2
46.0
314
21.5
18.2
85.9
100.0%
Average Tokens = 192
FastV
75.0
57.0
63.6
51.1
69.1
54.1
44.6
300
18.2
17.1
75.8
94.5%
PDrop
77.4
60.0
63.5
52.4
68.9
54.1
45.9
308
19.9
18.2
85.3
98.1%
PruMerge+
74.6
57.6
61.3
48.6
69.0
54.6
39.0
282
14.8
16.5
79.9
90.9%
Table 1: General multimodal benchmarks on LLaVA-1.5-7B. Methods are matched by layer-averaged visual-token budget. Within each budget, the best and second-best results are bold and underlined . Avg. averages benchmark-wise performance relative to the baseline.
Method
RefCOCO val
testA
testB
RefCOCO+ val
testA
testB
RefCOCOg val
test
Avg.
Baseline
56.2
64.4
47.5
50.1
59.4
38.9
48.8
48.4
100.0%
Average Tokens = 192
DART
32.9
36.5
30.7
28.5
33.0
26.8
29.6
29.4
60.2%
Nüwa
59.0
67.3
49.0
50.5
60.8
40.2
51.7
50.1
103.6%
HiPrune
5.3
5.5
7.2
4.5
4.8
5.6
5.2
4.8
10.6%
EADP
7.4
7.7
9.2
6.4
7.0
7.1
6.1
5.9
14.0%
Table 2: Visual grounding on LLaVA-1.5-7B. All entries are Acc @ 0.5 . Budget groups and highlighting follow Table 1 .
Token allocation
General
Reasoning
OCR
Hall.
RefCOCO+
Avg.
C
I
Q
GQA
MMB
SQA
AI2D
OCR-B
DocVQA
ChartQA
POPE
testA
testB
Gen.
REC
Baseline(576)
61.9
64.1
69.4
55.2
314
21.5
18.2
85.9
59.4
38.9
100.0%
100.0%
–
224
–
60.6
63.8
68.2
54.8
200
20.9
17.3
84.3
49.7
32.5
93.6%
83.6%
–
112
112
59.8
63.9
68.3
53.1
213
21.2
17.4
84.2
55.0
38.3
93.9%
95.5%
–
–
224
59.6
63.7
68.0
51.9
242
20.8
16.8
83.7
56.5
38.5
93.9%
97.0%
64
160
–
60.3
63.7
68.2
54.6
209
19.4
17.3
83.7
53.1
34.4
92.9%
89.0%
Table 4: Evidence ablation . C , I and Q are the numbers of coarse-context, image-intrinsic and query-focused tokens, all rows apply the same Stage-2 pruning.
Figure 5: Top: the fine evidence Bifocal selects for one image under different queries, image-intrinsic evidence and query-focused evidence. Bottom: grounding results of LLaVA-1.5-7B and pruning methods under the same budget, with the ground truth and the prediction.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Alignment of the image and text routes with the baseline’s attention. ∙ text route. ■ image route. Error bars are 95% confidence intervals.
Figure 7: Deletion test on the eight REC splits . The horizontal axis is the number of tokens removed from the unpruned 576 . The vertical axis is the change in Acc @ 0.5 relative to keeping all 576 , in percentage points. Each panel shows three curves: ∙ text route , ■ image route , and ▲ random .
Figure 8: Answers under single-route inputs. Each row is one question, with the ground truth shown by the icon on the right. The four columns are four visual token inputs: image route (CLS top- 144 ), text route (scoring-chain top- 144 ), mix (half from each route), and baseline (all 576 tokens). Within each panel, kept tokens retain their colour and discarded ones are dimmed.
Figure 9: Top row: autocorrelation against spatial lag, in cells. Bottom row: radially averaged power spectrum against spatial frequency, on a logarithmic vertical axis. Curves are the raw map sw , the map after the 3×3 mean filter s~w , and the removed component sw−s~w . Thin lines are individual samples and thick lines their mean. The dashed vertical line marks the median half-width of the ground-truth boxes.
Bias
Mechanism
Correction
Split
π
κ
r
r∖H
γ
Rρ2
R∥h∥2
ηa~
ηS
rs
GQA
22%
19 ×
0.53
0.49
0.96
0.67
0.17
75%
46%
0.87
TextVQA
17%
14 ×
0.30
0.25
0.96
0.59
0.16
64%
46%
0.92
RefCOCO testA
21%
18 ×
0.53
0.48
0.96
0.67
0.18
68%
46%
0.88
RefCOCO testB
22%
19 ×
0.54
0.49
0.96
0.63
0.14
75%
50%
0.87
Appendix
Table 6: Empirical check of the norm bias on LLaVA-1.5-7B, median over all images of each split. Symbols are defined in the text.
Figure 10: Three cases, one per row, five panels read left to right. (a) raw CLS attention a~ ; (b) the top- 80 cells kept by a~ ; (c) the 576 cells as points, abscissa the patch norm ν , ordinate a~ , with the least-squares baseline in dark — points are inside the box, outside it, and the eight cells of largest norm; (d) the residual S ; (e) the top- 80 cells kept by S . In (b) and (e) kept cells retain their colour and discarded ones are dimmed. The box is the ground truth.
Figure 11: One column per split, n=600 images each. The horizontal axis is the rank r under the score ( r=1 is the highest-scoring cell). Top row: the probability that the cell ranked r is one of the eight largest-norm cells. Bottom row: the probability that it falls inside the ground-truth box. Curves are raw CLS a~ and the debiased S , with the norm ν itself as a control in the bottom row. Bands are 95% confidence intervals of the mean and the dashed line is chance.
Method
RefCOCO val
test
testA
testB
RefCOCO+ val
testA
testB
RefCOCOg val
test
Avg.
Baseline
89.4
89.4
91.6
84.4
83.0
88.9
75.8
83.9
84.3
100.0%
Retain 50% Tokens
VisionZip
77.9
78.4
81.9
72.9
75.2
81.1
68.4
77.5
77.4
89.6%
DART
75.7
76.7
80.8
71.2
70.0
76.7
62.8
72.2
72.8
85.5%
Nüwa
86.8
87.6
89.2
83.0
79.8
85.4
73.6
81.9
81.8
97.2%
HiPrune
77.5
78.0
81.2
73.2
75.1
80.6
68.3
77.5
77.8
89.4%
Appendix
Table 7: Visual grounding on Qwen2.5-VL-7B. All entries are Acc @ 0.5 .
Method
General
Reasoning
OCR
Hall.
Avg.
GQA
MMB
VQA OK
SQA-I
AI2D
VQA T
OCR-B
DocVQA
ChartQA
POPE
Baseline
64.2
67.2
44.2
70.2
65.4
64.9
521
68.5
54.9
86.4
100.0%
Average Tokens ≈ 640
VisionZip
61.2
65.0
46.8
68.0
63.5
61.2
474
55.3
42.0
86.2
93.4%
SID
62.5
65.1
40.5
68.3
63.8
58.1
415
50.0
39.5
86.1
89.5%
Nüwa
63.4
64.6
43.7
68.8
64.0
63.0
376
60.9
48.5
85.6
93.5%
Appendix
Table 8: General benchmarks on LLaVA-NeXT-7B. Within each tier, the best and second-best results are bold and underlined . Avg. averages benchmark-wise performance relative to the baseline.
Method
RefCOCO val
testA
testB
RefCOCO+ val
testA
testB
RefCOCOg val
test
Avg.
Baseline
85.4
89.6
78.0
77.0
86.4
66.4
81.9
80.4
100.0%
Average Tokens ≈ 640
VisionZip
8.5
8.4
8.2
7.1
7.9
7.0
9.0
9.2
10.1%
SID
35.0
38.9
30.6
30.8
34.8
26.0
33.1
33.5
40.7%
Nüwa
63.9
67.7
57.4
56.3
63.2
46.8
57.7
56.9
72.8%
EADP
35.9
39.0
30.9
31.4
35.1
25.1
33.8
34.2
41.1%
Appendix
Table 9: Visual grounding on LLaVA-NeXT-7B ( anyres ). All entries are Acc @ 0.5 .
Method
Avg. Tokens
FLOPs (T)
Peak Mem. (GiB)
Prefill (ms)
TTFT (ms)
Speedup
LLaVA-1.5-7B
576
8.450
13.5
50.36
89.32
1.00 ×
VLM-Pruner
128
2.459
13.8
45.54
85.57
1.04 ×
EADP
128
2.454
13.5
25.41
81.67
1.09 ×
DART
128
2.483
13.4
35.86
75.00
1.19 ×
VisionZip
128
2.454
13.5
24.37
65.42
1.37 ×
HiPrune
128
2.454
13.5
24.40
64.75
1.38 ×
Appendix
Table 10: Efficiency on LLaVA-1.5-7B.
Figure 12: Grounding and OCR cases on Qwen2.5-VL-7B. In the grounding cases, is the ground truth and the prediction.
Figure 14: The two kinds of fine evidence under different queries. Each row is one POPE image. The first panel after the input shows the image-intrinsic evidence, and the remaining panels show the query-focused evidence for each query.
Visual token pruning reduces the inference overhead of multimodal large language models (MLLMs) by retaining only a subset of visual tokens. Existing methods usually select tokens based on importance or redundancy. However, we observe that these criteria produce stable spatial biases across inputs and do not always outperform simple Uniform Grid sampling, highlighting the value of broad spatial coverage. Motivated by this, we propose S2Prune, a training-free pruning method that preserves spatial coverage while adapting token density to local image structure. We first divide the image into regions and assign at least one token to each region to preserve coverage. The remaining token budget is then distributed according to Laplacian variation, giving more tokens to regions with richer structure. We then use Early Representation Change (ERC), computed from the first decoder block, to select representative tokens within each region. We evaluate S2Prune across diverse settings and two MLLM architectures. On Qwen2.5-VL-7B-Instruct, it achieves the highest average accuracy among the evaluated training-free pruning methods. With only 32 of the original 576 visual tokens, it still retains 79.3% of the full-model performance. Code is available at https://github.com/yuanyuanjia71-spec/S2Prune.
Yuanyuan Jia, Shunpu Tang, Qianqian Yang
College of Information Science and Electronic Engineering, Zhejiang University Hangzhou, China
Multimodal large language models (MLLMs) have achieved strong multimodal reasoning capabilities, but their efficiency is limited by the large number of visual tokens, which introduces substantial computational overhead. Visual token pruning offers a natural solution, yet existing methods are imperfect: attention-based criteria tend to retain redundant tokens, while diversity-based criteria are often agnostic to user instructions. Even methods that combine multiple criteria still lack a principled formulation of the intrinsic objective of token pruning. In this paper, we revisit visual token pruning from a first-principles perspective and formulate it as constructing Token Optimal Preservation Sets. Through a top-down information-theoretic analysis, we identify three fundamental principles for effective token selection: Task Relevance, Information Coverage, and Semantic Diversity. Based on these principles, we propose TOPS, a training-free and model-agnostic pruning module that can be applied to various MLLMs. Extensive experiments on 7 MLLM backbones and 14 benchmarks demonstrate that TOPS outperforms prior methods under diverse pruning settings. Notably, on LLaVA-NeXT, TOPS removes 77.8% of visual tokens while preserving 100.0% and 100.6% performance on its 7B and 13B models, respectively, suggesting that pruning redundant visual tokens can sometimes mitigate hallucination and inspire future lightweight MLLM design.
Tinghao Wang, Yichen Guo, Rui Huang +11
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Nanyang Technological University · University of Electronic Science and Technology of China +1
The high visual-token load in multimodal large language models (MLLMs) motivates training-free pruning to reduce later-layer computation, but under a fixed budget, pruning must preserve query-relevant evidence while avoiding redundancy. Existing methods rank tokens, diversify selected subsets, or optimize coverage without using a shared per-visual query utility to weight both visual targets and candidate representatives. We introduce QCPruner, which makes both roles query-conditioned through bilateral utility weighting. Using keyword-matched query anchors, QCPruner fuses two cross-modal cues into utility and applies it to both visual targets and candidate representatives within visual-affinity-based coverage. The resulting nonnegative facility-location objective is monotone and submodular, retains the standard (1-1/e) greedy guarantee, and requires no model training or parameter updates. Across LLaVA-1.5, LLaVA-NeXT, LLaVA-Video, and Qwen2.5-VL, QCPruner achieves the highest average relative performance among evaluated complete-system pruning methods at every reported token budget. At 32 of 576 tokens on LLaVA-1.5-7B, it retains 96.1% of unpruned performance, versus 93.9% for the strongest evaluated baseline. At 256 of 1296 tokens on Qwen2.5-VL-7B, the corresponding values are 96.7% and 92.5%.