Bifocal: Training-Free Mixed Granularity Visual Token Pruning for Multimodal LLMs

Peiran Yin; Hao Wu; Zihao Yang; Fengyu Zhou; Yijuan Liang; Haozhe Hu; Xudong Wang; Wei Zhang

cs.CVOct 13, 2026

Bifocal: Training-Free Mixed Granularity Visual Token Pruning for Multimodal LLMs

Authors: Peiran Yin, Hao Wu, Zihao Yang, Fengyu Zhou, Yijuan Liang, Haozhe Hu, Xudong Wang, Wei Zhang, +1 more

Organizations: EIT-NLP Lab, Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University · Uppsala University · University of Science and Technology of China

Abstract

Visual token pruning for multimodal large language models (MLLMs) appears highly effective, with existing methods retaining over 95% of unpruned performance on common VQA benchmarks using a quarter of the visual tokens or fewer. However, our extensive evaluation of ten methods across three MLLM families and five ability groups reveals a critical gap: strong general performance often comes at the cost of severe degradation on fine-grained tasks such as OCR and visual grounding. We trace these failures to two limitations of the prevailing paradigm of retaining the highest-ranked native patches. A single importance ranking overlooks the complementary evidence required by different tasks, while sparse patch selection disrupts both image content and spatial integrity. To address these limitations, we propose Bifocal, a training-free, two-stage pruning method built on a mixed-granularity framework. It allocates separate budgets to query-focused and image-intrinsic evidence, complemented by a pooled coarse representation of the entire image. Before the LLM, Bifocal matches question words to visual regions through a soft conjunction balanced against norm-debiased image saliency, while selecting image-intrinsic evidence to maximize representational coverage. Inside the LLM, it further prunes tokens using density-adjusted text-to-image attention. With roughly a quarter of the original visual tokens, Bifocal is the only evaluated method to retain over 90% of unpruned performance in every ability group across all three models, and ranks first on average in all nine model-budget settings. The code is available at: https://github.com/EIT-NLP/Bifocal

Figures & tables

Figure 1: Performance overview of Bifocal on LLaVA-1.5-7B. (a) General versus grounding performance of pruning methods in our evaluation, under matched budgets. (b, c) Radar charts over 20 evaluations in five ability groups, at an average of

128

visual tokens. Bifocal achieves state-of-the-art overall performance.

Figure 2: The image and text routes under a fixed token budget. image route; text route. Left: tokens selected by each route on two examples; marks tokens selected by both. Right: performance relative to the baseline on $20$ benchmarks at $144$ of $576$ tokens; half splits the budget evenly between the two routes, and random retains tokens at random.

Figure 3: Left: experiments testing positional and content integrity. Every row except ① keeps $144$ of the $576$ visual tokens; each cell reports RefCOCO testA / testB, with the best pruned result in bold and the second best underlined. Right: the tokens each row retains on one example. ground-truth box; predicted box; retained patch; extra native patch; pooled token, averaging its tile.

Figure 4: Overview of Bifocal , a training-free, two-stage visual token pruning method. Stage 1 runs before the LLM and retains a mixed-granularity representation: fine evidence at native-patch resolution, of two kinds, query-focused evidence selected for the question and image-intrinsic evidence selected from the image itself, and coarse tokens obtained by average pooling. Stage 2 keeps the top- $K$ tokens by text-to-image attention inside the LLM.

Method	General				Reasoning		OCR				Hall.	Avg.
Method	VQA v2	GQA	MMB	VQA OK	SQA-I	AI2D	VQA T	OCR-B	DocVQA	ChartQA	POPE
Baseline	78.5	61.9	64.1	53.5	69.4	55.2	46.0	314	21.5	18.2	85.9	100.0%
Average Tokens = 192
FastV	75.0	57.0	63.6	51.1	69.1	54.1	44.6	300	18.2	17.1	75.8	94.5%
PDrop	77.4	60.0	63.5	52.4	68.9	54.1	45.9	308	19.9	18.2	85.3	98.1%
PruMerge+	74.6	57.6	61.3	48.6	69.0	54.6	39.0	282	14.8	16.5	79.9	90.9%

Table 1: General multimodal benchmarks on LLaVA-1.5-7B. Methods are matched by layer-averaged visual-token budget. Within each budget, the best and second-best results are bold and underlined . Avg. averages benchmark-wise performance relative to the baseline.

Method	RefCOCO val	testA	testB	RefCOCO+ val	testA	testB	RefCOCOg val	test	Avg.
Baseline	56.2	64.4	47.5	50.1	59.4	38.9	48.8	48.4	100.0%
Average Tokens = 192
DART	32.9	36.5	30.7	28.5	33.0	26.8	29.6	29.4	60.2%
Nüwa	59.0	67.3	49.0	50.5	60.8	40.2	51.7	50.1	103.6%
HiPrune	5.3	5.5	7.2	4.5	4.8	5.6	5.2	4.8	10.6%
EADP	7.4	7.7	9.2	6.4	7.0	7.1	6.1	5.9	14.0%

Table 2: Visual grounding on LLaVA-1.5-7B. All entries are Acc @

0.5

. Budget groups and highlighting follow Table 1 .

Token allocation			General		Reasoning		OCR			Hall.	RefCOCO+		Avg.
$C$	$I$	$Q$	GQA	MMB	SQA	AI2D	OCR-B	DocVQA	ChartQA	POPE	testA	testB	Gen.	REC
Baseline(576)			61.9	64.1	69.4	55.2	314	21.5	18.2	85.9	59.4	38.9	100.0%	100.0%
–	224	–	60.6	63.8	68.2	54.8	200	20.9	17.3	84.3	49.7	32.5	93.6%	83.6%
–	112	112	59.8	63.9	68.3	53.1	213	21.2	17.4	84.2	55.0	38.3	93.9%	95.5%
–	–	224	59.6	63.7	68.0	51.9	242	20.8	16.8	83.7	56.5	38.5	93.9%	97.0%
64	160	–	60.3	63.7	68.2	54.6	209	19.4	17.3	83.7	53.1	34.4	92.9%	89.0%

Table 4: Evidence ablation .

C

I

and

Q

are the numbers of coarse-context, image-intrinsic and query-focused tokens, all rows apply the same Stage-2 pruning.

Figure 5: Top: the fine evidence Bifocal selects for one image under different queries, image-intrinsic evidence and query-focused evidence. Bottom: grounding results of LLaVA-1.5-7B and pruning methods under the same budget, with the ground truth and the prediction.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Alignment of the image and text routes with the baseline’s attention.

\bullet

text route.

\scriptstyle\blacksquare

image route. Error bars are

95\%

confidence intervals.

Figure 7: Deletion test on the eight REC splits . The horizontal axis is the number of tokens removed from the unpruned

576

. The vertical axis is the change in Acc @

0.5

relative to keeping all

576

, in percentage points. Each panel shows three curves:

\bullet

text route ,

\scriptstyle\blacksquare

image route , and

\scriptstyle\blacktriangle

random .

Figure 8: Answers under single-route inputs. Each row is one question, with the ground truth shown by the icon on the right. The four columns are four visual token inputs: image route (CLS top- $144$ ), text route (scoring-chain top- $144$ ), mix (half from each route), and baseline (all $576$ tokens). Within each panel, kept tokens retain their colour and discarded ones are dimmed.

Figure 9: Top row: autocorrelation against spatial lag, in cells. Bottom row: radially averaged power spectrum against spatial frequency, on a logarithmic vertical axis. Curves are the raw map $s_{w}$ , the map after the $3{\times}3$ mean filter $\tilde{s}_{w}$ , and the removed component $s_{w}-\tilde{s}_{w}$ . Thin lines are individual samples and thick lines their mean. The dashed vertical line marks the median half-width of the ground-truth boxes.

	Bias				Mechanism			Correction
Split	$\pi$	$\kappa$	$r$	$r_{\setminus\mathcal{H}}$	$\gamma$	$R^{2}_{\rho}$	$R^{2}_{\\|h\\|}$	$\eta_{\tilde{a}}$	$\eta_{S}$	$r_{s}$
GQA	22%	19 $\times$	0.53	0.49	0.96	0.67	0.17	75%	46%	0.87
TextVQA	17%	14 $\times$	0.30	0.25	0.96	0.59	0.16	64%	46%	0.92
RefCOCO testA	21%	18 $\times$	0.53	0.48	0.96	0.67	0.18	68%	46%	0.88
RefCOCO testB	22%	19 $\times$	0.54	0.49	0.96	0.63	0.14	75%	50%	0.87

Appendix

Table 6: Empirical check of the norm bias on LLaVA-1.5-7B, median over all images of each split. Symbols are defined in the text.

Figure 10: Three cases, one per row, five panels read left to right. (a) raw CLS attention $\tilde{a}$ ; (b) the top- $80$ cells kept by $\tilde{a}$ ; (c) the $576$ cells as points, abscissa the patch norm $\nu$ , ordinate $\tilde{a}$ , with the least-squares baseline in dark — points are inside the box, outside it, and the eight cells of largest norm; (d) the residual $S$ ; (e) the top- $80$ cells kept by $S$ . In (b) and (e) kept cells retain their colour and discarded ones are dimmed. The box is the ground truth.

Figure 11: One column per split, $n=600$ images each. The horizontal axis is the rank $r$ under the score ( $r=1$ is the highest-scoring cell). Top row: the probability that the cell ranked $r$ is one of the eight largest-norm cells. Bottom row: the probability that it falls inside the ground-truth box. Curves are raw CLS $\tilde{a}$ and the debiased $S$ , with the norm $\nu$ itself as a control in the bottom row. Bands are $95\%$ confidence intervals of the mean and the dashed line is chance.

Method	RefCOCO val	test	testA	testB	RefCOCO+ val	testA	testB	RefCOCOg val	test	Avg.
Baseline	89.4	89.4	91.6	84.4	83.0	88.9	75.8	83.9	84.3	100.0%
Retain 50% Tokens
VisionZip	77.9	78.4	81.9	72.9	75.2	81.1	68.4	77.5	77.4	89.6%
DART	75.7	76.7	80.8	71.2	70.0	76.7	62.8	72.2	72.8	85.5%
Nüwa	86.8	87.6	89.2	83.0	79.8	85.4	73.6	81.9	81.8	97.2%
HiPrune	77.5	78.0	81.2	73.2	75.1	80.6	68.3	77.5	77.8	89.4%

Appendix

Table 7: Visual grounding on Qwen2.5-VL-7B. All entries are Acc @

0.5

Method	General			Reasoning		OCR				Hall.	Avg.
Method	GQA	MMB	VQA OK	SQA-I	AI2D	VQA T	OCR-B	DocVQA	ChartQA	POPE
Baseline	64.2	67.2	44.2	70.2	65.4	64.9	521	68.5	54.9	86.4	100.0%
Average Tokens $\approx$ 640
VisionZip	61.2	65.0	46.8	68.0	63.5	61.2	474	55.3	42.0	86.2	93.4%
SID	62.5	65.1	40.5	68.3	63.8	58.1	415	50.0	39.5	86.1	89.5%
Nüwa	63.4	64.6	43.7	68.8	64.0	63.0	376	60.9	48.5	85.6	93.5%

Appendix

Table 8: General benchmarks on LLaVA-NeXT-7B. Within each tier, the best and second-best results are bold and underlined . Avg. averages benchmark-wise performance relative to the baseline.

Method	RefCOCO val	testA	testB	RefCOCO+ val	testA	testB	RefCOCOg val	test	Avg.
Baseline	85.4	89.6	78.0	77.0	86.4	66.4	81.9	80.4	100.0%
Average Tokens $\approx$ 640
VisionZip	8.5	8.4	8.2	7.1	7.9	7.0	9.0	9.2	10.1%
SID	35.0	38.9	30.6	30.8	34.8	26.0	33.1	33.5	40.7%
Nüwa	63.9	67.7	57.4	56.3	63.2	46.8	57.7	56.9	72.8%
EADP	35.9	39.0	30.9	31.4	35.1	25.1	33.8	34.2	41.1%

Appendix

Table 9: Visual grounding on LLaVA-NeXT-7B ( anyres ). All entries are Acc @

0.5

Method	Avg. Tokens	FLOPs (T)	Peak Mem. (GiB)	Prefill (ms)	TTFT (ms)	Speedup
LLaVA-1.5-7B	576	8.450	13.5	50.36	89.32	1.00 $\times$
VLM-Pruner	128	2.459	13.8	45.54	85.57	1.04 $\times$
EADP	128	2.454	13.5	25.41	81.67	1.09 $\times$
DART	128	2.483	13.4	35.86	75.00	1.19 $\times$
VisionZip	128	2.454	13.5	24.37	65.42	1.37 $\times$
HiPrune	128	2.454	13.5	24.40	64.75	1.38 $\times$

Appendix

Table 10: Efficiency on LLaVA-1.5-7B.

Figure 12: Grounding and OCR cases on Qwen2.5-VL-7B. In the grounding cases, is the ground truth and the prediction.

Figure 14: The two kinds of fine evidence under different queries. Each row is one POPE image. The first panel after the input shows the image-intrinsic evidence, and the remaining panels show the query-focused evidence for each query.

Explore similar work

S $^2$ Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models
Sep 1, 2026Yuanyuan Jia, Shunpu Tang, Qianqian YangEfficient VLM Inference Multimodal Large Language Models
College of Information Science and Electronic Engineering, Zhejiang University Hangzhou, China
TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference
Jun 25, 2026Tinghao Wang, Yichen Guo, Rui Huang +11Efficient VLM Inference Multimodal Large Language Models
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Nanyang Technological University · University of Electronic Science and Technology of China +1
QCPruner: Query-Conditioned Population Coverage for Visual Token Pruning
Sep 17, 2026Shengli He, Yongchao Liang, Roumeng He +4Multimodal Large Language Models Visual Token Pruning
Guizhou University · Shanghai Ocean University

Sep 1, 2026J/K move · Enter open · S save

S $^2$ Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models

Yuanyuan Jia, Shunpu Tang, Qianqian Yang

College of Information Science and Electronic Engineering, Zhejiang University Hangzhou, China

Efficient VLM Inference Multimodal Large Language Models Visual Token Pruning

Open paper AskPDF arXiv

Visual token pruning reduces the inference overhead of multimodal large language models (MLLMs) by retaining only a subset of visual tokens. Existing methods usually select tokens based on importance or redundancy. However, we observe that these criteria produce stable spatial biases across inputs and do not always outperform simple Uniform Grid sampling, highlighting the value of broad spatial coverage. Motivated by this, we propose S

^2

Prune, a training-free pruning method that preserves spatial coverage while adapting token density to local image structure. We first divide the image into regions and assign at least one token to each region to preserve coverage. The remaining token budget is then distributed according to Laplacian variation, giving more tokens to regions with richer structure. We then use Early Representation Change (ERC), computed from the first decoder block, to select representative tokens within each region. We evaluate S

^2

Prune across diverse settings and two MLLM architectures. On Qwen2.5-VL-7B-Instruct, it achieves the highest average accuracy among the evaluated training-free pruning methods. With only 32 of the original 576 visual tokens, it still retains 79.3% of the full-model performance. Code is available at https://github.com/yuanyuanjia71-spec/S2Prune.

Abstract

Explore similar work

S2^22Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models

TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference

QCPruner: Query-Conditioned Population Coverage for Visual Token Pruning

S $^2$ Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models