MiCo: Mutual Information Coverage Optimization through Semantic Erasure Modeling for Efficient MLLM Inference

Tinghao Wang; Yichen Guo; Qizhe Zhang; Yuan Zhang; Weimin Ouyang; Rui Huang; Jiajun Cao; Sixiang Chen

cs.CVSep 28, 2026

MiCo: Mutual Information Coverage Optimization through Semantic Erasure Modeling for Efficient MLLM Inference

Authors: Tinghao Wang, Yichen Guo, Qizhe Zhang, Yuan Zhang, Weimin Ouyang, Rui Huang, Jiajun Cao, Sixiang Chen, +6 more

Organizations: State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · University of Electronic Science and Technology of China · Nanyang Technological University · The University of Hong Kong · New York University

Abstract

Multimodal large language models (MLLMs) have demonstrated impressive performance in multimodal understanding, but processing large numbers of visual tokens results in high computational costs. While many methods have been proposed to reduce the number of visual tokens, most of them rely on heuristics and are prone to discarding substantial visual information during pruning, leading to degradation in model performance. In this work, by using a semantic erasure model, we derive a general mutual information coverage objective from task log-loss and propose MiCo, a training-free two-stage pruning method. MiCo first uses visual signals to select a representative candidate pool before visual tokens enter the language model, then performs task-aware subset selection within it. At each stage, suitable observable proxies instantiate the derived objective as a monotone submodular coverage function, which MiCo greedily optimizes under the token budget. MiCo is evaluated on diverse MLLMs ranging from 7B to 13B parameters across a broad range of image and video benchmarks spanning general visual reasoning, fine-grained OCR and grounding, hallucination detection, and long-video understanding. MiCo consistently achieves the best performance across nearly all evaluated models under all pruning ratios. On LLaVA-NEXT-13B, MiCo uses only 5.6% visual tokens, retains 97.5% of baseline performance, and achieves a 3.8-fold inference speedup. Our experiments demonstrate the effectiveness of MiCo and our mutual information coverage objective for visual token pruning.

Figures & tables

Panel of Figure 1 — Figure 1: MiCo: (a) two-stage pruning on LLaVA-1.5-7B at $T=64$ , retaining $576\!\rightarrow\!128\!\rightarrow\!46$ visual tokens for “What is the bird holding in its beak?”; (b) relative performance across eight image models and one video model in 22 model–budget settings, normalized to each model’s unpruned reference (video budgets are per frame); (c) POPE F1 retention relative to Vanilla versus one-token inference speedup on LLaVA-NeXT-13B.

Figure 2: Overview of MiCo’s two-stage pruning framework.

Method	LLaVA									Qwen-VL						InternVL
Model	1.5-7B			1.5-13B			NeXT-7B			2.5-VL-7B		3-VL-8B		3.5-9B		3-8B
Budget $T$	128	64	32	128	64	32	640	320	160	256	128	256	128	256	128	256	128
FastV	92.6	75.5	–	94.1	85.0	–	95.4	80.4	–	93.0	77.9	84.3	63.7	84.0	62.8	95.5	81.5
DivPrune	96.8	94.4	91.3	96.7	93.2	91.2	99.0	97.2	94.2	94.5	90.3	93.6	88.6	95.9	91.8	93.3	87.7
VisPruner	96.6	93.3	87.8	96.6	93.6	88.5	98.7	95.6	91.0	91.8	88.0	94.2	86.8	97.0	90.1	89.4	81.1
VScan	97.8	96.0	91.8	97.3	96.0	88.7	99.5	96.2	90.3	94.8	90.6	95.4	90.1	–	–	92.9	87.3

Table 2: Relative performance (%) across LLaVA, Qwen-VL, and InternVL models; Vanilla is 100%.

Figure 3: Performance–efficiency trade-offs (a, b) and MiCo’s latency by component with speedups over Vanilla (c, d); NeXT one-token, Video complete-answer latency. Details in Appendix D .

Retain 256 tokens (80.2% pruned)
	Qwen2.5-VL-7B			Qwen3-VL-8B
Method	RefCOCO	RefCOCO+	RefCOCOg	RefCOCO	RefCOCO+	RefCOCOg
Vanilla	86.8 (100.0%)	79.4 (100.0%)	85.5 (100.0%)	92.4 (100.0%)	88.7 (100.0%)	90.4 (100.0%)
FastV	33.0 (38.0%)	28.3 (35.6%)	35.5 (41.5%)	67.2 (72.7%)	59.0 (66.5%)	68.1 (75.3%)
VisPruner	22.3 (25.7%)	17.8 (22.4%)	21.7 (25.4%)	77.8 (84.2%)	71.7 (80.8%)	77.0 (85.2%)
ApET	44.0 (50.7%)	37.1 (46.7%)	42.6 (49.8%)	51.4 (55.6%)	47.1 (53.1%)	49.7 (55.0%)

Table 4: REC scores (%) on RefCOCO, RefCOCO+, and RefCOCOg.

	Qwen2.5-VL-7B						Qwen3-VL-8B
	$T=256$			$T=128$			$T=256$			$T=128$
Method	Text	Chart	Doc	Text	Chart	Doc	Text	Chart	Doc	Text	Chart	Doc
Vanilla	85.06	87.28	94.86	85.06	87.28	94.86	84.28	83.04	95.72	84.28	83.04	95.72
FastV	78.44	68.28	38.22	46.45	33.20	14.54	62.18	28.64	14.79	32.88	16.24	9.83
VisPruner	64.02	56.08	27.66	49.40	36.68	19.69	71.96	63.52	39.63	54.58	39.64	23.00
ApET	69.17	50.60	23.12	45.82	31.16	14.35	59.54	54.52	39.25	41.27	35.44	25.59

Table 5: TextVQA, ChartQA, and DocVQA scores (%).

Figure 5: Two-stage component comparison on Qwen2.5-VL-7B at

T=128

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Energy $\widehat{\mathcal{H}}_{v}$	Reliability $\widehat{\mathcal{R}}_{v}$	Task relevance $\widehat{\mathcal{P}}_{v}$
Stage 1	$\lVert x_{v}^{\mathrm{vis}}\rVert_{2}$	$\overline{A}^{\mathrm{vis}}_{\mathrm{CLS}\to v}$	$1$
Stage 2	$\lVert x_{v}^{(\ell_{\star})}\rVert_{2}$	$1$	$\overline{A}^{(\ell_{\star})}_{\mathrm{text}\to v}$

Appendix

Table 8: Stage-specific inputs to MiCo’s shared weighted-coverage selector.

Figure 6: Position-level validation of the Stage-1 information-amount proxy. Dense-label entropy from COCO-Stuff is compared with the mean raw Stage-1 feature norm at matched aspect bin and native position. The displayed fit uses a two-sided 1% trim ( $\rho=0.36832$ ); the pre-specified primary estimate is untrimmed ( $\rho=0.35943$ ). This is a geometry-conditioned $H(U_{v}\mid G)$ diagnostic rather than a direct measurement of prompt-conditioned entropy.

Figure 7: Three complementary diagnostics for Stage-1 reliability on LLaVA-NeXT-7B. (a) Category decoding is sensitive to target masking, but largely stable under context masking. (b) CLS attention has a positive concordance point estimate with decoding success under a linear probe. (c) The same direction persists with an MLP probe. Shading and error bars show image-bootstrap 95% confidence intervals over 200 test images and three fixed head seeds; concordance excludes strata without both successes and failures. The attention trends are suggestive, not conclusive, because the target-mask intervals include the 0.5 reference.

Figure 8: Local-range view of the Stage-2 task-relevance diagnostic on LLaVA-NeXT-7B, restricted to attention masses

0

–

0.08

. Points retain the original rank-decile means; the dashed line is the ideal calibration reference. The target is normalized RefCOCO box–cell overlap conditioned on the Stage-1 survivor pool. Higher-attention bins are outside the displayed range; the correlation reported in the text uses the complete held-out data.

Backbone	Decoder blocks $L$	Refinement $\ell_{\star}$	Stage-1 keep	Stage-2 keep $R$
LLaVA-1.5/NeXT-7B	32	7	$2T$	$\operatorname{round}(18T/25)$
LLaVA-1.5-13B	40	8	$2T$	$\operatorname{round}(3T/4)$
LLaVA-NeXT-13B	40	14	$2T$	$\operatorname{round}(6T/13)$
Qwen2.5-VL-7B	28	2	$2T$	$\operatorname{round}(12T/13)$
InternVL3-8B	28	4	$2T$	$\operatorname{round}(5T/6)$
Qwen3-VL-8B	36	6	$2T$	$\operatorname{round}(4T/5)$

Appendix

Table 9: MiCo budgets and refinement layers by architecture; video budgets are per frame.

Method	LLaVA-NeXT-7B			LLaVA-NeXT-13B
Method	Latency (ms) (Speedup)	Peak memory (GiB) (Reduction)	Logical FLOPs (T) (Reduction)	Latency (ms) (Speedup)	Peak memory (GiB) (Reduction)	Logical FLOPs (T) (Reduction)
Vanilla	290.0 (1.00 $\times$ )	15.8 (0.0%)	45.6 (0.0%)	484.1 (1.00 $\times$ )	28.4 (0.0%)	85.1 (0.0%)
Budget $T=640$ tokens per image
FastV	113.0 (2.57 $\times$ )	16.0 (-1.7%)	11.8 (74.1%)	154.9 (3.13 $\times$ )	28.1 (1.1%)	20.9 (75.4%)
SparseVLM	137.1 (2.11 $\times$ )	16.0 (-1.7%)	11.8 (74.2%)	176.3 (2.75 $\times$ )	28.1 (1.1%)	21.0 (75.3%)
DivPrune	108.7 (2.67 $\times$ )	14.2 (9.7%)	11.7 (74.4%)	154.2 (3.14 $\times$ )	25.9 (8.8%)	20.8 (75.6%)

Appendix

Table 10: LLaVA-NeXT-7B and LLaVA-NeXT-13B efficiency comparison (median latency; speedups and reductions are relative to each backbone’s Vanilla row).

Table 15

Component (GFLOPs)	$T=160$	$T=320$	$T=640$
Vision encoder	1836.289 (36.63%)	1836.289 (25.57%)	1836.289 (15.78%)
Multimodal projector	120.879 (2.41%)	120.879 (1.68%)	120.879 (1.04%)
Decoder layers and final normalization	3002.246 (59.90%)	5136.922 (71.54%)	9520.805 (81.81%)
Vocabulary output head	48.350 (0.96%)	78.496 (1.09%)	139.052 (1.19%)
Stage 1: scores, similarity and selection	3.729 (0.07%)	4.047 (0.06%)	4.684 (0.04%)
Stage 2: extra attention / relevance	0.113 (0.002%)	0.206 (0.003%)	0.391 (0.003%)

Appendix

Table 13: MiCo FLOPs breakdown on LLaVA-NeXT-7B (

K=7

); percentages are shares of total FLOPs. Pruning arithmetic is attributed to Stage 1 and Stage 2 separately.

Component (GFLOPs)	$T=16$	$T=32$	$T=64$
Vision encoder	41228.062 (69.53%)	41228.062 (54.73%)	41228.062 (37.19%)
Multimodal projector	1585.032 (2.67%)	1585.032 (2.10%)	1585.032 (1.43%)
Visual pooling / plumbing	0.155 (0.000%)	0.155 (0.000%)	0.155 (0.000%)
Decoder layers and final normalization	15444.005 (26.05%)	30518.109 (40.51%)	63934.284 (57.68%)
Vocabulary output head	929.368 (1.57%)	1738.100 (2.31%)	3357.739 (3.03%)
Pruning operators (Stage 1 and Stage 2, jointly counted)	109.016 (0.18%)	261.046 (0.35%)	745.496 (0.67%)

Appendix

Table 14: MiCo FLOPs breakdown on LLaVA-Video-7B (

K=6

) on the fixed 64-frame profiling cohort; percentages are shares of total FLOPs. The two pruning paths are counted jointly; the Triton greedy term is listed separately.

Figure 9: Additional component comparisons on Qwen2.5-VL-7B at

T=128

; these tasks are excluded from Acc 7 .

Method / relevance	$T$	AI2D	POPE	HallB	MME	MMB EN	MMB CN	MMStar	SQA IMG	Acc	Rel
MMTok (ICLR’26)	128	76.3	84.2	43.3	2120.7	79.8	77.5	53.7	81.4	75.3	90.8%
uniform	128	78.2	85.2	42.6	2149.09	80.2	78.9	56.3	83.8	76.6	92.3%
text-vision-sim	128	79.1	85.5	45.9	2142.01	79.7	79	56.5	84.3	77.1	93.0%
1-(text-vision-sim)	128	79.4	86.2	45.9	2170.42	80.3	78.7	55.5	83.4	77.2	93.1%
last-token-attn	128	80.5	85.6	48.3	2235.97	80.6	79	56.1	84.3	78.3	94.4%
MiCo (all-text-attn)	128	80.4	85.5	48.6	2252.37	80.8	79.1	56.5	84.6	78.5	94.7%

Appendix

Table 16: Stage-2 task-relevance ablation on Qwen2.5-VL-7B at

T=128

K=2

, and a Stage-1 pool of

N_{1}=2T

Figure 10: Stage-1 budget sensitivity on Qwen2.5-VL-7B at

T=128

: four illustrative tasks and the eight-task mean.

Figure 11: Stage-1 budget sensitivity on InternVL3-8B at

T=128

: four illustrative tasks and the eight-task mean.

Model	Refinement-layer configuration
LLaVA-1.5-7B	$K=7$
LLaVA-1.5-13B	$K=8$
LLaVA-NeXT-7B	$K=7$
LLaVA-NeXT-13B	$K=14$
Qwen3-VL-8B	$K=6$
InternVL3-8B	$K=4$

Appendix

Table 19: Refinement-layer configurations used in the reported experiments.

Figure 12: Refinement-layer selection on 100 OCRBench samples: LLaVA-1.5 at

T=64

and LLaVA-NeXT at

T=320

Panel of Figure 13 — Figure 13: Representative refinement-layer selection for Qwen3-VL-8B and InternVL3-8B on 100 OCRBench samples at $T=128$ .

Panel of Figure 17 — Figure 17: Layer-wise $\Delta$ KL on 100 OCRBench samples (lower is better); shading shows paired-bootstrap 95% intervals.

Method	GQA	SQA-IMG	TextVQA	POPE	MME	MMB-EN	MMB-CN	SEED	AI2D	MMMU	Acc	Rel
LLaVA-1.5-7B
Upper Bound — 576 tokens (no pruning)
Vanilla	61.9	69.5	58.2	85.9	1508.8	64.7	58.1	66.0	55.5	35.0	63.0	100.0%
Retain 128 tokens ( $\downarrow$ 77.8%)
FastV (ECCV’24)	54.0	69.2	56.4	68.2	1376.5	63.0	55.9	59.7	53.6	34.3	58.3	92.6%
SparseVLM (ICML’25)	57.3	69.0	56.3	83.1	1401.3	62.6	56.9	61.7	54.7	35.9	60.8	96.4%

Appendix

Table 20: Complete benchmark results across the LLaVA series, including MiCo-S1.

Method	AI2D	POPE	HallB	MME	MMB-EN	MMB-CN	MMStar	SQA-IMG	Acc	Rel
Qwen2.5-VL-7B
Vanilla	84.9	87.7	55.9	2302.0	84.8	82.9	65.5	86.8	83.0	100.0%
Retain 256 tokens (80.2% pruned)
FastV (ECCV’24)	78.4	83.0	49.1	2169.0	80.5	78.8	55.5	83.6	77.2	93.0%
SparseVLM (ICML’25)	77.6	82.9	46.4	2207.5	80.9	79.4	55.5	87.7	77.6	93.5%
DivPrune (CVPR’25)	81.2	85.3	46.6	2167.0	81.8	80.9	57.9	84.8	78.4	94.5%

Appendix

Table 21: Complete benchmark results for Qwen-VL and InternVL models, including MiCo-S1.

$K$	GQA	SQA IMG	TextVQA	POPE	MME	MMB EN	MMB CN	SEED	AI2D	MMMU	Acc	Rel
LLaVA-1.5-7B $T=32$
Vanilla	61.9	69.5	58.2	85.9	1508.8	64.7	58.1	66.0	55.5	35.0	63.0	100.0%
1	56.1	69.0	54.3	80.4	1295.0	58.9	52.2	57.5	52.7	33.3	57.9	91.9%
2	55.6	68.6	54.7	79.7	1298.2	58.4	52.2	57.8	53.3	34.4	58.0	92.0%
3	55.6	68.8	54.9	79.0	1336.5	59.7	53.0	57.9	53.0	34.2	58.3	92.5%
4	55.5	68.8	54.6	78.7	1323.3	59.5	52.7	57.9	53.0	35.2	58.2	92.4%

Appendix

Table 22: Complete refinement-layer sweeps for LLaVA-1.5 models across the three reported budgets.

$K$	GQA	SQA IMG	TextVQA	POPE	MME	MMB EN	MMB CN	SEED	AI2D	MMMU	Acc	Rel
LLaVA-NeXT-7B $T=160$
Vanilla	62.5	67.5	60.3	86.8	1511.8	65.8	57.3	69.7	64.7	35.2	64.5	100.0%
1	59.4	68.0	54.3	82.3	1379.2	62.4	54.3	63.4	62.8	36.8	61.3	94.9%
2	59.5	67.3	55.9	81.9	1388.0	62.6	54.9	63.4	63.0	37.0	61.5	95.3%
3	58.8	67.7	56.3	81.0	1442.7	62.8	55.6	63.5	62.6	35.3	61.6	95.4%
4	59.1	68.5	56.1	81.5	1419.2	62.3	55.1	63.5	62.9	35.3	61.5	95.3%

Appendix

Table 23: Complete refinement-layer sweeps for LLaVA-NeXT models across the three reported budgets.

Method	Chart QA	GQA	MMB CN	MMB EN	MME	MMStar	OCR Bench	POPE	Acc	Rel
Reference image-area reduction: 75.00%
Vanilla	88.5 (94.3) $\Delta\,5.8$	65.0 (88.2) $\Delta\,23.2$	67.4 (94.7) $\Delta\,27.3$	76.5 (97.9) $\Delta\,21.4$	70.4 (93.9) $\Delta\,23.5$	75.4 (86.7) $\Delta\,11.3$	83.8 (90.4) $\Delta\,6.6$	81.0 (96.7) $\Delta\,15.6$	76.0 (92.8) $\Delta\,16.8$	100.0 (100.0) $\Delta\,0.0$
FastV	57.8 ( 83.7 ) $\Delta\,25.9$	54.7 (77.3) $\Delta\,22.6$	58.7 (93.5) $\Delta\,34.8$	65.9 (95.9) $\Delta\,30.0$	62.0 (91.8) $\Delta\,29.9$	55.2 (76.2) $\Delta\,21.0$	43.0 (64.8) $\Delta\,21.8$	65.8 (91.5) $\Delta\,25.7$	57.9 (84.3) $\Delta\,26.5$	76.1 (90.8) $\Delta\,14.7$
SparseVLM	26.2 (51.5) $\Delta\,25.3$	50.4 (78.8) $\Delta\,28.4$	59.6 (93.7) $\Delta\,34.2$	71.0 (95.4) $\Delta\,24.5$	60.6 (92.5) $\Delta\,31.9$	53.0 (73.0) $\Delta\,20.0$	19.0 (44.3) $\Delta\,25.3$	70.2 (90.8) $\Delta\,20.6$	51.2 (77.5) $\Delta\,26.3$	67.4 (83.5) $\Delta\,16.1$
DivPrune	57.4 (78.0) $\Delta\,20.6$	60.3 ( 86.1 ) $\Delta\,25.7$	66.5 (93.9) $\Delta\,27.4$	68.7 (96.6) $\Delta\,27.9$	66.2 (92.4) $\Delta\,26.2$	59.6 ( 79.8 ) $\Delta\,20.3$	53.1 ( 72.3 ) $\Delta\,19.2$	75.4 ( 95.7 ) $\Delta\,20.3$	63.4 ( 86.8 ) $\Delta\,23.5$	83.4 ( 93.5 ) $\Delta\,10.1$
VisionZip	41.4 (65.9) $\Delta\,24.6$	57.6 (85.6) $\Delta\,28.0$	62.2 (93.7) $\Delta\,31.6$	70.0 (96.5) $\Delta\,26.4$	62.0 (92.5) $\Delta\,30.5$	55.7 (77.3) $\Delta\,21.5$	35.2 (60.4) $\Delta\,25.2$	64.6 (93.2) $\Delta\,28.7$	56.1 (83.1) $\Delta\,27.0$	73.8 (89.5) $\Delta\,15.7$

Appendix

Table 24: VTC-Bench scores (%) on Qwen2.5-VL-7B at

T=256

. Cells show Group A (Group B), with the absolute gap

\Delta

in percentage points below.

Method	Chart QA	GQA	MMB CN	MMB EN	MME	MMStar	OCR Bench	POPE	Acc	Rel
Reference image-area reduction: 75.00%
Vanilla	88.5 (94.3) $\Delta\,5.8$	65.0 (88.2) $\Delta\,23.2$	67.4 (94.7) $\Delta\,27.3$	76.5 (97.9) $\Delta\,21.4$	70.4 (93.9) $\Delta\,23.5$	75.4 (86.7) $\Delta\,11.3$	83.8 (90.4) $\Delta\,6.6$	81.0 (96.7) $\Delta\,15.6$	76.0 (92.8) $\Delta\,16.8$	100.0 (100.0) $\Delta\,0.0$
FastV	17.7 (40.1) $\Delta\,22.4$	37.3 (60.8) $\Delta\,23.5$	43.5 (82.3) $\Delta\,38.8$	49.3 (82.5) $\Delta\,33.2$	50.7 (77.8) $\Delta\,27.1$	34.4 (55.9) $\Delta\,21.5$	15.1 (26.9) $\Delta\,11.8$	49.6 (69.8) $\Delta\,20.2$	37.2 (62.0) $\Delta\,24.8$	48.9 (66.8) $\Delta\,17.8$
SparseVLM	10.0 (24.5) $\Delta\,14.4$	40.8 (68.5) $\Delta\,27.6$	50.9 (89.5) $\Delta\,38.6$	58.5 (89.7) $\Delta\,31.2$	57.7 (84.7) $\Delta\,26.9$	39.3 (61.4) $\Delta\,22.0$	11.7 (23.2) $\Delta\,11.5$	54.0 (80.3) $\Delta\,26.4$	40.4 (65.2) $\Delta\,24.8$	53.1 (70.2) $\Delta\,17.1$
DivPrune	35.4 ( 62.2 ) $\Delta\,26.8$	54.1 (82.9) $\Delta\,28.9$	59.1 (92.8) $\Delta\,33.7$	63.6 (94.7) $\Delta\,31.1$	67.6 (90.1) $\Delta\,22.5$	52.5 (74.0) $\Delta\,21.6$	31.3 ( 58.3 ) $\Delta\,27.0$	66.7 ( 94.2 ) $\Delta\,27.6$	53.8 ( 81.2 ) $\Delta\,27.4$	70.7 ( 87.4 ) $\Delta\,16.7$
VisionZip	24.3 (45.4) $\Delta\,21.0$	50.9 (81.3) $\Delta\,30.4$	53.9 (92.5) $\Delta\,38.6$	62.7 (94.8) $\Delta\,32.1$	62.0 (90.2) $\Delta\,28.2$	46.4 (70.7) $\Delta\,24.2$	17.3 (49.4) $\Delta\,32.0$	55.6 (91.1) $\Delta\,35.5$	46.6 (76.9) $\Delta\,30.3$	61.4 (82.8) $\Delta\,21.5$

Appendix

Table 25: VTC-Bench scores (%) on Qwen2.5-VL-7B at

T=128

. Cells show Group A (Group B), with the absolute gap

\Delta

in percentage points below.

Retain 256 tokens (80.2% pruned)
	Qwen2.5-VL-7B			Qwen3-VL-8B
Method	val	testA	testB	val	testA	testB
Vanilla	66.8 (100.0%)	47.3 (100.0%)	51.9 (100.0%)	49.1 (100.0%)	47.6 (100.0%)	47.8 (100.0%)
FastV (ECCV’24)	61.9 (92.8%)	27.4 (57.9%)	34.6 (66.7%)	55.3 (112.7%)	38.0 (79.9%)	39.6 (82.8%)
VisPruner (ICCV’25)	64.0 (95.8%)	35.8 (75.8%)	40.8 (78.5%)	52.0 (106.0%)	41.7 (87.5%)	42.9 (89.6%)
ApET (CVPR’26)	62.3 (93.3%)	27.6 (58.3%)	33.0 (63.6%)	50.0 (101.9%)	31.4 (66.0%)	35.1 (73.4%)

Appendix

Table 26: GREC scores (%) on gRefCOCO.

	LLaVA-NeXT-7B			LLaVA-NeXT-13B			Qwen2.5-VL-7B		InternVL3-8B
Method / $T$	640	320	160	640	320	160	256	128	256	128
Vanilla	50.7			51.1			87.7		88.1
FastV	37.0	11.9	–	38.9	18.9	–	70.6	39.6	55.0	25.0
DivPrune	38.2	30.1	23.8	39.5	31.7	26.2	75.1	66.7	44.9	32.5
VisPruner	43.6	34.2	25.8	47.8	37.9	31.6	71.5	65.5	26.9	6.9
MMTok	40.8	30.4	24.9	42.3	34.1	26.3	71.1	62.7	51.7	35.3

Appendix

Table 28: OCRBench scores (0–100, higher is better) on LLaVA-NeXT-7B/13B, Qwen2.5-VL-7B, and InternVL3-8B at token budget

T

Explore similar work

Jun 25, 2026cs.AI

TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference

Multimodal large language models (MLLMs) have achieved strong multimodal reasoning capabilities, but their efficiency is limited by the large number of visual tokens, which introduces substantial computational overhead. Visual token pruning offers a natural solution, yet existing methods are imperfect: attention-based criteria tend to retain redundant tokens, while diversity-based criteria are often agnostic to user instructions. Even methods that combine multiple criteria still lack a principled formulation of the intrinsic objective of token pruning. In this paper, we revisit visual token pruning from a first-principles perspective and formulate it as constructing Token Optimal Preservation Sets. Through a top-down information-theoretic analysis, we identify three fundamental principles for effective token selection: Task Relevance, Information Coverage, and Semantic Diversity. Based on these principles, we propose TOPS, a training-free and model-agnostic pruning module that can be applied to various MLLMs. Extensive experiments on 7 MLLM backbones and 14 benchmarks demonstrate that TOPS outperforms prior methods under diverse pruning settings. Notably, on LLaVA-NeXT, TOPS removes 77.8% of visual tokens while preserving 100.0% and 100.6% performance on its 7B and 13B models, respectively, suggesting that pruning redundant visual tokens can sometimes mitigate hallucination and inspire future lightweight MLLM design.

Tinghao Wang, Yichen Guo, Rui Huang +11

State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Nanyang Technological University · University of Electronic Science and Technology of China +1

Visual Token Pruning Multimodal Large Language Models LLM Inference Optimization Information-Theoretic Limits

Ask PDF

Sep 1, 2026cs.CV

S $^2$ Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models

Visual token pruning reduces the inference overhead of multimodal large language models (MLLMs) by retaining only a subset of visual tokens. Existing methods usually select tokens based on importance or redundancy. However, we observe that these criteria produce stable spatial biases across inputs and do not always outperform simple Uniform Grid sampling, highlighting the value of broad spatial coverage. Motivated by this, we propose S

^2

Prune, a training-free pruning method that preserves spatial coverage while adapting token density to local image structure. We first divide the image into regions and assign at least one token to each region to preserve coverage. The remaining token budget is then distributed according to Laplacian variation, giving more tokens to regions with richer structure. We then use Early Representation Change (ERC), computed from the first decoder block, to select representative tokens within each region. We evaluate S

^2

Prune across diverse settings and two MLLM architectures. On Qwen2.5-VL-7B-Instruct, it achieves the highest average accuracy among the evaluated training-free pruning methods. With only 32 of the original 576 visual tokens, it still retains 79.3% of the full-model performance. Code is available at https://github.com/yuanyuanjia71-spec/S2Prune.

Yuanyuan Jia, Shunpu Tang, Qianqian Yang

College of Information Science and Electronic Engineering, Zhejiang University Hangzhou, China

Visual Token Pruning Multimodal Large Language Models Geometric Bias

Ask PDF

Sep 17, 2026cs.CV

QCPruner: Query-Conditioned Population Coverage for Visual Token Pruning

The high visual-token load in multimodal large language models (MLLMs) motivates training-free pruning to reduce later-layer computation, but under a fixed budget, pruning must preserve query-relevant evidence while avoiding redundancy. Existing methods rank tokens, diversify selected subsets, or optimize coverage without using a shared per-visual query utility to weight both visual targets and candidate representatives. We introduce QCPruner, which makes both roles query-conditioned through bilateral utility weighting. Using keyword-matched query anchors, QCPruner fuses two cross-modal cues into utility and applies it to both visual targets and candidate representatives within visual-affinity-based coverage. The resulting nonnegative facility-location objective is monotone and submodular, retains the standard (1-1/e) greedy guarantee, and requires no model training or parameter updates. Across LLaVA-1.5, LLaVA-NeXT, LLaVA-Video, and Qwen2.5-VL, QCPruner achieves the highest average relative performance among evaluated complete-system pruning methods at every reported token budget. At 32 of 576 tokens on LLaVA-1.5-7B, it retains 96.1% of unpruned performance, versus 93.9% for the strongest evaluated baseline. At 256 of 1296 tokens on Qwen2.5-VL-7B, the corresponding values are 96.7% and 92.5%.

Shengli He, Yongchao Liang, Roumeng He +4

Guizhou University · Guizhou University, Guiyang, China · Shanghai Ocean University

Visual Token Pruning Multimodal Large Language Models Populations

Ask PDF

Abstract

Explore similar work

TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference

S2^22Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models

QCPruner: Query-Conditioned Population Coverage for Visual Token Pruning

S $^2$ Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models