Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs

Donghyeon Cho; Joowon Kim; Seungho Shin; Eunho Yang

cs.CVOct 27, 2026

Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs

Authors: Donghyeon Cho, Joowon Kim, Seungho Shin, Eunho Yang

Organizations: KAIST · AITRICS

Abstract

Diffusion vision-language models (DVLMs) process their entire input at every denoising step. Since visual tokens make up 89-97% of this input on the benchmarks we study, most of the decoding computation is spent repeatedly processing the same image. Visual token pruning methods partially relieve this burden by removing part of the visual tokens, but they lose the removed information and still process the remaining image at every step. To measure how often the image is actually needed, we compare LLaDA-V's predictions with and without the image at every decoding step and find that removing the image changes a token being unmasked at only 22.8% of steps on average and at as few as 7.9%. We therefore propose LOWN, a decoding method that decides at every step whether to process the image and how many tokens to unmask. LOWN predicts the masked tokens without the image and processes the image only when this prediction disagrees with the last image-conditioned prediction. Agreement certifies the earlier prediction for reuse, and the number of certified tokens sets how many are unmasked in that step. We further observe that the need for the image differs across benchmarks, so LOWN's adaptive decision is needed rather than a fixed schedule. Without training or a tuned threshold, LOWN retains 99.6% of LLaDA-V's score on ten benchmarks at 35.3% of the decoding FLOPs on average. LOWN also generalizes across LLaDA-based DVLMs, composes orthogonally with visual token pruning, and runs under key-value caching for further efficiency gains. Code and project page are available at https://hyeon9698.github.io/LOWN/

Figures & tables

Panel of Figure 1 — Figure 1: Look Only When Needed (LOWN). (a) Decoding with a diffusion VLM. Token pruning reduces the visual tokens processed during decoding, whereas LOWN decides at each step whether the visual tokens need to be processed at all and otherwise runs a TEXT-ONLY forward. (b) Composition of the input, averaged over ten benchmarks. Pruning reduces how many visual tokens each forward processes, LOWN whether a forward processes them at all. (c) Decoding FLOPs speedup. Each pruning method is shown at LOWN’s score (Appendix C.3 ), and LOWN + Token Pruning uses RedVTP ( $r{=}50\%$ ). (d) Quality relative to LLaDA-V when every method is run at the operating point closest to LOWN’s decoding FLOPs from above. LOWN keeps 99.6% on average.

Panel of Figure 2 — Figure 2: Visual dependence along the decoding trajectory of LLaDA-V. (a) Share of masked positions at which the FULL and TEXT-ONLY top-1 predictions disagree, over token position (front to back) and denoising step (early to late). (b) Share of steps on which the image changed at least one unmasked decision, per benchmark, with 95% confidence intervals.

Figure 3: Overview of LOWN. TEXT-ONLY agreement certifies anchor tokens for image-free unmasking, and an insufficient certified prefix triggers a FULL forward that refreshes the anchor and unmasks the positions where the two forwards disagree first.

Figure 4: Performance and efficiency on LLaDA-V . Gray dashed lines denote the baseline. Triangles and squares denote RedVTP and ZOO-Prune retention settings, respectively. Arrows connect each fixed-pruning setting to its composition with LOWN. KLASS is matched to LOWN’s FLOPs.

	Math	Science	Chart		MM	Doc	Cap	LLaVA-		DeCap
Method	Vista	QA	QA	MME	Bench	VQA	Arena	Bench	GQA	Bench	Avg.
LLaDA-V (Vanilla)	59.10	76.50	78.16	1719.49	70.88	83.91	$-$ 88.67	58.20	53.31	33.18	100.0%
FLOPs matched to LOWN
	51.70	70.70	74.32	1469.78	56.70	83.39	$-$ 91.00	56.90	50.97	33.03	93.0%
RedVTP	( $\times$ 0.354)	( $\times$ 0.286)	( $\times$ 0.641)	( $\times$ 0.256)	( $\times$ 0.280)	( $\times$ 0.520)	( $\times$ 0.411)	( $\times$ 0.505)	( $\times$ 0.260)	( $\times$ 0.411)	( $\times$ 0.392)
	53.70	73.87	76.44	1582.41	67.27	78.73	$-$ 93.17	54.30	50.16	31.02	94.2%

Table 1: Performance at matched decoding FLOPs on LLaDA-V . The upper block compares standalone methods at LOWN’s budget, and the lower block at the budget of LOWN with RedVTP (

r{=}50\%

). Competitors run on a discrete hyperparameter grid, so each cell uses the cheapest setting with at least LOWN’s FLOPs (Appendix C.3 ). Cells show scores with FLOPs relative to LLaDA-V below. Best scores are bold, second best underlined, and lowest FLOPs bold.

Method	MathVista					LLaVA-Bench					DeCapBench
Method	Score $\uparrow$	Time (s) $\downarrow$		FLOPs $\downarrow$		Score $\uparrow$	Time (s) $\downarrow$		FLOPs $\downarrow$		Score $\uparrow$	Time (s) $\downarrow$		FLOPs $\downarrow$
LLaDA-V	59.10	7.56		1.0000		58.20	6.99		1.0000		33.18	14.64		1.0000
Fast-dLLM	57.30	2.14		0.0856		63.00	2.02		0.0876		35.19	4.22		0.1104
Fast-dLLM + LOWN	57.10	1.23	( $-$ 42.4%)	0.0457	( $-$ 46.6%)	58.00	1.25	( $-$ 38.1%)	0.0489	( $-$ 44.1%)	33.35	2.10	( $-$ 50.1%)	0.0453	( $-$ 58.9%)

Table 2: Composition of LOWN and Fast-dLLM on LLaDA-V.

Figure 5: Performance and efficiency on LaViDa . Gray dashed lines denote the baseline. Triangles, squares, and diamonds denote RedVTP, ZOO-Prune, and D 3 ToM, respectively. Composition markers use RedVTP at 75% and ZOO-Prune at 33% retention.

	EgoSchema			MLVU
Method	Score $\uparrow$	FLOPs $\downarrow$		Score $\uparrow$	FLOPs $\downarrow$
VidLaDA	60.6	1.000		41.8	1.000
LOWN (Ours)	60.2	0.255	( $-$ 74.5%)	43.3	0.272	( $-$ 72.8%)

Table 3: Video-understanding results of LOWN on VidLaDA-8B.

	MathVista	LLaVA-Bench	DeCapBench
LOWN (Ours)	56.00 ( $\times$ 0.325)	57.40 ( $\times$ 0.454)	34.43 ( $\times$ 0.391)
Period 2	57.00 ( $\times$ 0.541)	56.10 ( $\times$ 0.516)	33.43 ( $\times$ 0.522)
Period 4	52.00 ( $\times$ 0.321)	49.00 ( $\times$ 0.274)	27.79 ( $\times$ 0.284)
Period 8	48.50 ( $\times$ 0.211)	41.30 ( $\times$ 0.154)	19.09 ( $\times$ 0.164)
Random	53.40 ( $\times$ 0.349)	53.80 ( $\times$ 0.456)	30.48 ( $\times$ 0.389)

Table 4: Comparison of LOWN with fixed and random image-access schedules on LLaDA-V. Cells report score and relative FLOPs.

	Adaptive	Math	Science	Chart		MM	Doc	Cap	LLaVA-		DeCap
Visual routing	acceptance	Vista	QA	QA	MME	Bench	VQA	Arena	Bench	GQA	Bench
✗	✗	59.10	76.50	78.16	1719.5	70.88	83.91	$-$ 88.67	58.20	53.31	33.18
[0.5pt/1.5pt]		57.60	77.39	77.76	1624.6	68.30	83.50	$-$ 86.83	57.90	52.53	33.02
✓	✗	0.5040 ( $-$ 49.6%)	0.3554 ( $-$ 64.5%)	0.7330 ( $-$ 26.7%)	0.3383 ( $-$ 66.2%)	0.3759 ( $-$ 62.4%)	0.6743 ( $-$ 32.6%)	0.6833 ( $-$ 31.7%)	0.7002 ( $-$ 30.0%)	0.3203 ( $-$ 68.0%)	0.6840 ( $-$ 31.6%)
[0.5pt/1.5pt]		56.00	78.83	78.36	1655.1	69.93	82.31	$-$ 85.67	57.40	53.24	34.43
✓	✓	0.3252 ( $-$ 35.5%)	0.2428 ( $-$ 31.7%)	0.5759 ( $-$ 21.4%)	0.2307 ( $-$ 31.8%)	0.2701 ( $-$ 28.1%)	0.4517 ( $-$ 33.0%)	0.3885 ( $-$ 43.1%)	0.4543 ( $-$ 35.1%)	0.2045 ( $-$ 36.2%)	0.3909 ( $-$ 42.9%)

Table 5: Ablation of visual routing and adaptive acceptance on LLaDA-V. Each setting reports scores and, below, relative FLOPs with the reduction from the preceding setting in parentheses.

Figure 6: Qualitative comparison with visual token pruning (RedVTP). In LOWN’s output, blue underlines mark tokens unmasked on FULL forward steps, while the remaining tokens are unmasked on TEXT-ONLY forward steps.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Visual tokens (%)	Unchanged decisions	Image-dependent steps	Already top-1
MathVista	89.0	89.0 [88.6, 89.3]	17.9 [17.2, 18.5]	90.2 [89.9, 90.6]
ScienceQA	91.6	91.8 [91.0, 92.5]	8.2 [7.5, 9.0]	88.2 [87.2, 89.0]
ChartQA	96.1	75.5 [74.2, 76.7]	24.5 [23.3, 25.8]	88.4 [87.4, 89.3]
MME	97.0	89.4 [88.5, 90.3]	10.6 [9.7, 11.5]	89.6 [88.7, 90.4]
MMBench	91.3	92.1 [91.3, 92.8]	7.9 [7.2, 8.7]	88.6 [87.6, 89.5]
DocVQA	96.6	71.8 [70.9, 72.7]	43.8 [42.4, 45.2]	86.4 [85.7, 87.1]

Appendix

Table 6: Per-benchmark observations on LLaDA-V. Visual tokens is their mean percentage of the FULL input. Unchanged decisions is the share of unmasked positions at which the FULL and TEXT-ONLY top-1 predictions agree. Image-dependent steps is the share of steps on which the image changed at least one unmasked decision. Already top-1 is the share of final tokens that were the top-1 prediction at their position before the step that unmasked them. Brackets give 95% confidence intervals, and the last row is the unweighted mean.

Panel of Figure 7 — Figure 7: Where and when the image changes a prediction, all ten benchmarks. Share of masked positions at which the FULL and TEXT-ONLY top-1 predictions disagree, over token position (front to back) and denoising step (early to late), as in Figure 2(a) .

Benchmark	Visual tokens (%)	Unchanged decisions	Image-dependent steps	Already top-1
MathVista	71.1	94.0 [93.8, 94.1]	10.4 [10.1, 10.7]	89.8 [89.6, 90.0]
ScienceQA	76.4	98.7 [98.6, 98.8]	1.3 [1.2, 1.4]	93.7 [93.5, 94.0]
ChartQA	88.8	83.6 [83.2, 84.0]	16.4 [16.0, 16.8]	92.6 [92.4, 92.9]
MME	87.9	96.1 [95.9, 96.3]	3.9 [3.7, 4.1]	93.7 [93.5, 94.0]
MMBench	78.4	96.8 [96.6, 96.9]	3.2 [3.1, 3.4]	93.7 [93.5, 93.9]
DocVQA	88.3	89.4 [89.3, 89.6]	15.2 [14.9, 15.4]	91.6 [91.4, 91.7]

Appendix

Table 7: Per-benchmark observations on LaViDa, in the format of Table 6 , with the response lengths and step counts of the LLaDA-V measurement and over the full evaluation sets. Brackets give 95% confidence intervals, and the last row is the unweighted mean.

Benchmark	Samples	Mean $\rho$ (%)
MathVista	1,000	10.07
ScienceQA	2,017	7.92
ChartQA	2,500	2.54
MME	2,374	3.15
MMBench	4,329	7.04
DocVQA	5,349	3.02

Appendix

Table 8: Mean TEXT-ONLY/FULL forward-cost ratio on LLaDA-V. For each sample, we compute

\rho_{i}=\mathcal{C}(p_{i}+o_{i})/\mathcal{C}(p_{i}+v_{i}+o_{i})

using its prompt length

p_{i}

, visual-token count

v_{i}

, and configured response length

o_{i}

. Each row averages these ratios over the benchmark’s samples, and

\mathcal{C}

is the analytical forward cost in Equation 4 .

FLOPs matched to LOWN
	MathVista	ScienceQA	ChartQA	MME	MMBench	DocVQA	CapArena	LLaVA-B.	GQA	DeCapB.
RedVTP $r$	30	20	60	20	20	50	40	50	20	40
ZOO-Prune $r$	30	20	80	30	20	60	50	60	20	50
KLASS $(\tau,\kappa)$	(0.2, 0.2)	(0.2, 1)	(0.2, 1)	(0.2, 0.3)	(0.2, 0.4)	(0.2, 0.1)	(0.2, 0.5)	(0.2, 1)	(0.2, 1)	(0.2, 0.4)
FLOPs matched to LOWN + RedVTP ( $r=50\%$ )
RedVTP $r$	20	10	30	10	20	25	20	25	10	20

Appendix

Table 9: Settings behind the competitor cells of Table 1 (FLOPs matched) and Table 10 (time matched). For RedVTP,

r

is the visual-token retention percentage. For ZOO-Prune, the released code sets the budget to

r\%

of a fixed 2,880 tokens rather than of each image’s visual tokens, so the fraction actually retained varies with the image. For KLASS,

(\tau,\kappa)

denotes its confidence and KL thresholds. The second and fourth blocks use the budget of LOWN composed with RedVTP at

r=50\%

	Math	Science	Chart		MM	Doc	Cap	LLaVA-		DeCap
Method	Vista	QA	QA	MME	Bench	VQA	Arena	Bench	GQA	Bench	Avg.
LLaDA-V (Vanilla)	59.10	76.50	78.16	1719.49	70.88	83.91	$-$ 88.67	58.20	53.31	33.18	100.0%
Time matched to LOWN
	51.70	65.49	72.20	1275.08	40.12	83.39	$-$ 88.50	57.60	48.05	33.88	88.7%
RedVTP	( $\times$ 0.394)	( $\times$ 0.374)	( $\times$ 0.674)	( $\times$ 0.300)	( $\times$ 0.366)	( $\times$ 0.566)	( $\times$ 0.507)	( $\times$ 0.612)	( $\times$ 0.318)	( $\times$ 0.504)	( $\times$ 0.461)
	53.70	72.48	76.00	1582.41	68.73	78.73	$-$ 94.67	56.70	50.50	31.15	94.6%

Appendix

Table 10: Performance at matched generation time on LLaDA-V . The upper block compares standalone methods at LOWN’s generation time, and the lower block at that of LOWN with RedVTP (

r{=}50\%

). Competitors run on a discrete hyperparameter grid, so each cell uses the setting with the shortest measured time at or above LOWN’s (Table 9 ), and LOWN gives the highest normalized average in both blocks. Cells show scores with mean generation time relative to LLaDA-V below. Best scores are bold, second best underlined, and lowest times bold.

LLaDA-V
Configuration before LOWN	FLOPs ratio $\downarrow$	Reduction $\uparrow$	Quality index $\uparrow$
LLaDA-V (unpruned)	0.353	64.7%	99.6
RedVTP (50% retention)	0.424	57.6%	100.9
ZOO-Prune (33% retention)	0.394	60.6%	98.7
LaViDa
LaViDa (unpruned)	0.174	82.6%	99.7

Appendix

Table 11: Relative gains from adding LOWN on LLaDA-V and LaViDa. Each row is measured against the listed configuration without LOWN. FLOPs ratio is the mean of the per-benchmark FLOPs ratios, and reduction is one minus this mean. Quality is the mean normalized index with the reference set to 100.

	CapArena		LLaVA-Bench		DeCapBench
Method	Score $\uparrow$	FLOPs $\downarrow$	Score $\uparrow$	FLOPs $\downarrow$	Score $\uparrow$	FLOPs $\downarrow$
LLaDA-V ( $T$ steps)	$-$ 88.67	1.000	58.20	1.000	33.18	1.000
LLaDA-V ( $T/2$ )	$-$ 92.33	0.500	45.40	0.500	32.17	0.500
LLaDA-V ( $T/4$ )	$-$ 99.00	0.250	26.20	0.250	25.84	0.250
LOWN (Ours)	$-$ 85.67	0.388	57.40	0.454	34.43	0.391

Appendix

Table 12: LLaDA-V with fewer denoising steps on CapArena, LLaVA-Bench, and DeCapBench.

T/2

and

T/4

run half and a quarter of the baseline’s

T

steps on the inputs of Table 1 . Scores use the judges of Table 1 , and FLOPs are relative to LLaDA-V. DeCapBench averages the captions its official evaluator judges valid, 399, 398, 392, and 399 of 400 for the four rows.

Figure 8: Performance and efficiency on LaViDa, all ten benchmarks. The markers follow Figure 5 .

	Math	Science	Chart		MM	Doc	Cap	LLaVA-		DeCap
Method	Vista	QA	QA	MME	Bench	VQA	Arena	Bench	GQA	Bench	Avg.
LaViDa (Vanilla)	43.80	72.34	67.16	1353.03	76.95	63.54	$-$ 93.17	38.80	56.69	30.18	100.0%
FLOPs matched to LOWN
	36.60	71.34	20.40	987.28	72.23	18.55	$-$ 99.33	28.60	50.14	22.58	73.9%
RedVTP	$r{=}25\%$ ( $\times$ 0.432)	$r{=}25\%$ ( $\times$ 0.448)	$r{=}10\%$ ( $\times$ 0.244)	$r{=}10\%$ ( $\times$ 0.248)	$r{=}25\%$ ( $\times$ 0.435)	$r{=}10\%$ ( $\times$ 0.225)	$r{=}20\%$ ( $\times$ 0.347)	$r{=}10\%$ ( $\times$ 0.226)	$r{=}25\%$ ( $\times$ 0.367)	$r{=}25\%$ ( $\times$ 0.389)	( $\times$ 0.336)
	39.20	71.05	39.56	1220.76	73.76	30.62	$-$ 95.67	30.20	52.23	28.55	84.3%

Appendix

Table 13: Performance at matched decoding FLOPs on LaViDa , in the format of Table 1 . Methods are compared at LOWN’s budget. Each cell shows the benchmark score, with the selected retention

r

and FLOPs relative to the LaViDa baseline below. Competitors run on a discrete grid of retention ratios, so each uses its cheapest measured setting whose FLOPs are at least LOWN’s (Appendix C.3 ). D 3 ToM uses constant 10% retention on MathVista, LLaVA-Bench, CapArena, and DeCapBench and its adaptive schedule on the remaining tasks. MME reports the perception score. Best scores are bold, second best underlined, and lowest FLOPs bold.

Figure 9: Additional qualitative comparisons with visual token pruning (RedVTP). The upper three examples are short-answer questions, and the lower three are open-ended captioning. Green marks correct answers and content consistent with the image, and red marks incorrect answers and content inconsistent with the image.

Figure 10: Qualitative comparison with visual token pruning (RedVTP) and its pruned image. The upper right darkens the visual tokens removed by RedVTP, and the inset enlarges the region around the x-axis tick 4.0. In LOWN’s output, blue underlines mark tokens unmasked on FULL forward steps, while the remaining tokens are unmasked on TEXT-ONLY forward steps.

Figure 11: A limitation example on DocVQA. In LOWN’s output, blue underlines mark tokens unmasked on FULL forward steps, while the remaining tokens are unmasked on TEXT-ONLY forward steps. FLOPs are relative to LLaDA-V on this input.

	Adaptive acceptance		Math	Science	Chart		MM	Doc	Cap	LLaVA-		DeCap
Visual routing	Disagreement	Whole prefix	Vista	QA	QA	MME	Bench	VQA	Arena	Bench	GQA	Bench
✗	✗	✗	59.10	76.50	78.16	1719.5	70.88	83.91	$-$ 88.67	58.20	53.31	33.18
[0.5pt/1.5pt]			57.60	77.39	77.76	1624.6	68.30	83.50	$-$ 86.83	57.90	52.53	33.02
✓	✗	✗	0.5040 ( $-$ 49.6%)	0.3554 ( $-$ 64.5%)	0.7330 ( $-$ 26.7%)	0.3383 ( $-$ 66.2%)	0.3759 ( $-$ 62.4%)	0.6743 ( $-$ 32.6%)	0.6833 ( $-$ 31.7%)	0.7002 ( $-$ 30.0%)	0.3203 ( $-$ 68.0%)	0.6840 ( $-$ 31.6%)
[0.5pt/1.5pt]			57.10	78.19	78.32	1688.5	69.85	82.33	$-$ 81.17	57.50	53.43	34.71
✓	✓	✗	0.3910 ( $-$ 22.4%)	0.3006 ( $-$ 15.4%)	0.5891 ( $-$ 19.6%)	0.2755 ( $-$ 18.6%)	0.3236 ( $-$ 13.9%)	0.4789 ( $-$ 29.0%)	0.4471 ( $-$ 34.6%)	0.5067 ( $-$ 27.6%)	0.2469 ( $-$ 22.9%)	0.4450 ( $-$ 34.9%)

Appendix

Table 14: Table 5 with adaptive acceptance split into disagreement-first order on FULL steps and whole-prefix acceptance on TEXT-ONLY steps (Section 4.2 ), on LLaDA-V. Each setting reports scores and, below, relative FLOPs with the reduction from the preceding setting in parentheses.

Benchmark	Coverage	$\epsilon(1)$	$\epsilon(2)$	$\epsilon(4)$	$\epsilon(8)$	$R_{\mathrm{ref}}$	Accept. error	$R_{\mathrm{LOWN}}$
MathVista	58.6	7.3	9.5	12.2	15.4	0.284	2.2	0.325
ScienceQA	63.5	6.4	8.3	9.2	8.8	0.218	4.7	0.243
ChartQA	47.0	10.6	14.0	16.6	15.2	0.490	4.2	0.576
MME	63.8	5.9	7.6	10.0	13.6	0.193	3.9	0.231
MMBench	65.3	6.3	7.3	7.5	6.9	0.217	3.9	0.270
DocVQA	43.4	11.0	15.5	20.2	25.3	0.517	3.0	0.452

Appendix

Table 15: Certificate reliability on LLaDA-V. Eligible positions are the masked positions that were already masked

\Delta

steps earlier. Coverage is the share of eligible positions whose TEXT-ONLY prediction agrees with a one-step-old anchor.

\epsilon(\Delta)

is the share of agreeing positions at which a fresh FULL forward predicts a different token when the anchor is

\Delta

steps old.

R_{\mathrm{ref}}=f^{*}+(1-f^{*})\rho+1/T

is the cost of the oracle schedule of Section 5.3 , which runs a FULL forward only on the fraction

f^{*}

of steps where the image changes an unmasked decision and pays

\rho

elsewhere, plus one initial FULL forward. For ChartQA, whose baseline stops at the end-of-sequence token after 7.0 of its 16 steps on average,

f^{*}

and

T

are taken over the steps the baseline actually runs (

f^{*}=32.3\%

). Acceptance error is the share of the tokens LOWN unmasks on TEXT-ONLY steps that a fresh FULL forward on the state they were accepted from would predict differently, and

R_{\mathrm{LOWN}}

is LOWN’s measured cost. Costs are relative to the baseline and rates are in percent.

Benchmark	Age 1	Age $\leq 2$	Max age
MathVista	93.8	99.5	5
ScienceQA	92.9	99.4	3
ChartQA	94.9	99.9	3
MME	94.2	99.5	4
MMBench	91.8	99.4	3
DocVQA	96.7	99.6	4

Appendix

Table 16: Anchor age of the tokens LOWN accepts on TEXT-ONLY steps, in LOWN’s own runs on LLaDA-V. Age is the number of LOWN steps since the FULL forward that produced the anchor. Shares are in percent, and the last row pools the accepted tokens of all benchmarks.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 487 papers

DiffPrune: differentiable information throttling for token pruning in vision-language models
Aug 3, 2026Landi He, Mingde Yao, Shawn Young +1Vision-Language Models Efficient VLM Inference
Shenzhen University of Advanced Technology · CUHK MMLab, CPII under InnoHK
DAVET: Denoising-Aware Visual Evidence Trajectory Allocation for Diffusion Vision-Language Models
Aug 3, 2026Yongkang Zhou, Xiang Xia, Cheng Yan +2Vision-Language Models Diffusion VLMs
University of Science and Technology of China Hefei, China
ACPruner: Visual Token Pruning as Biased Attention Coverage Maximization in LVLMs
Sep 28, 2026Xu Li, Yuxuan Liang, Yi Zheng +6Efficient VLM Inference Large Vision-Language Models
College of Computer Science and Artificial Intelligence Fudan University, Shanghai, China

Aug 3, 2026J/K move · Enter open · S save

DiffPrune: differentiable information throttling for token pruning in vision-language models

Landi He, Mingde Yao, Shawn Young, Lijian Xu

Shenzhen University of Advanced Technology · CUHK MMLab, CPII under InnoHK

Vision-Language Models Efficient VLM Inference Token Pruning Visual Token Pruning LVLM Pruning

Open paper AskPDF arXiv

Visual token pruning reduces the computational cost of Vision-Language Models (VLMs) by removing redundant visual tokens. The key is to learn a score that measures whether a token is useful. Existing methods typically rely on Gumbel-Softmax to approximate discrete selection during training. Such selectors make the score depend on the behavior of a relaxed pruning operator, not directly on the consequence of information loss. In this paper, we propose DiffPrune, which gives token scores a direct meaning. During training, DiffPrune keeps all tokens and weakens each token's information according to its score. If weakening a token hurts the task, the scorer is pushed to protect it; if not, the token can receive a lower score. Because the loss is differentiated through this actual information-throttling path, the scorer avoids the unstable surrogate path of relaxed token selection. DiffPrune implements this idea with an Information Throttler, which injects variance-preserving noise into visual tokens, where high-score tokens remain close to their original representations, while low-score tokens carry less original information. At inference, the throttler is removed, and hard top-K pruning is applied using the learned scores. Across ten VLM benchmarks, DiffPrune retains 96.5% of full-model accuracy while accelerating LLM prefill by 2.85x, with only 0.69 ms inference overhead. Code will be publicly available.

Abstract

Efficient VLM Inference

Explore similar work

DiffPrune: differentiable information throttling for token pruning in vision-language models

DAVET: Denoising-Aware Visual Evidence Trajectory Allocation for Diffusion Vision-Language Models

ACPruner: Visual Token Pruning as Biased Attention Coverage Maximization in LVLMs