SheafPrune: Sheaf-Guided Graph Propagation for Visual Token Pruning

Xiaoyang Ma; Chi Wang; Hong Sun

cs.CVOct 9, 2026

SheafPrune: Sheaf-Guided Graph Propagation for Visual Token Pruning

Authors: Xiaoyang Ma, Chi Wang, Hong Sun

Organizations: PHOENIX Lab Hangzhou Dianzi University Hangzhou, 310018, China

Abstract

Multimodal large language models process the complete visual token sequence, increasing prefill cost even when many tokens are irrelevant to the instruction. Token-wise saliency captures limited inter-token dependence; graph-based selection instead depends on how relations govern importance propagation and ranking. We study support-constrained propagation with node-degree correction. Inspired by sheaf geometry, SheafPrune uses modality-specific maps and a sparse text--visual graph to transfer relevance, then propagates scores on a separate visual graph and corrects them for degree. It retains a hard top- $k$ subset once before the frozen language model. The selector has about $4.5$ M parameters and is trained end-to-end through the frozen LLaVA-NeXT-8B backbone using only the language-modeling objective. Explicit retention-ratio conditioning lets one checkpoint serve every budget. Across four backbones, ten benchmarks and seven pruning ratios, with all baselines re-implemented under one protocol, SheafPrune achieves the best retention at $22$ of $28$ backbone--budget settings, with monotonically growing margins in the moderate-to-aggressive regime. On LLaVA-NeXT, retaining $10\%$ of visual tokens yields a $5.15\times$ net prefill speedup and an $84.0\%$ end-to-end FLOPs reduction. Together with component ablations, these results support graph-constrained importance propagation and degree correction as a design for visual token selection across the evaluated backbones and tokenization schemes, improving the trade-off between accuracy retention and inference cost. code:https://github.com/Bob0817912/SheafPrune_ICLR_2027

Figures & tables

Figure 1: Which regions survive pruning. Two VQAv2 examples on LLaVA-NeXT-8B at retention ratios $\rho\in\{0.5,0.3,0.1\}$ . FastV, SparseVLM and SheafPrune retain the same number of candidates; pruned positions are greyed out. The examples illustrate differences in retained regions, rather than establish the causal effect of question conditioning. Further samples appear in Appendix R .

Figure 2: Overview of SheafPrune . (a) Frozen encoders supply visual and textual representations. (b) Modality-specific projection and sparse cross-modal transfer produce relevance; support-constrained visual propagation and degree correction transform the fused initial score into a ranking. (c) Hard Top- $k$ selection physically removes unselected visual tokens before the frozen decoder. Matrices and scores are schematic.

Figure 3: Reported best-

\delta

performance envelope and prefill trade-offs. RA% averages ten benchmark retention ratios. The SheafPrune entries use per-point degree-exponent selection; they do not represent one fixed-

\delta

deployment configuration. Unavailable staged-method points reflect the layer-averaged budget limits described in Appendix L .

		General VQA		MLLM benchmarks			Text-oriented VQA			High-resolution		Avg
Model	Method	GQA	VQAv2	MME	POPE	MMB	ChartQA	DocVQA	TextVQA	HR-4K	HR-8K	acc	lat	RA%	LR%
LLaVA-NeXT-8B	No prune	65.0	82.1	38.4	83.6	77.0	69.7	72.6	66.7	51.2	46.1	65.2	149.4	–	–
	FastV	63.5	81.0	37.7	82.5	76.1	59.2	55.2	61.7	51.4	45.5	61.4	55.7	94.5	62.7
	SparseVLM	58.5	76.5	37.1	83.0	76.8	45.0	46.4	46.9	51.0	46.1	56.7	63.3	87.8	57.6
	PyramidDrop	62.2	80.8	37.8	85.0	76.1	45.4	49.3	55.5	52.1	45.6	59.0	63.0	91.0	57.8
	ToMe	64.5	80.8	37.8	82.4	76.3	52.1	48.7	60.5	50.0	42.5	59.6	52.6	91.6	64.8

Table 1: Comparison across four backbones (SheafPrune: reported best-

\delta

protocol) at

p=0.7

(

\rho=0.3

). acc: benchmark score (MME/28); lat: prefill latency including selection (ms); RA%: ten-benchmark macro-average retention; LR%: latency reduction, both against the unpruned backbone. Red/blue mark the best/second-best pruning method per column and backbone; Appendix K gives the profiling protocol and Table 9 the native scores and the remaining budgets.

Arm	Norm.	$\delta$	$p=0.5$	$p=0.7$	$p=0.9$		$p=0.9$
			RA (%)				Text
Random selection	—	—	92.12	86.32	73.38	${}_{\,-16.01}$	39.20	${}_{\,-35.77}$
Degree only	—	$1$	95.77	92.14	81.38	${}_{\,-8.01}$	57.09	${}_{\,-17.88}$
Dense normalization	dense	$1$	91.74	85.78	72.68	${}_{\,-16.71}$	38.10	${}_{\,-36.87}$
No degree correction	support	$\mathbf{0}$	91.88	85.97	72.80	${}_{\,-16.59}$	38.30	${}_{\,-36.67}$
Uniform initial score	support	$1$	98.50	96.72	88.19	${}_{\,-1.20}$	73.32	${}_{\,-1.65}$

Table 2: Fixed-configuration component ablation on LLaVA-NeXT-8B. Defaults:

t=3

q_{v}=0.05

\delta=1

, except as indicated. Norm.: the pair set used for normalization; Text: mean retention on ChartQA, DocVQA and TextVQA. Subscripts are differences from the full method at

p=0.9

. RA averages ten benchmark retention ratios. These entries use fixed configurations, not the best-

\delta

envelope; complete arms appear in Table 10 .

Figure 4: Local score–degree relationships. HR-Bench 8K examples at the $10\%$ retention budget. Degree and corrected score are negatively associated in the displayed cases (reported correlations $-0.854$ and $-0.931$ ). Retained regions are shown for visual comparison. The existing visualization uses $t=5$ , $\delta=1$ ; the default component table uses $t=3$ . Heatmaps are interpolated and scaled separately.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Symbol	Default	Component	Symbol	Default
training-interface dimension	$d$	$4096$	visual-graph relation ratio	$q_{v}$	$0.05$
text bottleneck	$d_{e}$	$256$	propagation steps	$t$	$3$
text-encoder layers	$L_{e}$	$2$	degree exponent	$\delta$	$1$
text-encoder heads	–	$4$	fusion mode	–	early
max selector text length	–	$1024$	visual cosine centering	–	false
comparison dimension	$h$	$128$	propagation normalization	–	masked

Appendix

Table 3: Default SheafPrune configuration.

Group	Benchmark	#Samples	Split	Metric
OCR / document / chart	ChartQA	2,500	test	relaxed accuracy
	DocVQA	5,349	validation	ANLS
	TextVQA	5,000	validation	VQA accuracy
	HR-Bench 4K	800	full	option accuracy
	HR-Bench 8K	800	full	option accuracy
General VQA / perception	GQA	12,578	testdev-balanced	exact-match accuracy

Appendix

Table 4: Evaluation benchmarks. All methods share the same sets and the same metric implementation.

Method	Role in the comparison
G-Prune	Visual graph propagation and ranking.
FastV	Decoder-attention-based token selection.
PyramidDrop	Staged removal within the decoder.
SparseVLM	Text-informed visual token sparsification.
ToMe	Token merging rather than pure deletion.
VisionZip	Visual importance and contextual compression.

Appendix

Table 5: Evaluated baseline roles. Implementations follow the corresponding open-source releases and method specifications; measurement and budget conventions are stated separately.

Method	Backbones	Budgets	Scope
FastV, ToMe, G-Prune	All four	$0.1$ – $0.9$	Evaluated across the reported grid.
PyramidDrop, SparseVLM	All four	$0.1$ – $0.7$	Under the implemented staged schedule, layer-averaged retention has a floor of $1/S=0.25$ .
FasterVLM, VisionZip, PruMerge	LLaVA-1.5	$0.1$ – $0.9$	Reported for this backbone; the present comparison does not evaluate their other backbone integrations.

Appendix

Table 6: Reported evaluation coverage. Budget ranges use pruning rate

p

Setting	Stage A: initial alignment	Stage B: ratio conditioning
Data	Strong-signal subset	Source-weighted $779$ K mixture
Duration	Approximately $0.3$ of an epoch	Approximately $2$ epochs
Retention	Fixed $\rho=0.9$	Four uniformly sampled steps from $\{0.5,0.3,0.1\}$ , then one additional step at $\rho=0.9$
STE temperature	Fixed $\beta=2.0$	Cosine annealing, $2.0\rightarrow 0.1$
Loss	Target-answer cross-entropy	Target-answer cross-entropy

Appendix

Table 7: Two-stage selector training. Retention is denoted by

\rho

; pruning rate is

p=1-\rho

. Approximate epoch counts follow the training configuration.

Category	Task characteristics	Weight
Strong	Fine-grained localization; OCR and text recognition; chart understanding; grounding and referring tasks	$2.5$
Regular	General VQA; GPT-4V-generated visual questions, answers and descriptions	$1.0$
Weak	Generic descriptions and summaries; dialogue with peripheral visual content	$0.5$
Excluded	Text-only samples without an image field	$0$

Appendix

Table 8: Relative sampling weights by task category. Descriptions identify task types rather than an inventory of dataset names or splits.

Method	$p$	General VQA		MLLM benchmarks			Text-oriented VQA			High-resolution		Aggregate		Efficiency
	(%)	GQA	VQA2	MME	POPE	MMB	Chart	Doc	Text	HR-4K	HR-8K	Avg	RA	lat	$\times$	LR	tok/s	FLOPs
LLaVA-NeXT-8B
No prune	$0$	65.0	82.1	1074.21	83.6	77.0	69.7	72.6	66.7	51.2	46.1	65.2	–	149.4	–	–	66.5	0.0
Random	$10$	64.9	82.4	1076.44	83.0	77.2	67.8	70.8	66.3	51.5	46.4	64.9	99.5	–	–	–	–	–
Random	$30$	64.5	82.1	1056.92	82.9	76.8	62.4	62.1	64.0	50.5	45.2	62.8	96.4	–	–	–	–	–
Random	$50$	64.4	80.0	1075.43	82.2	77.3	56.5	51.3	57.0	47.1	44.6	59.9	92.1	–	–	–	–	–

Appendix

Table 9: Complete results. Every backbone, method and pruning rate in one table. Benchmark columns are the original scores of the final result workbook: Chart, Doc, Text, MMB and VQA2 are ChartQA (relaxed accuracy), DocVQA (ANLS), TextVQA, MMBench-EN and VQAv2; POPE is F1, MME keeps its native scale, the remaining benchmarks use accuracy-based metrics, and PruMerge is LLaVA-PruMerge. “Avg” is the ten-benchmark mean with MME rescaled as score

/28

; “RA” is the ten-benchmark macro-average retention (%) relative to the unpruned model of the same backbone, i.e. each benchmark is normalized first and the ratios are then averaged, which is the quantity every claim in the paper is stated in. “lat” is the measured prefill latency (ms) of the complete method — the forward pass plus that method’s own selection cost — and “LR” the latency reduction (%) it gives against the unpruned prefill of the same backbone; “tok/s” is greedy decoding throughput and “FLOPs” the end-to-end FLOPs reduction (%) including the frozen vision tower. Efficiency was profiled at

p\in\{0.5,0.7,0.9\}

on one

60

-sample batch shared by all four backbones, so it is a per-backbone quantity and “–” in those columns marks a budget that was not profiled. SparseVLM and PyramidDrop prune inside the decoder in stages and their layer-averaged retention cannot fall below

1/S=25\%

, so they have no

p\in\{0.8,0.9\}

entry at all; FasterVLM, VisionZip and LLaVA-PruMerge are reported only on LLaVA-1.5 in this evaluation; other backbone integrations are not evaluated here. Best pruning method per backbone, pruning rate and column in bold.

	Configuration				Ten-benchmark RA (%)			At $p{=}0.9$
Arm	$s^{(0)}$	Prop.	Norm.	$\delta$	$0.5$	$0.7$	$0.9$	Text	OCR	Gen.
Random selection	—	$\times$	—	—	92.12	86.32	73.38	39.20	54.98	91.77
Degree only	uniform	$\times$	—	$1$	95.77	92.14	81.38	57.09	69.03	93.74
Visual prior only	$\overline{n}$	$\times$	—	$0$	91.13	84.40	70.23	34.10	49.88	90.57
Relevance only †	$\widehat{r}$	$\times$	—	$0$	91.81	85.80	72.58	38.00	53.78	91.37
Uniform initial score	uniform	$\checkmark$	support	$1$	98.50	96.72	88.19	73.32	81.54	94.63

Appendix

Table 10: Complete component ablation of SheafPrune on LLaVA-NeXT-8B. The default configuration matches Table 2 ; “Text” averages ChartQA, DocVQA and TextVQA, “OCR” adds HR-Bench 4K and 8K, and “Gen.” averages GQA, VQAv2, MMBench-EN, POPE and MME. Rows marked

\dagger

give additional initialization variants.

Configuration	$\rho{=}0.9$	$0.7$	$0.5$	$0.4$	$0.3$	$0.2$	$0.1$
B: no norm, no graph	99.24	96.66	94.71	92.75	89.54	85.85	77.60
N: norm, no graph	99.52	97.00	89.38	83.23	74.62	59.48	53.72
U: uniform, graph	99.41	99.38	98.40	97.88	96.06	93.47	82.39
F: norm, graph	99.43	99.30	98.43	97.88	95.96	93.48	82.39
F $-$ U (value of the norm prior)	$+0.02$	$-0.07$	$+0.03$	$-0.00$	$-0.10$	$+0.01$	$-0.00$
U $-$ B (value of the diffusion)	$+0.17$	$+2.71$	$+3.69$	$+5.12$	$+6.52$	$+7.62$	$+4.79$

Appendix

Table 11: Component ablation of the norm-plus-graph baseline family on LLaVA-1.5-7B: accuracy retention (%) averaged over ten benchmarks. These are arms of that family, not of SheafPrune. B = random selection, N = feature norm without a graph, U = uniform initialization with diffusion, F = norm initialization with diffusion.

(a) Propagation depth		(b) Similarity threshold
$t$	Mean	$s$	Mean
$1$	54.06	$0.3$	49.11
$3$	54.23	$0.4$	53.61
$5$	53.85	$0.5$	53.85
$10$	53.85	$0.6$	51.18
$20$	53.06	$0.7$	44.90

Appendix

Table 12: Sensitivity of the propagation depth

t

and of the similarity threshold

s

, measured as the mean score over four benchmarks at

\downarrow 90\%

on Qwen2.5-VL. These are point estimates, without a measured noise threshold.

	$N_{v}$	Random	Ours
GQA	$345$	$0.500$	$0.417$
VQAv2	$345$	$0.432$	$0.333$
DocVQA	$4032$	$0.859$	$0.771$
HR-Bench	$4096$	$1.000$	$0.844$

Appendix

Table 13: Grid coverage at

\downarrow 90\%

: fraction of

8\times 8

cells that keep a token.

Figure 5: Additional qualitative samples. Same protocol as Figure 1 : FastV, SparseVLM and SheafPrune at $\rho\in\{0.5,0.3,0.1\}$ on LLaVA-NeXT-8B, with pruned positions greyed out.

Visual Token Pruning

Visual token pruning reduces the image and video patches processed by transformers to lower inference cost while preserving task-relevant information.

See all 276 papers

StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Models
Sep 12, 2026Hansen Zhang, Landi He, Mingde Yao +1Efficient VLM Inference Visual Token Pruning
Shenzhen University of Advanced Technology, Shenzhen, China · School of Computer Science, Nanjing University, Nanjing, China · CUHK MMLab, CPII under InnoHK, Hong Kong, China
From Token Importance to Conditional Removability: Rethinking Visual Token Pruning in Multimodal Large Language Models
Sep 22, 2026Shengli He, Yongchao Liang, Roumeng He +5Visual Token Pruning Multimodal Token Compression
Guizhou University, Guiyang, China. · Shanghai Ocean University, Shanghai, China.
S $^2$ Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models
Sep 1, 2026Yuanyuan Jia, Shunpu Tang, Qianqian YangEfficient VLM Inference Multimodal Large Language Models
College of Information Science and Electronic Engineering, Zhejiang University Hangzhou, China

Sep 12, 2026J/K move · Enter open · S save

StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Models

Hansen Zhang, Landi He, Mingde Yao, Lijian Xu

Shenzhen University of Advanced Technology, Shenzhen, China · School of Computer Science, Nanjing University, Nanjing, China · CUHK MMLab, CPII under InnoHK, Hong Kong, China

Efficient VLM Inference Visual Token Pruning LVLM Pruning Multimodal Large Language Models

Open paper AskPDF arXiv

Visual prefixes account for a major portion of the per-layer computation in multimodal large language models (MLLMs), making visual-token pruning a direct approach to accelerating inference. Existing top-K methods typically evaluate tokens independently and apply a uniform budget to all inputs, overlooking both selection-dependent interactions and variations in visual complexity across samples. In contrast, we propose StepPrune, which formulates visual-token pruning as an adaptive sequential decision process. Conditioned on previously selected tokens and textual context, StepPrune progressively constructs the retained subset and automatically determines its size through a learned STOP action. During training, a variance-preserving noise gate provides a differentiable surrogate for the discrete selection process, whereas during inference, unselected tokens are physically removed before language-model prefill. A grouped selection mechanism further extends StepPrune to high-resolution inputs. Experiments across LLaVA-1.5, LLaVA-NeXT, Qwen2.5-VL, and InternVL3 show that StepPrune achieves the best average normalized performance retention across all evaluated pruning rates on LLaVA-1.5, Qwen2.5-VL, and InternVL3, while remaining competitive on the substantially longer AnyRes prefixes of LLaVA-NeXT. On LLaVA-1.5, StepPrune retains 94.6% of the full-prefix normalized performance while pruning 88.9% of the visual tokens. At a mean retained count of 64, StepPrune reduces prefill latency from 59.95 ms to 40.05 ms, corresponding to a 1.50x prefill speed-up.

Abstract

Visual Token Pruning

Explore similar work

StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Models

From Token Importance to Conditional Removability: Rethinking Visual Token Pruning in Multimodal Large Language Models

S2^22Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models

S $^2$ Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models