Vision language models (VLMs) incur substantial inference cost because every visual token is processed by the attention and MLP projections of every decoder layer, even when token-specific visual computation is unnecessary at many depths. We introduce Patch-to-Prune (P2P), inspired by Mechanistic Interpretability, a training-free framework that converts activation patching from a diagnostic tool into an inference-time computation bypass. P2P performs validation-guided forward and backward layer sweeps to identify decoder regions whose visual-token projection outputs can be replaced by fixed neutral proxy activation vectors within a user-specified accuracy tolerance. Unlike conventional token-pruning methods, P2P preserves the sequence length, token order, positional information, attention mask, and residual pathways, thereby pruning computation without removing tokens or modifying the pretrained model weights. We evaluate P2P on four VLMs from the Qwen2.5-VL and LLaVA families across seven multi-modal benchmarks using mutually disjoint calibration, validation, and test partitions. P2P at a 3% tolerance retains around 94% of dense accuracy while reducing FLOPs by 55%. Beyond these efficiency gains, our layer-wise analysis suggests that visual processing in VLMs is non-uniformly distributed across decoder depth: early and late layers often require little token-specific visual computation, whereas intermediate layers appear to perform most task-relevant visual integration, enabling later reasoning to rely largely on visual information already embedded in shared residual and textual representations. This makes P2P both an efficient inference framework and a causal lens into visual information processing in VLMs.
Figures & tables
Figure 1: Framework of Patch-to-Prune
Figure 2: Layer-wise sensitivity on POPE under cumulative forward and backward mean patching of image, text, and joint image–text token streams.
Figure 3: Tolerance sensitivity analysis.
Method
Metric
Tol.
wco
wvg
pope
sqa
gqa
tvqa
cvb
ai2d
LLaVA-v1.6-Vicuna-7B
Vanilla
Acc.
–
0.576
0.58
0.848
0.647
0.639
0.601
0.598
0.642
FLOPs (T)
32.6
30
32.5
23.6
32.2
34
31.2
30.1
Latency (s)
0.218
0.199
0.22
0.163
0.531
1.07
0.213
0.201
FastV
Acc.
–
0.559
0.561
0.842
0.646
0.636
0.596
0.605
0.634
FLOPs (T)
17.1
15.8
16.9
12.6
16.7
18
16.4
15.8
Table 1: Absolute accuracy, FLOPs (teraFLOPs), and average inference latency (seconds per sample) for LLaVA-7B and 13B across seven benchmarks. We compare Vanilla inference, FastV, PyramidDrop, and P2P at 1%, 3%, and 5% tolerance levels.
Figure 4: Patch-type ablation using cumulative forward pruning of image-stream attention and MLP computations with mean, zero, noise, or random-token activation replacement.
Figure 5: Proxy-estimation set-size ablation using cumulative forward mean patching of image-stream attention and MLP computations with 1, 25, 50, 100, 250, or 500 statistics examples.
Method
Tol.
sqa
tvqa
cvb
Vanilla
–
0.647
0.601
0.598
P2P
1%
0.646
0.543
0.556
P2P+LoRA
1%
0.665
0.59
0.566
P2P
3%
0.624
0.524
0.506
P2P+LoRA
3%
0.629
0.589
0.628
P2P
5%
0.570
0.454
0.505
Table 2: Full-test accuracy for LLaVA-7B comparing frozen Vanilla, frozen P2P, and P2P+LoRA fine-tuned under mean patching at 1%, 3%, and 5% tolerances
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Metric
Tol.
wco
wvg
pope
sqa
gqa
tvqa
cvb
ai2d
Qwen2.5-VL-3B-Instruct
Vanilla
Acc.
–
0.848
0.839
0.875
0.752
0.6
0.773
0.696
0.771
Vanilla
FLOPs (T)
–
2.39
1.93
2.27
1.6
2.22
6.31
5.71
3.16
FastV
Acc.
–
0.828
0.808
0.875
0.752
0.582
0.765
0.687
0.753
FastV
FLOPs (T)
–
1.46
1.22
1.32
1.02
1.28
3.46
2.59
1.82
PDrop
Acc.
–
0.801
0.77
0.848
0.735
0.568
0.737
0.663
0.717
PDrop
FLOPs (T)
–
1.34
1.13
1.2
0.954
1.16
3.14
2.37
1.67
Appendix
Table 3: Full-test accuracy for LLaVA-7B comparing frozen Vanilla, frozen P2P, and P2P+LoRA fine-tuned under mean patching at 1%, 3%, and 5% tolerances.
Figure 6: Layer-wise sensitivity on WhatsUp’s COCOQA under cumulative forward and backward mean patching of image, text, and joint image–text token streams.
Figure 7: Layer-wise sensitivity on ScienceQA under cumulative forward and backward mean patching of image, text, and joint image–text token streams.
Vision-Language Models (VLMs) have exhibited impressive performance across diverse visual scenarios. However, this success comes at the cost of explosive growth in visual tokens, which imposes substantial memory and computational overhead during inference, ultimately increasing latency. To improve VLM inference efficiency, a typical class of visual token pruning methods estimates token importance by aggregating attention scores across all heads in the pruning layer of the Large Language Model (LLM) backbone and prunes tokens based on aggregated scores. However, in this paper, we reveal a compelling phenomenon: the capability to pinpoint critical visual tokens is concentrated within a small fraction of heads. Aggregation exclusively on these heads can improve task performance. Inspired by this observation, we propose ProViP, a training-free progressive visual token pruning framework. ProViP first removes redundant visual tokens based on the embedding similarity of input tokens before reasoning of the LLM backbone, and then further prunes tokens during reasoning via head-aware pruning. Experiments demonstrate that ProViP delivers outstanding task performance and inference efficiency. For instance, when applied to LLaVA-1.5-7B, ProViP retains 95.9% of the original performance and achieves 1.62x inference speedup under an 88.9% pruning ratio.
Chaofang Ma, Lin Jiang, Carol Jingyi Li +4
The Hong Kong University of Science and Technology · Northeastern University · The Hong Kong University of Science and Technology (GZ)
Visual token pruning is a promising approach to reducing the inference cost of large vision-language models (LVLMs), yet aggressive token reduction often causes substantial performance degradation. We identify three key factors behind this degradation: text-guided selection bias, information loss from discarded tokens, and positional distortion caused by sequence compaction. Based on these observations, we propose VPRune, a training-free pre-LLM pruning framework consisting of visual-only diversity selection, similarity-guided token recycling, and position-preserving restoration. Experiments on FastVLM-1.5B across multiple vision-language benchmarks demonstrate that VPRune achieves a favorable accuracy--compression trade-off, with particularly pronounced advantages under aggressive compression. Furthermore, evaluations on edge-device show that VPRune effectively reduces end-to-end inference latency while maintaining superior task performance, demonstrating its practicality for resource-constrained LVLM deployment.
Guangchuan Lv, Dianxing Shi, Dingjie Fu
Northeastern University · Beihang University · Huazhong University of Science and Technology
Vision-Language Models (VLMs) have recently demonstrated remarkable capabilities in visual understanding and reasoning, but they also impose significant computational burdens due to long visual sequence inputs. Recent works address this issue by pruning unimportant visual tokens, achieving substantial computational reduction while maintaining model performance. The core of token pruning lies in determining token importance, with current approaches primarily relying on attention scores from vision encoders or Large Language Models (LLMs). In this paper, we analyze the effectiveness of attention mechanisms in both vision encoders and LLMs. We find that vision encoders suffer from attention sink, leading to poor focus on informative foreground regions, while in LLMs, although prior studies have identified attention bias toward token positions, text-to-vision attention demonstrates resistance to this bias and enables effective pruning guidance in middle layers. Based on these observations, we propose LearnPruner, a two-stage token pruning framework that first removes redundant vision tokens via a learnable pruning module after the vision encoder, then retains only task-relevant tokens in the LLM's middle layer. Experimental results show that our LearnPruner can preserve approximately 95% of the original performance while using only 5.5% of vision tokens, and achieve 3.2× inference acceleration, demonstrating a superior accuracy-efficiency trade-off.