While Large Vision-Language Models (LVLMs) demonstrate exceptional multi-modal capabilities, the quadratic computational cost of processing high-resolution visual tokens remains a critical bottleneck. Though recent token reduction strategies attempt to accelerate inference, such methods inadequately exploit attention values and fail to address token redundancy. More critically, they overlook the ``attention shift'' phenomenon inherent in LVLMs, which skews token attention scores. In this work, we propose ASAP, a novel training-free, KV-Cache-compatible pruning recipe that comprehensively addresses these limitations. First, we mitigate the attention shift by utilizing a dynamic bidirectional soft attention mask, ensuring the selection of genuinely informative tokens rather than naive attention-based selection. Second, we posit that high semantic redundancy within the token set degrades performance. We therefore introduce a weighted soft merging component that merges semantically similar tokens, preserving only the most feature-dense visual patches for subsequent layers. ASAP achieves virtually lossless compression of visual context, retaining 99.02% of the original LLaVA-NeXT-7B performance while aggressively slashing computational FLOPs by ~80%.
Figures & tables
Figure 1: ASAP integration within the LVLM architecture. The plug-and-play pruning module is embedded directly within the language model backbone, operating between standard decoder layers to compress the visual token sequence during the inference forward pass.
Figure 2: Qualitative comparison of token selection. Retained tokens (top 128) are visible, while dropped tokens are darkened. FastV demonstrates a bottom spatial bias, missing the distant third car. ASAP successfully mitigates this bias, preserving critical background features to accurately count all cars.
Figure 3: The two-stage ASAP architecture. After decoder layer L , a salience-guided bidirectional matrix retains top- k salient visual tokens. A feature-based similarity module then consolidates redundant tokens via salience-weighted merging and a salvage mechanism. This dual-filtering pipeline passes a compact, feature-dense visual set to layer L+1 , strictly preserving all text and system prompts.
Method
FLOPs
Ratio
VQA \scalebox0.6V2
GQA
SQA
VQA \scalebox0.6Text
MME
MMB
MMB \scalebox0.6CN
Avg
LLaVA-1.5-13B
3.817
100%
80.0
63.25
71.6
61.19
1529.93
67.71
63.60
100
Retain 192 tokens ( ↓ 66.67%)
FastV [ECCV’24]
1.259
32.99
75.42
58.67
70.64
57.23
1442.31
66.43
61.05
95.37
PruMerge+ [ICCV’25]
1.253
32.83
76.97
59.72
71.10
57.12
1451.31
66.32
62.54
95.94
SparseVLM [ICML’25]
1.253
32.83
78.31
57.80
73.08
51.93
1485.27
67.61
61.77
95.75
DivPrune [CVPR’25]
1.253
32.83
78.42
59.12
73.08
58.84
1482.01
65.95
59.02
96.68
Table 1: Performance comparison of ASAP with other pruning strategies on LLaVA-1.5-13B [ 31 ] . Ratio represents the FLOPs ratio (percentage of vanilla FLOPs retained). Avg refers to the average percentage performance across all benchmarks compared to the vanilla model. Best scores are highlighted in red .
Method
FLOPs
VQA \scalebox0.6V2
GQA
VQA \scalebox0.6Text
MMB
MMVet
Avg
LLaVA-NeXT-7B
20.825
81.2
62.9
59.6
65.8
40.0
100.0
Retain 960 tokens ( ↓ 66.67%)
FastV [ECCV’24]
6.477
80.33
61.98
59.32
65.12
39.4
98.89
VisPruner [ICCV’25]
6.458
80.0
62.20
58.81
65.29
38.4
98.26
ASAP (Ours)
6.258
81.19
62.29
59.67
66.41
40.1
100.06
Retain 640 tokens ( ↓ 77.78%)
Table 2: Performance comparison of ASAP with other pruning strategies on LLaVA-NeXT-7B [ 30 ] . Avg refers to the average percentage performance across all benchmarks compared to the vanilla model. Best scores are highlighted in red .
Method
FLOPs
Ratio (%)
GQA
SQA \scalebox0.6IMG
POPE
MME
MMB \scalebox0.6CN
Avg
InternVL 2.5 8B
12.575
100.0
63.40
97.81
90.59
1696.26
82.60
100.00
Retain 510 tokens
FastV [ECCV’24]
3.969
31.56
61.44
96.03
89.10
1614.42
80.32
97.17
ASAP (Ours)
3.969
31.56
62.31
97.07
90.08
1645.32
82.47
98.76
Retain 340 tokens
FastV [ECCV’24]
2.663
21.17
59.17
91.51
88.47
1603.51
79.33
95.02
Table 3: Performance comparison of ASAP with other pruning strategies on InternVL 2.5 8B. Avg refers to the average percentage performance across all benchmarks compared to the vanilla model. Best scores within each token budget are highlighted in red .
Figure 4: Ablation of ASAP modules on TextVQA and MME. FastV prunes naively using Top- k selection. Bidirectional uses Salience-Guided Bidirectional attention values. ASAP adds Salience-Weighted Consolidation and Budget Reallocation. The progressive gains confirm that combining attention-driven selection with feature consolidation maximizes semantic density.
Method
# Token
FLOPs (T)
Percption score
Storage (MB)
GPU Memory (GB)
CUDA Time (ms)
LLaVA-1.5-7B
576
3.817
1506.46
321.39
14.33
38.27
Retain 192 tokens
FastV [ECCV’24]
192
1.259
1437.45
129.73
13.98
33.65
ASAP (Ours)
192
1.259
1466.19
116.64
13.96
35.13
Retain 128 tokens
FastV [ECCV’24]
128
0.836
1365.435
97.33
13.97
31.45
Table 4: System efficiency comparison on LLaVA-1.5-7B. We evaluate the computational cost and memory footprint during the prefill phase.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Impact of integrating SG-BiMask in existing pruning frameworks. Our SG-BiMask component consistently improves performance when integrated with the standard causal attention baseline (FastV) across all evaluated benchmarks.
Vision-language models (VLMs) rely on long visual token sequences for visual understanding, making the prefill stage expensive in both computation and memory. Most existing pruning methods follow an absolute-ranking paradigm, assigning importance scores to visual tokens and retaining a fixed top-K subset. In this work, we argue that this paradigm is fundamentally brittle: attention sinks distort token importance rankings, while image redundancy and query-dependent visual evidence make fixed token budgets unreliable across inputs. We propose OccamToken, a training-free framework that replaces absolute token ranking with register-anchored relative evidence testing. Instead of asking which tokens are globally important, OccamToken evaluates whether a visual token provides information beyond a register-based reference. Our key insight is that register tokens naturally absorb low-information attention patterns, making them a stable reference for identifying genuinely informative visual evidence. Based on this principle, OccamToken performs both image-adaptive redundancy pruning and query-adaptive relevance pruning through dynamic thresholds derived from register attention. Across LLaVA-NeXT, LLaVA-v1.5, and Qwen3-VL, OccamToken consistently improves the accuracy-efficiency trade-off without additional training. Notably, on LLaVA-NeXT, it reduces 2,880 visual tokens to approximately 40 while preserving over 93% of the original accuracy, enabling stable visual token compression even in the extreme 1.4% retention regime.
Visual token pruning is a promising approach to reducing the inference cost of large vision-language models (LVLMs), yet aggressive token reduction often causes substantial performance degradation. We identify three key factors behind this degradation: text-guided selection bias, information loss from discarded tokens, and positional distortion caused by sequence compaction. Based on these observations, we propose VPRune, a training-free pre-LLM pruning framework consisting of visual-only diversity selection, similarity-guided token recycling, and position-preserving restoration. Experiments on FastVLM-1.5B across multiple vision-language benchmarks demonstrate that VPRune achieves a favorable accuracy--compression trade-off, with particularly pronounced advantages under aggressive compression. Furthermore, evaluations on edge-device show that VPRune effectively reduces end-to-end inference latency while maintaining superior task performance, demonstrating its practicality for resource-constrained LVLM deployment.
Guangchuan Lv, Dianxing Shi, Dingjie Fu
Northeastern University · Beihang University · Huazhong University of Science and Technology
Large Vision-Language Models (LVLMs) face significant computational inefficiencies caused by the large number of visual tokens. Existing visual token pruning methods mainly focus on either retaining individually important tokens or selecting mutually diverse ones. In this work, we revisit visual token pruning from a coverage perspective and formulate it as a biased attention coverage maximization problem. The key idea is to select a compact token subset whose encoder-side outgoing attention can jointly cover the image while assigning higher coverage priority to more informative regions. From this perspective, we propose ACPruner, a training-free visual token pruning framework for efficient LVLM inference. ACPruner first estimates token importance by combining intra-modal saliency and inter-modal relevance, then derives token-wise coverage from attention patterns within the vision encoder, and finally performs greedy selection to maximize the proposed coverage objective. Extensive experiments across multiple LVLM backbones, including LLaVA-1.5-7B/13B, LLaVA-NeXT-7B/13B, Qwen2.5-VL-7B, and LLaVA-OneVision-7B, show that ACPruner consistently achieves strong performance retention while delivering substantial end-to-end inference speedups.
Xu Li, Yuxuan Liang, Yi Zheng +6
College of Computer Science and Artificial Intelligence Fudan University, Shanghai, China