High-resolution Large Vision-Language Models (LVLMs) preserve critical, fine-grained visual details by encoding a global thumbnail alongside multiple local crops. However, the resulting thousands of vision tokens introduce severe computational bottlenecks during Large Language Model (LLM) inference. Existing token pruning methods rely primarily on query-agnostic visual saliency, while current query-aware approaches either operate at a coarse crop-level resolution or compute token relevance too late, after vision tokens have already entered the heavy LLM layers. To address this, we propose a training-free, hierarchical token pruning framework that utilizes query guidance prior to executing any LLM blocks. Our top-down hierarchy first operates at the instance-level, dynamically scaling the vision token retention ratio of each sample based on its visual complexity. Next, at the crop-level, we distribute this sample budget across local crops using sentence-level cross-modal similarity. Finally, at the token-level, we exploit projected vision tokens alongside textual query token embeddings in the shared space to measure fine-grained alignment, applying a text-debiasing normalization to produce high-fidelity query scores. These query-guided budgets and scores are fused with local and global visual saliency metrics to select the most critical vision tokens. Extensive experiments on LLaVA-NeXT-7B demonstrate that our plug-and-play framework retains 98.0% and 94.8% of unpruned performance at 25% and 10% vision token retention, respectively. Notably, at 25% retention, our method delivers a 2.0x inference throughput speedup and even surpasses the unpruned baseline on POPE by 0.7 points.
Figures & tables
Fig. 1: Pruning masks on a POPE sample at a 25% retention ratio. GlobalCom 2 uses an image-only mask that is unchanged by the query; for the chair question shown here, it removes most chair evidence before the LLM and answers incorrectly. Our query-aware mask retains the relevant chair tokens and answers correctly.
Fig. 2: Adaptive per-sample budget allocation on POPE examples. Our query-aware framework dynamically scales token retention according to scene complexity. The visually redundant snow scene (left) retains only 516 of 2,880 visual tokens, whereas the cluttered baseball scene (right) dynamically allocates 860 tokens to preserve fine-grained details needed for accurate question answering.
Fig. 3: Overview of our training-free token pruning framework for high-resolution LVLMs. An input image is processed into a full-image view and multiple sub-images, which are passed through a Visual Encoder and Projector to produce visual tokens ( V ). A text query is tokenized into embeddings ( E ). The pruning process consists of three main stages: (1) Adaptive Budget Allocation: A two-level budget is computed. At the Sample-level , a token retention ratio ( Rj ) is dynamically set for each image based on its visual token diversity ( Hj ). At the Crop-level , this budget is distributed among the sub-images based on the query-crop relevance. (2) Query Score Algorithm: A fine-grained relevance score is computed between the query embeddings ( E ) and the visual tokens ( V ). Raw cross-modal similarities are standardized using a text-debiasing z-score to produce a robust query score ( qsi ) for each token. (3) Fuse & Select: The final token selection score is calculated by fusing the Query Score with two visual saliency cues: a local score from the sub-image’s own [CLS] attention ( ai ) and a global score derived from the full-image view ( gi ). The top- bi tokens are selected and fed to LLM for inference.
Method
VizWiz
GQA
SQA-I
TextVQA
MME-P
MMB
AI2D
POPE
Avg. (%)
Retention ratio R=25%
LLaVA-NeXT-7B
60.6
64.3
70.3
61.4
1519.3
67.1
65.3
87.6
100.0
FastV [ 8 ]
54.2
60.4
68.8
58.4
1477.3
65.6
63.9 ∗
83.1
95.5
SparseVLM [ 19 ]
56.0
59.9
67.5
58.3
1465.9
65.6
64.2 ∗
85.0
95.8
PyramidDrop [ 20 ]
56.8
60.3
68.5
59.8
1473.7
65.6
63.9 ∗
85.5
96.6
FasterVLM [ 9 ]
55.4
61.3
67.1
58.8
1454.6
66.0
62.6 ∗
87.2
95.9
TABLE I: Comparison with existing methods on LLaVA-NeXT-7B. The retention ratio R is defined with respect to the 2,880 token positions including padding. Avg. is the mean score relative to the unpruned model. Baseline results without a mark are taken from GlobalCom 2 [ 11 ] ; ∗ denotes results obtained by running the released code. The best and second-best results among pruning methods are in bold and underlined.
Sample alloc.
Query score
MME-P
AI2D
✓
1501.4
63.4
✓
1506.5
63.6
✓
✓
1514.1
63.7
TABLE II: Module ablation on LLaVA-NeXT-7B at R=25% .
α
0.0
0.1
0.3
0.5
0.7
0.9
1.0
MMB
65.4
65.3
65.3
65.2
65.6
65.0
65.1
POPE
88.1
88.1
88.2
88.2
88.3
88.3
87.7
TABLE III: Ablation on α at R=25% ( β=0.5 ). α=1 uses only the global score; the default is underlined.
β
0.0
0.1
0.3
0.5
0.7
0.9
1.0
TextVQA
59.1
59.2
59.4
59.4
59.4
59.2
59.3
MMB
65.3
65.0
65.2
65.6
65.5
65.3
65.3
TABLE IV: Ablation on β at R=25% ( α=0.7 ). β=0 uses only the query score as the local score; the default is underlined. β=1 removes the query score.
Method
TFLOPs ↓
Mem. ↓ (GB)
Thr. ↑ (q/s)
Perf. ↑
LLaVA-NeXT-7B
32.78
16.55
3.15 (1.0 × )
100.0%
HERO [ 12 ]
10.88
15.44
4.87 (1.5 × )
97.8%
Ours (fixed R )
10.88
14.55
6.34 (2.0 × )
97.8%
Ours
10.88
14.84
6.35 (2.0 × )
98.0%
TABLE V: Inference efficiency of LLaVA-NeXT-7B on POPE at R=25% . TFLOPs: LLM prefill FLOPs. Memory: peak GPU memory. Throughput: questions per second. Performance: average relative score in Table I .
Large Vision-Language Models (LVLMs) face significant computational inefficiencies caused by the large number of visual tokens. Existing visual token pruning methods mainly focus on either retaining individually important tokens or selecting mutually diverse ones. In this work, we revisit visual token pruning from a coverage perspective and formulate it as a biased attention coverage maximization problem. The key idea is to select a compact token subset whose encoder-side outgoing attention can jointly cover the image while assigning higher coverage priority to more informative regions. From this perspective, we propose ACPruner, a training-free visual token pruning framework for efficient LVLM inference. ACPruner first estimates token importance by combining intra-modal saliency and inter-modal relevance, then derives token-wise coverage from attention patterns within the vision encoder, and finally performs greedy selection to maximize the proposed coverage objective. Extensive experiments across multiple LVLM backbones, including LLaVA-1.5-7B/13B, LLaVA-NeXT-7B/13B, Qwen2.5-VL-7B, and LLaVA-OneVision-7B, show that ACPruner consistently achieves strong performance retention while delivering substantial end-to-end inference speedups.
Xu Li, Yuxuan Liang, Yi Zheng +6
College of Computer Science and Artificial Intelligence Fudan University, Shanghai, China
Large Vision-Language Models (LVLMs) rely on dense visual tokens to capture fine-grained visual information, but processing all these tokens incurs substantial computational and memory overhead during inference. To address this issue, we propose ResPrune, a training-free visual token pruning framework that enables efficient LVLM inference by selecting a compact yet informative subset of visual tokens. ResPrune formulates visual token pruning as a subspace reconstruction problem and employs a greedy subspace expansion strategy guided by residual energy, allowing it to preserve the geometric structure of the original visual token space. To further incorporate cross modal alignment, the selection process is conditioned on textual relevance, encouraging the retention of tokens that are both informative and instruction-relevant. The proposed method is lightweight and model-agnostic, and can be seamlessly integrated into existing LVLM pipelines without retraining or architectural modifications. Extensive experiments on multiple LVLM backbones, including LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL, demonstrate that ResPrune consistently outperforms existing pruning approaches across a wide range of benchmarks, while achieving effective reductions in computation, memory consumption, and inference latency.
Xu Li, Yi Zheng, Yuxuan Liang +5
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai 200433, China
While Large Vision-Language Models (LVLMs) demonstrate exceptional multi-modal capabilities, the quadratic computational cost of processing high-resolution visual tokens remains a critical bottleneck. Though recent token reduction strategies attempt to accelerate inference, such methods inadequately exploit attention values and fail to address token redundancy. More critically, they overlook the ``attention shift'' phenomenon inherent in LVLMs, which skews token attention scores. In this work, we propose ASAP, a novel training-free, KV-Cache-compatible pruning recipe that comprehensively addresses these limitations. First, we mitigate the attention shift by utilizing a dynamic bidirectional soft attention mask, ensuring the selection of genuinely informative tokens rather than naive attention-based selection. Second, we posit that high semantic redundancy within the token set degrades performance. We therefore introduce a weighted soft merging component that merges semantically similar tokens, preserving only the most feature-dense visual patches for subsequent layers. ASAP achieves virtually lossless compression of visual context, retaining 99.02% of the original LLaVA-NeXT-7B performance while aggressively slashing computational FLOPs by ~80%.