High-resolution Large Vision-Language Models (LVLMs) preserve critical, fine-grained visual details by encoding a global thumbnail alongside multiple local crops. However, the resulting thousands of vision tokens introduce severe computational bottlenecks during Large Language Model (LLM) inference. Existing token pruning methods rely primarily on query-agnostic visual saliency, while current query-aware approaches either operate at a coarse crop-level resolution or compute token relevance too late, after vision tokens have already entered the heavy LLM layers. To address this, we propose a training-free, hierarchical token pruning framework that utilizes query guidance prior to executing any LLM blocks. Our top-down hierarchy first operates at the instance-level, dynamically scaling the vision token retention ratio of each sample based on its visual complexity. Next, at the crop-level, we distribute this sample budget across local crops using sentence-level cross-modal similarity. Finally, at the token-level, we exploit projected vision tokens alongside textual query token embeddings in the shared space to measure fine-grained alignment, applying a text-debiasing normalization to produce high-fidelity query scores. These query-guided budgets and scores are fused with local and global visual saliency metrics to select the most critical vision tokens. Extensive experiments on LLaVA-NeXT-7B demonstrate that our plug-and-play framework retains 98.0% and 94.8% of unpruned performance at 25% and 10% vision token retention, respectively. Notably, at 25% retention, our method delivers a 2.0x inference throughput speedup and even surpasses the unpruned baseline on POPE by 0.7 points.
Figures & tables
Fig. 1: Pruning masks on a POPE sample at a 25% retention ratio. GlobalCom 2 uses an image-only mask that is unchanged by the query; for the chair question shown here, it removes most chair evidence before the LLM and answers incorrectly. Our query-aware mask retains the relevant chair tokens and answers correctly.
Fig. 2: Adaptive per-sample budget allocation on POPE examples. Our query-aware framework dynamically scales token retention according to scene complexity. The visually redundant snow scene (left) retains only 516 of 2,880 visual tokens, whereas the cluttered baseball scene (right) dynamically allocates 860 tokens to preserve fine-grained details needed for accurate question answering.
Fig. 3: Overview of our training-free token pruning framework for high-resolution LVLMs. An input image is processed into a full-image view and multiple sub-images, which are passed through a Visual Encoder and Projector to produce visual tokens ( V ). A text query is tokenized into embeddings ( E ). The pruning process consists of three main stages: (1) Adaptive Budget Allocation: A two-level budget is computed. At the Sample-level , a token retention ratio ( Rj ) is dynamically set for each image based on its visual token diversity ( Hj ). At the Crop-level , this budget is distributed among the sub-images based on the query-crop relevance. (2) Query Score Algorithm: A fine-grained relevance score is computed between the query embeddings ( E ) and the visual tokens ( V ). Raw cross-modal similarities are standardized using a text-debiasing z-score to produce a robust query score ( qsi ) for each token. (3) Fuse & Select: The final token selection score is calculated by fusing the Query Score with two visual saliency cues: a local score from the sub-image’s own [CLS] attention ( ai ) and a global score derived from the full-image view ( gi ). The top- bi tokens are selected and fed to LLM for inference.
Method
VizWiz
GQA
SQA-I
TextVQA
MME-P
MMB
AI2D
POPE
Avg. (%)
Retention ratio R=25%
LLaVA-NeXT-7B
60.6
64.3
70.3
61.4
1519.3
67.1
65.3
87.6
100.0
FastV [ 8 ]
54.2
60.4
68.8
58.4
1477.3
65.6
63.9 ∗
83.1
95.5
SparseVLM [ 19 ]
56.0
59.9
67.5
58.3
1465.9
65.6
64.2 ∗
85.0
95.8
PyramidDrop [ 20 ]
56.8
60.3
68.5
59.8
1473.7
65.6
63.9 ∗
85.5
96.6
FasterVLM [ 9 ]
55.4
61.3
67.1
58.8
1454.6
66.0
62.6 ∗
87.2
95.9
TABLE I: Comparison with existing methods on LLaVA-NeXT-7B. The retention ratio R is defined with respect to the 2,880 token positions including padding. Avg. is the mean score relative to the unpruned model. Baseline results without a mark are taken from GlobalCom 2 [ 11 ] ; ∗ denotes results obtained by running the released code. The best and second-best results among pruning methods are in bold and underlined.
Sample alloc.
Query score
MME-P
AI2D
✓
1501.4
63.4
✓
1506.5
63.6
✓
✓
1514.1
63.7
TABLE II: Module ablation on LLaVA-NeXT-7B at R=25% .
α
0.0
0.1
0.3
0.5
0.7
0.9
1.0
MMB
65.4
65.3
65.3
65.2
65.6
65.0
65.1
POPE
88.1
88.1
88.2
88.2
88.3
88.3
87.7
TABLE III: Ablation on α at R=25% ( β=0.5 ). α=1 uses only the global score; the default is underlined.
β
0.0
0.1
0.3
0.5
0.7
0.9
1.0
TextVQA
59.1
59.2
59.4
59.4
59.4
59.2
59.3
MMB
65.3
65.0
65.2
65.6
65.5
65.3
65.3
TABLE IV: Ablation on β at R=25% ( α=0.7 ). β=0 uses only the query score as the local score; the default is underlined. β=1 removes the query score.
Method
TFLOPs ↓
Mem. ↓ (GB)
Thr. ↑ (q/s)
Perf. ↑
LLaVA-NeXT-7B
32.78
16.55
3.15 (1.0 × )
100.0%
HERO [ 12 ]
10.88
15.44
4.87 (1.5 × )
97.8%
Ours (fixed R )
10.88
14.55
6.34 (2.0 × )
97.8%
Ours
10.88
14.84
6.35 (2.0 × )
98.0%
TABLE V: Inference efficiency of LLaVA-NeXT-7B on POPE at R=25% . TFLOPs: LLM prefill FLOPs. Memory: peak GPU memory. Throughput: questions per second. Performance: average relative score in Table I .