Vision Token Pruning via Query-Vision Interaction Decomposition

Harshithanjani Athi; Sravan Kumar Ankireddy; Jianzhong Charlie Zhang; Hyeji Kim

cs.LGOct 23, 2026

Vision Token Pruning via Query-Vision Interaction Decomposition

Authors: Harshithanjani Athi, Sravan Kumar Ankireddy, Jianzhong Charlie Zhang, Hyeji Kim

Organizations: 1The University of Texas at Austin · 2Samsung Research America

Abstract

Large vision-language models (VLMs), which process both visual inputs and text queries, incur high inference costs due to the large number of visual tokens, making visual token pruning a natural approach for improving efficiency. Recent work has leveraged query information to guide token selection, often through heuristics derived from attention patterns or token dynamics. However, these approaches do not explicitly exploit the underlying structure of query-vision interactions. We address this gap by observing that the query-vision interaction matrix, naturally induced by the query and key projections already present in transformer attention, is effectively low-rank, with a small number of dominant latent interaction modes capturing query-relevant semantics. Motivated by this observation, we propose Query-Vision Decomposition (QViD), a novel training-free query-aware visual token pruning method that exploits this structure. Extensive experiments on both image and video understanding benchmarks show that QViD consistently improves performance under matched token budgets, with the largest gains appearing under aggressive compression regimes.

Figures & tables

Figure 1: Comparison of Visual-Token Scoring. Compared with prior methods such as V 2 Drop and SparseVLM, QViD concentrates high scores on query-relevant image regions, leading to more effective visual-token pruning.

Figure 2: QViD Pipeline. QViD performs query-aware visual-token pruning by explicitly modeling query–vision interactions. At pruning layer $\ell$ , it forms the interaction matrix $M=Q_{\text{txt}}K_{\text{vis}}^{\top}$ , extracts its dominant low-rank modes with SVD, and scores each visual token by its participation in these modes. The top- $K$ tokens are retained for subsequent reasoning, preserving query-relevant visual evidence while reducing sequence length. Integer values in the illustrative matrices are used only for ease of presentation.

Panel of Figure 3 — Figure 3: Qualitative Observations. (a) The query–vision interaction matrix has a rapidly decaying singular-value spectrum. (b) Dominant SVD components emphasize query-relevant tokens, while residual components are less aligned with the task.

Methods	GQA	SQA	TextVQA	POPE	MME	MMB	Average
Upper Bound, 576 Tokens
LLaVA-1.5-7B (Liu et al. 2024)	61.9	69.5	58.2	85.9	1862	64.6	100.0%
Average Retain 192 Tokens ( $\downarrow$ 66.7%)
QViD (ours)	59.3	69.0	57.8	87.4	1793	63.2	98.1%
V 2 Drop [CVPR’26]	58.5	69.3	55.6	85.1	1826	63.7	97.6%
PDrop [CVPR’25]	57.1	68.8	56.1	82.3	1766	63.2	96.0%

Table 1: Image Understanding Results on LLaVA-1.5-7B. QViD achieves the best average performance at all retention ratios, with the largest gain under aggressive compression at 64 tokens. Methods are sorted by average performance within each retention setting.

Method	AI2D	GQA	TextVQA	POPE	MME-P	MME-C	SQA	MMB	Avg.
Baseline	80.7	61.9	81.5	88.2	1684.0	635.7	85.3	84.9	100.0%
Retain 33.3% ( $\downarrow$ 66.7%)
QViD (ours)	76.6	58.6	79.0	85.6	1610.6	493.6	81.2	81.0	93.4%
V 2 Drop [CVPR’26]	76.7	52.8	53.4	85.4	1589.0	600.4	80.9	82.0	90.4%
Retain 22.2% ( $\downarrow$ 77.8%)
QViD (ours)	73.8	56.0	77.1	83.7	1577.9	446.1	79.0	78.8	90.1%

Table 2: Image Understanding Results on Qwen2-VL-7B. QViD improves average performance at all retention ratios, with the largest gains at 11.1% retention. Avg. is mean normalized performance relative to the full-token baseline.

Method	MVBench	VideoMME				Avg.
		Overall	Short	Med.	Long
Baseline	58.4	58.2	70.3	56.4	47.9	100.0%
Retention = 25%
QViD	56.6	57.2	68.3	54.4	49.2	97.7%
V 2 Drop [CVPR’26]	56.5	57.1	68.1	53.6	49.8	97.5%
Retention = 15%

Table 3: Video Understanding and Compute Comparison. (a) Results on LLaVA-OV-7B show that QViD consistently outperforms V 2 Drop under matched token budgets, with the largest gains under aggressive compression. Avg. denotes the mean normalized performance relative to the full-token baseline. (b) Total TFLOPs on LLaVA-1.5-7B include pruning overhead; reduction is computed relative to the full-token baseline.

Retain 64 Tokens ( $\downarrow$ 88.9%)
Method	GQA	SQA	TextVQA	POPE	MME	Avg.
Baseline	61.9	69.5	58.2	85.9	1862.0	100.0%
QViD	52.2	68.3	55.5	80.8	1553.0	91.1%
Column norm	42.2	64.0	42.6	54.7	1009.0	70.3%
Retain 128 Tokens ( $\downarrow$ 77.8%)
QViD	57.7	68.7	57.1	85.5	1726.0	96.5%

Table 4: Ablations on QViD Scoring with LLaVA-1.5-7B. (a) SVD scoring outperforms column-norm scoring from the same query–vision interaction matrix. (b) Using only a small number of dominant SVD components recovers most of the benefit of QViD, indicating that token relevance is concentrated in a few interaction modes.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Layer $\ell$	AI2D	GQA	TextVQA	POPE	MME	SQA	Avg.
Baseline	55.2	61.9	58.2	85.9	1862.0	69.5	100.0%
1	53.7	56.7	56.7	85.1	1718.8	68.9	96.1%
2	54.0	57.9	56.8	85.6	1710.9	68.6	96.5%
3	54.1	57.7	57.1	85.5	1726.3	68.7	96.7%
4	53.5	57.2	57.2	85.4	1719.3	69.0	96.4%
5	53.7	57.0	57.3	85.4	1729.6	68.6	96.5%

Appendix

Table 5: Layer Selection Ablation on LLaVA-1.5-7B. We vary the scoring layer

\ell

while retaining 128 visual tokens. MME denotes the sum of MME-Perception and MME-Cognition. Avg. denotes the mean normalized performance relative to the full-token baseline across all reported metrics. Bold denotes the best value in each column.

Layer $\ell$	AI2D	GQA	TextVQA	POPE	MME-P	MME-C	SQA	MMB	Avg.
Baseline	80.7	61.9	81.5	88.2	1684.0	635.7	85.3	84.9	100.0%
Retain 33.3% Tokens ( $\downarrow 66.7\%$ )
1	74.8	57.2	78.5	84.9	1604.6	477.9	81.4	79.1	92.1%
2	75.7	58.0	79.4	85.9	1618.4	470.7	81.7	80.8	92.9%
3	76.6	58.6	79.0	85.6	1610.6	493.6	81.2	81.0	93.4%
4	75.9	59.1	77.9	86.7	1596.0	502.5	81.5	80.5	93.4%

Appendix

Table 6: Full Layer Sweep on Qwen2-VL-7B. We vary the QViD scoring layer

\ell

across three retention ratios. Avg. is computed as the mean normalized performance relative to the full-token baseline. Bold denotes the best value in each column within a retention block.

Retain 64 Tokens ( $\downarrow$ 88.9%)
Method	GQA	SQA	TextVQA	POPE	MME	Avg.
Baseline	61.9	69.5	58.2	85.9	1862	100.0%
V 2 Drop	50.5	68.9	51.8	75.1	1470.0	87.2%
$n_{\mathrm{svd}}=1$	42.1	63.8	42.6	20.7	1002.7	62.2%
$n_{\mathrm{svd}}=2$	47.4	67.5	48.3	54.8	1362.3	78.7%
$n_{\mathrm{svd}}=4$	53.5	68.1	54.2	74.0	1599.5	89.9%

Appendix

Table 7: Ablation on the Number of SVD Components for LLaVA-v1.5-7B.

n_{\mathrm{svd}}

denotes the number of singular-vector components used to compute the visual-token importance scores. We vary

n_{\mathrm{svd}}

, the number of singular-vector components used for QViD scoring, across three token budgets. Avg. denotes the mean normalized performance relative to the full-token baseline across all reported metrics.

Retain 64 Tokens ( $\downarrow$ 88.9%)
Method	GQA	SQA	TextVQA	POPE	MME	Avg.
Baseline	61.9	69.5	58.2	85.9	1862.0	100.0%
Column norm	42.2	64.0	42.6	54.7	1009.0	70.3%
Sigma-weighted	42.2	64.0	42.6	54.7	1009.0	70.3%
Per-head max	53.2	68.1	55.7	82.1	1581.0	92.0%
QViD	52.2	68.3	55.5	80.8	1553.0	91.1%

Appendix

Table 8: Scoring-Rule Ablation on LLaVA-1.5-7B. We compare different scoring rules derived from the query–vision interaction matrix at matched token budgets. Avg. denotes the mean normalized performance relative to the full-token baseline across all reported metrics.

Method	Tokens	Lat. (ms)	Tput. (img/s)	Speedup	TFLOPs
Baseline	576	100.1	8.60	1.00 $\times$	8.86
QViD	64	80.1	10.40	1.21 $\times$	2.53
QViD	128	81.3	10.27	1.19 $\times$	3.42
QViD	192	81.5	10.26	1.19 $\times$	4.32
V 2 Drop	64	77.7	10.66	1.24 $\times$	1.97
V 2 Drop	128	78.6	10.55	1.23 $\times$	2.85

Appendix

Table 9: Detailed Efficiency Results on LLaVA-1.5-7B. We report measured latency, throughput, speedup, and total TFLOPs for different pruning methods and retained-token budgets. TFLOPs include pruning overhead.

Figure 4: Accuracy–Compute Tradeoff on LLaVA-1.5-7B. We compare V 2 Drop, and QViD at 64, 128, and 192 retained visual tokens on POPE, GQA, and TextVQA. The gray dashed line and star indicate the full-token baseline with 576 visual tokens. QViD achieves a stronger accuracy–compute tradeoff, especially under aggressive pruning, by retaining more query-relevant visual tokens at comparable TFLOPs.

Method	$n$	AI2D	ChartQA	MMMU	OCRBench	SEEDBench	Avg.
Baseline	576	55.2	18.2	36.1	31.5	66.2	100.0%
Retain 64 Tokens ( $\downarrow 88.9\%$ )
V 2 Drop	64	52.3	13.2	33.3	14.4	48.0	75.5%
QViD	64	52.2	16.6	35.0	26.5	54.6	89.9%
Retain 128 Tokens ( $\downarrow 77.8\%$ )
V 2 Drop	128	53.5	15.0	35.1	26.2	59.4	89.9%

Appendix

Table 10: Additional Multimodal Benchmark Results on LLaVA-v1.5-7B. We compare QViD with V 2 Drop at matched token budgets on AI2D, ChartQA, MMMU, OCRBench, and SEEDBench. Avg. denotes the mean normalized performance relative to the full-token baseline.

Method	Retention	MMMU-Pro
Baseline	100%	33.8
V 2 Drop	33.3%	32.3
QViD	33.3%	31.7
V 2 Drop	22.2%	30.2
QViD	22.2%	29.9
V 2 Drop	11.1%	23.6

Appendix

Table 11: MMMU-Pro Standard Results on Qwen2-VL-7B. We compare QViD with V 2 Drop under matched retention ratios on the 1,730-sample standard-format split.

All Tokens (100%)
Method	MVBench	VideoMME				Avg.
		Overall	Short	Med.	Long
Baseline	42.9	36.2	41.2	35.0	32.4	100.0%
Retention = 25%
QViD	41.5	37.7	43.2	36.1	33.8	100.4%
V 2 Drop	41.2	36.3	40.1	35.1	33.7	98.2%

Appendix

Table 12: Video Understanding Results on Video-LLaVA-7B. We compare QViD with V 2 Drop on MVBench and VideoMME under token compression. Avg. denotes the mean normalized performance over MVBench and VideoMME Overall relative to the full-token baseline.

Retain 64 Tokens ( $\downarrow 88.9\%$ )
Method	$n$	HallusionBench			COCO Captioning
		aAcc	fAcc	qAcc	CIDEr
Baseline	576	47.9	20.5	12.3	1.087
V 2 Drop	64	46.8	15.0	9.5	0.591
QViD	64	47.8	19.7	12.7	0.782
Retain 128 Tokens ( $\downarrow 77.8\%$ )

Appendix

Table 13: HallusionBench and COCO Captioning Results on LLaVA-v1.5-7B. We compare QViD with V 2 Drop at matched token budgets. HallusionBench is reported using aAcc, fAcc, and qAcc, while COCO Captioning is reported using CIDEr. QViD improves both hallucination-related metrics and caption quality under visual-token compression.

Method	Retention	HallusionBench			COCO Captioning
		aAcc	fAcc	qAcc	CIDEr
Baseline	100%	67.5	38.7	42.2	0.977
Retain 33.3% Tokens ( $\downarrow 66.7\%$ )
V 2 Drop	33.3%	61.8	33.8	33.6	0.563
QViD	33.3%	62.3	35.3	35.2	0.922
Retain 22.2% Tokens ( $\downarrow 77.8\%$ )

Appendix

Table 14: HallusionBench and COCO Captioning Results on Qwen2-VL-7B. We compare QViD with V 2 Drop at matched retention ratios where available. HallusionBench is reported using aAcc, fAcc, and qAcc, while COCO Captioning is reported using CIDEr. QViD preserves stronger hallucination robustness and substantially better captioning quality under compression.

Figure 5: Low-Rank Structure of Query–Vision Interactions. We plot the normalized singular-value spectrum

\sigma_{k}/\sigma_{1}

M=Q_{\mathrm{txt}}K_{\mathrm{vis}}^{\top}

at the pruning layer of LLaVA-1.5-7B. Across datasets, the spectra decay rapidly, showing that query–vision interactions are concentrated in a few dominant directions. Faint curves show individual samples, solid curves show the mean, and shaded bands indicate variation across heads.

Figure 6: Dominant SVD Components Capture Task-Relevant Structure. Top SVD components emphasize task-critical query tokens, while residual components are less aligned with the question. The spectra show that query–vision interactions are concentrated in a few dominant directions.

Retention	Method	POPE	GQA	TextVQA	MMStar	AI2D	MME
–	Baseline	$87.3{\pm}0.7$	$61.9{\pm}0.8$	$81.2{\pm}1.0$	$57.6{\pm}2.5$	$80.7{\pm}1.4$	$2320{\pm}113$
5.56%	QViD	$59.9{\pm}0.9$	$45.2{\pm}0.9$	$64.0{\pm}1.3$	$34.8{\pm}2.4$	$66.6{\pm}1.7$	$1520{\pm}153$
	FastV	$46.3{\pm}1.0$	$37.7{\pm}0.8$	$42.5{\pm}1.3$	$25.7{\pm}2.2$	$63.7{\pm}1.7$	$1076{\pm}159$
	V 2 Drop	$57.6{\pm}1.0$	$39.7{\pm}0.9$	$44.9{\pm}1.3$	$27.8{\pm}2.3$	$64.7{\pm}1.7$	$1090{\pm}159$
11.1%	QViD	$73.5{\pm}0.8$	$50.7{\pm}0.9$	$72.2{\pm}1.2$	$39.0{\pm}2.5$	$69.5{\pm}1.6$	$1771{\pm}148$
	FastV	$58.0{\pm}1.0$	$43.2{\pm}0.9$	$55.3{\pm}1.3$	$33.5{\pm}2.4$	$64.8{\pm}1.7$	$1479{\pm}149$

Appendix

Table 15: 95% confidence intervals on Qwen2-VL-7B over three independent random seeds.

Figure 7: Additional Qualitative Token-Score Visualizations. We compare token-score maps from QViD, V 2 Drop, and SparseVLM across multiple image-query pairs. Brighter regions indicate higher token scores. QViD tends to assign high scores to query-relevant visual regions, while the baseline methods often produce more diffuse score maps.

Figure 8: Additional Query-Token Score Visualizations. For each example, we compare query-token scores obtained from the dominant SVD components with scores from the residual components. The dominant components place higher weight on semantically informative query tokens, while residual components are less aligned with the task-relevant words.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

May 28, 2026

OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning

Vision-language models (VLMs) rely on long visual token sequences for visual understanding, making the prefill stage expensive in both computation and memory. Most existing pruning methods follow an absolute-ranking paradigm, assigning importance scores to visual tokens and retaining a fixed top-K subset. In this work, we argue that this paradigm is fundamentally brittle: attention sinks distort token importance rankings, while image redundancy and query-dependent visual evidence make fixed token budgets unreliable across inputs. We propose OccamToken, a training-free framework that replaces absolute token ranking with register-anchored relative evidence testing. Instead of asking which tokens are globally important, OccamToken evaluates whether a visual token provides information beyond a register-based reference. Our key insight is that register tokens naturally absorb low-information attention patterns, making them a stable reference for identifying genuinely informative visual evidence. Based on this principle, OccamToken performs both image-adaptive redundancy pruning and query-adaptive relevance pruning through dynamic thresholds derived from register attention. Across LLaVA-NeXT, LLaVA-v1.5, and Qwen3-VL, OccamToken consistently improves the accuracy-efficiency trade-off without additional training. Notably, on LLaVA-NeXT, it reduces 2,880 visual tokens to approximately 40 while preserving over 93% of the original accuracy, enabling stable visual token compression even in the extreme 1.4% retention regime.

Geng Li, Guohao Chen, Ting Chen +6

1Nanyang Technological University (NTU)

Efficient VLM Inference Large Vision-Language Models Visual Token Pruning Visual Token Compression LVLM Pruning

Ask PDF

Oct 16, 2026

Hierarchical and Query-Aware Visual Token Pruning for High-Resolution LVLMs

High-resolution Large Vision-Language Models (LVLMs) preserve critical, fine-grained visual details by encoding a global thumbnail alongside multiple local crops. However, the resulting thousands of vision tokens introduce severe computational bottlenecks during Large Language Model (LLM) inference. Existing token pruning methods rely primarily on query-agnostic visual saliency, while current query-aware approaches either operate at a coarse crop-level resolution or compute token relevance too late, after vision tokens have already entered the heavy LLM layers. To address this, we propose a training-free, hierarchical token pruning framework that utilizes query guidance prior to executing any LLM blocks. Our top-down hierarchy first operates at the instance-level, dynamically scaling the vision token retention ratio of each sample based on its visual complexity. Next, at the crop-level, we distribute this sample budget across local crops using sentence-level cross-modal similarity. Finally, at the token-level, we exploit projected vision tokens alongside textual query token embeddings in the shared space to measure fine-grained alignment, applying a text-debiasing normalization to produce high-fidelity query scores. These query-guided budgets and scores are fused with local and global visual saliency metrics to select the most critical vision tokens. Extensive experiments on LLaVA-NeXT-7B demonstrate that our plug-and-play framework retains 98.0% and 94.8% of unpruned performance at 25% and 10% vision token retention, respectively. Notably, at 25% retention, our method delivers a 2.0x inference throughput speedup and even surpasses the unpruned baseline on POPE by 0.7 points.

Luther Wei, Ling Zou, Kang-Yang Huang +2

Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan · MediaTek Inc., Hsinchu, Taiwan

Visual Token Pruning Efficient VLM Inference LVLM Pruning

Ask PDF

Apr 27, 2026

LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models

Vision-Language Models (VLMs) have recently demonstrated remarkable capabilities in visual understanding and reasoning, but they also impose significant computational burdens due to long visual sequence inputs. Recent works address this issue by pruning unimportant visual tokens, achieving substantial computational reduction while maintaining model performance. The core of token pruning lies in determining token importance, with current approaches primarily relying on attention scores from vision encoders or Large Language Models (LLMs). In this paper, we analyze the effectiveness of attention mechanisms in both vision encoders and LLMs. We find that vision encoders suffer from attention sink, leading to poor focus on informative foreground regions, while in LLMs, although prior studies have identified attention bias toward token positions, text-to-vision attention demonstrates resistance to this bias and enables effective pruning guidance in middle layers. Based on these observations, we propose LearnPruner, a two-stage token pruning framework that first removes redundant vision tokens via a learnable pruning module after the vision encoder, then retains only task-relevant tokens in the LLM's middle layer. Experimental results show that our LearnPruner can preserve approximately 95% of the original performance while using only 5.5% of vision tokens, and achieve 3.2

\times

inference acceleration, demonstrating a superior accuracy-efficiency trade-off.

Rinyoichi Takezoe, Yaqian Li, Zihao Bo +3

Li Auto Inc.

Efficient VLM Inference Attention Sinks Visual Token Pruning

Ask PDF

Abstract

Efficient VLM Inference

Explore similar work

OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning

Hierarchical and Query-Aware Visual Token Pruning for High-Resolution LVLMs

LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models