Don't Ask the LLM: Visual Token Pruning with Spectral Structure and Semantic Relevance

Intaek Shin; Geuntaek Lim; Sungjune Park; Yukyung Choi

cs.CVOct 28, 2026

Don't Ask the LLM: Visual Token Pruning with Spectral Structure and Semantic Relevance

Authors: Intaek Shin, Geuntaek Lim, Sungjune Park, Yukyung Choi

Organizations: Sejong University, South Korea

Abstract

Large vision-language models (LVLMs) represent images and videos as long visual-token sequences, substantially increasing inference costs. These costs motivate visual-token pruning, whose effectiveness depends on both what signal is used for selection and when that signal becomes available. Our analysis reveals complementary limitations: visual-side [CLS] attention produces scattered, background-biased selections under aggressive compression, while stronger query-conditioned signals become available after substantial computation within the large language model (LLM). This raises a natural question: can structurally coherent, query-aware token selection be performed before consulting the LLM? We therefore propose ST-Pruner (Spectral-Textual Pruner), a training-free framework for pre-LLM visual-token selection. It combines attention-independent spectral saliency with inverted patch-text similarity through parameter-free multiplicative fusion. Across diverse LVLMs, ST-Pruner retains over 95% of full-token performance using at most 11.1% of visual tokens and achieves a 3.2x end-to-end speedup on LLaVA-NeXT. Code is available at https://github.com/sejong-rcv/ST-Pruner.

Figures & tables

Figure 1: Structural Saliency Comparison. (a) Selection of $K=64$ tokens using [CLS] attention and our spectral criterion. (b) F1 score on POPE across token budgets using each structural signal.

Figure 2: Why Not Ask the LLM? (a) Pre-LLM inverted patch–text similarity ( $-\mathrm{sim}$ ) and in-LLM query-to-patch attention at representative decoder layers, visualized by their top- $K$ token selections. (b–c) Evaluations on GQA ( $n=12{,}578$ ). (b) Re-inference accuracy when retaining only the top- $K$ tokens selected by each LLM layer, with the full-token model and pre-LLM $-\mathrm{sim}$ as references. (c) Layer-wise overlap between the pre-LLM $-\mathrm{sim}$ and in-LLM top- $K$ token sets, normalized by the expected overlap of two random $K$ -token subsets, such that $1\times$ denotes random overlap.

Figure 3: Overview of ST-Pruner. Given pre-LLM visual features $\mathbf{Z}$ , the spectral branch derives its saliency score from a randomized low-rank SVD, while the semantic branch computes inverted patch–text relevance using a query embedding from the corresponding frozen text encoder of the paired vision–language model. Their parameter-free multiplicative fusion determines the top- $K$ visual tokens, which are retained in their original sequence order before LLM inference. The textual query is independently tokenized for the standard language-model input.

Upper Bound, 576 Tokens ( 100% )
Method	VQA V2	GQA	VizWiz	SQA	VQA Text	POPE	MME P	MMB	Avg.
LLaVA-1.5-7B	78.5	61.9	50.1	69.5	58.2	85.9	1506.5	64.7	100.0
Retain 192 Tokens ( $\downarrow 66.7\%$ )
FastV (ECCV’24)	67.1	52.7	50.8	67.3	52.5	64.8	–	61.2	89.9
SparseVLM (ICML’25)	75.6	57.6	50.5	69.1	56.1	83.6	–	62.5	97.1
VisionZip (CVPR’25)	76.8	59.3	50.5	68.9	57.3	85.3	1432.7	63.0	98.0

Table 1: Performance comparison on LLaVA-1.5-7B. Avg. denotes mean performance retention relative to the displayed full-token upper bound over the available metrics.

Upper Bound, 2880 Tokens ( 100% )
Method	VQA V2	GQA	VizWiz	SQA	VQA Text	POPE	MME P	Avg.
LLaVA-NeXT-7B	81.3	62.7	54.7	72.1	60.3	86.5	1501.1	100.0
Retain 640 Tokens ( $\downarrow 77.8\%$ )
FastV	77.0	58.9	53.9	67.4	58.1	79.5	1412.6	94.7
SparseVLM	79.2	61.2	53.6	67.6	59.7	85.3	1456.8	97.3
VisionZip	79.1	61.2	57.1	68.1	59.9	86.0	1493.4	98.9

Table 2: Performance comparison on LLaVA-NeXT-7B. Avg. denotes mean retention relative to the reproduced full-token upper bound over the displayed metrics. LLaVA-NeXT uses a variable number of views, and the displayed token counts denote maxima for five-view inputs.

Upper Bound, 2048 Tokens ( 100% )
Method	TGIF	MSVD	MSRVTT	Avg.
Video-LLaVA-7B	47.1	69.8	56.7	100.0
Retain 136 Tokens ( $\downarrow 93.4\%$ )
FastV (ECCV’24)	23.1	38.0	19.3	45.8
SparseVLM (ICML’25)	44.7	68.2	31.0	82.4
VisionZip (CVPR’25)	42.4	63.5	52.1	91.0

Table 3: Performance comparison on Video-LLaVA-7B. Avg. denotes mean performance retention over the three video benchmarks relative to the displayed full-token reference.

Upper Bound, 1296 Tokens ( 100% )
Method	GQA	SQA	POPE	Avg.
Qwen2.5-VL-7B	62.0	87.8	88.2	100.0
Retain 512 Tokens ( $\downarrow 60.5\%$ )
VisionZip (CVPR’25)	58.1	86.2	83.3	95.4
DivPrune (CVPR’25)	61.3	87.9	87.4	99.4
VisPruner (ICCV’25)	60.8	87.3	86.7	98.6

Table 4: Cross-architecture comparison on Qwen2.5-VL-7B. ST-Pruner (Spectral) denotes the proposed spectral criterion without the semantic branch.

Retain 128 Tokens
Variant	GQA	POPE	MME P	VQA Text
Semantic-only ( $+\mathrm{sim}$ )	39.5	1.5	626.0	42.7
Semantic-only ( $-\mathrm{sim}$ )	59.6	86.9	1407.0	53.9
Spectral-only	58.5	85.9	1377.2	57.0
ST-Pruner (Ours)	59.4	87.3	1432.3	57.1
Retain 64 Tokens

Table 5: Signal ablation on LLaVA-1.5-7B.

+\mathrm{sim}

and

-\mathrm{sim}

denote conventional and inverted patch–text similarity, respectively. Spectral-only denotes the spectral branch of ST-Pruner.

Fusion Strategy
Setting	$K=128$	$K=64$
Entropy-weighted ( $\tau=0.5$ )	97.2	94.4
Direct product (Ours)	97.4	94.5
Spectral Rank $R$ with Direct Product
$R=18$	96.2	93.6
$R=36$	97.0	94.2

Table 6: Robustness to fusion strategy and spectral rank. Values denote mean performance retention over GQA, POPE, TextVQA, MME P , and MMB, normalized by the matched full-token reproduction used for these ablations. For entropy-weighted fusion, we report the strongest setting (

\tau=0.5

) from the full temperature sweep. Complete per-benchmark results are provided in the supplementary material.

Method	Retain	FLOPs (T)	Prefill (ms)	Decode (ms)	Total (ms)	KV (MB)	POPE F1
LLaVA-NeXT-7B	2880	34.92	679.2	48.8	727.9	1542	86.5
SparseVLM (ICML’25)	320 †	7.81 ( $\times 4.5$ )	299.3 ( $\times 2.3$ )	44.1 ( $\times 1.1$ )	343.4 ( $\times 2.1$ )	359	76.9
VisionZip (CVPR’25)	320	4.53 ( $\times 7.7$ )	142.5 ( $\times 4.8$ )	36.0 ( $\times 1.4$ )	178.4 ( $\times 4.1$ )	200	82.3
DivPrune (CVPR’25)	320	4.53 ( $\times 7.7$ )	187.5 ( $\times 3.6$ )	33.6 ( $\times 1.5$ )	221.1 ( $\times 3.3$ )	200	83.4
VisPruner (ICCV’25)	320	4.53 ( $\times 7.7$ )	163.0 ( $\times 4.2$ )	34.9 ( $\times 1.4$ )	197.9 ( $\times 3.7$ )	200	81.3
CDPruner (NeurIPS’25)	320	4.53 ( $\times 7.7$ )	177.7 ( $\times 3.8$ )	34.2 ( $\times 1.4$ )	211.9 ( $\times 3.4$ )	200	85.8

Table 7: Inference efficiency on LLaVA-NeXT-7B. Results are measured over the full POPE evaluation set of 8,910 samples on a single NVIDIA V100 GPU under FP16. The cross-method comparison uses a 320-token target budget. Prefill and total latency include method-specific pruning and scoring overhead. Total denotes end-to-end generation latency with a maximum generation length of 16 tokens. Values in parentheses denote speedup or FLOPs reduction factors relative to the unpruned model.

Retain 192 Tokens
Setting	GQA	POPE	TextVQA	MME P	MMB	Avg.
Entropy-weighted ( $\tau=0.1$ )	60.23	87.24	57.46	1462.73	62.24	98.23
Entropy-weighted ( $\tau=0.25$ )	60.23	87.22	57.42	1439.85	62.41	97.96
Entropy-weighted ( $\tau=0.5$ )	60.33	87.37	57.43	1430.01	62.33	97.88
Entropy-weighted ( $\tau=1.0$ )	60.34	87.34	57.49	1435.92	62.24	97.95
Entropy-weighted ( $\tau=2.0$ )	60.32	87.30	57.45	1438.44	62.24	97.95

Table S1 : Fine-grained fusion-strategy results. Avg. is the mean benchmark-wise performance retention relative to the matched full-token reproduction.

Retain 192 Tokens
Rank	GQA	POPE	TextVQA	MME P	MMB	Avg.
$R=18$	60.15	86.31	55.03	1449.01	62.84	97.16
$R=36$	60.43	87.03	57.15	1448.93	62.93	98.17
$R=72$ (Ours)	60.26	87.29	57.59	1449.94	62.50	98.21
$R=144$	60.25	87.07	57.17	1455.17	62.16	97.98
$R=288$	60.10	87.07	56.87	1435.42	62.59	97.70

Table S2 : Fine-grained spectral-rank sweep. Avg. is the mean benchmark-wise performance retention relative to the matched full-token reproduction.

Figure S1 : Qualitative comparison across vision encoders and spectral ranks. Two representative examples are shown. For each example, rows correspond to different vision encoders (CLIP, SigLIP, and SigLIP2), and columns show the original image and top- $K$ selections obtained with full-rank scoring and reduced-rank approximations ( $R=144,72,36$ ). Across encoder families and rank settings, the selections remain broadly concentrated on similar salient object and scene regions, with $R=72$ showing patterns close to full-rank scoring.

Retain 192 Tokens
Feature space	POPE	GQA	TextVQA	MME P	MMB
ViT-1024 (Ours)	87.29	60.26	57.59	1449.94	62.50
Projector-4096	87.02	60.42	57.55	1452.84	62.93
Retain 128 Tokens
ViT-1024 (Ours)	87.26	59.38	57.06	1432.26	62.16
Projector-4096	87.13	59.54	56.98	1437.73	61.99

Table S3 : Feature-space ablation on LLaVA-1.5-7B. We compare spectral scoring on 1024-dimensional pre-projector ViT features and 4096-dimensional projector outputs while keeping all other settings fixed.

Retain 192 Tokens
Weighting	GQA	POPE	TextVQA	MME P	MMB	Avg.
$p=0$ (uniform)	60.28	87.22	57.32	1446.14	62.41	98.03
$p=0.5$ (Ours)	60.26	87.29	57.59	1449.94	62.50	98.21
$p=1$ (linear)	60.34	87.25	57.23	1450.04	62.67	98.16
$p=2$ (quadratic)	56.61	82.98	49.53	1269.44	51.79	87.54
Retain 128 Tokens

Table S4 : Singular-value weighting ablation on LLaVA-1.5-7B. The exponent

p

controls the strength of singular-value weighting in

\sum_{k\leq R}\sigma_{k}^{p}U_{ik}^{2}

. Avg. is the mean benchmark-wise performance retention relative to the matched full-token reproduction.

Figure S2 : Spectral hierarchy on general visual content. Three examples span different scene structures. Columns show the original image and top- $K$ selections from component bands $1$ – $72$ , $73$ – $144$ , $145$ – $288$ , and $289$ – $576$ . Leading bands emphasize coherent foreground structure, whereas lower-energy bands become increasingly dispersed.

Figure S3 : Spectral hierarchy on text-bearing visual content. Three OCR-relevant examples use the same component bands as Figure S2 . Leading bands preserve localized text-bearing regions, whereas later bands become increasingly dispersed.

Method	Tokens	FLOPs (T)	Prefill (ms)	Decode (ms)	Total (ms)	KV (MB)	F1
LLaVA-NeXT-7B	2880	34.92	679.2	48.8	727.9	1542	86.5
Retain 640 Tokens
SparseVLM (ICML’25)	640 †	11.05 ( $\times 3.2$ )	368.4 ( $\times 1.8$ )	44.8 ( $\times 1.1$ )	413.1 ( $\times 1.8$ )	505	85.3
VisionZip (CVPR’25)	640	8.33 ( $\times 4.2$ )	222.7 ( $\times 3.1$ )	34.5 ( $\times 1.4$ )	257.2 ( $\times 2.8$ )	368	86.0
DivPrune (CVPR’25)	640	8.33 ( $\times 4.2$ )	286.5 ( $\times 2.4$ )	32.5 ( $\times 1.5$ )	319.0 ( $\times 2.3$ )	368	86.0
VisPruner (ICCV’25)	640	8.33 ( $\times 4.2$ )	267.9 ( $\times 2.5$ )	36.8 ( $\times 1.3$ )	304.7 ( $\times 2.4$ )	368	85.9

Table S5 : Complete inference-efficiency sweep on LLaVA-NeXT-7B. Values in parentheses denote speedup or FLOPs reduction factors relative to the 2880-token baseline. All latencies are measured over the full POPE set. For SparseVLM, FLOPs and KV cache use its actual layer-wise token trajectory rather than the final retained token count.

Visual Token Pruning

Visual token pruning reduces the image and video patches processed by transformers to lower inference cost while preserving task-relevant information.

See all 278 papers

ACPruner: Visual Token Pruning as Biased Attention Coverage Maximization in LVLMs
Sep 28, 2026Xu Li, Yuxuan Liang, Yi Zheng +6Efficient VLM Inference Large Vision-Language Models
College of Computer Science and Artificial Intelligence Fudan University, Shanghai, China
Hierarchical and Query-Aware Visual Token Pruning for High-Resolution LVLMs
Oct 16, 2026Luther Wei, Ling Zou, Kang-Yang Huang +2Visual Token Pruning Efficient VLM Inference
Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan · MediaTek Inc., Hsinchu, Taiwan
VPRune: Efficient Training-free Pre-LLM Visual Token Pruning
Sep 21, 2026Guangchuan Lv, Dianxing Shi, Dingjie FuEfficient VLM Inference Large Vision-Language Models
Northeastern University · Beihang University · Huazhong University of Science and Technology

Sep 28, 2026J/K move · Enter open · S save

ACPruner: Visual Token Pruning as Biased Attention Coverage Maximization in LVLMs

Xu Li, Yuxuan Liang, Yi Zheng, Zhe Liu+5

College of Computer Science and Artificial Intelligence Fudan University, Shanghai, China

Efficient VLM Inference Large Vision-Language Models Visual Token Pruning LVLM Pruning

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) face significant computational inefficiencies caused by the large number of visual tokens. Existing visual token pruning methods mainly focus on either retaining individually important tokens or selecting mutually diverse ones. In this work, we revisit visual token pruning from a coverage perspective and formulate it as a biased attention coverage maximization problem. The key idea is to select a compact token subset whose encoder-side outgoing attention can jointly cover the image while assigning higher coverage priority to more informative regions. From this perspective, we propose ACPruner, a training-free visual token pruning framework for efficient LVLM inference. ACPruner first estimates token importance by combining intra-modal saliency and inter-modal relevance, then derives token-wise coverage from attention patterns within the vision encoder, and finally performs greedy selection to maximize the proposed coverage objective. Extensive experiments across multiple LVLM backbones, including LLaVA-1.5-7B/13B, LLaVA-NeXT-7B/13B, Qwen2.5-VL-7B, and LLaVA-OneVision-7B, show that ACPruner consistently achieves strong performance retention while delivering substantial end-to-end inference speedups.

Abstract

Visual Token Pruning

Explore similar work

ACPruner: Visual Token Pruning as Biased Attention Coverage Maximization in LVLMs

Hierarchical and Query-Aware Visual Token Pruning for High-Resolution LVLMs

VPRune: Efficient Training-free Pre-LLM Visual Token Pruning