SVD-Prune: Spectral Vision Token Pruning under Extreme Compression

Yvon Apedo; Martyna Poreba; Michal Szczepanski; Samia Bouchafa

cs.CVNov 1, 2026

SVD-Prune: Spectral Vision Token Pruning under Extreme Compression

Authors: Yvon Apedo, Martyna Poreba, Michal Szczepanski, Samia Bouchafa

Organizations: Universit´e Paris-Saclay, Univ Evry, IBISC, 91020, Evry-Courcouronnes, France · Universit´e Paris-Saclay, CEA, List, F-91120 Palaiseau, France

Abstract

Vision-Language Models (VLMs) and Vision-Language-Action (VLA) models process hundreds of visual tokens, making visual representations a major computational bottleneck. We argue that visual token pruning should be formulated not as identifying individually important tokens, but as preserving the dominant low-rank spectral structure of the visual representation. Based on this formulation, we introduce SVD-Prune, a training-free framework that leverages Singular Value Decomposition (SVD) and statistical leverage scores for globally informed token selection. SVD-Prune is architecture-agnostic and can be applied at different stages of the multimodal pipeline, including encoder-side, decoder-side, and hybrid configurations Extensive experiments across VLM and VLA benchmarks demonstrate strong robustness under extreme compression. SVD-Prune reduces the visual token. SVD-Prune cuts the visual token budget by up to 99.3%, while, at a 16-token budget, encoder-side pruning lowers inference FLOPs by 82.4% and achieves a 4.18x end-to-end speedup, including pruning overhead.

Figures & tables

Figure 1: Overview of SVD-Prune. Visual representations are decomposed using SVD to identify a dominant spectral subspace, and tokens are ranked using statistical leverage scores to retain a compact representative subset. $N$ denotes the number of tokens, $d$ the feature dimension, and $m$ the truncation rank.

Retain 192 Tokens (33.3%)
Methods	Location	SQA [ 19 ]	VizWiz [ 7 ]	POPE [ 14 ]	GQA [ 9 ]	TextVQA [ 22 ]	MME [ 6 ]
Vanilla	–	70.43	50.00	86.96	61.90	58.20	1862.00
FastV [ 5 ]	Decoder	69.10	50.80	64.80	52.88	52.50	1605.00
PDrop [ 26 ]	Decoder	70.02	51.10	82.30	57.30	56.50	1766.00
SparseVLM [ 31 ]	Decoder	69.10	52.90	83.60	57.60	56.10	1721.00
VisionZip [ 27 ]	Encoder	68.90	54.10	85.30	59.30	57.30	1782.60

Table 1: Response quality of LLaVA-1.5-7B with SVD-Prune across multimodal benchmarks and token budgets. Decoder-side pruning is applied after the 8th layer, while hybrid pruning first reduces visual tokens to 64 after the encoder, followed by a second pruning stage after the 8th decoder layer. Best and second-best results are marked in bold and underlined , respectively. Results marked with ∗ correspond to SVD-Prune applied after the 12th decoder layer.

Figure 2: Qualitative visualization of SVD-Prune on representative examples from MME (top), TextVQA (middle), and POPE (bottom). From left to right: input image with the associated prompt, token leverage-score heatmap, and encoder-side pruning results retaining 32, 16, 8, and 4 visual tokens, respectively. Green markers indicate the spatial locations of the selected tokens.

Method	Location	Latency	FPS	FLOPs
		[ms]		[T]
Vanilla	–	930.00	1.06	3.450
FastV [ 5 ]	Decoder	332.14	3.01	1.232
SparseVLM [ 31 ]	Decoder	372.00	2.69	1.380
VisionZip [ 27 ]	Encoder	184.52	5.42	0.604
SVD-Prune (ours)	Encoder	222.56	4.33	0.606

Table 2: Performance comparison on an RTX 3080 16GB GPU. All pruning methods are evaluated using the same final visual token budget of 16 tokens; the vanilla model uses its original 576 visual tokens.

Latency [ms]
Component	Pruning Location
	Encoder	Decoder	Hybrid
Encoder	22.88	22.88	22.88
SVD-Prune overhead
SVD	23.94	43.50	23.94 + 5.45
Leverage Scoring	0.53	0.53	0.53 + 0.15

Table 3: Latency breakdown and computational overhead of different SVD-Prune deployment modes with a 16-token budget. Decoder-side pruning is applied after LLM layer 8 in both the decoder and hybrid modes. For the hybrid mode, latency and FLOPs are reported as encoder-side + decoder-side SVD-Prune costs.

Method	# Tokens	TextVQA	SQA	MME
Vanilla	1024	63.44	58.37	1702.84
SVD-Prune (ours)	900	62.84	57.37	1716.38
	784	61.80	58.94	1679.76
	484	53.46	59.61	1650.04
	256	37.88	57.14	1624.31

Table 4: Performance of QwenVL with encoder-side SVD-Prune across different visual token budgets.

Tokens	LIBERO-Spatial			LIBERO-Goal			LIBERO-Object			LIBERO-10
Tokens	SR (%)	GFLOPs	Lat. (ms)	SR (%)	GFLOPs	Lat. (ms)	SR (%)	GFLOPs	Lat. (ms)	SR (%)	GFLOPs	Lat. (ms)
256	78.0	4376.05	1099.60	75.2	4213.82	1015.32	72.4	4254.36	1436.51	55.8	4294.92	3213.72
192	79.0	3935.92	1044.56	78.4	3774.09	1014.45	71.6	3814.53	1441.28	54.6	3854.89	2922.52
128	80.8	3075.68	982.76	76.8	3336.51	983.44	73.0	3376.85	1399.32	51.8	3417.31	2631.32
56	79.6	2349.89	900.96	79.6	2846.80	988.74	71.0	2887.03	988.13	47.0	2927.27	2304.10

Table 5: Task success rate and inference efficiency of OpenVLA with SVD-Prune on LIBERO-Spatial, LIBERO-Goal, LIBERO-Object, and LIBERO-10. Each suite comprises 10 tasks with 50 trials per task. Decoder-side pruning is applied after the 16th decoder layer.

Figure 3: Qualitative visualization of SVD-Prune on LIBERO-Spatial (top) and LIBERO-Object (bottom), at selected frame indices. Input frames and corresponding decoder-side pruning results retaining 56 visual tokens are shown, with frame numbers indicated below each example.

$\varepsilon$	192	16	192	16	192	16
	GQA [ 9 ]		POPE [ 14 ]		MME [ 6 ]
0.10	38.30	38.30	50.53	48.93	1123.71	1151.58
0.70	40.61	38.67	52.22	52.80	1150.03	1167.41
0.75	45.95	40.87	86.30	71.91	1579.33	1283.55
0.80	59.88	52.80	87.75	72.68	1788.00	1364.13
0.85	59.90	52.86	87.76	72.77	1789.01	1364.24

Table 6: Effect of the spectral energy retention threshold

\varepsilon

on the performance of LLaVA-1.5-7B across multiple benchmarks and token budgets.

Scoring function	TextVQA	POPE	MME
Energy-weighted	42.07	76.16	1204.48
Whitening-based	42.00	83.16	1536.33
AVI	53.23	83.68	1539.07
Leverage	55.14	83.87	1575.00

Table 7: Ablation of token scoring functions on LLaVA-1.5-7B with encoder-side pruning and a fixed budget of 64 visual tokens.

Visual Token Pruning

Visual token pruning reduces the image and video patches processed by transformers to lower inference cost while preserving task-relevant information.

See all 282 papers

Beyond Attention Scores: SVD-Based Vision Token Pruning for Efficient Vision-Language Models
Apr 13, 2026Yvon Apedo, Martyna Poreba, Michal Szczepanski +1Vision-Language Models Efficient VLM Inference
Universit´e Paris-Saclay, Univ Evry, IBISC, 91020, Evry-Courcouronnes, France · Universit´e Paris-Saclay, CEA, List, F-91120, Palaiseau, France
Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity
Mar 10, 2026Zhengyao Fang, Pengyuan Lyu, Chengquan Zhang +3Visual Token Pruning Visual Token Compression
Harbin Institute of Technology, Shenzhen · Independent Researcher · Peng Cheng Laboratory
VPRune: Efficient Training-free Pre-LLM Visual Token Pruning
Sep 21, 2026Guangchuan Lv, Dianxing Shi, Dingjie FuEfficient VLM Inference Large Vision-Language Models
Northeastern University · Beihang University · Huazhong University of Science and Technology

Apr 13, 2026J/K move · Enter open · S save

Beyond Attention Scores: SVD-Based Vision Token Pruning for Efficient Vision-Language Models

Yvon Apedo, Martyna Poreba, Michal Szczepanski, Samia Bouchafa

Universit´e Paris-Saclay, Univ Evry, IBISC, 91020, Evry-Courcouronnes, France · Universit´e Paris-Saclay, CEA, List, F-91120, Palaiseau, France

Vision-Language Models Efficient VLM Inference Visual Token Pruning

Open paper AskPDF arXiv

Vision-Language Models (VLMs) have revolutionized multi-modal learning by jointly processing visual and textual information. Yet, they face significant challenges due to the high computational and memory demands of processing long sequences of vision tokens. Many existing methods rely on local heuristics, such as attention scores or token norms. However, these criteria suffer from positional bias and information dispersion, limiting their ability to preserve essential content at high pruning ratios and leading to performance degradation on visually detailed images. To address these issues, we propose SVD-Prune, a training-free, plug-and-play token pruning method based on Singular Value Decomposition. It decomposes the vision token feature matrix and selects the top-k tokens using statistical leverage scores, ensuring only tokens contributing most to the dominant global variance are preserved. Experiments show that SVD-Prune consistently outperforms prior pruning methods under extreme vision token budgets, maintaining strong performance even with 32 and 16 vision tokens.

Abstract

Visual Token Pruning

Explore similar work

Beyond Attention Scores: SVD-Based Vision Token Pruning for Efficient Vision-Language Models

Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity

VPRune: Efficient Training-free Pre-LLM Visual Token Pruning