Clustering and Token Denoising for Faster and More Robust VLMs

Baptiste Rossigneux; Inna Kucher; Vincent Lorrain; Emmanuel Casseau

cs.CVAug 19, 2026

Clustering and Token Denoising for Faster and More Robust VLMs

Authors: Baptiste Rossigneux, Inna Kucher, Vincent Lorrain, Emmanuel Casseau

Organizations: Universit´e Paris-Saclay, CEA, LIST, France · Univ. Rennes, CNRS, INRIA, IRISA, France · Vsora, France

Abstract

Recent Visual-Language Models (VLMs) have enhanced the capabilities of pre-trained LLMs by adding vision tokens alongside text, with approaches like LLaVA showing impressive results. However, the computational burden of processing up to 576 or 729 visual tokens makes edge deployment challenging. While various token pruning techniques require retraining, some are training-free and thus can easily adapt to architecture changes. We introduce ClustRS, a two-part, training-free algorithm for robust token pruning. Its first component is an attention-weighted, clustering algorithm that selects representative tokens from each semantic cluster. The second component, Residual Shrinkage, is a one-pass denoising step on the selected tokens. These training-free lightweight steps make LLaVA ready for real-world data, improving robustness to a wide range of image-noise types and intensities. Experimental results on the ScienceQA-IMG and MM-VET benchmarks show our method outperforms attention- and diversity-based methods by up to 20% under extreme noise and token conditions (reducing tokens by 97%, down to 16 tokens) on LLaVA 1.5 7b and achieves exceptional results on LLaVA-OneVision, where we match baseline performance with fewer than one-third of their tokens under mild noise conditions. Our study demonstrates a simple yet powerful alternative to both score-only and diversity-only pruning rules, paving the way for compute-efficient and noise-resilient VLM deployment.

Figures & tables

Panel of Figure 1 — Figure 1: Two views of the token distribution before and after corruption on the inputs of a LLaVA-1.5-7b. (a,c) PCA and (b,d) UMAP plot the same 576 visual tokens in 2-D; gray dots denote all tokens, coloured dots denote those retained by competing pruning rules. Top row: clean image. Bottom row: the same image with heavy Gaussian noise ( $\sigma\!=\!3.0$ ). Note how noise collapses several semantic groupings into one, illustrating why a purely diversity-based selector can over-allocate tokens under severe corruption.

Noise level $\sigma$	Avg. rank change (top–15)	Top–15 overlap
$0.0$	$0$	$100\,\%$
$0.1$	$288$	$13\,\%$
$1.0$	$321$	$0\,\%$
$3.0$	$308$	$0\,\%$

Table 1: Instability of FasterVLM’s token ranking under increasing Gaussian noise. Avg. rank change (top-15) : The average absolute change in rank of importance for the original top-15 tokens after noise is applied. For example, a token ranked 1st by FasterVLM on a clean image could drop to rank 300 even with a small noise. Top-15 overlap : The percentage of tokens that remain in the top-15 set after noise is applied.

Panel of Figure 2 — Figure 2: High-level comparison of the token–selection pipelines. Left: FasterVLM keeps the $k$ tokens with the highest [CLS] self-attention scores immediately after the vision encoder. Right: our C + H module first applies attention-weighted Huber $k$ -means to cluster the projected tokens and then retains a single, most important representative from each cluster, with the help of the [CLS] self-attention scores from the pre-projection representations. This reduced set is then enhanced with RRS to denoise each token with its cluster statistics, before being forwarded to the language model.

Panel of Figure 3 — Figure 3: Comparison of full‐image vs. cluster‐focused token movement after RRS. ( a ) Global PCA on all $N$ tokens highlights which selected tokens have moved (red $\rightarrow$ blue) in context. ( b ) Local PCA fit on one cluster’s tokens shows fine‐grained per‐cluster shifts.

Panel of Figure 4 — Figure 4: MM-VET accuracy of LLaVA-1.5-7B under additive Gaussian noise ( $\sigma\!\in\![0.1,3.0]$ ) for three token budgets ( $k{=}16,50,144$ ).

Panel of Figure 5 — Figure 5: MM-VET accuracy of LLaVA-1.5-7B under salt–&–pepper noise ( $\rho\!\in\![0.01,0.50]$ ) for the three token budgets evaluated in the Chapter. Gaussian-noise counterparts are shown in Figure 4 .

Noise $\sigma$	Token Budget ( $k$ )	FasterVLM	DivPrune	VisionZip	ClustRS
Clean	16	0.211	0.232	0.239	0.271
	50	0.290	0.284	0.285	0.304
	144	0.322	0.318	0.322	0.320
0.1	16	0.218	0.211	0.210	0.270
	50	0.310	0.280	0.300	0.316
	144	0.329	0.290	0.329	0.333

Table 2: MM-VET scores for LLaVA-1.5-7B under Gaussian noise with standard deviation

\sigma

. We compare the training-free token pruning methods FasterVLM, DivPrune, VisionZip, and ClustRS at different retained-token budgets

k

. The best result in each row is shown in bold.

$k=16$
Setting	FasterVLM	DivPrune	VisionZip	ClustRS
Brightness and contrast	0.203	0.218	0.225	0.250
Cutout	0.183	0.213	0.210	0.214
$k=50$
Brightness and contrast	0.280	0.265	0.268	0.292
Cutout	0.213	0.223	0.228	0.233

Table 4: Mean MM-VET scores under brightness and contrast transformations and cutout corruption. Brightness-and-contrast results are averaged over six configurations: brightness factors

b\in\{0.5,1.5,2.0\}

and contrast factors

c\in\{0.5,1.5,2.0\}

. Cutout results are averaged over three severity levels,

p\in\{0.1,0.25,0.4\}

, denoting the fraction of the image area masked.

Noise ( $\rho$ )	Token Budget ( $k$ )	FasterVLM	DivPrune	VisionZip	ClustRS
0.0 (Clean)	16	67.8	69.2	68.9	69.0
	50	69.2	68.5	68.9	69.2
	144	67.9	68.3	68.4	68.8
0.1	16	67.5	68.6	68.3	68.6
	50	68.5	67.6	67.8	68.7
	144	68.4	67.6	68.0	68.0

Table 5: ScienceQA scores under Gaussian noise for three token budgets

k\in\{16,50,144\}

Method	k (Tokens Kept)	Total Visual Overhead (ms)
FasterVLM	144 and 16	33-34
DivPrune	144 and 16	39-40
VisionZip	144 and 16	34-38
ClustRS (Ours)	16	35
ClustRS (Ours)	144	52

Table 7: Comparison of total visual overhead for different token pruning methods.

Corruption Type	Level	FasterVLM	DivPrune	ClustRS (Ours)
Motion Blur	High (B=31)	0.184	0.219	0.222
Motion Blur	Medium (B=15)	0.218	0.235	0.254
JPEG Artifacts	Medium (q=60)	0.250	0.261	0.284
JPEG Artifacts	Low (q=30)	0.240	0.230	0.282

Table 8: MM-VET scores of LLaVA-1.5-7b with 16 visual tokens under different image corruptions. Here B is the size of the blurring kernel, and q is the quality factor of the PIL library (going from 1 -worst- to 95 -best-)

Tokens	Noise	Clustering	C+H	ClustRS	Gain
( $k$ )	( $\sigma$ )		only		(RRS)
16	0.5	0.233	0.254	0.272	+0.018
	1.0	0.195	0.223	0.250	+0.027
	2.0	0.185	0.193	0.222	+0.029
	3.0	0.151	0.156	0.193	+0.037
50	0.5	0.284	0.312	0.290	-0.022

Table 9: Ablation study on MM-VET under Gaussian noise, showing the distinct contributions of our proposed components. We compare standard k-means++ Clustering ( Clustering ), robust clustering with Huber distance ( C+H ), and our full ClustRS method which adds Residual Shrinkage (RRS). The results demonstrate that while clustering provides a baseline improvement, the Huber distance is critical for robustness, and RRS provides further gains, especially in low-token regimes. All scores are accuracy on the LLaVA-1.5-7B model.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Noise ( $\sigma$ )	Token Budget ( $k$ )	FasterVLM	DivPrune	ClustRS (salience)
0.0 (Clean)	16	0.138	0.164	0.272
	50	0.145	0.257	0.347
	180	0.293	0.334	0.418
0.1	16	0.123	0.166	0.284
	50	0.130	0.215	0.348
	180	0.296	0.289	0.363

Appendix

Table 10: MM-VET scores under Gaussian noise (

\sigma

) for LLaVA OneVision 7b across different token budgets (

k

). The

\sigma=0.0

row represents performance on clean images. Scores compare FasterVLM, DivPrune, and our combined C+H (salience) + RRS method.

Panel of Figure 6 — Figure 6: MM-VET accuracy of LLaVA 7b Onevision under additive Gaussian noise ( $\sigma\!\in\![0.1,3.0]$ ) for token budgets $k\in\{16,50,180\}$ .

Condition	Token Budget ( $k$ )	$\tau=0.25$	$\tau=0.5$	$\tau=1.0$	$\tau=2.0$	$\tau=4.0$
$k=16$	Clean	0.250	0.260	0.273	0.261	0.259
	Gaussian ( $\sigma=0.5$ )	0.220	0.240	0.240	0.230	0.220
	Gaussian ( $\sigma=2.0$ )	0.167	0.173	0.161	0.165	0.157
	Salt-and-pepper ( $\rho=0.10$ )	0.230	0.252	0.256	0.248	0.238
$k=50$	Clean	0.283	0.294	0.302	0.307	0.298
	Gaussian ( $\sigma=0.5$ )	0.252	0.260	0.270	0.266	0.254

Appendix

Table 12: Sensitivity analysis of the Huber threshold

\tau

on MM-VET with LLaVA-1.5-7B. The default value is

\tau=1

. We see a broad performance plateau for

\tau\in[0.5,2]

, with stronger degradation only for very small or large thresholds.

Condition	Token Budget ( $k$ )	No RRS	$[0,1]$	$[0.05,0.95]$	$[0.10,0.95]$	$[0.20,0.90]$	$[0.30,0.80]$
$k=16$	Clean	0.268	0.252	0.275	0.272	0.266	0.260
	Gaussian ( $\sigma=0.5$ )	0.238	0.232	0.249	0.246	0.246	0.230
	Gaussian ( $\sigma=2.0$ )	0.174	0.151	0.171	0.160	0.171	0.174
	S&P ( $\rho=0.10$ )	0.240	0.244	0.254	0.251	0.250	0.240
$k=50$	Clean	0.306	0.281	0.308	0.306	0.307	0.298
	Gaussian ( $\sigma=0.5$ )	0.265	0.259	0.272	0.274	0.272	0.263

Appendix

Table 13: Sensitivity analysis of the clipping interval applied to the RRS shrinkage coefficient

\alpha_{i}

on MM-VET with LLaVA-1.5-7B. “No RRS” uses C+H token selection without residual shrinkage. The default clipping interval is

[0.10,0.95]

. Clipping is essential for preventing unstable shrinkage, whereas performance is relatively insensitive to the precise clipped interval. RRS improves performance under mild and sparse corruption but can be detrimental under severe dense Gaussian noise, where retaining the original selected tokens is preferable.

Visual Token Pruning

Visual token pruning reduces the image and video patches processed by transformers to lower inference cost while preserving task-relevant information.

See all 276 papers

OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning
May 28, 2026Geng Li, Guohao Chen, Ting Chen +6Efficient VLM Inference Large Vision-Language Models
1Nanyang Technological University (NTU)
VPRune: Efficient Training-free Pre-LLM Visual Token Pruning
Sep 21, 2026Guangchuan Lv, Dianxing Shi, Dingjie FuEfficient VLM Inference Large Vision-Language Models
Northeastern University · Beihang University · Huazhong University of Science and Technology
ResPrune: Text-Conditioned Subspace Reconstruction for Visual Token Pruning in Large Vision-Language Models
Mar 22, 2026Xu Li, Yi Zheng, Yuxuan Liang +5Visual Token Pruning Efficient VLM Inference
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai 200433, China

May 28, 2026J/K move · Enter open · S save

OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning

Geng Li, Guohao Chen, Ting Chen, Shilin Shan+5

1Nanyang Technological University (NTU)

Efficient VLM Inference Large Vision-Language Models Visual Token Pruning Visual Token Compression LVLM Pruning

Open paper AskPDF arXiv

Vision-language models (VLMs) rely on long visual token sequences for visual understanding, making the prefill stage expensive in both computation and memory. Most existing pruning methods follow an absolute-ranking paradigm, assigning importance scores to visual tokens and retaining a fixed top-K subset. In this work, we argue that this paradigm is fundamentally brittle: attention sinks distort token importance rankings, while image redundancy and query-dependent visual evidence make fixed token budgets unreliable across inputs. We propose OccamToken, a training-free framework that replaces absolute token ranking with register-anchored relative evidence testing. Instead of asking which tokens are globally important, OccamToken evaluates whether a visual token provides information beyond a register-based reference. Our key insight is that register tokens naturally absorb low-information attention patterns, making them a stable reference for identifying genuinely informative visual evidence. Based on this principle, OccamToken performs both image-adaptive redundancy pruning and query-adaptive relevance pruning through dynamic thresholds derived from register attention. Across LLaVA-NeXT, LLaVA-v1.5, and Qwen3-VL, OccamToken consistently improves the accuracy-efficiency trade-off without additional training. Notably, on LLaVA-NeXT, it reduces 2,880 visual tokens to approximately 40 while preserving over 93% of the original accuracy, enabling stable visual token compression even in the extreme 1.4% retention regime.

Abstract

Visual Token Pruning

Explore similar work

OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning

VPRune: Efficient Training-free Pre-LLM Visual Token Pruning

ResPrune: Text-Conditioned Subspace Reconstruction for Visual Token Pruning in Large Vision-Language Models