Image Clustering with Vision-Language Model Embeddings

João P. Matos-Carvalho; Nuno Fachada

cs.CVOct 23, 2026

Image Clustering with Vision-Language Model Embeddings

Authors: João P. Matos-Carvalho, Nuno Fachada

Organizations: LASIGE, Faculdade de Ciências, Universidade de Lisboa, Lisboa, 1749-016, Portugal · COPELABS, Lusófona University, Lisboa, 1700-921, Portugal · INESC INOV, Lisboa, 1000-029, Portugal

Abstract

Pretrained vision-language models provide image representations whose suitability for clustering may vary with feature extraction, image domain, and partitioning algorithm. We compare nine representation pipelines across seven image benchmarks and four clustering configurations, yielding 252 baseline combinations. Eight pipelines extract visual features, including two vision-only references; the ninth generates descriptions and embeds the resulting text. With agglomerative clustering (Ward linkage) fixed, Qwen3-VL achieves the highest adjusted Rand index (ARI) on five datasets, including 0.925 on MNIST, while BLIP-2 leads on CIFAR-10 and CIFAR-100 with ARIs of 0.925 and 0.591, respectively. A separate sensitivity analysis compares raw and L2-normalized embeddings at their original dimensionality across all 63 dataset-representation pairs. Normalization preserves the highest-ARI representation in every dataset, increases ARI in 35 pairs, decreases it in 26, and leaves two numerically unchanged. On the SVHN dataset, the highest ARI remains low, rising from 0.097 to 0.098 after normalization. The evaluated caption-mediated pipeline obtains relatively high ARI on isolated digits but does not consistently exceed direct visual features. These findings support selecting and evaluating complete representation-clustering pipelines for the intended domain, with explicit attention to extraction and normalization choices.

Figures & tables

Study/ Model	Ref.	Year	Representation or grouping strategy	Additional training or processing	Relation to this study
VLM2Vec	[ 9 ]	2025	Instruction-conditioned multimodal embeddings	Contrastive embedding training	Learns task-oriented embedding spaces
mmE5	[ 10 ]	2025	Multimodal and multilingual embeddings	Training on synthetic and labeled data	Studies embedding training-data quality
SCAN	[ 1 ]	2020	Self-supervised features and learned cluster assignments	Representation learning, clustering, and self-labeling	Reference for learned image clustering
SCP	[ 17 ] ⋆	2025	Frozen visual features and positive pairs	Small clustering-head training	Fixed encoder with learned assignments
FUNGI	[ 18 ]	2024	Pretrained embeddings augmented with projected gradients	Gradient-based feature construction; no weight updates	Augments features; our L2 analysis preserves dimension
TGIC	[ 8 ]	2024	Generated captions/VQA outputs encoded as text	Text gen. and encoding with pretrained models	Direct antecedent of our caption route

Table 1: Selected related work compared in terms of representation or grouping strategy, additional training or processing, and relationship to the present study.

Panel of Figure 1 — Figure 1: Experimental workflow across seven datasets, nine representation pipelines, and four clustering configurations ( $7\times 9\times 4=252$ baseline combinations). The visual route includes six vision–language models and two vision-only models (DINOv3 and DiT), while the caption-mediated route generates and encodes image descriptions. Each representation is clustered independently, with the number of clusters $K$ determined by the reference class count. Reference labels are used only for evaluation and to determine $K$ , not for feature extraction or fitting cluster assignments. The baseline uses raw, single-precision embeddings at their original dimensionality; a separate sensitivity analysis compares raw and L2-normalized embeddings with AHC-Ward clustering fixed.

Dataset	Ref.	Images	Classes
MNIST	[ 19 ]	10,000	10
USPS	[ 20 ]	2,007	10
Fashion-MNIST	[ 21 ]	10,000	10
EMNIST (balanced)	[ 22 ]	18,800	47
SVHN	[ 23 ]	26,032	10
CIFAR-10	[ 24 ]	10,000	10

Table 2: Evaluated test partitions of the seven image benchmarks, showing the number of images and reference classes.

Representation	Checkpoint identifier
CLIP	openai/clip-vit-base-patch32
SigLIP2	google/siglip2-base-patch16-224
BLIP	Salesforce/blip-image-captioning-base
BLIP-2	Salesforce/blip2-opt-2.7b
LLaVA	llava-hf/llava-1.5-7b-hf
Qwen3-VL	Qwen/Qwen3-VL-8B-Instruct

Table 3: Pretrained checkpoint identifiers. The extracted components and embedding dimensions are specified in Table 4 .

Representation	Component and aggregation	Dim.
CLIP	Visual patch-state mean, followed by vision post-layer normalization and visual projection	512
SigLIP2	Mean of all visual patch states	768
BLIP	Visual-state mean after the first token	768
BLIP-2	Visual-backbone state mean after the first token, rather than language-model states	1,408
LLaVA	Mean after the first entry of the returned image-feature sequence	4,096
Qwen3-VL	Mean of all tokens in each returned image-feature sequence	4,096

Table 4: Representation components and aggregation used in the main comparison. Dimensions refer to the vectors supplied to clustering. The baseline uses no additional vector normalization; the sensitivity analysis applies per-image L2 normalization. The routes describe the evaluated implementations, not an optimization over alternative outputs of each model.

Representation	MNIST	Fashion	EMNIST	USPS	SVHN	CIFAR-10	CIFAR-100
CLIP	0.511	0.534	0.283	0.426	0.013	0.508	0.186
SigLIP2	0.479	0.529	0.332	0.687	0.012	0.575	0.311
BLIP	0.614	0.515	0.324	0.579	0.010	0.453	0.216
BLIP-2	0.588	0.526	0.327	0.685	0.033	0.925	0.591
LLaVA	0.176	0.237	0.106	0.266	0.005	0.340	0.168
Qwen3-VL	0.925	0.573	0.434	0.717	0.097	0.777	0.475

Table 5: Adjusted Rand index (ARI) with a fixed AHC-Ward clustering algorithm. Bold values indicate the best representation for each dataset.

Representation	Mean rank	Mean ARI	Median ARI	Wins
Qwen3-VL	1.286	0.571	0.573	5
BLIP-2	2.714	0.525	0.588	2
SigLIP2	3.714	0.418	0.479	0
BLIP	5.000	0.387	0.453	0
CLIP	5.000	0.352	0.426	0
DINOv3	5.286	0.403	0.468	0

Table 6: Descriptive summary of representation performance with AHC-Ward clustering held fixed. Ranks are computed within each dataset (1 indicates the best performance) and averaged equally across the seven datasets. Wins indicate the number of datasets on which each representation achieves the highest ARI. Mean and median ARI are reported as descriptive summaries of the evaluated benchmarks and should be interpreted in light of differences in dataset characteristics and clustering difficulty.

Representation	MNIST	Fashion	EMNIST	USPS	SVHN	CIFAR-10	CIFAR-100
CLIP	0.474	0.517	0.284	0.472	0.011	0.411	0.189
SigLIP2	0.534	0.508	0.339	0.675	0.023	0.611	0.310
BLIP	0.670	0.488	0.301	0.555	0.014	0.486	0.213
BLIP-2	0.507	0.564	0.333	0.716	0.035	0.911	0.584
LLaVA	0.312	0.451	0.159	0.372	0.010	0.391	0.200
Qwen3-VL	0.933	0.583	0.440	0.841	0.098	0.909	0.516

Table 7: ARI with AHC-Ward after per-image L2 normalization, retaining the original embedding dimensions. Bold marks the highest value in each dataset before rounding.

Representation	Raw ARI	L2 ARI	$\Delta$	Higher	Lower
CLIP	0.352	0.337	-0.0147	3	4
SigLIP2	0.418	0.428	+0.0104	4	3
BLIP	0.387	0.390	+0.0022	3	4
BLIP-2	0.525	0.521	-0.0035	4	3
LLaVA	0.185	0.271	+0.0850	7	0
Qwen3-VL	0.571	0.617	+0.0460	7	0

Table 8: Mean ARI with AHC-Ward, equally weighted across seven datasets.

\Delta

is L2 minus raw. Higher/lower/equal counts compare L2 with raw within the same representation and dataset, using a numerical tolerance of

10^{-12}

Representation	K-means++	K-means (random)	AHC-Ward	Fuzzy c-means	Mean
CLIP	0.314	0.308	0.352	0.062	0.259
SigLIP2	0.386	0.400	0.418	0.084	0.322
BLIP	0.378	0.385	0.387	0.080	0.308
BLIP-2	0.474	0.506	0.525	0.081	0.397
LLaVA	0.170	0.170	0.185	0.053	0.145
Qwen3-VL	0.546	0.588	0.571	0.085	0.447

Table 9: ARI averaged equally across seven datasets, separately for each clustering configuration. The final column also averages equally across configurations; bold indicates the highest mean ARI for each representation among the four clustering configurations, selected before rounding. These averages summarize performance across datasets with differing characteristics and clustering difficulty.

Dataset	Representation	Clustering config.	ARI	V-measure	F1 (greedy)
MNIST	Qwen3-VL	K-means++	0.941	0.933	0.973
Fashion-MNIST	Qwen3-VL	K-means (random)	0.630	0.728	0.751
EMNIST	Qwen3-VL	AHC-Ward	0.434	0.687	0.554
USPS	Qwen3-VL	K-means (random)	0.759	0.834	0.813
SVHN	Qwen3-VL	AHC-Ward	0.097	0.158	0.217
CIFAR-10	BLIP-2	K-means (random)	0.970	0.964	0.986

Table 10: Observed maximum-ARI configuration per dataset, selected retrospectively among 36 configurations. V-measure and weighted F1 refer to that same configuration.

Representation	Dimension	Recorded ms/image
CLIP	512	1.49
SigLIP2	768	4.18
BLIP	768	17.91
BLIP-2	1,408	66.07
LLaVA	4,096	10.63
Qwen3-VL	4,096	10.40

Table 11: Pooled embedding extraction time (total recorded seconds divided by total images), with embedding dimensions repeated from Table 4 for ease of analysis. Local timings exclude model loading, while API timings include caption generation and network latency. Local extraction was performed on an NVIDIA DGX Spark; timings reflect the respective local and remote execution conditions.

Panel of Figure 2 — Figure 2: Illustrative PCA–t-SNE projections selected retrospectively from the quantitative results. Top: Qwen3-VL on MNIST (left) and SVHN (right), contrasting domains for a fixed representation. Bottom: BLIP-2 (left) and caption-mediated embeddings (right) on CIFAR-10, contrasting representation routes for a fixed dataset. Colors denote reference classes, not predicted clusters. Panels are independently projected and rescaled; visible separation does not validate clustering in the original embedding spaces.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Representation	MNIST	Fashion	EMNIST	USPS	SVHN	CIFAR-10	CIFAR-100
CLIP	0.630	0.675	0.557	0.578	0.023	0.641	0.552
SigLIP2	0.672	0.688	0.611	0.752	0.025	0.705	0.657
BLIP	0.757	0.659	0.601	0.713	0.018	0.605	0.548
BLIP-2	0.726	0.720	0.603	0.740	0.084	0.938	0.822
LLaVA	0.348	0.413	0.357	0.398	0.022	0.494	0.530
Qwen3-VL	0.934	0.698	0.687	0.824	0.158	0.860	0.781

Appendix

Table 12: V-measure with a fixed AHC-Ward clustering algorithm. Bold values indicate the best representation for each dataset.

Representation	MNIST	Fashion	EMNIST	USPS	SVHN	CIFAR-10	CIFAR-100
CLIP	0.622	0.656	0.561	0.629	0.021	0.608	0.559
SigLIP2	0.704	0.679	0.604	0.732	0.045	0.726	0.654
BLIP	0.780	0.651	0.590	0.685	0.024	0.612	0.554
BLIP-2	0.685	0.714	0.605	0.770	0.079	0.928	0.823
LLaVA	0.514	0.627	0.431	0.508	0.020	0.564	0.562
Qwen3-VL	0.939	0.718	0.688	0.872	0.183	0.910	0.786

Appendix

Table 13: V-measure with AHC-Ward after per-image L2 normalization, retaining the original embedding dimensions. Bold marks the highest value in each dataset before rounding.

Representation	Algorithm	ARI	V-measure	F1 (greedy)	Silhouette
CLIP	K-means++	0.436	0.520	0.656	0.111
	K-means (random)	0.373	0.498	0.591	0.118
	AHC-Ward	0.511	0.630	0.636	0.088
	Fuzzy c-means	0.062	0.126	0.049	0.157
SigLIP2	K-means++	0.488	0.616	0.697	0.167
	K-means (random)	0.488	0.623	0.643	0.165

Appendix

Table 14: Detailed results for MNIST for all clustering configurations. Bold marks column maxima before rounding for ARI, V-measure, and F1 (greedy centroid mapping). Silhouette is not compared across feature spaces.

Representation	Algorithm	ARI	V-measure	F1 (greedy)	Silhouette
CLIP	K-means++	0.481	0.638	0.527	0.182
	K-means (random)	0.515	0.636	0.644	0.159
	AHC-Ward	0.534	0.675	0.651	0.160
	Fuzzy c-means	0.145	0.319	0.051	0.180
SigLIP2	K-means++	0.534	0.668	0.699	0.167
	K-means (random)	0.515	0.660	0.687	0.166

Appendix

Table 15: Detailed results for Fashion-MNIST for all clustering configurations. Bold marks column maxima before rounding for ARI, V-measure, and F1 (greedy centroid mapping). Silhouette is not compared across feature spaces.

Representation	Algorithm	ARI	V-measure	F1 (greedy)	Silhouette
CLIP	K-means++	0.228	0.497	0.365	0.068
	K-means (random)	0.237	0.498	0.409	0.073
	AHC-Ward	0.283	0.557	0.394	0.041
	Fuzzy c-means	0.011	0.069	0.002	-0.076
SigLIP2	K-means++	0.280	0.545	0.448	0.087
	K-means (random)	0.282	0.549	0.451	0.091

Appendix

Table 16: Detailed results for EMNIST for all clustering configurations. Bold marks column maxima before rounding for ARI, V-measure, and F1 (greedy centroid mapping). Silhouette is not compared across feature spaces.

Representation	Algorithm	ARI	V-measure	F1 (greedy)	Silhouette
CLIP	K-means++	0.390	0.527	0.576	0.169
	K-means (random)	0.354	0.530	0.482	0.172
	AHC-Ward	0.426	0.578	0.542	0.147
	Fuzzy c-means	0.104	0.159	0.037	0.162
SigLIP2	K-means++	0.472	0.646	0.584	0.141
	K-means (random)	0.547	0.667	0.695	0.196

Appendix

Table 17: Detailed results for USPS for all clustering configurations. Bold marks column maxima before rounding for ARI, V-measure, and F1 (greedy centroid mapping). Silhouette is not compared across feature spaces.

Representation	Algorithm	ARI	V-measure	F1 (greedy)	Silhouette
CLIP	K-means++	0.016	0.023	0.138	0.042
	K-means (random)	0.017	0.024	0.143	0.043
	AHC-Ward	0.013	0.023	0.143	0.007
	Fuzzy c-means	0.009	0.009	0.064	0.033
SigLIP2	K-means++	0.014	0.024	0.138	0.059
	K-means (random)	0.014	0.025	0.138	0.059

Appendix

Table 18: Detailed results for SVHN for all clustering configurations. Bold marks column maxima before rounding for ARI, V-measure, and F1 (greedy centroid mapping). Silhouette is not compared across feature spaces.

Representation	Algorithm	ARI	V-measure	F1 (greedy)	Silhouette
CLIP	K-means++	0.457	0.605	0.705	0.098
	K-means (random)	0.465	0.605	0.704	0.101
	AHC-Ward	0.508	0.641	0.689	0.075
	Fuzzy c-means	0.095	0.190	0.045	0.106
SigLIP2	K-means++	0.617	0.731	0.805	0.142
	K-means (random)	0.632	0.743	0.786	0.144

Appendix

Table 19: Detailed results for CIFAR-10 for all clustering configurations. Bold marks column maxima before rounding for ARI, V-measure, and F1 (greedy centroid mapping). Silhouette is not compared across feature spaces.

Representation	Algorithm	ARI	V-measure	F1 (greedy)	Silhouette
CLIP	K-means++	0.186	0.536	0.347	0.050
	K-means (random)	0.198	0.538	0.361	0.050
	AHC-Ward	0.186	0.552	0.351	0.032
	Fuzzy c-means	0.006	0.072	0.001	-0.029
SigLIP2	K-means++	0.297	0.634	0.461	0.075
	K-means (random)	0.318	0.641	0.486	0.079

Appendix

Table 20: Detailed results for CIFAR-100 for all clustering configurations. Bold marks column maxima before rounding for ARI, V-measure, and F1 (greedy centroid mapping). Silhouette is not compared across feature spaces.

Image Clustering

Image clustering groups images by visual or semantic similarity without requiring labels for each image.

See all 24 papers

VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training
Sep 29, 2026Yuanwei Hu, Bo Peng, Yuheng Jia +3Clustering Deep Clustering
University College London · The University of Queensland · Southeast University +2
Language-Assisted Image Clustering Guided by Discriminative Relational Signals and Adaptive Semantic Centers
Mar 25, 2026Jun Ma, Xu Zhang, Zhengxing Jiao +4Image Clustering Clustering
School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Computing Information Sciences, Saint Francis University, Hong Kong, China · Department of Computer Science, City University of Hong Kong, Hong Kong, China
TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment
Jun 5, 2026Sweta Mahajan, Sukrut Rao, Jiahao Xie +2Vision-Language Retrieval Vision-Language Alignment
Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany · Department of Language Science and Technology, Saarland University, Saarbrücken, Germany

Sep 29, 2026J/K move · Enter open · S save

VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training

Yuanwei Hu, Bo Peng, Yuheng Jia, Xinting Hu+2

University College London · The University of Queensland · Southeast University+2

Clustering Deep Clustering Image Clustering Vision-Language Pretraining

Open paper AskPDF arXiv

Vision-language pre-training has reshaped image clustering, giving rise to language-assisted image clustering (LaIC), which leverages textual semantics to complement visual representations. Despite the rapid proliferation of LaIC methods, it remains unclear how much LaIC has actually advanced image clustering, as existing studies generally suffer from major limitations, including inconsistent experimental settings, inadequate dataset selection, and limited evaluation dimensions. To address this gap, we introduce VLM4Cluster, a comprehensive benchmark for image clustering in the era of pre-trained vision-language models (VLMs). VLM4Cluster implements 17 representative methods spanning classical, deep, and language-assisted image clustering, and evaluates them on 20 datasets covering classical, challenging, fine-grained, large-scale, and out-of-distribution settings. Beyond effectiveness, VLM4Cluster systematically investigates image clustering along three complementary dimensions: robustness to adversarial perturbations, generalization under distribution shifts, and computational efficiency. Our study shows that LaIC substantially advances the clustering performance frontier on many semantically demanding benchmarks, generally exhibits stronger generalization under distribution shifts, and achieves a more favorable effectiveness-efficiency trade-off. However, its gains become less consistent on large-scale and fine-grained datasets, while language assistance does not systematically reduce sensitivity to adversarial perturbations. VLM4Cluster is released at https://github.com/YuanweiHuu/VLM4Cluster.

Abstract

Image Clustering

Explore similar work

VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training

Language-Assisted Image Clustering Guided by Discriminative Relational Signals and Adaptive Semantic Centers

TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment