Language-Assisted Image Clustering Guided by Discriminative Relational Signals and Adaptive Semantic Centers

Jun Ma; Xu Zhang; Zhengxing Jiao; Yaxin Hou; Hui Liu; Junhui Hou; Yuheng Jia

cs.LGMar 25, 2026

Language-Assisted Image Clustering Guided by Discriminative Relational Signals and Adaptive Semantic Centers

Authors: Jun Ma, Xu Zhang, Zhengxing Jiao, Yaxin Hou, Hui Liu, Junhui Hou, Yuheng Jia

Organizations: School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Computing Information Sciences, Saint Francis University, Hong Kong, China · Department of Computer Science, City University of Hong Kong, Hong Kong, China

Abstract

Language-Assisted Image Clustering (LAIC) augments the input images with additional texts with the help of vision-language models (VLMs) to promote clustering performance. Despite recent progress, existing LAIC methods often overlook two issues: (i) textual features constructed for each image are highly similar, leading to weak inter-class discriminability; (ii) the clustering step is restricted to pre-built image-text alignments, limiting the potential for better utilization of the text modality. To address these issues, we propose a new LAIC framework with two complementary components. First, we exploit cross-modal relations to produce more discriminative self-supervision signals for clustering, as it compatible with most VLMs training mechanisms. Second, we learn category-wise continuous semantic centers via prompt learning to produce the final clustering assignments. Extensive experiments on eight benchmark datasets demonstrate that our method achieves an average improvement of 2.6% over state-of-the-art methods, and the learned semantic centers exhibit strong interpretability. Code is available in the supplementary material.

Figures & tables

Figure 1: (a) Analysis of the phenomena in the first step of existing LAICs and our method on the ImageNet-10 dataset. (i) Jointly PCA results shows text features of noun from candidate set exhibit a more compact distribution, which are closed to each other compared to image features. (ii) Due to the phenomenon in (i), the similarity of text features across different samples are overall highly similar, leading to weak inter-class discriminability. (iii) Rows of our learned image-text representation matrix performs as new representation, showing better inter-class discriminability with in-class consistency retained. (b) Classification scores of three images (from DTD dataset) with respect to different semantic centers obtained by different methods. Zero-shot CLIP relies on semantic centers using ground-truth class names, which may not always capture accurate semantics (e.g, an image of motorboat is incorrectly assigned to “Surfing”. Our learned semantic center can be even better than class names, showing stronger discriminability.

Figure 2: Accuracy of each sample and its

\hat{k}

-nn (

\hat{k}=10

) belonging to the same ground-truth class on different space. We compare the results by computing similarity on images features from CLIP and rows of

\mathbf{C}

, showing our image-text representation matrix has better neighbor consistency.

Datasets		ImageNet-Dogs			ImageNet-10			STL-10			CIFAR-10			Avg.
Methods	Backbone	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	Avg.
DEC ( Xie et al., 2016 )	ResNet	12.2	19.5	7.9	28.2	38.1	20.3	13.6	18.5	5.0	25.7	30.1	16.1	19.6
IIC ( Ji et al., 2019 )	ResNet	-	-	-	-	-	-	49.6	59.6	39.7	51.3	61.7	41.1	-
DCCM ( Wu et al., 2019 )	ResNet	32.1	38.3	18.2	60.8	71.0	55.5	37.6	48.2	26.2	49.6	62.3	40.8	45.1
PICA ( Huang et al., 2020 )	ResNet	33.6	32.4	17.9	78.2	85.0	73.3	59.2	69.3	50.4	56.1	64.5	46.7	55.6
BYOL ( Grill et al., 2020 )	ResNet	69.7	72.9	60.9	88.4	94.7	88.9	75.4	86.1	71.5	78.0	87.5	75.2	79.1

Table 1: Clustering performance (in percent %) across four widely used datasets. ZS-CLIP means zero-shot CLIP.

Datasets		Flowers-102			CIFAR-100			DTD			UCF-101			Avg.
Methods	Backbone	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	Avg.
SCAN ( Van Gansbeke et al., 2020 )	ResNet	77.9	56.5	50.9	55.4	38.7	25.1	59.4	46.4	31.7	79.7	61.1	53.1	53.0
LFSS ( Li et al., 2025b )	ResNet	79.2	55.4	51.8	66.0	52.4	37.6	58.7	45.5	31.9	58.9	34.5	21.6	49.5
SIC ( Cai et al., 2023 )	ViT-B/32	86.9	70.2	64.2	63.5	48.3	34.7	59.6	45.9	30.5	81.0	61.9	52.4	58.3
TAC ( Li et al., 2024 )	ViT-B/32	87.7	71.8	64.3	68.1	55.4	42.4	62.1	50.1	34.4	82.8	68.9	60.6	62.4
PRO-DSC ( Meng et al., 2025 )	ViT-B/32	83.2	69.8	59.6	68.2	55.7	40.2	57.6	48.1	31.3	81.0	61.9	52.8	59.1

Table 2: Clustering performance (in percent %) across four datasets with larger cluster numbers.

Panel of Figure 3 — Figure 3: Visualization of image-text representation matrix $\mathbf{C}$ .

ImageNet-10		STL-10
Most similar noun (ours)	GT class name	Most similar noun (ours)	GT class name
sports_car	Sports Car	draft_horse	Horse
airline	Airliner	tabby_cat	Cat
soccer_ball	Soccer Ball	container_ship	Ship
snow_leopard	Snow Leopard	sports_car	Car
king_penguin	King Penguin	riflebird	Bird

Table 3: Comparisons between most similar nouns for semantic centers and corresponding ground-truth (noted as GT) class names.

$\mathcal{L_{\text{sup}}}$	$\mathcal{L_{\text{con}}}$	$\mathcal{L_{\text{ent}}}$	ImageNet-Dogs			DTD
$\mathcal{L_{\text{sup}}}$	$\mathcal{L_{\text{con}}}$	$\mathcal{L_{\text{ent}}}$	NMI	ACC	ARI	NMI	ACC	ARI
✓			82.4	81.7	75.2	60.5	51.8	34.1
	✓		8.1	14.3	3.6	10.2	12.4	8.6
		✓	9.1	14.6	5.7	13.5	16.2	9.7
✓	✓		86.0	87.6	81.7	62.6	51.9	34.3
✓		✓	85.9	87.4	82.1	63.3	53.2	35.5

Table 4: Ablation results with different loss combinations.

Panel of Figure 4 — Figure 4: K-means performances comparisons on our method (Ours (no-train)) and two different baseline methods (CLIP K-means and TAC (no-train))across four datasets.

Figure 5: Accuracy gains achieved through neighborhood consistency filtering on seven datasets. Our strategy effectively mitigates clustering noise and ensures reliable supervision.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	#Samples	#Classes
ImageNet-Dogs ( Chang et al., 2017 )	19,500	15
ImageNet-10 ( Chang et al., 2017 )	13,000	10
STL-10 ( Coates et al., 2011 )	13,000	10
CIFAR-10 ( Krizhevsky, 2009 )	60,000	10
Flowers-102 ( Nilsback and Zisserman, 2008 )	8189	102
CIFAR-100 ( Krizhevsky, 2009 )	60,000	100

Appendix

Table 5: A summary of benchmark datasets used for evaluation.

Datasets	ImageNet-Dogs			ImageNet-10			STL-10			CIFAR-10
Methods	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
SIC ( Cai et al., 2023 )	69.0 $\pm$ 1.6	69.7 $\pm$ 1.1	55.8 $\pm$ 1.5	96.9 $\pm$ 0.2	98.3 $\pm$ 0.1	96.2 $\pm$ 0.2	95.3 $\pm$ 0.1	98.1 $\pm$ 0.1	95.9 $\pm$ 0.1	84.7 $\pm$ 0.1	92.6 $\pm$ 0.1	84.4 $\pm$ 0.1
MCA ( Qiu et al., 2024 )	73.3 $\pm$ 1.5	74.9 $\pm$ 2.5	61.6 $\pm$ 2.5	-	-	-	95.5 $\pm$ 0.1	98.1 $\pm$ 0.1	96.0 $\pm$ 0.1	84.9 $\pm$ 0.2	92.7 $\pm$ 0.2	84.6 $\pm$ 0.2
TAC ( Li et al., 2024 )	80.9 $\pm$ 1.0	83.3 $\pm$ 1.4	72.0 $\pm$ 1.0	98.2 $\pm$ 0.1	99.2 $\pm$ 0.1	98.3 $\pm$ 0.1	95.6 $\pm$ 0.2	98.3 $\pm$ 0.1	96.1 $\pm$ 0.1	83.4 $\pm$ 0.2	92.1 $\pm$ 0.2	83.4 $\pm$ 0.3
LFSS ( Li et al., 2025b )	61.9 $\pm$ 1.9	69.3 $\pm$ 1.7	53.1 $\pm$ 1.6	85.2 $\pm$ 0.6	93.8 $\pm$ 0.5	85.4 $\pm$ 0.8	77.3 $\pm$ 1.1	86.1 $\pm$ 0.9	74.0 $\pm$ 0.8	84.4 $\pm$ 0.6	92.2 $\pm$ 0.4	84.6 $\pm$ 0.5
PRO-DSC ( Meng et al., 2025 )	-	-	-	98.0 $\pm$ 0.3	99.0 $\pm$ 0.2	97.8 $\pm$ 0.2	95.4 $\pm$ 0.2	98.1 $\pm$ 0.4	95.9 $\pm$ 0.2	79.6 $\pm$ 0.4	87.1 $\pm$ 0.3	80.2 $\pm$ 0.4

Appendix

Table 6: Average clustering performance (in percent %) across four widely used benchmarks datasets.

Datasets	Flowers-102			CIFAR-100			DTD			UCF-101
Methods	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
SIC ( Cai et al., 2023 )	86.9 $\pm$ 0.9	70.2 $\pm$ 0.8	64.2 $\pm$ 0.6	63.5 $\pm$ 0.8	48.3 $\pm$ 0.7	34.7 $\pm$ 0.9	59.9 $\pm$ 0.8	45.5 $\pm$ 0.8	30.8 $\pm$ 0.9	81.7 $\pm$ 0.8	62.1 $\pm$ 1.1	52.5 $\pm$ 0.9
TAC ( Li et al., 2024 )	87.7 $\pm$ 0.8	70.8 $\pm$ 0.9	63.9 $\pm$ 0.5	68.1 $\pm$ 0.2	55.4 $\pm$ 0.7	42.4 $\pm$ 0.6	61.8 $\pm$ 0.5	50.9 $\pm$ 0.6	34.8 $\pm$ 0.7	82.5 $\pm$ 0.4	69.1 $\pm$ 0.8	60.2 $\pm$ 0.7
LFSS ( Li et al., 2025b )	79.2 $\pm$ 1.1	55.4 $\pm$ 1.2	51.8 $\pm$ 0.9	66.0 $\pm$ 0.6	52.4 $\pm$ 0.8	37.6 $\pm$ 0.8	58.7 $\pm$ 1.1	45.5 $\pm$ 1.2	31.9 $\pm$ 0.9	58.9 $\pm$ 0.9	34.5 $\pm$ 1.2	21.6 $\pm$ 1.1
PRO-DSC ( Meng et al., 2025 )	83.2 $\pm$ 1.0	69.8 $\pm$ 0.8	59.6 $\pm$ 0.7	68.2 $\pm$ 0.5	55.7 $\pm$ 0.6	40.2 $\pm$ 0.3	57.6 $\pm$ 0.7	48.1 $\pm$ 0.9	31.3 $\pm$ 0.8	81.0 $\pm$ 0.8	61.9 $\pm$ 0.9	52.8 $\pm$ 0.7
Ours	89.6 $\pm$ 0.8	78.6 $\pm$ 0.7	72.9 $\pm$ 0.4	68.8 $\pm$ 0.2	58.1 $\pm$ 0.4	43.4 $\pm$ 0.3	63.8 $\pm$ 0.6	53.7 $\pm$ 0.6	36.2 $\pm$ 0.5	84.4 $\pm$ 0.5	70.2 $\pm$ 0.7	61.9 $\pm$ 0.9

Appendix

Table 7: Average clustering performance (in percent %) across four benchmarks datasets with larger cluster numbers.

Figure 6: Clustering performances with different

\theta

on CIFAR-10 and DTD.

Figure 7: Clustering performances with different

\gamma

on CIFAR-10 and DTD.

Figure 8: Clustering performances and percent of selected samples to total samples with different

\hat{k}

on CIFAR-10 and DTD.

		CIFAR-10			DTD-47			UCF-101
Methods	Backbone	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
TAC (no-train) ( Li et al., 2024 )	BLIP ViT-B/16	79.9	88.9	77.5	62.3	52.1	35.9	76.1	60.9	51.4
Ours (no-train)	BLIP ViT-B/16	83.6	92.1	83.5	65.1	57.2	41.7	78.9	62.9	53.6

Appendix

Table 8: Comparisons with various methods on clustering performance (in percent %) across three benchmarks datasets.

Image Clustering

Image clustering groups images by visual or semantic similarity without requiring labels for each image.

See all 22 papers

VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training
Sep 29, 2026Yuanwei Hu, Bo Peng, Yuheng Jia +3Clustering Deep Clustering
University College London · The University of Queensland · Southeast University +2
Hierarchical Textual Knowledge for Enhanced Image Clustering
Apr 13, 2026Yijie Zhong, Yunfan Gao, Weipeng Jiang +1Image Clustering Knowledge Representation
Tongji University · Huawei Technologies Ltd.
Learning Deep Modality-Shared Self-Expressiveness for Image Clustering with Textual Information
Aug 9, 2026Xianghan Meng, Wei He, Zhiyuan Huang +1Cross-Modal Representation Learning Clustering
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, P.R. China

Sep 29, 2026J/K move · Enter open · S save

VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training

Yuanwei Hu, Bo Peng, Yuheng Jia, Xinting Hu+2

University College London · The University of Queensland · Southeast University+2

Clustering Deep Clustering Image Clustering Vision-Language Pretraining

Open paper AskPDF arXiv

Vision-language pre-training has reshaped image clustering, giving rise to language-assisted image clustering (LaIC), which leverages textual semantics to complement visual representations. Despite the rapid proliferation of LaIC methods, it remains unclear how much LaIC has actually advanced image clustering, as existing studies generally suffer from major limitations, including inconsistent experimental settings, inadequate dataset selection, and limited evaluation dimensions. To address this gap, we introduce VLM4Cluster, a comprehensive benchmark for image clustering in the era of pre-trained vision-language models (VLMs). VLM4Cluster implements 17 representative methods spanning classical, deep, and language-assisted image clustering, and evaluates them on 20 datasets covering classical, challenging, fine-grained, large-scale, and out-of-distribution settings. Beyond effectiveness, VLM4Cluster systematically investigates image clustering along three complementary dimensions: robustness to adversarial perturbations, generalization under distribution shifts, and computational efficiency. Our study shows that LaIC substantially advances the clustering performance frontier on many semantically demanding benchmarks, generally exhibits stronger generalization under distribution shifts, and achieves a more favorable effectiveness-efficiency trade-off. However, its gains become less consistent on large-scale and fine-grained datasets, while language assistance does not systematically reduce sensitivity to adversarial perturbations. VLM4Cluster is released at https://github.com/YuanweiHuu/VLM4Cluster.

Abstract

Image Clustering

Explore similar work

VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training

Hierarchical Textual Knowledge for Enhanced Image Clustering

Learning Deep Modality-Shared Self-Expressiveness for Image Clustering with Textual Information