VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training

Yuanwei Hu; Bo Peng; Yuheng Jia; Xinting Hu; Yadan Luo; Wenjie Zhu

cs.CVSep 29, 2026

VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training

Authors: Yuanwei Hu, Bo Peng, Yuheng Jia, Xinting Hu, Yadan Luo, Wenjie Zhu

Organizations: University College London · The University of Queensland · Southeast University · University of Science and Technology of China · Auckland University of Technology

Abstract

Vision-language pre-training has reshaped image clustering, giving rise to language-assisted image clustering (LaIC), which leverages textual semantics to complement visual representations. Despite the rapid proliferation of LaIC methods, it remains unclear how much LaIC has actually advanced image clustering, as existing studies generally suffer from major limitations, including inconsistent experimental settings, inadequate dataset selection, and limited evaluation dimensions. To address this gap, we introduce VLM4Cluster, a comprehensive benchmark for image clustering in the era of pre-trained vision-language models (VLMs). VLM4Cluster implements 17 representative methods spanning classical, deep, and language-assisted image clustering, and evaluates them on 20 datasets covering classical, challenging, fine-grained, large-scale, and out-of-distribution settings. Beyond effectiveness, VLM4Cluster systematically investigates image clustering along three complementary dimensions: robustness to adversarial perturbations, generalization under distribution shifts, and computational efficiency. Our study shows that LaIC substantially advances the clustering performance frontier on many semantically demanding benchmarks, generally exhibits stronger generalization under distribution shifts, and achieves a more favorable effectiveness-efficiency trade-off. However, its gains become less consistent on large-scale and fine-grained datasets, while language assistance does not systematically reduce sensitivity to adversarial perturbations. VLM4Cluster is released at https://github.com/YuanweiHuu/VLM4Cluster.

Figures & tables

Algorithms
Classical Image Clustering	$k$ -means, Spectral Clustering (SC), EnSC-ORGEN, SSC-OMP
Deep Image Clustering	IDC, SCAN, CPP, TEMI, PRO-DSC
Language-assisted Image Clustering (LaIC)	SIC, TAC, TAC++, SAC, GradNorm, NTK-SC, SEIC, MAGIC
Datasets
Classical	CIFAR-10, CIFAR-20, STL-10, ImageNet-10, ImageNet-Dogs
Challenging	DTD, UCF-101, CIFAR-100

Table 1: Overview of the VLM4Cluster benchmark. The benchmark covers 17 clustering algorithms, 20 datasets, and VLM configurations spanning vision backbones, pretraining data, and training objectives. Evaluation encompasses effectiveness, robustness, generalization, and efficiency.

Classical Image Clustering
Dataset	STL-10			CIFAR-10			CIFAR-20			ImageNet-10			ImageNet-Dogs
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	93.0	96.1	93.4	74.8	73.5	63.6	59.2	48.6	35.6	94.2	96.0	93.3	64.0	60.3	48.1
SC	90.2	95.3	90.1	73.9	78.2	67.5	53.4	52.8	39.8	94.1	96.4	92.4	44.1	43.9	27.6
SSC-OMP	78.6	83.4	72.9	71.7	74.0	62.5	55.5	47.6	35.2	89.6	94.0	87.4	30.3	34.3	15.6
EnSC-ORGEN	78.8	83.8	73.0	75.8	82.2	70.8	58.9	52.1	40.3	93.3	96.4	92.3	34.9	42.4	19.7

Table 2: Clustering results on classical datasets using CLIP ViT-B/32 pretrained on LAION-400M. The best value in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	Aircraft			Cars			Flowers			Food			Pets
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	48.2	21.6	11.7	80.4	53.4	46.7	88.3	72.4	68.6	74.7	57.6	54.1	79.9	68.2	60.9
SC	45.2	21.6	11.3	75.0	45.4	39.2	86.4	72.9	70.6	66.1	55.3	44.4	71.5	67.1	60.9
SSC-OMP	41.8	17.9	6.6	62.4	37.4	21.8	51.5	31.4	22.0	47.9	36.2	23.0	42.9	30.5	17.6
EnSC-ORGEN	45.8	22.3	9.8	77.3	55.8	41.3	86.0	71.7	65.4	71.7	63.9	48.2	61.2	54.0	37.3

Table 5: Clustering results on fine-grained datasets using CLIP ViT-B/32 pretrained on LAION-400M. The best value in each metric column is highlighted in bold .

Deep Image Clustering
Dataset	ImageNet-A			ImageNet-C			ImageNet-R			ImageNet-V2			ImageNet-S			Average			Effective Robustness
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
IDC	47.6	12.2	4.1	66.9	34.8	12.8	53.2	31.8	14.2	76.8	38.2	18.1	61.0	25.1	2.8	61.1	28.4	10.4	+1.59	+0.39	-0.98
SCAN	49.0	12.3	4.1	69.2	39.0	11.0	53.1	32.1	9.6	78.2	41.0	23.1	59.3	24.6	1.9	61.8	29.8	9.9	-1.32	-0.92	-5.60
CPP	36.4	11.4	2.1	63.8	31.8	9.0	34.7	20.9	5.1	74.6	35.8	7.3	55.8	20.7	3.7	53.1	24.1	5.4	-2.34	-2.02	-0.54
TEMI	50.3	12.4	4.2	72.3	45.7	23.2	54.7	33.6	14.7	80.1	46.6	27.1	64.5	30.7	5.0	64.4	33.8	14.8	-3.34	-2.86	-5.16

Table 6: Clustering performance on OOD datasets using CLIP ViT-B/32 pretrained on LAION-400M. The best result in each metric column is highlighted in bold . Positive and negative effective robustness values are shown in red and green , respectively.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Training Split	Test Split	# of Training	# of Test	# of Classes
Classical Datasets
STL-10	Train	Test	5000	8000	10
CIFAR-10	Train	Test	50000	10000	10
CIFAR-20	Train	Test	50000	10000	20
ImageNet-10	Train	Test	13000	500	10
ImageNet-Dogs	Train	Test	19500	750	15

Appendix

Table 7: A summary of datasets used for evaluation.

Method	Hyperparameter	Values
General Settings	$K$	# of classes
	$M$	# of training sample
	optimizer	Adam, AdamW, SGD
	learning rate	$1e-5,5e-5,1e-4,5e-4,1e-3,5e-3,1e-2$
	weight decay	$0,1e-4,1e-3,2e-3,3e-3,\ldots,1e-2$
	epochs	10, 20, 50, 100, 200, 500, 800

Appendix

Table 8: Hyperparameter search spaces for different methods. For details on the meaning of these hyper-parameters, please refer to Appendix B .

Classical Image Clustering
Dataset	STL-10			CIFAR-10			CIFAR-20			ImageNet-10			ImageNet-Dogs			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	95.6	98.2	96.0	78.7	80.8	71.3	61.6	55.7	40.2	97.4	98.4	96.5	64.8	61.2	47.2	79.6	78.9	70.2
SC	95.3	98.1	95.9	77.8	81.5	72.2	54.1	53.5	39.2	96.6	98.0	95.7	46.7	45.1	30.0	74.1	75.2	66.6
SSC-OMP	82.4	84.1	76.9	73.9	76.7	66.2	58.0	50.8	37.6	86.7	90.0	81.8	34.0	36.9	18.4	67.0	67.7	56.2
EnSC-ORGEN	83.4	88.3	78.8	78.6	83.5	73.8	62.3	53.3	39.3	79.9	89.2	77.9	49.4	53.7	34.5	70.7	73.6	60.9

Appendix

Table 9: Clustering results on classical datasets using CLIP ViT-B/16 pre-trained on LAION-400M. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	CIFAR-100			DTD			UCF101			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
zero-shot	75.2	71.4	53.4	62.7	50.9	35.3	80.6	67.0	53.1	72.8	63.1	47.3
$k$ -means	70.9	55.2	42.5	67.5	54.7	39.7	83.3	62.8	55.8	73.9	57.6	46.0
SC	69.2	55.4	44.9	66.3	56.6	42.8	82.9	64.9	56.5	72.8	59.0	48.0
SSC-OMP	62.4	46.4	31.9	58.9	47.3	31.3	66.4	40.0	28.0	62.6	44.6	30.4

Appendix

Table 10: Clustering results on challenging datasets with CLIP ViT-B/16 pre-trained on LAION-400M. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	Places			ImageNet			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	58.8	30.9	18.8	77.0	46.4	35.3	67.9	38.7	27.1
SC	57.4	30.8	19.0	75.4	43.3	33.1	66.4	37.0	26.0
SSC-OMP	52.0	24.3	11.2	66.1	31.7	13.7	59.1	28.0	12.5
EnSC-ORGEN	61.2	32.8	20.7	75.4	47.1	27.7	68.3	40.0	24.2

Appendix

Table 11: Clustering results on large-scale datasets with CLIP ViT-B/16 pre-trained on LAION-400M. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	Aircraft			Flowers			Food			Cars			Pets			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	52.2	25.7	15.4	91.5	76.6	75.9	80.9	64.6	63.4	84.0	58.8	51.7	81.4	71.4	63.1	78.0	59.4	53.9
SC	47.4	22.8	13.3	88.2	77.4	75.0	74.2	63.0	54.4	81.1	54.4	49.4	63.0	50.7	43.3	70.8	53.7	47.1
SSC-OMP	43.2	19.4	8.0	51.1	30.7	19.4	54.0	41.5	28.3	66.2	40.5	25.6	44.9	32.2	19.1	51.9	32.9	20.1
EnSC-ORGEN	49.1	25.0	12.7	87.9	73.8	65.9	79.1	73.8	61.1	80.0	61.5	47.1	64.3	54.5	39.2	72.1	57.7	45.2

Appendix

Table 12: Clustering results on fine-grained datasets with CLIP ViT-B/16 pre-trained on LAION-400M. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	STL-10			CIFAR-10			CIFAR-20			ImageNet-10			ImageNet-Dogs			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	97.4	99.0	97.8	83.7	86.5	78.5	64.2	56.6	42.9	98.6	99.2	98.2	74.2	68.5	58.0	83.6	81.9	75.1
SC	97.0	98.8	97.3	82.5	82.9	75.8	59.2	58.5	44.2	98.7	99.4	98.7	56.8	56.0	42.3	78.8	79.1	71.7
SSC-OMP	89.3	93.8	87.6	84.4	85.9	79.5	62.8	56.0	42.2	96.2	97.8	95.2	40.3	38.8	21.7	74.6	74.5	65.3
EnSC-ORGEN	86.0	90.3	81.0	78.5	84.0	77.7	60.1	53.8	40.5	91.1	94.2	88.0	52.0	57.9	36.2	73.5	76.0	64.7

Appendix

Table 13: Clustering results on classical datasets with CLIP ViT-L/14 pre-trained on LAION-400M. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	CIFAR-100			DTD			UCF101			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	75.3	58.4	48.9	67.9	57.4	42.5	84.7	65.2	57.6	76.0	60.3	49.7
SC	74.7	59.9	51.9	69.8	61.4	45.8	85.8	69.4	62.6	76.7	63.6	53.4
SSC-OMP	68.7	54.6	40.7	63.0	51.4	35.1	70.0	43.5	33.5	67.2	49.8	36.5
EnSC-ORGEN	73.8	58.3	43.0	66.5	56.4	38.9	79.1	53.6	44.2	73.1	56.1	42.0

Appendix

Table 14: Clustering results on challenging datasets with CLIP ViT-L/14 pre-trained on LAION-400M. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	Places			ImageNet			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	59.3	32.1	19.8	79.8	50.5	40.6	69.5	41.3	30.2
SC	57.0	31.0	19.2	79.0	47.6	38.8	68.0	39.3	29.0
SSC-OMP	53.4	26.3	13.0	70.5	38.1	17.7	62.0	32.2	15.3
EnSC-ORGEN	64.1	34.6	24.3	77.8	50.1	31.0	71.0	42.4	27.6

Appendix

Table 15: Clustering results on large-scale datasets with CLIP ViT-L/14 pre-trained on LAION-400M. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	Aircraft			Flowers			Food			Cars			Pets			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	56.5	30.6	19.8	93.2	77.6	77.6	85.3	78.3	70.3	89.0	67.5	64.0	85.9	74.9	70.6	82.0	65.8	60.4
SC	52.6	28.3	19.7	92.7	83.9	83.3	82.0	70.6	64.4	86.8	62.2	60.0	71.2	57.0	52.7	77.1	60.4	56.0
SSC-OMP	47.9	23.8	11.9	62.0	41.8	34.8	68.0	54.5	42.1	75.6	53.6	41.2	54.9	46.1	31.8	61.7	44.0	32.4
EnSC-ORGEN	52.9	27.8	17.0	91.0	77.6	72.9	82.5	76.3	63.8	85.6	66.7	55.8	71.3	62.6	48.5	76.7	62.2	51.6

Appendix

Table 16: Clustering results on fine-grained datasets with CLIP ViT-L/14 pre-trained on LAION-400M. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	STL-10			CIFAR-10			CIFAR-20			ImageNet-10			ImageNet-Dogs			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	95.1	98.0	95.6	80.9	86.7	75.0	63.4	55.2	42.3	95.6	96.2	92.6	67.1	66.9	51.8	80.4	80.6	71.5
SC	94.0	97.5	94.5	79.3	82.7	74.6	56.5	55.2	42.3	95.6	97.2	94.1	50.4	49.6	34.0	75.1	76.4	67.9
SSC-OMP	79.5	86.1	74.5	77.8	83.0	73.2	59.5	52.6	39.8	90.7	94.4	88.4	39.3	47.6	24.2	69.4	72.7	60.0
EnSC	82.6	84.0	76.0	78.9	83.0	74.0	62.4	58.9	42.4	83.8	88.6	78.8	51.0	52.3	34.2	71.7	73.3	61.1

Appendix

Table 17: Clustering results on classical datasets with CLIP ViT-B/32 pre-trained on LAION-2B. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	CIFAR-100			DTD			UCF101			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	73.1	59.4	45.7	66.7	57.1	40.7	80.9	59.8	51.8	73.5	58.8	46.0
SC	72.2	59.6	49.5	66.4	55.1	42.8	81.8	63.2	55.6	73.4	59.3	49.3
SSC-OMP	63.9	49.3	34.8	59.4	48.1	31.6	63.5	37.7	25.1	62.3	45.0	30.5
EnSC	71.9	58.4	43.4	64.8	56.0	36.9	77.2	51.9	41.0	71.3	55.4	40.4

Appendix

Table 18: Clustering results on challenging datasets with CLIP ViT-B/32 pre-trained on LAION-2B. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	Places-365			ImageNet-1K			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	58.4	31.2	18.3	75.1	43.3	31.7	66.7	37.2	25.0
SC	56.6	29.9	18.1	74.1	40.4	30.5	65.4	35.1	24.3
SSC-OMP	52.3	25.0	12.0	65.9	31.1	12.7	59.1	28.1	12.4
EnSC	57.0	30.2	15.2	74.9	46.1	26.9	65.9	38.1	21.0

Appendix

Table 19: Clustering results on large-scale datasets with CLIP ViT-B/32 pre-trained on LAION-2B. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	Aircraft			Flowers			Food			Cars			Pets			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	55.3	27.2	17.9	89.2	72.2	70.2	75.5	68.5	54.5	81.0	55.2	47.8	81.5	70.4	63.6	76.5	58.7	50.8
SC	49.6	24.8	15.7	85.6	74.5	74.0	69.0	58.8	47.8	77.4	49.4	44.1	74.4	68.8	63.5	71.2	55.3	49.0
SSC-OMP	44.4	20.3	8.7	54.5	34.6	23.1	50.9	36.4	24.3	66.6	41.5	27.3	50.6	37.5	23.8	53.4	34.1	21.4
EnSC	50.4	25.7	13.3	86.5	76.9	66.2	73.7	65.9	51.4	78.8	59.1	45.8	67.1	57.9	44.8	71.3	57.1	44.3

Appendix

Table 20: Clustering results on fine-grained datasets with CLIP ViT-B/32 pre-trained on LAION-2B. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	STL-10			CIFAR-10			CIFAR-20			ImageNet-10			ImageNet-Dogs			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	92.2	94.2	88.6	77.7	78.8	68.8	58.1	51.1	36.8	98.6	99.0	98.2	77.0	74.5	63.6	80.7	79.5	71.2
SC	88.1	91.2	84.7	74.2	78.1	67.9	51.9	49.5	36.6	98.2	99.0	97.8	61.7	63.1	48.5	74.8	76.2	67.1
SSC-OMP	81.7	86.2	76.6	74.2	84.5	70.4	55.8	50.8	37.3	95.8	97.8	95.2	44.3	50.3	31.6	70.4	73.9	62.2
EnSC-ORGEN	87.9	85.9	80.5	83.0	90.8	80.9	59.9	49.5	38.2	98.4	99.2	98.2	66.9	73.2	55.5	79.2	79.7	70.7

Appendix

Table 21: Clustering results on classical datasets with SigLIP ViT-B/16. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	CIFAR-100			DTD			UCF101			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	68.3	51.5	36.7	67.8	54.7	40.6	82.8	63.4	54.4	73.0	56.5	43.9
SC	70.4	57.6	46.4	68.5	57.2	45.2	85.2	68.8	62.2	74.7	61.2	51.3
SSC-OMP	63.3	46.0	34.0	61.1	45.9	32.3	69.4	43.6	32.3	64.6	45.1	32.9
EnSC-ORGEN	71.4	57.8	43.4	66.5	55.6	39.1	81.7	58.1	49.0	73.2	57.2	43.9

Appendix

Table 22: Clustering results on challenging datasets with SigLIP ViT-B/16. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	Places			ImageNet			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	60.0	32.8	20.2	80.9	52.5	42.6	70.4	42.6	31.4
SC	58.2	32.6	20.2	80.5	48.7	40.4	69.3	40.6	30.3
SSC-OMP	54.6	26.8	13.2	73.7	42.6	21.4	64.2	34.7	17.3
EnSC-ORGEN	58.9	32.7	17.2	80.9	55.2	35.7	69.9	44.0	26.5

Appendix

Table 23: Clustering results on large-scale datasets with SigLIP ViT-B/16. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	Aircraft			Flowers			Food			Cars			Pets			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	71.8	44.0	37.2	94.4	79.7	80.5	86.7	80.0	71.7	91.0	69.9	66.4	87.2	78.1	73.2	86.2	70.3	65.8
SC	63.6	39.2	33.2	94.4	87.3	87.9	83.7	75.4	68.7	88.4	63.7	62.9	72.3	57.5	53.3	80.5	64.6	61.2
SSC-OMP	55.2	34.1	21.1	69.9	47.3	41.6	73.2	60.7	49.2	79.8	58.2	47.2	63.9	56.0	42.6	68.4	51.2	40.3
EnSC-ORGEN	63.9	39.0	28.9	90.9	71.1	70.3	85.7	78.3	69.6	91.1	74.1	69.8	85.1	78.0	69.8	83.3	68.1	61.7

Appendix

Table 24: Clustering results on fine-grained datasets with SigLIP ViT-B/16. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	Pre-attack			Post-attack			Change (%)
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	73.7	58.8	50.0	56.5	44.7	32.2	23.3	24.0	35.6
SC	69.5	57.8	48.5	53.2	41.8	30.7	23.5	27.7	36.7
SSC-OMP	58.0	44.5	31.8	45.0	33.4	20.4	22.4	24.9	35.8
EnSC-ORGEN	68.3	56.9	43.7	54.0	43.7	28.8	20.9	23.2	34.1

Appendix

Table 25: Average pre-attack and post-attack clustering performance across 15 datasets using CLIP ViT-B/32 pretrained on LAION-400M. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	CIFAR-10			CIFAR-20			STL-10			ImageNet-10			ImageNet-Dogs			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	11.8	23.4	7.0	14.8	20.9	6.3	71.2	76.4	61.3	88.6	91.8	83.5	51.5	49.7	31.4	47.6	52.4	37.9
SC	11.8	23.7	7.4	13.6	20.5	6.5	69.3	75.1	61.1	86.5	91.0	81.4	41.2	41.7	24.5	44.5	50.4	36.2
SSC-OMP	11.5	22.8	7.8	11.7	17.9	6.1	67.9	73.6	59.5	81.2	89.0	78.1	29.1	32.4	14.0	40.3	47.1	33.1
EnSC-ORGEN	14.1	25.7	9.2	17.2	21.9	7.8	61.8	68.9	49.0	83.2	90.0	78.4	41.0	44.5	23.7	43.5	50.2	33.6

Appendix

Table 26: Post-attack clustering performance on classical datasets using CLIP ViT-B/32 pretrained on LAION-400M. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	CIFAR-100			DTD			UCF-101			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	27.6	13.3	4.3	57.4	46.1	28.6	69.4	46.2	33.5	51.5	35.2	22.2
SC	27.1	13.5	5.7	58.2	48.0	33.1	68.8	45.5	35.2	51.3	35.7	24.7
SSC-OMP	23.1	10.4	3.8	49.5	36.0	20.6	60.2	33.9	20.4	44.3	26.8	14.9
EnSC-ORGEN	28.9	14.9	5.6	55.1	43.0	25.7	67.3	41.6	28.5	50.4	33.2	19.9

Appendix

Table 27: Post-attack clustering performance on challenging datasets using CLIP ViT-B/32 pretrained on LAION-400M. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	Places365			ImageNet-1K			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	54.1	25.8	13.5	68.1	32.7	20.4	61.1	29.2	16.9
SC	52.4	24.9	14.6	67.5	31.9	21.4	60.0	28.4	18.0
SSC-OMP	47.8	20.7	8.9	60.8	25.1	8.6	54.3	22.9	8.7
EnSC-ORGEN	52.9	26.6	11.7	69.1	38.0	19.7	61.0	32.3	15.7

Appendix

Table 28: Post-attack clustering performance on large-scale datasets using CLIP ViT-B/32 pretrained on LAION-400M. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	Aircraft			Cars			Flowers			Food			Pets			Average
Metric	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$k$ -means	43.3	18.5	8.4	73.3	44.6	35.0	81.5	65.5	58.0	64.3	57.1	40.9	71.4	59.0	50.1	66.7	48.9	38.5
SC	40.6	17.0	7.7	69.2	38.8	31.7	80.5	66.8	61.8	57.1	46.5	34.2	54.9	42.6	34.8	60.5	42.4	34.1
SSC-OMP	39.2	15.9	5.2	57.8	30.6	16.6	53.5	32.2	22.0	41.8	31.2	17.8	40.0	29.0	15.9	46.5	27.8	15.5
EnSC-ORGEN	42.4	19.4	7.6	70.5	47.2	32.1	82.7	65.8	58.8	62.8	55.1	38.1	61.4	53.4	35.8	64.0	48.2	34.5

Appendix

Table 29: Post-attack clustering performance on fine-grained datasets using CLIP ViT-B/32 pretrained on LAION-400M. The best result in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	STL-10			CIFAR-10			CIFAR-20			ImageNet-10			ImageNet-Dogs			Average
Metric	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$
$k$ -means	0.207	2.590	419.44	0.172	2.903	441.43	0.104	3.250	156.82	0.320	2.059	36.12	0.085	3.140	17.44	0.178	2.789	214.25
SC	0.203	2.622	415.40	0.164	2.974	430.53	0.071	3.583	142.61	0.325	2.020	37.51	0.045	3.214	14.78	0.162	2.883	208.17
SSC-OMP	0.160	3.405	367.07	0.120	3.777	383.19	0.056	4.154	128.14	0.297	2.113	34.72	-0.030	5.255	8.38	0.120	3.741	184.30
EnSC	0.170	3.372	369.31	0.162	3.272	416.86	0.044	3.850	138.56	0.315	2.019	36.47	0.028	3.678	13.19	0.144	3.238	194.88

Appendix

Table 30: Discriminability analysis on classical datasets using CLIP ViT-B/32 pretrained on LAION-400M.

\uparrow

indicates larger values are better and vice versa. The best value in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	CIFAR-100			DTD			UCF101			Average
Metric	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$
$k$ -means	0.105	3.140	58.57	0.138	2.771	21.08	0.211	2.307	33.63	0.151	2.739	37.76
SC	0.084	3.222	51.19	0.122	2.874	18.92	0.205	2.310	32.33	0.137	2.802	34.15
SSC-OMP	0.011	4.827	40.01	0.045	3.613	14.70	0.017	3.670	16.38	0.024	4.037	23.70
EnSC	0.048	3.430	51.17	0.064	3.277	17.26	0.104	2.643	26.47	0.072	3.117	31.63

Appendix

Table 31: Discriminability analysis on challenging datasets using CLIP ViT-B/32 pretrained on LAION-400M.

\uparrow

indicates larger values are better and vice versa. The best value in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	Places-365			ImageNet-1K			Average
Metric	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$
$k$ -means	0.070	3.303	58.90	0.047	3.162	39.10	0.058	3.232	49.00
SC	0.039	3.134	52.33	0.038	3.032	44.77	0.038	3.083	48.55
SSC-OMP	-0.050	5.137	35.44	-0.143	4.939	21.12	-0.096	5.038	28.28
EnSC	0.013	3.536	50.94	-0.021	3.390	34.26	-0.004	3.463	42.60

Appendix

Table 32: Discriminability analysis on large-scale datasets using CLIP ViT-B/32 pretrained on LAION-400M.

\uparrow

indicates larger values are better and vice versa. The best value in each metric column is highlighted in bold .

Classical Image Clustering
Dataset	Aircraft			Flowers			Food			Cars			Pets			Average
Metric	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$	SIL $\uparrow$	DBI $\downarrow$	CHI $\uparrow$
$k$ -means	0.059	3.136	20.02	0.216	2.361	69.61	0.123	3.034	148.57	0.124	2.790	35.86	0.117	3.228	64.55	0.128	2.910	67.72
SC	0.011	3.411	15.97	0.201	2.537	63.29	0.096	3.475	125.24	0.091	2.744	31.15	0.080	3.232	53.94	0.096	3.080	57.92
SSC-OMP	-0.095	5.041	9.44	-0.104	5.811	18.91	-0.024	6.891	72.35	-0.058	4.271	18.08	-0.039	5.502	31.99	-0.064	5.503	30.15
EnSC	-0.038	3.717	15.19	0.137	2.625	60.39	0.075	3.350	133.45	0.074	2.906	32.10	0.037	3.329	52.94	0.057	3.185	58.82

Appendix

Table 33: Discriminability analysis on fine-grained datasets using CLIP ViT-B/32 pretrained on LAION-400M.

\uparrow

indicates larger values are better and vice versa. The best value in each metric column is highlighted in bold .

Explore similar work

Dynamic Cluster Data Sampling for Efficient and Long-Tail-Aware Vision-Language Pre-training
Apr 30, 2026Mingliang Liang, Zhuoran Liu, Arjen P. de Vries +1Recent Vision-Language Models Cluster
Institute for Computing and Information Sciences, Radboud University, Nijmegen, the Netherlands · University of Amsterdam, Amsterdam, the Netherlands · Radboud University
DataComp-VLM: Improved Open Datasets for Vision-Language Models
Jun 26, 2026Matteo Farina, Vishaal Udandarao, Thao Nguyen +33Ava-Vlm Multimodal Benchmarks
University of Trento · 2Tübingen AI Center, University of Tübingen · University of Cambridge +12
Learning Deep Modality-Shared Self-Expressiveness for Image Clustering with Textual Information
Aug 9, 2026Xianghan Meng, Wei He, Zhiyuan Huang +1Multimodal Representations Modalities
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, P.R. China

Apr 30, 2026, cs.CVJ/K move · Enter open · S save

Dynamic Cluster Data Sampling for Efficient and Long-Tail-Aware Vision-Language Pre-training

Mingliang Liang, Zhuoran Liu, Arjen P. de Vries, Martha Larson

Institute for Computing and Information Sciences, Radboud University, Nijmegen, the Netherlands · University of Amsterdam, Amsterdam, the Netherlands · Radboud University

Recent Vision-Language Models Cluster

Open paper AskPDF arXiv

The computational cost of training a vision-language model (VLM) can be reduced by sampling the training data. Previous work on efficient VLM pre-training has pointed to the importance of semantic data balance, adjusting the distribution of topics in the data to improve VLM accuracy. However, existing efficient pre-training approaches may disproportionately remove rare concepts from the training corpus. As a result, long-tail concepts remain insufficiently represented in the training data and are not effectively captured during training. In this work, we introduce a dynamic cluster-based sampling approach (DynamiCS) that downsamples large clusters of data and upsamples small ones. We first demonstrate the advantage of our cluster-scaling approach, which maintains the relative order of semantic clusters in the data and emphasizes the long-tail. This approach contrasts with current work, which focuses only on flattening the semantic distribution of the data. Then, we show the importance of dynamic sampling, which applies sampling at each epoch to improve cross-epoch data diversity and make upsampling practical. Our experiments show that DynamiCS reduces the computational cost of VLM training and provides a performance advantage for long-tail concepts. Code available at https://github.com/MingliangLiang3/DynamiCS.

Abstract

Explore similar work

Dynamic Cluster Data Sampling for Efficient and Long-Tail-Aware Vision-Language Pre-training

DataComp-VLM: Improved Open Datasets for Vision-Language Models

Learning Deep Modality-Shared Self-Expressiveness for Image Clustering with Textual Information