CAESAR: Clustering via Autonomous Embedding-Space Agglomerative Reorganization

Ilan Bacry; Rémi Devaux; Antoine Jardin

cs.LGSep 26, 2026

CAESAR: Clustering via Autonomous Embedding-Space Agglomerative Reorganization

Authors: Ilan Bacry, Rémi Devaux, Antoine Jardin

Organizations: Arlequin AI

Abstract

Clustering algorithms that operate on nearest-neighbor graphs, such as FINCH (First Integer Neighbor Clustering Hierarchy), depend heavily on the quality of the embedding space they are given. However, pretrained vision and language model embeddings are not optimized for this purpose. We propose CAESAR, a method that reorganizes a pretrained embedding space: a reorganization network is trained to pull mutual nearest neighbors together and push non-neighbors apart, yielding a reorganized embedding space substantially better suited to clustering. CAESAR offers a second major advantage: it never requires the number of clusters $K$ . This matters because in realistic unsupervised settings, $K$ is typically unknown and discovering it is often part of the problem, yet most strong clustering methods take it as an input. We therefore design the entire CAESAR pipeline to infer $K$ rather than assume it is known. Empirically, reorganizing the embeddings consistently improves clustering over the raw space on both text and image datasets. Since the few deep clustering methods that also infer $K$ do not release their code, we complement controlled comparisons with methods that infer $K$ on the same embedding space by comparisons with strong deep clustering methods that are given the true $K$ , giving them a substantial oracle advantage. Even so, CAESAR outperforms all of them on text, achieves the best results on the most challenging image benchmark and remains competitive on the others. Reorganizing pretrained embeddings thus emerges as a simple and powerful route to clustering realistic data, where classes overlap and the number of clusters is unknown.

Figures & tables

Figure 1: Overview of CAESAR. Starting from a pretrained embedding space $X$ , we build a mutual $k$ -NN graph (Section 3.2 ) and train a reorganization network $f_{\theta}$ (Section 3.3 ) with a pull–push objective (Section 3.4 ). The resulting embeddings $Z=f_{\theta}(X)$ are partitioned by FINCH (Section 3.5 ). No stage of the pipeline uses the number of clusters.

Panel of Figure 2 — Figure 2: Effect of the pull–push objective on AG News, visualized with t-SNE ( Van der Maaten and Hinton, 2008 ) . Colors denote the four ground-truth classes and are used only for visualization, never during training. In the pretrained embedding space (left) the classes overlap; in the reorganized embedding space (right) they form compact, well-separated groups.

Method	NMI	ACC
FINCH (raw, sil.)	33.6	0 4.5
EVoC	36.2	21.8
FINCH (raw, $K$ ) †	49.6	58.5
CAESAR (ours)	61.5	78.3

Table 1: Clustering performance on AG News (text,

\times 100

). All methods use the same pretrained embedding space. Without using the number of clusters, CAESAR outperforms all methods, including the oracle FINCH and recovers a number of clusters close to the true

K

(Appendix B ). raw: FINCH on the pretrained embedding space. sil.: level selected by silhouette.

K

: level whose number of clusters is closest to the true

K

\dagger

: oracle method, knows the true number of clusters

K

. Bold : best.

	CIFAR-20		CIFAR-10		STL-10
Method	NMI	ACC	NMI	ACC	NMI	ACC
FINCH (raw, sil.)	29.6	14.1	39.1	19.9	44.7	20.0
CC †	43.1	42.9	70.5	79.0	76.4	85.0
SPICE †	44.8	46.8	73.4	83.8	81.7	90.8
SCAN †	48.6	50.7	79.7	88.3	69.8	80.9
FINCH (raw, $K$ ) †	49.2	38.4	63.3	60.5	82.1	84.1

Table 2: Comparison with deep image clustering methods (

\times 100

). All baselines except EVoC and FINCH (raw, sil.) are oracle methods (

\dagger

): they know the true number of clusters

K

. Although CAESAR uses neither, it obtains the best NMI and ACC on CIFAR-20 and remains competitive in NMI on CIFAR-10 and STL-10, where it outperforms CC, SPICE and SCAN. Literature methods use ResNet or CLIP encoders and their scores are taken from the published papers ( Li et al., 2023 ) ; CAESAR, FINCH and EVoC use Qwen3-VL-Embedding features. raw: FINCH on the pretrained embedding space. sil.: level selected by silhouette.

K

: level whose number of clusters is closest to the true

K

(Appendix B ). Bold : best.

Configuration	NMI	ARI	ACC
FINCH (raw, sil.)	36.8	26.7	14.6
+ $f_{\theta}$ (pull only)	0 2.4	0 0.7	15.6
+ push (in-batch)	45.6	30.3	41.6
+ memory bank ( ours )	71.9	52.9	65.9

Table 3: Cumulative ablation of CAESAR’s components (

\times 100

, mean over three seeds, averaged over CIFAR-20, CIFAR-10, STL-10 and AG News). Starting from FINCH on the pretrained embedding space, each row adds one component; the last row is our complete method.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Modality	Encoder	$K$	Train	Val	Test
AG News	Text	Qwen3-Embedding-8B	$4$	$108{,}000$	$12{,}000$	$7{,}600$
CIFAR-10	Image	Qwen3-VL-Embedding-8B	$10$	$45{,}000$	$5{,}000$	$10{,}000$
CIFAR-20	Image	Qwen3-VL-Embedding-8B	$20$	$45{,}000$	$5{,}000$	$10{,}000$
STL-10	Image	Qwen3-VL-Embedding-8B	$10$	$4{,}500$	$500$	$8{,}000$

Appendix

Table 4: Datasets used in our main experiments. All embeddings are

4096

-dimensional and kept fixed.

Hyperparameter	Value
Architecture of $f_{\theta}$	$d\rightarrow 1024\rightarrow 512\rightarrow 256$
Activations	SELU after each hidden layer, linear output
Output dimension $d^{\prime}$	$256$
Optimizer	AdamW, learning rate $3\times 10^{-4}$ , weight decay $10^{-4}$
Batch size	$256$
Epochs	at most $600$ , early stopping (patience $20$ )

Appendix

Table 5: Hyperparameters of CAESAR. All values are shared across datasets and encoders, except

k

, which is estimated from each dataset.

Method	NMI	ACC	ARI
FINCH (raw, sil.)	33.6	0 4.5	0 7.6
EVoC	36.2	21.8	30.6
FINCH (raw, $K$ ) †	49.6	58.5	42.2
CAESAR (ours)	61.5	78.3	58.3

Appendix

Table 6: Clustering performance on AG News (text,

\times 100

), ordered by increasing NMI.

\dagger

: given the true number of clusters. Bold : best in each column.

Method	CIFAR-20	CIFAR-10	STL-10
CAESAR (ours)	38.7	54.4	60.3

Appendix

Table 7: ARI of CAESAR (

\times 100

) on the three image datasets: CIFAR-20 (the

20

superclasses of CIFAR-100), CIFAR-10 and STL-10, all encoded with Qwen3-VL-Embedding-8B.

	AG News	CIFAR-20	CIFAR-10	STL-10
True $K$	$4$	$20$	$10$	$10$
Selected clusters	$5$	$21$	$9$	$8$

Appendix

Table 8: Number of clusters selected by the silhouette criterion on the FINCH hierarchy of CAESAR, compared with the true number of classes

K

. The number of clusters is never given to the method.

$\lambda$	0.1	0.2	0.5	0.8
NMI	41.4	44.5	68.8	62.0

Appendix

Table 9: Effect of the push weight

\lambda

(NMI

\times 100

on validation splits, averaged over AG News, CIFAR-10, STL-10 and MNIST).

Encoder	Method	CIFAR-20	CIFAR-10	STL-10
Qwen3-VL-Embedding	FINCH (raw, sil.)	29.6	39.1	44.7
Qwen3-VL-Embedding	CAESAR (ours)	61.5	81.5	83.0
CLIP ViT-B/32	FINCH (raw, sil.)	24.3	32.5	44.8
CLIP ViT-B/32	CAESAR (ours)	56.5	80.0	90.1

Appendix

Table 10: Robustness to the encoder (NMI

\times 100

). With both a large (

4096

-d Qwen3-VL-Embedding) and a much smaller (

512

-d CLIP) encoder, CAESAR yields a large improvement over silhouette-selected FINCH on the pretrained embedding space, confirming that the gain comes from the reorganization rather than from the encoder.

Cluster Number Estimation

See all 13 papers

CLUBench: A Clustering Benchmark
May 28, 2026Feng Xiao, Dazhi Fu, Chris Ding +1Benchmark Design Clustering
The Chinese University of Hong Kong (Shenzhen)
VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training
Sep 29, 2026Yuanwei Hu, Bo Peng, Yuheng Jia +3Clustering Deep Clustering
University College London · The University of Queensland · Southeast University +2
Cluster-R1: Large Reasoning Models Are Instruction-following Clustering Agents
Mar 6, 2026Peijun Qing, Puneet Mathur, Nedim Lipka +5Clustering Unsupervised Clustering
Dartmouth College, USA · Adobe Research, San Jose, USA

May 28, 2026J/K move · Enter open · S save

CLUBench: A Clustering Benchmark

Feng Xiao, Dazhi Fu, Chris Ding, Jicong Fan

The Chinese University of Hong Kong (Shenzhen)

Benchmark Design Clustering Unsupervised Clustering Deep Clustering

Open paper AskPDF arXiv

Clustering is a fundamental problem in data science with a long-standing research history, yielding numerous insightful algorithms. Despite this progress, a systematic and large-scale empirical evaluation that jointly considers conventional algorithms, deep learning-based methods, and recent foundation model-based clustering remains largely absent, leading to limited guidance on algorithm selection and deployment. To address this gap, we introduce CLUBench, a comprehensive clustering benchmark comprising 24 algorithms of diverse principles evaluated on 131 datasets across tabular, text, and image data, involving 178,815 experiments. Importantly, our analyses of (i) the impact of hyperparameter tuning,(ii) the impact of data types and characteristics,(iii) the impact of pretrained embeddings,(iv) large language model-based clustering,(v) the similarity of algorithms, and (vi) the low-rank structures of performance matrices, yield meaningful insights and promising pathways for clustering research. For instance, our study reveals that: 1) All evaluated deep clustering methods do not exhibit a significant advantage compared with the top-performing conventional clustering algorithms (e.g., KMeans, SpeClu) in terms of average performance; 2) For image and text clustering tasks, combining pretrained embeddings with conventional clustering algorithms (e.g., KMeans, SpeClu) offers effective and efficient clustering; 3) Clustering remains a challenging and nontrivial problem, even in the era of increasingly dominant foundation models. Moreover, we propose to use the low-rank structure in cross-model performance matrices to efficiently approximate the overall performance evaluation in practical applications. We further demonstrate the feasibility of model selection based on the performance matrices across all hyperparameter configurations.

Abstract

Cluster Number Estimation

Explore similar work

CLUBench: A Clustering Benchmark

VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training

Cluster-R1: Large Reasoning Models Are Instruction-following Clustering Agents