Hierarchical Textual Knowledge for Enhanced Image Clustering

Yijie Zhong; Yunfan Gao; Weipeng Jiang; Haofen Wang

cs.CVApr 13, 2026

Hierarchical Textual Knowledge for Enhanced Image Clustering

Authors: Yijie Zhong, Yunfan Gao, Weipeng Jiang, Haofen Wang

Organizations: Tongji University · Huawei Technologies Ltd.

Abstract

Image clustering aims to group images in an unsupervised fashion. Traditional methods focus on knowledge from visual space, making it difficult to distinguish between visually similar but semantically different classes. Recent advances in vision-language models enable the use of textual knowledge to enhance image clustering. However, most existing methods rely on coarse class labels or simple nouns, overlooking the rich conceptual and attribute-level semantics embedded in textual space. In this paper, we propose a knowledge-enhanced clustering (KEC) method that constructs a hierarchical concept-attribute structured knowledge with the help of large language models (LLMs) to guide clustering. Specifically, we first condense redundant textual labels into abstract concepts and then automatically extract discriminative attributes for each single concept and similar concept pairs, via structured prompts to LLMs. This knowledge is instantiated for each input image to achieve the knowledge-enhanced features. The knowledge-enhanced features with original visual features are adapted to various downstream clustering algorithms. We evaluate KEC on 20 diverse datasets, showing consistent improvements across existing methods using additional textual knowledge. KEC without training outperforms zero-shot CLIP on 14 out of 20 datasets. Furthermore, the naive use of textual knowledge may harm clustering performance, while KEC provides both accuracy and robustness.

Figures & tables

Figure 1: Motivation and intuition behind KEC. We first contrast the type of knowledge used by existing methods versus our method. It also illustrates that class labels alone often fail to identify objects of similar categories. Building on prior work showing that CLIP attends to targeted semantical regions [ 40 ] , we visualize the attention maps derived from nouns and discriminative attributes using the method from [ 4 ] . The results motivate us to mine discriminative attributes to construct structured textual knowledge for guiding image clustering.

Figure 2: Overview of KEC. Image-Text Mapping first aligns visual features with the textual space using noun features from WordNet. We then construct a hierarchical concept-attribute structured knowledge in the textual space. Representative concepts are distilled from semantically redundant nouns and abstracted using an LLM. Discriminative attributes are generated based on either single concepts or similar concept pairs. After constructing this structured knowledge, KEC instantiates features for each input image to obtain knowledge-enhanced features. These are combined with visual features and can be flexibly integrated into various downstream clustering methods.

Dataset	Average			Flowers			Pets			CLEVR
Metrics	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
CLIP (k-means)	55.6	53.6	34.8	86.5	71.5	67.7	65.0	51.3	40.5	18.9	27.6	8.1
TAC (no train)	55.3	54.2	35.1	84.5	69.4	64.8	79.2	65.8	57.8	11.8	23.6	5.4
$\mathtt{KEC}$ (no train)	58.0	56.6	38.5	87.3	72.8	67.5	81.2	67.8	63.3	16.3	26.6	8.9
SIC	54.9	53.0	35.6	67.7	43.1	34.0	67.7	51.6	42.6	5.2	20.6	1.5
TAC	55.2	55.4	37.2	80.4	64.2	59.6	83.6	77.9	69.1	8.4	22.4	3.6

Table 1: KEC outperforms existing methods across diverse settings. The first three lines reflect the advantages of KEC under training-free settings. The last line uses ground-truth class labels as a loosely constrained upper bound. The remaining lines highlight the superiority of our hierarchical textual knowledge under different training strategies. Bold and underlined numbers denote the best and second-best results. Average refers to the average performance across all 20 datasets.

Figure 3: Comparisons of different types of textual knowledge. We categorize the impact of textual knowledge into two groups by comparing the performance gap between clustering without textual knowledge and with class label knowledge. KEC consistently achieves better or comparable performance by incorporating hierarchical knowledge. In contrast, TAC shows limited improvements on a subset of datasets, and its performance trend closely mirrors that of zero-shot CLIP, suggesting that simply using coarse nouns or labels is insufficient and even degrades performance.

Con.	Des.	UA	BA	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
				Average			Flowers			Pets			CLEVR
✓				56.5	54.9	36.9	84.9	68.9	62.3	78.9	66.0	61.1	14.2	23.2	5.5
	✓			56.2	54.8	36.4	85.0	68.6	62.0	78.9	64.9	60.9	15.4	24.2	6.3
✓	✓			56.7	55.4	37.2	85.5	70.8	63.6	79.0	64.7	60.5	15.8	25.8	8.0
✓	✓	✓		57.8	56.3	38.3	87.3	72.6	67.6	80.5	67.3	63.1	16.0	26.6	8.6
✓	✓		✓	57.8	56.2	38.1	86.8	72.2	65.2	80.2	66.7	62.8	16.3	26.6	8.9

Table 2: Ablation studies of different levels of the knowledge and their components in KEC. Con. and Des. mean the name of the concept and its corresponding description. UA and BA mean the uni-concept attribute and bi-concept attribute, respectively.

Figure 4: Analyses on time consumption and hyperparameters. (a) Runtime comparison between KEC and TAC in the knowledge construction stage across datasets with varying numbers of classes and images. We break down the runtime of KEC into three components (Concept, Uni-Concept Attribute, and Bi-Concept Attribute) and include K-Means as a lightweight downstream clustering algorithm for reference. (b) and (c) show the impact of the number of attributes on clustering performance (measured by ARI) across datasets of different scales. The size of each point reflects the corresponding runtime, reflecting the relation between attribute numbers and computational cost.

Symbol	Meaning
Initializing Image-Text Mapping
$\mathcal{I},I_{i}$	the collection of input images and a single input image
$\mathcal{W},w_{i}$	the collection of nouns and a single noun
$N_{v},N_{t}$	the number of input images and nouns
$X,\mathbf{x}_{i}$	the collection of visual features and the feature of image $I_{i}$
$T,\mathbf{t}_{i}$	the collection of noun features and the feature of noun $w_{i}$

Table 3: The summary of the used symbols.

Dataset	Number fo Classes	Train Size	Test Size	Dataset	Number fo Classes	Train Size	Test Size
CIFAR-10	10	50,000	10,000	DTD	47	3,760	1,880
CIFAR-100	100	50,000	10,000	SUN397	397	19,850	19,850
STL-10	10	5,000	8,000	EuroSAT	10	10,000	5,000
ImageNet	1000	1,281,167	50,000	Resisc45	45	25,200	6,300
Food101	101	75,750	25,250	GTSRB	43	26,640	12,630
Flowers	102	2,040	6,149	PCAM	2	294,192	32,768

Table 4: Details of 20 datasets.

Simple ImageNet Templates:

1. itap of a [class].

2. a bad photo of the [class].

3. a origami [class].

4. a photo of the large [class].

5. a [class] in a video game.

6. art of the [class].

Table 5: The prompts used for zero-shot CLIP.

Dataset	CIFAR-10			CIFAR-100			STL-10			DTD			UCF-101
Metrics	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
CLIP (k-means)	73.7	80.4	64.6	59.7	43.4	29.1	91.8	94.3	89.2	58.6	45.4	29.0	80.7	59.5	51.4
TAC (no train)	81.4	90.4	80.3	65.0	50.1	35.7	92.5	94.8	89.9	60.3	48.2	31.4	80.9	61.8	52.0
$\mathtt{KEC}$ (no train)	81.9	90.7	80.6	66.4	51.8	37.3	95.1	97.9	95.5	60.7	47.4	31.5	81.9	62.5	53.2
SIC	84.7	92.6	84.4	63.2	47.6	34.8	95.3	98.1	95.9	59.6	45.9	30.5	81.4	65.3	56.7
TAC	82.9	91.5	82.3	67.5	56.1	40.8	95.6	98.2	96.1	60.8	47.8	32.4	81.3	67.4	58.2

Table 6: Comparison results across all datasets.

Con.	Des.	UA	BA	CIFAR-10			CIFAR-100			STL-10			DTD			UCF-101
Con.	Des.	UA	BA	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
✓				80.9	89.7	79.7	64.5	49.4	36.0	94.0	97.0	94.5	58.0	43.6	26.4	80.5	60.2	51.8
	✓			80.1	89.0	78.3	64.2	47.9	33.4	93.7	96.7	94.0	57.4	46.3	28.8	80.7	60.7	52.7
✓	✓			81.4	90.1	80.5	64.6	49.1	36.0	92.4	96.5	94.4	58.2	46.8	28.7	80.7	61.2	51.3
✓	✓	✓		81.9	90.5	80.2	66.0	50.3	36.6	95.1	97.9	95.5	59.8	46.2	30.1	81.9	60.8	52.0
✓	✓		✓	81.9	90.6	80.3	65.7	50.9	36.4	95.1	97.9	95.5	60.2	46.8	31.1	81.7	61.8	53.1

Table 7: Ablation results across all datasets. Con. and Des. represent the name and its description of a concept, respectively. UA and BA represent the uni-concept attribute and bi-concept attribute.

Methods	Aircraft			Cars
Methods	NMI	ACC	ARI	NMI	ACC	ARI
CLIP (zero-shot)	48.8	22.0	10.7	77.4	59.3	43.3
KEC	50.6	22.5	13.3	68.3	37.6	26.6
KEC +	52.1	24.2	15.8	74.8	53.7	42.6

Table 8: Results of KEC with targeted orientation.

TopK	Flowers			Pets			CLEVR
TopK	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
1	86.9	72.1	69.2	80.1	67.9	62.1	17.0	27.0	7.8
3	87.0	72.6	66.9	80.0	68.2	62.7	16.0	26.6	8.6
5	87.3	72.8	67.5	81.2	67.8	63.3	16.3	26.6	8.9
10	87.7	73.6	68.2	80.0	67.5	62.6	16.1	26.4	8.7

Table 9: Results for different numbers of selected nouns.

	Flowers			Pets			CLEVR
	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
$N_{v}/50$	87.0	72.5	67.2	81.3	67.9	63.5	16.5	26.7	8.9
$N_{v}/100$	87.4	72.7	67.3	81.2	67.8	63.2	16.3	26.7	8.9
$N_{v}/300$	87.3	72.8	67.5	81.2	67.8	63.3	16.3	26.6	8.9
$N_{v}/500$	87.6	73.0	67.8	81.1	67.5	62.9	15.8	26.1	8.1

Table 10: Results for different cluster numbers in the initial Image-Text Mapping.

	Flowers			Pets			CLEVR
$\alpha$	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
1.0	86.5	72.9	67.9	78.2	65.3	58.6	15.5	25.6	6.9
0.9	87.8	73.1	68.2	81.0	67.7	62.9	15.1	25.4	6.7
0.8	87.3	72.8	67.5	81.2	67.8	63.3	16.3	26.6	8.9
0.7	86.5	72.9	67.9	80.5	67.0	62.7	16.0	26.6	8.5
0.6	84.0	67.7	64.5	75.5	62.3	55.1	15.1	25.8	8.0

Table 11: Performance analysis based on varying the weight value of visual and textual similarity. The bold numbers indicate the best results. In practice, we set

\alpha

to 0.8.

	Flowers			Pets			CLEVR
$\beta$	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
0.9	86.7	71.4	65.9	79.9	64.9	60.7	15.4	24.8	6.7
0.8	87.3	72.8	67.5	81.2	67.8	63.3	16.3	26.6	8.9
0.7	87.5	71.7	67.0	80.0	68.8	62.8	14.7	25.4	6.5
0.6	86.3	71.2	64.9	80.5	67.2	62.1	14.5	24.2	5.9
0.5	83.5	67.2	63.8	78.8	66.1	60.9	14.0	24.4	6.1

Table 12: Performance analysis based on varying the threshold value used in KEC. The bold numbers indicate the best results. The threshold

\beta

is empirically set to 0.8.

	Average
	NMI	ACC	ARI
K-Means	58.0	56.6	38.5
Spectral Clustering	57.0	55.8	38.4
Agglomerative Clustering	58.2	56.9	38.2
Bisecting K-Means	57.2	56.2	37.6

Table 13: Clustering performance of KEC using different downstream traditional clustering algorithms.

Model	Method	Average			Flowers			Pets			CLEVR
Model	Method	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
CLIP ResNet-50	CLIP (K-Means)	50.5	47.9	28.4	83.6	70.4	63.6	55.9	38.8	27.5	7.9	20.2	2.5
	TAC (no train)	50.9	48.4	29.1	83.3	67.0	60.4	76.0	60.1	52.1	6.8	20.0	1.8
	KEC (no train)	51.9	49.3	30.5	84.1	71.1	64.2	78.6	67.9	58.1	7.1	20.6	2.0
CLIP ViT-B/16	CLIP (K-Means)	58.1	56.0	38.0	86.1	72.1	68.4	69.1	53.3	44.8	12.6	26.0	5.8
	TAC (no train)	56.5	54.2	36.5	86.2	66.1	62.3	83.1	70.2	64.0	9.7	23.2	4.0

Table 14: Performance of KEC under different pretrained visual-language models. We report average results across 20 datasets, along with detailed scores on three representative datasets. Bold numbers indicate the best performance within each setting.

LLMs	Flowers			Pets			CLEVR
LLMs	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
Claude-3-7-Sonnet	86.5	72.9	67.9	79.8	67.1	59.6	16.7	26.8	8.6
Gemini-2.0-Flash	86.5	72.9	67.9	81.0	67.8	62.7	17.0	26.8	9.0
Deepseek-V3	80.1	65.9	60.6	86.0	72.7	66.0	16.1	26.0	8.1
Qwen-Turbo	80.6	70.8	62.4	86.1	70.3	64.9	16.0	27.4	8.8
Qwen2.5-7B-Instruct	81.0	71.9	61.8	86.5	72.9	67.9	15.9	26.2	8.1

Table 15: Performance comparison of KEC with different LLMs for knowledge construction. The LLMs are from different providers and with various model sizes. The results demonstrate the robustness of KEC.

Figure 5: KEC effectively reduces semantic redundancy compared to naive use of textual knowledge.

Figure 6: t-SNE visualization of features before and after knowledge enhancement across six representative datasets. The left column shows CLIP visual features, and the right column shows knowledge-enhanced features obtained by KEC. For datasets with large numbers of classes, only the first 10 classes are visualized for clarity.

Dataset	Hierarchical Knowledge
CIFAR10	Concepts:
	Commercial and Utility Vehicles, Bird Species, Water and Land Transportation, aviation and air travel, Automobiles, Maritime Vessels, Musical Heritage, Equestrian Sports, Ecological Diversity of Amphibians and Reptiles, Pets, Feline and Animal, Antelope and Deer Species, emergency response vehicles, Equine Management and Culture, Aircraft, Dog Breeds, Maritime Vessels and Shipping, Fish Species, aviation and avian agility, Plant Ecology
	Uni-Concept Attributes:
	Prominent cargo area or flatbed design, Feathers covering the body, colors on animal fur or skin, Beak shape and size, Streamlined fuselage with a pointed nose, Large and swept-back wings with distinctive winglets, wheels or tracks on vehicles, Large hull structure with a streamlined shape, sails or masts or other rigging elements, Bright and high-visibility color schemes
	Bi-Concept Attributes:
	Feathered or fur-covered bodies, Color Variation, Mode of travel, Number of wheels, cargo or shipping containers on the vessels, Size and domestication level, representations and cultural symbols, Body structure and limb configuration, Color scheme and markings, landmark achievements

Table 16: Examples of the hierarchical knowledge constructed by KEC. We show 20 representative concepts, 10 Uni-Concept and Bi-Concept attributes, respectively. These knowledge elements are automatically derived via LLMs and serve to inject structured semantic guidance for clustering.

Image Clustering

Image clustering groups images by visual or semantic similarity without requiring labels for each image.

See all 22 papers

Universal Guideline-Driven Image Clustering via a Hybrid LLM Agent
Jun 23, 2026Wenliang Zhong, Rob Barton, Lucas Goncalves +7Clustering Unsupervised Clustering
1The University of Texas at Arlington · 2Amazon
Language-Assisted Image Clustering Guided by Discriminative Relational Signals and Adaptive Semantic Centers
Mar 25, 2026Jun Ma, Xu Zhang, Zhengxing Jiao +4Image Clustering Clustering
School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Computing Information Sciences, Saint Francis University, Hong Kong, China · Department of Computer Science, City University of Hong Kong, Hong Kong, China
VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training
Sep 29, 2026Yuanwei Hu, Bo Peng, Yuheng Jia +3Clustering Deep Clustering
University College London · The University of Queensland · Southeast University +2

Jun 23, 2026J/K move · Enter open · S save

Universal Guideline-Driven Image Clustering via a Hybrid LLM Agent

Wenliang Zhong, Rob Barton, Lucas Goncalves, Kushal Kumar+6

1The University of Texas at Arlington · 2Amazon

Clustering Unsupervised Clustering Image Clustering Semantic Clustering

Open paper AskPDF arXiv

Unifying image clustering across different clustering scenarios remains challenging due to fundamental gaps among tasks. We introduce a Guideline-Driven Image Clustering Agent, the first universal framework that bridges these gaps through textual guidelines. To incorporate complex guidelines without task-specific training, we propose Generative Concept Proxy Modeling, which generates guideline-aware embeddings via concept proxy extraction. For scenarios requiring automatic cluster discovery, we introduce LLM Traversal based on Minimum Spanning Tree that selectively applies LLM reasoning for complex semantic judgments. Our method generalizes across diverse clustering scenarios spanning from general to fine-grained categorization, from global to local criteria, and from balanced to long-tail distributions. Our framework consistently outperforms specialized methods across diverse clustering tasks.

Abstract

Image Clustering

Explore similar work

Universal Guideline-Driven Image Clustering via a Hybrid LLM Agent

Language-Assisted Image Clustering Guided by Discriminative Relational Signals and Adaptive Semantic Centers

VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training