Multi-Modal Representation Learning via Semi-Supervised Rate Reduction for Generalized Category Discovery

Wei He; Xianghan Meng; Zhiyuan Huang; Xianbiao Qi; Rong Xiao; Chun-Guang Li

cs.CVFeb 23, 2026

Multi-Modal Representation Learning via Semi-Supervised Rate Reduction for Generalized Category Discovery

Authors: Wei He, Xianghan Meng, Zhiyuan Huang, Xianbiao Qi, Rong Xiao, Chun-Guang Li

Organizations: School of Artificial Intelligence, Beijing University of Posts and Telecommunications · Intellifusion Inc., Shenzhen, P.R. China

Abstract

Generalized Category Discovery (GCD) aims to identify both known and unknown categories, with only partial labels given for the known categories, posing a challenging open-set recognition problem. State-of-the-art approaches for GCD task are usually built on multi-modality representation learning, which is heavily dependent upon inter-modality alignment. However, few of them cast a proper intra-modality alignment to generate a desired underlying structure of representation distributions. In this paper, we propose a novel and effective multi-modal representation learning framework for GCD via Semi-Supervised Rate Reduction, called SSR $^2$ -GCD, to learn cross-modality representations with desired structural properties based on emphasizing to properly align intra-modality relationships. Moreover, to boost knowledge transfer, we integrate prompt candidates by leveraging the inter-modal alignment offered by Vision Language Models. We conduct extensive experiments on generic and fine-grained benchmark datasets demonstrating superior performance of our approach.

Figures & tables

Figure 1: Illustration of our proposed framework: SSR 2 -GCD.

		ImageNet-100			ImageNet-1k
Method		All	Old	New	All	Old	New
DINOv1	GCD	74.1	89.8	66.3	52.5	72.5	42.2
	SimGCD	83.0	93.1	77.9	57.1	77.3	46.9
	SPTNet	85.4	93.2	81.4	-	-	-
	SelEx	83.1	93.6	77.8	-	-	-
	Hyp-SelEx	86.8	94.6	82.8	-	-	-

Table 1: The average ACC (%) on ImageNet datasets. “ † ” denotes the reproduced results using CLIP backbone.

		CIFAR-10			CIFAR-100			CUB			Stanford Cars			Oxford Pets			Flowers102
Method		All	Old	New	All	Old	New	All	Old	New	All	Old	New	All	Old	New	All	Old	New
DINOv1	GCD	91.5	97.9	88.2	73.0	76.2	66.5	51.3	56.6	48.7	39.0	57.6	29.9	80.2	85.1	77.6	74.4	74.9	74.1
	GPC	92.2	98.2	89.1	77.9	85.0	63.0	55.4	58.2	53.1	42.8	59.2	32.8	-	-	-	-	-	-
	SimGCD	97.1	95.1	98.1	80.1	81.2	77.8	60.3	65.6	57.7	53.8	71.9	45.0	87.7	85.9	88.6	71.3	80.9	66.5
	PromptCAL	97.9	96.6	98.5	81.2	84.2	75.3	62.9	64.4	62.1	50.2	70.1	40.6	-	-	-	-	-	-
	SPTNet	97.3	95.0	98.6	81.3	84.3	75.6	65.8	68.8	65.1	59.0	79.2	49.3	-	-	-	-	-	-

Table 2: The average ACC (%) on generic and fine-grained datasets. “ † ” denotes the reproduction of using CLIP backbone.

Rep. Losses	Inter	Intra	CIFAR-10	CIFAR-100	CUB	Stanford Cars	Oxford Pets	Flowers102
N/A	$\times$	$\times$	97.9	84.1	74.5	86.0	91.9	87.4
$\mathcal{L}_{\text{CLIP}}$	$\checkmark$	$\times$	98.3	86.0	76.7	87.0	94.1	89.7
$\mathcal{L}_{\text{con}}$	$\times$	$\checkmark$	98.4	86.7	77.5	87.9	94.9	91.8
$\mathcal{L}_{\text{SSR$ {}^{2} $}}$	$\times$	$\checkmark$	98.5	86.4	78.3	89.2	95.7	93.5
$\mathcal{L}_{\text{CLIP}}$ + $\mathcal{L}_{\text{con}}$	$\checkmark$	$\checkmark$	98.2	86.3	78.0	86.7	95.0	90.9
$\mathcal{L}_{\text{CLIP}}$ + $\mathcal{L}_{\text{SSR$ {}^{2} $}}$	$\checkmark$	$\checkmark$	98.3	86.1	77.2	88.1	95.0	92.9

Table 3: Evaluation of different representation learning methods. Average ACC (%) on “All” categories is reported. “N/A”: frozen CLIP.

Figure 2: Distributions of pairwise similarities on Flowers102. (a): at the beginning of training. (b)-(c): training with $\mathcal{L}_{\mathrm{SSR^{2}}}$ at epochs 10 and 200. (d)-(e): training with $\mathcal{L}_{\text{CLIP}}$ at epochs 10 and 200.

Panel of Figure 3 — Figure 3: Consistency measure $\rho$ as a function of training epoch when trained with different losses on different datasets. (a)-(b): on Flowers102. (c)-(d): on Stanford Cars.

Panel of Figure 4 — Figure 4: Mean ranks of image embeddings.

Panel of Figure 5 — Figure 5: Visualizations of image embeddings.

Figure 6: Cosine similarity (ordered by ground-truths) and cluster ranks on Flowers102.

Dual	RTA	$\mathcal{L}_{\text{SSR$ {}^{2} $}}$	Stanford Cars			Flowers102
Dual	RTA	$\mathcal{L}_{\text{SSR$ {}^{2} $}}$	All	Old	New	All	Old	New
$\times$	$\times$	$\times$	75.2	85.4	71.8	78.3	88.1	72.5
$\checkmark$	$\times$	$\times$	81.7	90.3	77.1	83.9	88.3	81.3
$\checkmark$	$\checkmark$	$\times$	86.0	91.0	83.4	87.4	90.8	85.5
$\checkmark$	$\times$	$\checkmark$	85.5	91.7	82.2	89.1	91.8	88.0
$\checkmark$	$\checkmark$	$\checkmark$	89.2	93.1	87.3	93.5	93.3	93.9

Table 4: Ablation study of different components.

Panel of Table 5 — Table 5: Effect of the hyper-parameter $\alpha$ in RTA strategy.

Panel of Figure 7 — Figure 7: Effect of the number of candidates in RTA strategy.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Labelled		Unlabelled
Dataset	#Image	#Class	#Image	#Class
CIFAR10	12.5K	5	37.5K	10
CIFAR100	20.0K	80	30.0K	100
ImageNet-100	31.9K	50	95.3K	100
ImageNet-1K	321K	500	960K	1000
CUB	1.5K	100	4.5K	200

Appendix

Table A.1 : Statistics of the benchmark datasets.

Figure A.1 : An overview of Retrieval-based Text Aggregation (RTA).

CLIP	Last residual attention block: $\mathbb{R}^{512}\rightarrow\mathbb{R}^{512}$
CLIP	Image/text projector: $\mathbb{R}^{512}\rightarrow\mathbb{R}^{512}$
Cls.	Linear projection: $\mathbb{R}^{512}\rightarrow\mathbb{R}^{K}$
Cls.	Softmax function

Appendix

Table A.2 : Model architectures. “ CLIP ” denotes the learnable parameters in CLIP and “ Cls ” denotes the classifiers.

from torchvision.transforms import *
Compose([
RandomResizedCrop(32, BILINEAR)
RandomHorizontalFlip(p=0.5),
ColorJitter()
ToTensor(),
Normalize([0.485, 0.456, 0.406],

Appendix

Table A.3 : Python code for the image augmentation.

Input: text
For each word in text :
If len(word) $\geq$ 3 :
index $\leftarrow$ random(1, len(word) $-$ 2)
action $\leftarrow$ random({replace, delete, add, none})
Case action :
replace : word $\leftarrow$ replace random char at index

Appendix

Table A.4 : Pseudo-code for the text augmentation.

		CIFAR-10			CIFAR-100			ImageNet-100			ImageNet-1k
Method		All	Old	New	All	Old	New	All	Old	New	All	Old	New
	TextGCD	98.2 $\scriptstyle\pm 0.0$	98.0 $\scriptstyle\pm 0.2$	98.6 $\scriptstyle\pm 0.1$	85.7 $\scriptstyle\pm 0.9$	86.3 $\scriptstyle\pm 0.6$	84.6 $\scriptstyle\pm 2.2$	88.0 $\scriptstyle\pm 0.6$	92.4 $\scriptstyle\pm 0.9$	85.2 $\scriptstyle\pm 1.2$	64.8 $\scriptstyle\pm 0.2$	77.8 $\scriptstyle\pm 0.5$	58.3 $\scriptstyle\pm 0.4$
	GET	97.2 $\scriptstyle\pm 0.1$	94.6 $\scriptstyle\pm 0.1$	98.5 $\scriptstyle\pm 0.1$	82.1 $\scriptstyle\pm 0.4$	85.5 $\scriptstyle\pm 0.5$	75.5 $\scriptstyle\pm 0.5$	91.7 $\scriptstyle\pm 0.3$	95.7 $\scriptstyle\pm 0.0$	89.7 $\scriptstyle\pm 0.4$	62.4 $\scriptstyle\pm 0.0$	74.0 $\scriptstyle\pm 0.2$	56.6 $\scriptstyle\pm 0.1$
	Ours	98.5 $\scriptstyle\pm 0.2$	98.3 $\scriptstyle\pm 0.2$	98.6 $\scriptstyle\pm 0.3$	86.4 $\scriptstyle\pm 0.5$	86.2 $\scriptstyle\pm 0.6$	86.9 $\scriptstyle\pm 1.6$	92.1 $\scriptstyle\pm 0.3$	96.0 $\scriptstyle\pm 0.6$	90.2 $\scriptstyle\pm 0.8$	66.7 $\scriptstyle\pm 0.1$	77.3 $\scriptstyle\pm 0.3$	61.1 $\scriptstyle\pm 0.2$

Appendix

Table A.5: The

\text{mean}\pm\text{std}

ACC (%) of TextGCD, GET and our approach on generic datasets.

		CUB			Stanford Cars			Oxford Pets			Flowers102
Method		All	Old	New	All	Old	New	All	Old	New	All	Old	New
	TextGCD	76.6 $\scriptstyle\pm 0.6$	80.6 $\scriptstyle\pm 2.0$	74.7 $\scriptstyle\pm 1.7$	86.1 $\scriptstyle\pm 0.9$	91.8 $\scriptstyle\pm 0.4$	83.9 $\scriptstyle\pm 1.3$	93.7 $\scriptstyle\pm 0.6$	93.2 $\scriptstyle\pm 1.1$	94.0 $\scriptstyle\pm 0.9$	87.2 $\scriptstyle\pm 2.3$	90.7 $\scriptstyle\pm 1.3$	85.4 $\scriptstyle\pm 3.8$
	GET	77.0 $\scriptstyle\pm 0.5$	78.1 $\scriptstyle\pm 1.6$	76.4 $\scriptstyle\pm 1.2$	78.5 $\scriptstyle\pm 1.3$	86.8 $\scriptstyle\pm 1.5$	74.5 $\scriptstyle\pm 2.2$	91.1 $\scriptstyle\pm 1.0$	89.7 $\scriptstyle\pm 1.6$	92.4 $\scriptstyle\pm 1.2$	85.5 $\scriptstyle\pm 0.5$	90.8 $\scriptstyle\pm 1.5$	81.3 $\scriptstyle\pm 1.7$
	Ours	78.3 $\scriptstyle\pm 0.8$	78.5 $\scriptstyle\pm 1.2$	78.2 $\scriptstyle\pm 0.9$	89.2 $\scriptstyle\pm 0.3$	93.1 $\scriptstyle\pm 0.9$	87.3 $\scriptstyle\pm 0.2$	95.7 $\scriptstyle\pm 0.2$	95.1 $\scriptstyle\pm 0.5$	96.0 $\scriptstyle\pm 0.4$	93.5 $\scriptstyle\pm 0.4$	93.3 $\scriptstyle\pm 0.8$	93.9 $\scriptstyle\pm 1.0$

Appendix

Table A.6: The

\text{mean}\pm\text{std}

ACC (%) of TextGCD, GET and our approach on fine-grained datasets.

Dataset	# Unknown categories	# Removed tags
CUB-200	100	90
Stanford Cars	98	83
Flowers102	51	43
Oxford Pets	19	17

Appendix

Table B.7: The number of identical tags between the candidate pool and the unknown categories in four benchmark datasets.

Datasets	CUB	Cars	Pets	Flowers
TexGCD [ 32 ]	$76.6$	$86.1$	$93.7$	$87.2$
TextGCD (w/o)	$75.7$	$85.5$	$91.5$	$84.0$
SSR 2 -GCD (w/o)	$\bf 77.6$	$\bf 88.1$	$\bf 94.0$	$\bf 91.6$

Appendix

Table B.8: Effect of retrieval-based methods with or without (w/o) prompts from unknown categories on four test datasets. ACC (%) on “All” categories is reported.

Rep. Losses	Inter	Intra	C-10	C-100	CUB	Cars	Pets	Flowers
N/A	$\times$	$\times$	97.9	84.1	74.5	86.0	91.9	87.4
$\mathcal{L}_{\text{CLIP}}$	$\checkmark$	$\times$	98.3	86.0	76.6	86.7	93.9	89.7
$\mathcal{L}_{\text{CICO}}$	$\checkmark$	$\times$	98.0	85.0	76.4	86.1	94.9	87.2
$\mathcal{L}_{\text{con}}$	$\times$	$\checkmark$	98.4	86.7	77.5	87.9	94.9	91.8
$\mathcal{L}_{\mathrm{SSR^{2}}}$	$\times$	$\checkmark$	98.5	86.4	78.3	89.2	95.7	93.5
$\mathcal{L}_{\text{CLIP}}$ + $\mathcal{L}_{\text{con}}$	$\checkmark$	$\checkmark$	98.2	86.3	78.0	86.7	95.0	90.9

Appendix

Table B.9: Evaluation of different representation learning methods. Average ACC (%) on “All” categories is reported. “N/A” denotes using a frozen CLIP.

Text retrieval models	Prompt qualities	Stanford Cars			Flowers102
Text retrieval models	Prompt qualities	$\mathcal{L}_{\text{SSR}^{2}}$	$\mathcal{L}_{\text{SSR}^{2}}+\mathcal{L}_{\text{CLIP}}$	$\Delta$	$\mathcal{L}_{\text{SSR}^{2}}$	$\mathcal{L}_{\text{SSR}^{2}}+\mathcal{L}_{\text{CLIP}}$	$\Delta$
CLIP-H/14	High	$89.2$	$88.1$	$1.1$ $\downarrow$	$93.5$	$92.9$	$0.6$ $\downarrow$
CLIP-L/14	Medium	$87.5$	$86.1$	$1.4$ $\downarrow$	$92.4$	$91.8$	$0.6$ $\downarrow$
CLIP-B/16	Low	$85.2$	$84.0$	$1.2$ $\downarrow$	$90.1$	$89.8$	$0.3$ $\downarrow$

Appendix

Table B.10: Accuracy comparison of models using different feature learning methods under various text prompt qualities.

\Delta

denotes the accuracy difference before and after adding the inter-modal alignment loss

\mathcal{L}_{\text{CLIP}}

, and “

\downarrow

” indicates accuracy drop.

	CIFAR-10			CIFAR-100			CUB			Stanford Cars			Oxford Pets			Flowers102
Method	All	Old	New	All	Old	New	All	Old	New	All	Old	New	All	Old	New	All	Old	New
GCD	91.5	97.9	88.2	73.0	76.2	66.5	51.3	56.6	48.7	39.0	57.6	29.9	80.2	85.1	77.6	74.4	74.9	74.1
GCD+SSR 2	92.5	96.4	91.6	73.9	79.0	63.2	51.9	55.0	47.1	47.9	56.1	47.3	83.6	87.7	79.8	80.0	83.3	78.5
SimGCD	97.1	95.1	98.1	80.1	81.2	77.8	60.3	65.6	57.7	53.8	71.9	45.0	87.7	85.9	88.6	71.3	80.9	66.5
SimGCD+SSR 2	97.6	97.5	97.7	81.1	82.5	78.9	60.8	64.7	59.0	57.1	66.8	53.9	90.0	89.8	91.2	81.6	83.5	80.1

Appendix

Table B.11: Comparison to uni-modal counterparts. ACC (%) on generic and fine-grained datasets is reported.

Figure B.2 : Mean accuracy with standard deviation over 3 trials of “All” categories with varying penalty weight $\nu$ on Flowers102.

Panel of Figure 26 — Figure B.3 : Consistency measure $\rho$ as a function of training epoch under different losses on Stanford Cars.

Figure B.4 : ACC (%) with varying $\epsilon$ in SSR 2 across six benchmark datasets.

Panel of Figure 28 — Figure B.5 : Visualization of image embeddings ( top ) and text embeddings ( bottom ) on Oxford Pets.

Panel of Figure 29 — Figure B.6 : ACC curves on benchmark datasets.

Methods	Running Time (sec/iter)			Memory (MB)
Methods	Forward	Backward	Overall	w/o RTA †	w/ RTA
TextGCD [ 32 ]	$0.34$	$6.7\times 10^{-3}$	$0.69$	7,622	-
SSR 2 -GCD	$0.57$	$7.5\times 10^{-2}$	$0.92$	8,035	11,080

Appendix

Table B.12 : Running time and memory costs on Flowers102. “

\dagger

” denotes to use the method in TextGCD [ 32 ] to produce text features.

Figure B.7 : Evaluation on Clustering performance ACC, NMI and ARI of our SSR 2 -GCD with varying $d_{\text{cls}}$ on Flowers102.

Generalized Category Discovery

Generalized category discovery learns known classes from labeled examples while grouping unlabeled data into known and previously unseen categories.

See all 26 papers

SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery
Feb 19, 2026Lorenzo Caselli, Marco Mistretta, Simone Magistri +1Cross-Modal Learning Cross-Modal Representation Learning
University of Florence, Media Integration and Communication Center (MICC), Italy
OmniGCD: Abstracting Generalized Category Discovery for Modality Agnosticism
Apr 16, 2026Jordan Shipard, Arnold Wiliem, Kien Nguyen Thanh +2Generalized Category Discovery
SAIVT, QUT, Australia · Shield AI, Australia · La Trobe University, Australia
Synergistic Dual-Branch Adaptation for Multi-modal Generalized Category Discovery
Jun 19, 2026Yuxun Qu, Minyu Zhou, Yongqiang Tang +2Cross-Modal Alignment Multimodal Classification
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 101408, China

Feb 19, 2026J/K move · Enter open · S save

SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery

Lorenzo Caselli, Marco Mistretta, Simone Magistri, Andrew D. Bagdanov

University of Florence, Media Integration and Communication Center (MICC), Italy

Cross-Modal Learning Cross-Modal Representation Learning Cross-Modal Alignment Generalized Category Discovery

Open paper AskPDF arXiv

Generalized Category Discovery (GCD) aims to identify novel categories in unlabeled data while leveraging a small labeled subset of known classes. Training a parametric classifier solely on image features often leads to overfitting to old classes, and recent multimodal approaches improve performance by incorporating textual information. However, they treat modalities independently and incur high computational cost. We propose SpectralGCD, an efficient and effective multimodal approach to GCD that uses CLIP cross-modal image-concept similarities as a unified cross-modal representation. Each image is expressed as a mixture over semantic concepts from a large task-agnostic dictionary, which anchors learning to explicit semantics and reduces reliance on spurious visual cues. To maintain the semantic quality of representations learned by an efficient student, we introduce Spectral Filtering which exploits a cross-modal covariance matrix over the softmaxed similarities measured by a strong teacher model to automatically retain only relevant concepts from the dictionary. Forward and reverse knowledge distillation from the same teacher ensures that the cross-modal representations of the student remain both semantically sufficient and well-aligned. Across six benchmarks, SpectralGCD delivers accuracy comparable to or significantly superior to state-of-the-art methods at a fraction of the computational cost. The code is publicly available at: https://github.com/miccunifi/SpectralGCD.

Abstract

Generalized Category Discovery

Explore similar work

SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery

OmniGCD: Abstracting Generalized Category Discovery for Modality Agnosticism

Synergistic Dual-Branch Adaptation for Multi-modal Generalized Category Discovery