The Devil Is in Gradient Entanglement: Energy-Aware Gradient Coordinator for Robust Generalized Category Discovery

Haiyang Zheng; Nan Pu; Yaqi Cai; Teng Long; Wenjing Li; Nicu Sebe; Zhun Zhong

cs.LGMar 25, 2026

The Devil Is in Gradient Entanglement: Energy-Aware Gradient Coordinator for Robust Generalized Category Discovery

Authors: Haiyang Zheng, Nan Pu, Yaqi Cai, Teng Long, Wenjing Li, Nicu Sebe, Zhun Zhong

Organizations: University of Trento, Italy · Hefei University of Technology, China

Abstract

Generalized Category Discovery (GCD) leverages labeled data to categorize unlabeled samples from known or unknown classes. Most previous methods jointly optimize supervised and unsupervised objectives and achieve promising results. However, inherent optimization interference still limits their ability to improve further. Through quantitative analysis, we identify a key issue, i.e., gradient entanglement, which 1) distorts supervised gradients and weakens discrimination among known classes, and 2) induces representation-subspace overlap between known and novel classes, reducing the separability of novel categories. To address this issue, we propose the Energy-Aware Gradient Coordinator (EAGC), a plug-and-play gradient-level module that explicitly regulates the optimization process. EAGC comprises two components: Anchor-based Gradient Alignment (AGA) and Energy-aware Elastic Projection (EEP). AGA introduces a reference model to anchor the gradient directions of labeled samples, preserving the discriminative structure of known classes against the interference of unlabeled gradients. EEP softly projects unlabeled gradients onto the complement of the known-class subspace and derives an energy-based coefficient to adaptively scale the projection for each unlabeled sample according to its degree of alignment with the known subspace, thereby reducing subspace overlap without suppressing unlabeled samples that likely belong to known classes. Experiments show that EAGC consistently boosts existing methods and establishes new state-of-the-art results. Code is available at https://haiyangzheng.github.io/EAGC.

Figures & tables

Figure 1: (a) $\mathbb{GDC}$ and $\mathbb{SOC}$ measure Gradient Entanglement (GE) from the gradient-direction and feature-subspace perspectives, respectively. Both indices reveal that existing GCD methods (e.g., SimGCD [ 74 ] ) suffer from non-negligible optimization conflicts. (b) We attribute this issue to two main factors. I: Gradient Direction Deviation. Unsupervised gradients aim to discover novel categories but are inherently noisy and unreliable, which distort the optimization direction dictated by the supervised objective that focuses on known categories. II: Subspace Overlap. Noisy and unreliable unsupervised objectives produce highly divergent gradients, whereas gradients of the supervised objective remain relatively consistent. Joint optimization inevitably lets supervised gradients dominate, which progressively attract novel-class representations toward the known-class subspace. (c) Our method effectively mitigates GE, yielding substantial performance gains.

Figure 2: Framework Overview of the proposed Energy-Aware Gradient Coordinator ( EAGC ). EAGC consists of two components: (1) Anchor-based Gradient Alignment ( AGA ), which uses feature anchors from a reference model to generate the alignment term $\nabla g_{\text{align}}$ to stabilize the updates of $\mathcal{L}_{\text{sup}}$ , and (2) Energy-aware Elastic Projection ( EEP ), which generates the projection term $\nabla g_{\text{proj}}$ by projecting the unlabeled-gradient component $\nabla\mathcal{L}_{\text{unsup}}$ onto the complement of the known-class subspace $\mathcal{S}^{\text{known}}$ using a sample-adaptive elastic weight $\tau$ .

Method	CUB			Stanford-Cars			Aircraft			CIFAR-100			ImageNet-100			Average
Method	All	Old	New	All	Old	New	All	Old	New	All	Old	New	All	Old	New	All	Old	New
ORCA [ 2 ]	36.3	43.8	32.6	31.6	32.0	31.4	31.9	42.2	26.9	73.5	92.6	63.9	81.8	86.2	79.6	51.0	59.4	46.9
GCD [ 63 ]	51.3	56.6	48.7	39.0	57.6	29.9	45.0	41.1	46.9	73.0	76.2	66.5	74.1	89.8	66.3	56.5	64.3	51.7
GPC [ 88 ]	52.0	55.5	47.5	38.2	58.9	27.4	43.3	40.7	44.8	75.4	84.6	60.1	75.3	93.4	66.7	56.8	66.6	49.3
XCon [ 10 ]	52.1	54.3	51.0	40.5	58.8	31.7	47.7	44.4	49.4	74.2	81.2	60.3	77.6	93.5	69.7	58.4	66.4	52.4
PromptCAL [ 85 ]	62.9	64.4	62.1	50.2	70.1	40.6	52.2	52.2	52.3	81.2	84.2	75.3	83.1	92.7	78.3	65.9	72.7	61.7

Table 1: Comparison with state-of-the-art GCD methods. Best and second-best results are bold and underlined , respectively.

Method	CUB			Stanford Cars			Aircraft
Method	All	Old	New	All	Old	New	All	Old	New
SimGCD	71.5	78.1	68.3	71.5	81.9	66.6	63.9	69.9	60.9
+ EAGC	77.9	79.9	76.9	81.5	90.9	76.9	74.6	74.9	74.4
SelEx	87.4	85.1	88.5	82.2	93.7	76.7	79.8	82.3	78.6
+ EAGC	89.4	82.3	92.9	85.3	92.7	81.7	84.9	81.8	86.5
Avg. $\Delta$	+4.2	-0.5	+6.5	+6.6	+4.0	+7.7	+7.9	+2.3	+10.7

Table 2: Comparison with DINOv2 as the backbone.

Method	AGA	EEP	CUB			Stanford-Cars
Method	AGA	EEP	All	Old	New	All	Old	New
SimGCD	x	x	60.3	65.6	57.7	53.8	71.9	45.0
I	✓	x	65.6	71.1	62.9	62.2	75.9	55.6
II	x	✓	65.2	71.1	62.2	57.7	74.6	49.6
+ EAGC	✓	✓	66.5	71.0	64.3	62.9	76.0	56.6
SelEx	x	x	73.6	75.3	72.8	58.5	75.6	50.3

Table 3: Ablation study of the proposed components. AGA denotes Anchor-based Gradient Alignment, and EEP denotes Energy-aware Elastic Projection.

Method	CUB		Stanford Cars		Aircraft
Method	$\mathbb{GDC}$ $\downarrow$	$\mathbb{SOC}$ $\downarrow$	$\mathbb{GDC}$ $\downarrow$	$\mathbb{SOC}$ $\downarrow$	$\mathbb{GDC}$ $\downarrow$	$\mathbb{SOC}$ $\downarrow$
SimGCD [ 74 ]	0.2669	0.6478	0.0909	0.8110	0.0935	0.8643
+EAGC	0.1525	0.4923	0.0500	0.6208	0.0375	0.7139
LegoGCD [ 3 ]	0.2646	0.6468	0.0905	0.8112	0.0922	0.8642
+EAGC	0.1594	0.4844	0.0515	0.6135	0.0376	0.7112
SelEx [ 50 ]	0.0005	0.5902	0.0007	0.6022	0.0313	0.6033

Table 4: Quantitative comparison of the Gradient Deviation Coefficient (

\mathbb{GDC}

) and Subspace Overlap Coefficient (

\mathbb{SOC}

Method	CUB			Stanford Cars			Aircraft
Method	All	Old	New	All	Old	New	All	Old	New
$w/o$ EEP	80.9	74.5	84.2	59.3	76.6	50.9	61.7	57.9	63.6
Ours	83.2	73.9	87.9	65.7	83.7	57.0	65.7	65.6	65.7

Table 5: Evaluation of the gradient projection strategy based on SelEx. “w/o EEP” indicates a variant that uniformly projects all unlabeled gradients away from the known-class subspace without applying energy-aware weighting.

Method	CUB			Stanford Cars
Method	All	Old	New	All	Old	New
PCA-16	82.7	74.5	86.8	64.9	84.0	55.7
PCA-32	80.9	73.5	84.7	64.5	84.1	55.0
PCA-64	82.1	73.7	86.3	64.0	83.3	54.6
Ours	83.2	73.9	87.9	65.7	83.7	57.0

Table 6: Comparison of subspace construction strategies. “PCA-K” refers to a hard subspace built from the top-

K

principal components of labeled data.

Figure 3: Effects of the aperture $\eta$ and projection-strength hyperparameters $\lambda_{a}$ and $\lambda_{p}$ .

Dataset	Labeled		Unlabeled
Dataset	# Images	# Classes	# Images	# Classes
CUB [ 66 ]	1,498	100	4,496	200
Stanford Cars [ 20 ]	2,000	98	6,144	196
Aircraft [ 38 ]	1,666	50	5,001	100
CIFAR - 100 [ 21 ]	20,000	80	30,000	100
ImageNet - 100 [ 8 ]	31,860	50	95,255	100

Table A : Dataset statistics and splits.

Data	Method	All ACC	Peak Mem	Inv. Cost	Time Costs (s)
Data	Method	All ACC	(MB)	( ms/ep )	Avg. Epoch	$\mathcal{E}_{r}(\cdot)$ Constr.
CUB	SimGCD	60.3	4289.4	-	55.9	-
CUB	+ EAGC	66.5	4927.2	2.9	63.8	467.9
IN-100	SimGCD	83.0	4288.1	-	947.7	-
IN-100	+ EAGC	83.5	4921.3	3.5	1262.2	6899.1

Table B : Computational overhead analysis on an NVIDIA A100.

Method	SimGCD			SelEx
Method	All	Old	New	All	Old	New
Baseline	44.0	58.0	36.4	39.6	54.9	31.3
+ EAGC	48.4	59.8	42.4	42.0	55.0	35.0

Table C : Extended results on the long-tailed Herbarium19 dataset.

Method	CUB			Stanford Cars			ImageNet-100
Method	All	Old	New	All	Old	New	All	Old	New
SimGCD [ 74 ]	61.5	66.4	59.1	49.1	65.1	41.3	81.7	91.2	76.8
+EAGC	66.5	69.4	65.0	63.3	74.8	57.8	82.2	94.1	76.2
SelEx [ 50 ]	72.0	72.3	71.9	58.7	75.3	50.8	85.4	94.0	81.0
+EAGC	83.1	75.4	86.9	64.2	79.1	56.9	88.9	95.5	85.5
Avg. $\triangle$	+8.1	+3.1	+10.5	+9.9	+6.8	+11.3	+2.0	+2.2	+2.0

Table D: Comparison without Number of Categories

K

Batch	CUB				Stanford Cars				Aircraft
Size	$\mathcal{L}_{\text{cls}}$	All	Old	New	$\mathcal{L}_{\text{cls}}$	All	Old	New	$\mathcal{L}_{\text{cls}}$	All	Old	New
32	0.021	66.5	71.0	64.3	0.034	62.9	76.0	56.6	0.042	57.7	60.4	56.3
128	0.035	68.6	71.3	67.3	0.052	58.2	71.8	51.7	0.554	50.3	58.3	46.3

Table E : Effect of batch size for training the reference model.

Training	CUB				Stanford Cars				Aircraft
Epoch	$\mathcal{L}_{\text{cls}}$	All	Old	New	$\mathcal{L}_{\text{cls}}$	All	Old	New	$\mathcal{L}_{\text{cls}}$	All	Old	New
20	0.046	67.8	71.8	65.9	0.130	55.1	71.8	47.0	0.232	55.6	60.9	52.9
30	0.021	66.5	71.0	64.3	0.034	62.9	76.0	56.6	0.042	57.7	60.4	56.3
100	0.014	65.7	67.5	64.8	0.018	54.9	69.0	48.1	0.011	57.9	58.3	57.7

Table F : Effect of the number of training epochs for the reference model.

Unfrozen	CUB				Stanford Cars
Blocks	$\mathcal{L}_{\text{cls}}$	All	Old	New	$\mathcal{L}_{\text{cls}}$	All	Old	New
1	0.021	66.5	71.0	64.3	0.034	62.9	76.0	56.6
2	0.015	65.5	73.3	61.5	0.028	56.9	74.9	48.2
3	0.013	66.2	71.2	63.7	0.025	60.6	72.1	55.1
4	0.012	67.3	69.6	66.1	0.016	66.4	74.7	62.4
5	0.012	66.6	69.7	65.1	0.019	57.7	71.4	51.1

Table G: Effect of the number of unfrozen ViT blocks in the reference model.

Method	CUB			Stanford Cars
Method	All	Old	New	All	Old	New
SelEx [ 50 ]	73.6	75.3	72.8	58.5	75.6	50.3
+EAGC	83.2	73.9	87.9	65.7	83.7	57.0
+EAGC ${}_{\text{loss}}$	82.1	74.5	85.9	65.1	83.9	56.1

Table H : Evaluation on loss-based proximal regularization (replacing gradient-level AGA with the explicit loss

\mathcal{L}_{\text{sup}}^{\prime}

)

Figure A : Comparison of two relation-based distillation variants: (a) Gram-matrix distillation; (b) sample-level relation distillation.

Method	CUB			Stanford Cars
Method	All	Old	New	All	Old	New
SelEx [ 50 ]	73.6	75.3	72.8	58.5	75.6	50.3
+EAGC	83.2	73.9	87.9	65.7	83.7	57.0
+RelDistill ${}_{\text{Gram}}$	80.4	68.5	86.3	59.8	79.7	50.1
+RelDistill ${}_{\text{sample}}$	82.9	79.5	84.6	60.0	81.9	49.5

Table I : Comparison of relation-based distillation variants and gradient-level AGA.

Figure B : Empirical validation of gradient entanglement. (a) Cosine similarity between the reference supervised gradient ( $g_{L}$ ) and various mixed gradients. (b) The evolution of gradient dominance ( $\rho_{\text{grad}}$ ) and representation collapse coefficient ( $\rho_{\text{in}}$ ) during the joint training process.

Figure C : Qualitative comparison using t-SNE visualizations.

Figure D : Visualization of attention maps of baselines and EAGC-integrated baselines on CUB.

Figure E : Additional Visualization of attention maps of baselines and EAGC-integrated baselines on CUB.

Figure F : Visualization of attention maps of baselines and EAGC-integrated baselines on Stanford Cars.

Figure G : Visualization of attention maps of baselines and EAGC-integrated baselines on Aircraft.

Generalized Category Discovery

Generalized category discovery learns known classes from labeled examples while grouping unlabeled data into known and previously unseen categories.

See all 25 papers

May 10, 2026

Relational Retrieval: Leveraging Known-Novel Interactions for Generalized Category Discovery

In this study, we tackle Generalized Category Discovery (GCD) via a Relational Retrieval perspective, explicitly coupling labeled and unlabeled data through bidirectional knowledge transfer. While existing methods treat these sources separately, missing valuable interaction opportunities, we propose Relational Pattern Consistency (RPC) that enables mutual enhancement. RPC employs One-vs-All classifiers for soft ID/OOD decomposition, then introduces two mechanisms: (i) for known-class preservation, we transfer semantic behavioral alignment; (ii) for category discovery, we leverage the insight that samples from the same category maintain invariant relationships with known-class prototypes, transforming unreliable pseudo-labeling into well-defined relational pattern matching. This bidirectional design allows labeled data to guide unlabeled learning while discovering novel categories through their collective relational signatures. Extensive experiments demonstrate RPC achieves state-of-the-art performance on both generic and fine-grained benchmarks.

Yulin Xu, Chunqi Guo, Yuanzhen Shuai +1

University College London London, UK · Juniata College Huntingdon, USA

Semi-Supervised Learning Generalized Category Discovery

Ask PDF

Sep 9, 2026

CoGe-GCD: Reframing Generalized Category Discovery with Compositional Generalization

Generalized Category Discovery (GCD) assigns unlabeled instances, mixed with labeled data, to known or novel categories, requiring human-like compositional reasoning: reusing primitives learned from known classes and deciding when new combinations imply new categories. Existing GCD methods operate on unstructured token features and struggle to extrapolate to novel compositions. We propose CoGe-GCD, which rethinks GCD through compositional generalization with two coupled stages. (i) Compositional Perception structures patch tokens by mapping them to a small vocabulary of primitives and refining token embeddings via competitive token-primitive assignment and information passing, yielding coherent groups for discovery. (ii) Generalizing Induction exploits the induced geometric structure and applies a structure-preserving calibration over spatial relations, maintaining probabilistic semantics while improving extrapolation to unseen primitive combinations. CoGe-GCD is implemented as an inductive-bias module between backbone and projection head, without modifying heads or losses, and can be plugged into diverse GCD frameworks. On standard benchmarks, it consistently improves all-class accuracy, unknown-class number estimation, and geometric quality, with marginal computational overhead. Code is available at https://github.com/lytang63/CoGe-GCD.

Luyao Tang, Jiewei Zheng, Kunze Huang +3

The University of Hong Kong, Hong Kong SAR, China · Xiamen University, Xiamen, China · Shenzhen University, Shenzhen, China.

Geometric Representation Learning Classification Compositional Generalization Generalized Category Discovery

Ask PDF

May 5, 2026

Sparsity Hurts: Simple Linear Adapter Can Boost Generalized Category Discovery

Generalized Category Discovery (GCD) seeks to identify novel categories from unlabeled data while retaining the classification ability of seen categories. Prior GCD methods commonly leverage transferable representations from pre-trained models, adapting to downstream datasets via partial fine-tuning (updating only the final ViT block) and visual prompt tuning (appending learnable vectors to inputs). However, conventional partial fine-tuning offers limited flexibility, as it fails to adapt the entire model; meanwhile, visual prompt tuning is prone to overfitting, due to its sensitivity to initialization and inherently constrained capacity. To address these limitations, we propose LAGCD, a simple yet effective GCD approach that embeds a residual linear adapter into each ViT block. From the perspective of feature sparsity, we systematically show that non-linearity in conventional adapters impairs performance, whereas our linear adapter enhances it by enabling more flexible model capacity. We further introduce an auxiliary distribution alignment loss to mitigate the negative impact of biased predictions between seen and novel categories. Extensive experiments on both generic and fine-grained datasets confirm that LAGCD consistently improves performance over many sophisticated baselines. The source code is available at https://github.com/yebo0216best/LAGCD

Bo Ye, Kai Gan, Tong Wei +1

School of Computer Science and Engineering, Southeast University, Nanjing 210096, China, and the Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China.

Vision Foundation Model Adaptation Parameter-Efficient Fine-Tuning Generalized Category Discovery

Ask PDF

Abstract

Generalized Category Discovery

Explore similar work

Relational Retrieval: Leveraging Known-Novel Interactions for Generalized Category Discovery

CoGe-GCD: Reframing Generalized Category Discovery with Compositional Generalization

Sparsity Hurts: Simple Linear Adapter Can Boost Generalized Category Discovery