Hierarchical Semantic Tree Anchoring for CLIP-Based Class-Incremental Learning

Tao Hu; Lan Li; Zhen-Hao Xie; Da-Wei Zhou

cs.CVNov 19, 2025

Hierarchical Semantic Tree Anchoring for CLIP-Based Class-Incremental Learning

Authors: Tao Hu, Lan Li, Zhen-Hao Xie, Da-Wei Zhou

Organizations: School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University

Abstract

Class-Incremental Learning (CIL) enables models to learn new classes continually while preserving past knowledge. Recently, vision-language models like CLIP offer transferable features via multimodal pre-training, making them well-suited for CIL. However, real-world visual and linguistic concepts are inherently hierarchical: a textual concept like "dog" subsumes fine-grained categories such as "Labrador" and "Golden Retriever," and each category is associated with corresponding images. Nevertheless, existing CLIP-based CIL methods fail to explicitly capture this inherent hierarchy, leading to feature drift of fine-grained classes during incremental updates and ultimately to catastrophic forgetting. To address this challenge, we propose HASTEN (Hierarchical Semantic Tree Anchoring), a hierarchy-aware framework that uses semantic structure to stabilize CLIP-based CIL. Rather than treating classes as isolated labels, HASTEN leverages external hierarchical knowledge as structured supervision to organize visual and textual features in hyperbolic space, helping maintain parent--child relations as new tasks arrive and mitigating feature drift. Since the shared hyperbolic mapper is updated across tasks, we further stabilize it by constraining its updates to a null space induced by prior-task features, reducing interference with previous mappings while retaining adaptability to new classes. Extensive experiments on nine benchmarks show that HASTEN consistently outperforms existing methods while reducing feature drift and catastrophic forgetting. Code is available at: https://github.com/LAMDA-CL/NeurIPS2026-HASTEN.

Figures & tables

Figure 1: Effect of feature hierarchy. Top: Without hierarchy, fine-grained features tend to drift as new classes emerge. Bottom: With a hierarchical constraint, features are better anchored and relations are more stable during incremental learning.

Figure 2: Illustration of Hasten . Left : Hierarchical semantic tree building and hyperbolic projection. We use GPT-5 to generate a tree-structured semantic hierarchy and design task-specific hierarchical perception modules to meet downstream knowledge requirements. Euclidean features are projected into hyperbolic space via a global hyperbolic mapping layer. Top-Right : Null space projection of the TP layer, ensuring TP does not interfere with the outputs of old tasks during updates. Bottom-Right : Illustration of the entailment loss in $\mathcal{B}^{2}$ . This loss pushes the embedding $\mathbf{z}_{c}$ within an imaginary cone projected by its paired parent embedding $\mathbf{z}_{p}$ .

Method	Aircraft				CIFAR100				Cars
	B0 Inc10		B50 Inc10		B0 Inc10		B50 Inc10		B0 Inc10		B50 Inc10
	$\bar{\mathcal{A}}$	${\mathcal{A}_{B}}$	$\bar{\mathcal{A}}$	${\mathcal{A}_{B}}$	$\bar{\mathcal{A}}$	${\mathcal{A}_{B}}$	$\bar{\mathcal{A}}$	${\mathcal{A}_{B}}$	$\bar{\mathcal{A}}$	${\mathcal{A}_{B}}$	$\bar{\mathcal{A}}$	${\mathcal{A}_{B}}$
Finetune	3.16	0.96	1.72	1.05	7.84	4.44	5.30	2.46	3.14	1.10	1.54	1.13
SimpleCIL [ 88 ]	59.24	48.09	53.05	48.09	84.15	76.63	80.20	76.63	92.04	86.85	88.96	86.85
ZS-CLIP [ 56 ]	26.66	17.22	21.70	17.22	81.81	71.38	76.49	71.38	82.60	76.37	78.32	76.37
L2P [ 76 ]	47.19	28.29	44.07	32.13	82.74	73.03	81.14	73.61	76.63	61.82	76.37	65.64

Table 1: Average and last performance comparison of different methods. The best performance is shown in bold. All methods are initialized with the same pre-trained CLIP for a fair comparison.

Panel of Figure 3 — Figure 3: Incremental performance of different methods. We report the performance gap after the last incremental stage of Hasten and the runner-up method at the end of the line. All methods use the same pre-trained CLIP weights. More results are in the supplementary.

Panel of Figure 4 — Figure 4: (a) Ablation study and (b) parameter sensitivity analysis. (c)(d) t-SNE [ 68 ] visualizations of visual and textual features on CIFAR100 B0 Inc5. We show the feature distributions of old classes and new classes without (c) and with (d) hierarchy.

Panel of Figure 5 — Figure 5: Image traversals with Hasten and CLIP. Hasten ’s path through hyperbolic space reveals a smooth transition from specific to generic text descriptions, demonstrating a learned hierarchy. In contrast, CLIP’s flat representation typically yields fewer descriptions along the path.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Visualization of the semantic tree structure derived from the output JSON on CIFAR100. The root node “entity” branches into abstract categories (e.g., “organism”), which further subdivide until reaching the specific leaf classes from the input list.

Panel of Figure 7 — Figure 7: Comprehensive analysis of Hasten on different benchmarks. Our method demonstrates consistent superiority and robustness across various random seeds, pre-trained checkpoints, and LLMs.

Panel of Figure 8 — Figure 8: Additional ablation studies on ImageNet-R, Cars, and UCF. The full Hasten model consistently outperforms variants without hierarchical supervision or TP projection, confirming the complementary roles of hierarchy preservation and shared-mapper stabilization.

Variant	Cars		CIFAR100		UCF101
	$\bar{\mathcal{A}}$	$\mathcal{A}_{B}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{B}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{B}$
Hasten (Full)	93.06	88.93	88.64	82.51	96.49	92.69
w/o $\mathcal{L}_{\mathrm{hier}}^{\mathrm{txt}\rightarrow\mathrm{txt}}$	91.99	87.56	86.75	80.48	94.21	91.80
w/o $\mathcal{L}_{\mathrm{hier}}^{\mathrm{txt}\rightarrow\mathrm{img}}$	92.38	88.25	87.37	81.33	95.84	92.54
w/o parent–child separation term	92.93	88.73	88.68	82.62	96.16	92.57
w/o $\mathcal{L}_{\mathrm{tp}}$	90.98	86.25	85.70	80.55	94.06	90.46

Appendix

Table 2: Component analysis of Hasten under the B0 Inc10 setting. Each entry reports average accuracy

\bar{\mathcal{A}}

and final-stage accuracy

\mathcal{A}_{B}

Method	Food		Cars		UCF		SUN
Method	B0 Inc10	B50 Inc10	B0 Inc10	B50 Inc10	B0 Inc10	B50 Inc10	B0 Inc30	B150 Inc30
ZS-CLIP [ 56 ]	5.22	2.84	6.41	3.14	10.77	4.74	10.16	4.97
SimpleCIL [ 88 ]	6.22	3.76	5.11	2.51	4.86	2.31	8.12	3.51
L2P [ 76 ]	24.69	20.26	7.45	4.22	8.15	6.22	22.87	17.57
DualPrompt [ 75 ]	24.28	20.77	7.35	4.71	9.26	7.95	23.54	17.63
CODA-Prompt [ 64 ]	25.82	22.41	6.14	4.59	9.80	9.21	22.71	17.69

Appendix

Table 3: Forgetting measure

F_{B}

of different methods under different incremental settings. Lower

F_{B}

indicates less forgetting, and the best result in each column is highlighted in bold.

Figure 9: Robustness to noisy hierarchical supervision on Cars and UCF. We corrupt parent-child relations in the semantic tree and report final-stage accuracy. Clean hierarchies consistently perform best, showing the benefit of accurate hierarchical supervision.

Panel of Figure 10 — Figure 10: Parameter sensitivity analysis on CUB B0 Inc20. (a) Impact of the text-to-image hierarchical loss weight $\lambda_{2}$ and the parent-child separation margin $\delta$ . (b) Impact of the foundational geometric priors: hyperbolic curvature $c$ and cone aperture $\kappa$ . (c) Number of trainable parameters on ImageNet-R B0 Inc20.

Panel of Figure 11 — Figure 11: Efficiency analysis. (a) Running time comparison across methods. (b) Trade-off between accuracy and the number of trainable parameters on ImageNet-R B0 Inc20.

Method	$\bar{\mathcal{A}}$	Total Time (min) $\downarrow$	Train Time (min) $\downarrow$	Eval Time (min) $\downarrow$	Peak GPU (GB) $\downarrow$	Infer. (ms/img) $\downarrow$
SimpleCIL [ 88 ]	83.81	1.24	0.55	0.69	0.92	3.20
DualPrompt [ 75 ]	69.49	61.77	59.82	1.95	6.95	9.06
CODA-Prompt [ 64 ]	73.27	66.43	64.26	2.17	15.53	10.09
L2P [ 76 ]	70.57	73.66	71.60	2.07	2.85	9.60
RAPF [ 30 ]	79.26	16.95	16.08	0.87	12.25	4.06
PROOF [ 90 ]	82.86	46.15	44.57	1.58	1.30	7.34

Appendix

Table 4: Overhead comparison on CUB B0 Inc20 . All methods start from the same pre-trained CLIP backbone and are evaluated under the same hardware environment, using a single RTX 3090 GPU per method. “Total Time” denotes the end-to-end running time from the start of continual training to the completion of the final evaluation. “Peak GPU” denotes the peak training GPU memory of the current process, and “Infer.” reports the average evaluation latency per test image aggregated over all sessions. Lower is better for all overhead metrics, while higher is better for

\bar{\mathcal{A}}

Panel of Figure 12 — Figure 12: Incremental performance of different methods on half-base setting. We report the performance gap after the last incremental stage of Hasten and the runner-up method at the end of the line. All methods use the same CLIP pre-trained weights.

Panel of Figure 13 — Figure 13: Incremental performance of different methods on B0 setting. We report the performance gap after the last incremental stage of Hasten and the runner-up method at the end of the line. All methods use the same CLIP pre-trained weights.

Panel of Figure 14 — Figure 14: Image traversals with Hasten and CLIP: Hasten ’s path through hyperbolic space reveals a smooth transition from specific to generic text descriptions, reflecting a more systematic and fine-grained visual-textual semantic hierarchy than CLIP.

Dataset	Average Radius by Tree Depth						Average Radius by Node Type			Structural Distance
Dataset	D 1	D 2	D 3	D 4	D 5	D 6	Virtual	Real	Image	Sibling	Non-Rel
CIFAR-100	0.13	0.29	0.29	0.67	0.68	–	0.26	0.30	0.68	0.27	0.32
CUB-200	0.10	0.12	0.48	0.80	0.83	–	0.32	0.53	0.82	0.51	0.54
UCF-101	0.14	0.17	0.35	0.72	0.75	0.78	0.25	0.44	0.75	0.42	0.47

Appendix

Table 5: Quantitative hierarchy verification computed after the final incremental task. The radius is defined as the geodesic distance to the Lorentz origin

\mathbf{o}=[\mathbf{0},1]

via Eq. 4. Structural distances compare the pairwise geodesic proximity of sibling nodes versus non-relative nodes. (A dash (–) indicates that the dataset tree does not reach this depth.)

Class-Incremental Learning

Class-incremental learning studies how models can add new categories over time while retaining the ability to recognize previously learned classes.

See all 160 papers

HERMAN: Hierarchical Representation Matching for CLIP-based Class-Incremental Learning
Sep 26, 2025Zhen-Hao Xie, Yan Wang, Lan Li +3Class-Incremental Learning Vision-Language Models
School of Artificial Intelligence and the State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China
Visual Branch is What You Need for CLIP-based Class-Incremental Learning
Sep 29, 2026Tao Hu, Zhen-Hao Xie, Jingcai Guo +2Class-Incremental Learning Modality Gap in VLMs
School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University · Hong Kong Polytechnic University
BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning
Nov 14, 2025Lan Li, Tao Hu, Da-Wei Zhou +3Class-Incremental Learning VLM Adaptation
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China

Sep 26, 2025J/K move · Enter open · S save

HERMAN: Hierarchical Representation Matching for CLIP-based Class-Incremental Learning

Zhen-Hao Xie, Yan Wang, Lan Li, Han-Jia Ye+2

School of Artificial Intelligence and the State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China

Class-Incremental Learning Vision-Language Models VLM Adaptation Hierarchical Representation Learning Incremental Learning Continual Learning for VLMs

Open paper AskPDF arXiv

Class-Incremental Learning (CIL) aims to endow models with the ability to continuously adapt to evolving data streams. Recent advances in pre-trained vision-language models (e.g., CLIP) provide a powerful foundation for this task. However, existing approaches often rely on simplistic templates, such as "a photo of a [CLASS]", which overlook the hierarchical nature of visual concepts. For example, recognizing "cat" versus "car" depends on coarse-grained cues, while distinguishing "cat" from "lion" requires fine-grained details. Similarly, the current feature mapping in CLIP relies solely on the representation from the last layer, neglecting the hierarchical information contained in earlier layers. In this work, we introduce HiErarchical Representation MAtchiNg (HERMAN) for CLIP-based CIL. Our approach leverages LLMs to recursively generate discriminative textual descriptors, thereby augmenting the semantic space with explicit hierarchical cues. These descriptors are matched to different levels of the semantic hierarchy and adaptively routed based on task-specific requirements, enabling precise discrimination while alleviating catastrophic forgetting in incremental tasks. Extensive experiments on multiple benchmarks demonstrate that our method consistently achieves state-of-the-art performance.

Abstract

Class-Incremental Learning

Explore similar work

HERMAN: Hierarchical Representation Matching for CLIP-based Class-Incremental Learning

Visual Branch is What You Need for CLIP-based Class-Incremental Learning

BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning