Class-Incremental Learning (CIL) enables models to learn new classes continually while preserving past knowledge. Recently, vision-language models like CLIP offer transferable features via multimodal pre-training, making them well-suited for CIL. However, real-world visual and linguistic concepts are inherently hierarchical: a textual concept like "dog" subsumes fine-grained categories such as "Labrador" and "Golden Retriever," and each category is associated with corresponding images. Nevertheless, existing CLIP-based CIL methods fail to explicitly capture this inherent hierarchy, leading to feature drift of fine-grained classes during incremental updates and ultimately to catastrophic forgetting. To address this challenge, we propose HASTEN (Hierarchical Semantic Tree Anchoring), a hierarchy-aware framework that uses semantic structure to stabilize CLIP-based CIL. Rather than treating classes as isolated labels, HASTEN leverages external hierarchical knowledge as structured supervision to organize visual and textual features in hyperbolic space, helping maintain parent--child relations as new tasks arrive and mitigating feature drift. Since the shared hyperbolic mapper is updated across tasks, we further stabilize it by constraining its updates to a null space induced by prior-task features, reducing interference with previous mappings while retaining adaptability to new classes. Extensive experiments on nine benchmarks show that HASTEN consistently outperforms existing methods while reducing feature drift and catastrophic forgetting. Code is available at: https://github.com/LAMDA-CL/NeurIPS2026-HASTEN.
Figures & tables
Figure 1: Effect of feature hierarchy. Top: Without hierarchy, fine-grained features tend to drift as new classes emerge. Bottom: With a hierarchical constraint, features are better anchored and relations are more stable during incremental learning.
Figure 2: Illustration of Hasten . Left : Hierarchical semantic tree building and hyperbolic projection. We use GPT-5 to generate a tree-structured semantic hierarchy and design task-specific hierarchical perception modules to meet downstream knowledge requirements. Euclidean features are projected into hyperbolic space via a global hyperbolic mapping layer. Top-Right : Null space projection of the TP layer, ensuring TP does not interfere with the outputs of old tasks during updates. Bottom-Right : Illustration of the entailment loss in B2 . This loss pushes the embedding zc within an imaginary cone projected by its paired parent embedding zp .
Method
Aircraft
CIFAR100
Cars
B0 Inc10
B50 Inc10
B0 Inc10
B50 Inc10
B0 Inc10
B50 Inc10
Aˉ
AB
Aˉ
AB
Aˉ
AB
Aˉ
AB
Aˉ
AB
Aˉ
AB
Finetune
3.16
0.96
1.72
1.05
7.84
4.44
5.30
2.46
3.14
1.10
1.54
1.13
SimpleCIL [ 88 ]
59.24
48.09
53.05
48.09
84.15
76.63
80.20
76.63
92.04
86.85
88.96
86.85
ZS-CLIP [ 56 ]
26.66
17.22
21.70
17.22
81.81
71.38
76.49
71.38
82.60
76.37
78.32
76.37
L2P [ 76 ]
47.19
28.29
44.07
32.13
82.74
73.03
81.14
73.61
76.63
61.82
76.37
65.64
Table 1: Average and last performance comparison of different methods. The best performance is shown in bold. All methods are initialized with the same pre-trained CLIP for a fair comparison.
Figure 3: Incremental performance of different methods. We report the performance gap after the last incremental stage of Hasten and the runner-up method at the end of the line. All methods use the same pre-trained CLIP weights. More results are in the supplementary.
Figure 4: (a) Ablation study and (b) parameter sensitivity analysis. (c)(d) t-SNE [ 68 ] visualizations of visual and textual features on CIFAR100 B0 Inc5. We show the feature distributions of old classes and new classes without (c) and with (d) hierarchy.
Figure 5: Image traversals with Hasten and CLIP. Hasten ’s path through hyperbolic space reveals a smooth transition from specific to generic text descriptions, demonstrating a learned hierarchy. In contrast, CLIP’s flat representation typically yields fewer descriptions along the path.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Visualization of the semantic tree structure derived from the output JSON on CIFAR100. The root node “entity” branches into abstract categories (e.g., “organism”), which further subdivide until reaching the specific leaf classes from the input list.
Figure 7: Comprehensive analysis of Hasten on different benchmarks. Our method demonstrates consistent superiority and robustness across various random seeds, pre-trained checkpoints, and LLMs.
Figure 8: Additional ablation studies on ImageNet-R, Cars, and UCF. The full Hasten model consistently outperforms variants without hierarchical supervision or TP projection, confirming the complementary roles of hierarchy preservation and shared-mapper stabilization.
Variant
Cars
CIFAR100
UCF101
Aˉ
AB
Aˉ
AB
Aˉ
AB
Hasten (Full)
93.06
88.93
88.64
82.51
96.49
92.69
w/o Lhiertxt→txt
91.99
87.56
86.75
80.48
94.21
91.80
w/o Lhiertxt→img
92.38
88.25
87.37
81.33
95.84
92.54
w/o parent–child separation term
92.93
88.73
88.68
82.62
96.16
92.57
w/o Ltp
90.98
86.25
85.70
80.55
94.06
90.46
Appendix
Table 2: Component analysis of Hasten under the B0 Inc10 setting. Each entry reports average accuracy Aˉ and final-stage accuracy AB .
Method
Food
Cars
UCF
SUN
B0 Inc10
B50 Inc10
B0 Inc10
B50 Inc10
B0 Inc10
B50 Inc10
B0 Inc30
B150 Inc30
ZS-CLIP [ 56 ]
5.22
2.84
6.41
3.14
10.77
4.74
10.16
4.97
SimpleCIL [ 88 ]
6.22
3.76
5.11
2.51
4.86
2.31
8.12
3.51
L2P [ 76 ]
24.69
20.26
7.45
4.22
8.15
6.22
22.87
17.57
DualPrompt [ 75 ]
24.28
20.77
7.35
4.71
9.26
7.95
23.54
17.63
CODA-Prompt [ 64 ]
25.82
22.41
6.14
4.59
9.80
9.21
22.71
17.69
Appendix
Table 3: Forgetting measure FB of different methods under different incremental settings. Lower FB indicates less forgetting, and the best result in each column is highlighted in bold.
Figure 9: Robustness to noisy hierarchical supervision on Cars and UCF. We corrupt parent-child relations in the semantic tree and report final-stage accuracy. Clean hierarchies consistently perform best, showing the benefit of accurate hierarchical supervision.
Figure 10: Parameter sensitivity analysis on CUB B0 Inc20. (a) Impact of the text-to-image hierarchical loss weight λ2 and the parent-child separation margin δ . (b) Impact of the foundational geometric priors: hyperbolic curvature c and cone aperture κ . (c) Number of trainable parameters on ImageNet-R B0 Inc20.
Figure 11: Efficiency analysis. (a) Running time comparison across methods. (b) Trade-off between accuracy and the number of trainable parameters on ImageNet-R B0 Inc20.
Method
Aˉ
Total Time (min) ↓
Train Time (min) ↓
Eval Time (min) ↓
Peak GPU (GB) ↓
Infer. (ms/img) ↓
SimpleCIL [ 88 ]
83.81
1.24
0.55
0.69
0.92
3.20
DualPrompt [ 75 ]
69.49
61.77
59.82
1.95
6.95
9.06
CODA-Prompt [ 64 ]
73.27
66.43
64.26
2.17
15.53
10.09
L2P [ 76 ]
70.57
73.66
71.60
2.07
2.85
9.60
RAPF [ 30 ]
79.26
16.95
16.08
0.87
12.25
4.06
PROOF [ 90 ]
82.86
46.15
44.57
1.58
1.30
7.34
Appendix
Table 4: Overhead comparison on CUB B0 Inc20 . All methods start from the same pre-trained CLIP backbone and are evaluated under the same hardware environment, using a single RTX 3090 GPU per method. “Total Time” denotes the end-to-end running time from the start of continual training to the completion of the final evaluation. “Peak GPU” denotes the peak training GPU memory of the current process, and “Infer.” reports the average evaluation latency per test image aggregated over all sessions. Lower is better for all overhead metrics, while higher is better for Aˉ .
Figure 12: Incremental performance of different methods on half-base setting. We report the performance gap after the last incremental stage of Hasten and the runner-up method at the end of the line. All methods use the same CLIP pre-trained weights.
Figure 13: Incremental performance of different methods on B0 setting. We report the performance gap after the last incremental stage of Hasten and the runner-up method at the end of the line. All methods use the same CLIP pre-trained weights.
Figure 14: Image traversals with Hasten and CLIP: Hasten ’s path through hyperbolic space reveals a smooth transition from specific to generic text descriptions, reflecting a more systematic and fine-grained visual-textual semantic hierarchy than CLIP.
Dataset
Average Radius by Tree Depth
Average Radius by Node Type
Structural Distance
D 1
D 2
D 3
D 4
D 5
D 6
Virtual
Real
Image
Sibling
Non-Rel
CIFAR-100
0.13
0.29
0.29
0.67
0.68
–
0.26
0.30
0.68
0.27
0.32
CUB-200
0.10
0.12
0.48
0.80
0.83
–
0.32
0.53
0.82
0.51
0.54
UCF-101
0.14
0.17
0.35
0.72
0.75
0.78
0.25
0.44
0.75
0.42
0.47
Appendix
Table 5: Quantitative hierarchy verification computed after the final incremental task. The radius is defined as the geodesic distance to the Lorentz origin o=[0,1] via Eq. 4. Structural distances compare the pairwise geodesic proximity of sibling nodes versus non-relative nodes. (A dash (–) indicates that the dataset tree does not reach this depth.)
Class-Incremental Learning (CIL) aims to endow models with the ability to continuously adapt to evolving data streams. Recent advances in pre-trained vision-language models (e.g., CLIP) provide a powerful foundation for this task. However, existing approaches often rely on simplistic templates, such as "a photo of a [CLASS]", which overlook the hierarchical nature of visual concepts. For example, recognizing "cat" versus "car" depends on coarse-grained cues, while distinguishing "cat" from "lion" requires fine-grained details. Similarly, the current feature mapping in CLIP relies solely on the representation from the last layer, neglecting the hierarchical information contained in earlier layers. In this work, we introduce HiErarchical Representation MAtchiNg (HERMAN) for CLIP-based CIL. Our approach leverages LLMs to recursively generate discriminative textual descriptors, thereby augmenting the semantic space with explicit hierarchical cues. These descriptors are matched to different levels of the semantic hierarchy and adaptively routed based on task-specific requirements, enabling precise discrimination while alleviating catastrophic forgetting in incremental tasks. Extensive experiments on multiple benchmarks demonstrate that our method consistently achieves state-of-the-art performance.
Zhen-Hao Xie, Yan Wang, Lan Li +3
School of Artificial Intelligence and the State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China
Class-Incremental Learning (CIL) requires models to recognize new classes over time without forgetting previously learned ones. With the rise of vision-language pre-training, CLIP has become a strong foundation for CIL. A common design in CLIP-based CIL is to construct textual classifier weights by encoding class-name templates with the CLIP text encoder, and then classify visual features by image-text cosine similarity. This design is appealing: since CLIP aligns images and text in a shared embedding space, textual weights appear to provide an off-the-shelf classifier for incremental classes. However, we show that this seemingly natural design is not always beneficial, as a modality gap can still separate the two modalities and make textual classifier weights deviate from visual class distributions. Empirically, under identical task-wise CIL training, initializing the cosine classifier with visual class centers yields lower loss and better incremental accuracy than using CLIP textual features. Motivated by these observations, we propose VIS, a visual-only method for CLIP-based CIL that removes the deployed textual branch and constructs the incremental classifier entirely in the visual space. To obtain stronger task-adaptive visual representations, VIS uses only base-session data to enhance CLIP's final visual representation with informative visual-layer features. Built on the enhanced visual representation, VIS employs a simple kernelized incremental least-squares SVM, whose classifier weights are solved in closed form from additive sufficient statistics. When new classes arrive, VIS accumulates their sufficient statistics and recomputes the classifier weights for all seen classes, enabling efficient incremental updates while preserving historical class knowledge. Extensive experiments show that VIS achieves state-of-the-art performance without a textual branch.
Tao Hu, Zhen-Hao Xie, Jingcai Guo +2
School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University · Hong Kong Polytechnic University
Class-Incremental Learning (CIL) aims to continually learn new categories without forgetting previously acquired knowledge. Vision-language models such as CLIP offer strong transferable representations via multi-modal supervision, making them promising for CIL. However, applying CLIP to CIL poses two major challenges: (1) adapting to downstream tasks often requires additional learnable modules, increasing model complexity and susceptibility to forgetting; and (2) while multi-modal representations offer complementary strengths, existing methods have yet to fully realize their potential in effectively integrating visual and textual modalities. To address these issues, we propose BOFA (Bridge-layer Orthogonal Fusion for Adaptation), a novel framework for CIL. BOFA confines all model adaptation exclusively to CLIP's existing cross-modal bridge-layer, thereby adding no extra parameters or inference cost. To prevent forgetting within this layer, it leverages Orthogonal Low-Rank Fusion, a mechanism that constrains parameter updates to a low-rank ``safe subspace" mathematically constructed to be orthogonal to past task features. This ensures stable knowledge accumulation without data replay. Furthermore, BOFA employs a cross-modal hybrid prototype that synergizes stable textual prototypes with visual counterparts derived from our stably adapted bridge-layer, enhancing classification performance. Extensive experiments on standard benchmarks show that BOFA achieves superior accuracy and efficiency compared to existing methods.
Lan Li, Tao Hu, Da-Wei Zhou +3
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China