Semantic-Guided Dynamic Sparsification for Pre-Trained Model-based Class-Incremental Learning
Authors: Ruiqi Liu, Boyu Diao, Zijia An, Runjie Shao, Feixiang Ge, Yongjun Xu
Organizations: Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Class-Incremental Learning (CIL) requires a model to continually learn new classes without forgetting old ones. A common and efficient solution freezes a pre-trained model and employs lightweight adapters, whose parameters are often forced to be orthogonal to prevent inter-task interference. However, we argue that this parameter-constraining method is detrimental to plasticity. To this end, we propose Semantic-Guided Dynamic Sparsification (SGDS), a novel method that proactively guides the activation space by governing the orientation and rank of its subspaces through targeted sparsification. Specifically, SGDS promotes knowledge transfer by encouraging similar classes to share a compact activation subspace, while simultaneously preventing interference by assigning non-overlapping activation subspaces to dissimilar classes. By sculpting class-specific sparse subspaces in the activation space, SGDS effectively mitigates interference without imposing rigid constraints on the parameter space. Extensive experiments on various benchmark datasets demonstrate the state-of-the-art performance of SGDS.
Figures & tables
Figure 1: Comparison between parameter-constraining and our activation-guiding methods. (a) Constraining adapter parameters (e.g., via orthogonality ⊥ ) to reduce interference often harms model plasticity. (b) Our method guides the adapter’s input activations into orthogonal subspaces. This separation mitigates inter-task interference while preserving the model’s plasticity, as its parameters remain unconstrained.
Figure 2: An overview of SGDS. Operating at the input of each backbone adapter, SGDS proactively guides the activation space. The process begins by selecting a learning strategy based on semantic similarity: Knowledge Reuse (top path) to foster knowledge transfer, or New Subspace Allocation (bottom path) to prevent interference. This decision then guides the Semantic Exploration phase, which governs the subspace’s orientation by either aligning it with an existing subspace or creating a new orthogonal one. Subsequently, the Activation Compaction phase reduces the subspace’s rank by enforcing targeted sparsity, making it more compact. This entire process delivers a strategically structured activation stream to each adapter, freeing it from inter-task interference and thereby achieving a superior stability-plasticity balance.
Method
CIFAR-100
ImageNet-R
ImageNet-A
ObjectNet
Aˉ (%)
AT (%)
Aˉ (%)
AT (%)
Aˉ (%)
AT (%)
Aˉ (%)
AT (%)
Finetune
38.90
20.17
32.32
22.78
24.28
14.51
19.14
8.73
L2P ( Wang et al. 2022c )
85.94
79.93
75.46
69.77
49.39
41.71
63.78
52.19
DualPrompt ( Wang et al. 2022b )
87.87
81.15
73.10
67.18
53.71
41.67
59.27
49.33
CODA-Prompt ( Smith et al. 2023 )
89.11
81.96
77.97
72.27
53.54
42.73
66.07
53.29
SLCA ( Zhang et al. 2023 )
92.49
88.55
81.17
77.00
68.66
58.74
72.55
61.30
Table 1: Average ( Aˉ ) and final task ( AT ) Top-1 accuracy comparison on four challenging class-incremental learning benchmarks. All methods use ViT-B/16-IN21K as the backbone and are evaluated in the exemplar-free setting. The specific task configurations (e.g., number of base classes and incremental steps) are detailed in the experimental setup section. The best performance in each column is highlighted in bold .
Figure 3: Performance comparison with rehearsal-based methods on CIFAR-100 (left), ImageNet-R (center), and ObjectNet (right). The plots track the average accuracy as the model learns from a sequence of tasks. SGDS consistently outperforms strong baselines, and the performance gap widens over time, indicating superior knowledge retention.
Figure 4: t-SNE visualizations of parameter and activation distributions on ImageNet-R. Colors represent tasks, and numbers denote layers. (a) The down-projection parameters ( Wdown ) cluster tightly by layer regardless of the task, indicating they learn a shared, task-agnostic function. (b) In contrast, the up-projection parameters ( Wup ) are dispersed and form task-specific clusters, revealing their role in specializing for each task. (c) Similarly, the input activations are intermingled in early layers but become clearly separated by task in later layers.
SE
AC
Aˉ (%)
AT (%)
(Exploration)
(Compaction)
✗
✗
83.42
79.27
✗
✓
84.53
79.23
✓
✗
84.66
79.35
✓
✓
85.41
80.52
Table 2: Ablation study on ImageNet-R. We analyze the contribution of the two core components of our proposed SGDS: Semantic Exploration (SE) and Activation Compaction (AC). The best performance in each column is highlighted in bold . The results confirm that both modules are crucial for optimal performance.
Strategy
Aˉ (%)
AT (%)
Parameter-Space Regularization
Regularize Wdown only
83.33
79.12
Regularize Wup only
84.37
79.87
Regularize Both ( Wdown&Wup )
83.22
78.55
Activation-Space Guidance (Ours)
SGDS on All Layers (1-12)
84.59
80.10
Table 3: Comparison of Activation Guidance and Parameter Constraints. We compare two distinct strategies for mitigating interference on ImageNet-R. The first group applies orthogonality regularization directly to the adapter’s parameter space. The second group implements SGDS by guiding the activation space at different network depths. The results show that guiding activations, particularly in later layers, yields superior performance.
Figure 5: Hyperparameter Robustness on ImageNet-R. The plots show the average incremental accuracy as a function of (a) the compaction strength γ and exploration balance β , and (b) the sparsity ratio k . SGDS demonstrates stable performance across a wide range of values for all key hyperparameters.
Figure 6: t-SNE Visualization of Activation Frequencies on CIFAR-100. The visualization shows activation frequencies from the final network layer. The baseline model (a) shows mixed activations with no clear structure. In contrast, SGDS (b) organizes activations into distinct and compact clusters, grouping semantically similar classes (e.g., vehicles, natural objects) together.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Dynamic Adaptation of Learning Strategies in a Long Task Sequence (CIFAR-100, 20 Tasks). This figure illustrates how SGDS dynamically adjusts its learning strategy. (a) The left panel shows the per-task count of classes assigned to Knowledge Reuse versus New Subspace Allocation . Early tasks (e.g., 1-5) are dominated by New Subspace Allocation , demonstrating high plasticity as the model builds its initial semantic foundation. Later tasks show an increasing prevalence of Knowledge Reuse . (b) The right panel tracks the ratio of classes designated for Knowledge Reuse . The ratio steadily increases, indicating that as the model’s knowledge base expands, it increasingly and efficiently leverages existing subspaces. This automatic shift from a plasticity-dominant to a stability-focused strategy is crucial for SGDS’s strong performance in long-sequence CIL scenarios.
Figure A2: Performance stability across multiple random seeds on ImageNet-R (10-task setting). The plot displays the average accuracy as the number of learned tasks increases. All results shown are the mean performance calculated over five different random seeds, which vary the order of incoming classes. The consistent and superior performance of SGDS highlights its robustness against variations in the incremental learning sequence.
Figure A3: Incremental accuracy trends of SGDS compared to SOTA CIL methods on four standard benchmarks: CIFAR-100, ImageNet-R, ImageNet-A, and ObjectNet. Across all scenarios, SGDS demonstrates consistently superior performance. This highlights its robust ability to mitigate catastrophic forgetting and maintain a clear advantage as the number of learned classes increases.
Class-Incremental Learning (CIL) requires models to learn new classes without forgetting old ones. A common method is to freeze a pre-trained model and train a new, lightweight adapter for each task. While this prevents forgetting, it treats the learned knowledge as a simple, unstructured collection and fails to use the relationships between tasks. To this end, we propose the Semantic-guided Adaptive Expert Forest (SAEF), a new method that organizes adapters into a structured hierarchy for better knowledge sharing. SAEF first groups tasks into conceptual clusters based on their semantic relationships. Then, within each cluster, it builds a balanced expert tree by creating new adapters from merging the adapters of similar tasks. At inference time, SAEF finds and activates a set of relevant experts from the forest for any given input. The final prediction is made by combining the outputs of these activated experts, weighted by how confident each expert is. Experiments on several benchmark datasets show that SAEF achieves SOTA performance.
Ruiqi Liu, Boyu Diao, Hangda Liu +3
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Class Incremental Learning (CIL) aims to learn new concepts consistently from a data stream without forgetting. Unlike typical CIL methods which need to learn a model from scratch, pre-trained model (PTM) can easily adapt to a new task with fine-tuning. However, existing PTM-based CIL methods fail to achieve a trade-off between performance and computational expenditure, i.e., they either adopt the same parameter space so that leading catastrophic forgetting, or expand a new branch for each task but adding more computational cost. To this end, we propose MetrIc Learning with Expandable Subspace (Miles) to harness the prior information within pre-trained knowledge, thereby orchestrating an efficient expansion of the parameter space through guided optimization. Specifically, it decouples the learnable modules with the pre-trained model, exploiting prior information from intermediate features of the backbone network to enable more flexible parameter expansion. Then, a central loss is adopted to guide the new category to cluster towards the corresponding prototype in the new task subspace while incorporating an auxiliary distance regularization term to maintain metric equilibrium across tasks. Extensive experiments on six benchmark datasets demonstrate that Miles achieves state-of-the-art performance in various CIL settings.
Kai Jiang, Zisong Lin, Hongyuan Zhang +2
National Key Laboratory of Radar Signal Processing, Xidian University, Xi’an 710071, China · The University of Hong Kong, Hong Kong SAR, China · Institute of Artificial Intelligence (TeleAI) of China Telecom, China
Class-Incremental Learning (CIL) aims to continuously learn new classes without forgetting previously acquired knowledge. Parameter-efficient fine-tuning with pre-trained models reduces parameter overhead but can suffer from cumulative interference and suboptimal alignment between inference samples and specialized modules. We propose Dynamic LoRA-Experts and Prototype-Ensemble Matching (DLEPEM), a two-stage rehearsal-free framework. DLEPEM allocates a task-specific LoRA-Expert for each incremental task to reduce cross-task interference, then combines frozen pre-trained-model prototypes with task-adaptive LoRA-Expert prototypes for reliable task-level discrimination. Experiments on standard CIL and Few-Shot CIL benchmarks demonstrate strong performance under the evaluated protocols.
Hongwei Zhao, Rui Liu, Yansong Liu
School of Computer Science and Engineering, Beihang University