Continual learning aims to acquire new knowledge while retaining previously learned knowledge. However, updating shared parameters when learning new tasks often overwrites existing knowledge, causing catastrophic forgetting. Memory replay methods mitigate catastrophic forgetting but rely on previous samples, raising privacy, memory, and scalability concerns. On the other hand, parameter isolation-based methods preserve learned knowledge but may limit knowledge transfer, whereas regularization promotes adaptability but remains susceptible to interference between old and new tasks. To maintain the stability-plasticity trade-off, this paper proposes Stability-Plasticity-Aware Region Selection (SPARS), a replay-free hybrid framework combining global parameter isolation with localized parameter regularization. SPARS estimates parameter importance using either the Fisher Information Matrix (FIM) or a novel Confidence Score (CS) estimator that leverages logit sensitivity and achieves comparable performance to FIM without requiring labelled data. Based on parameter importance, SPARS partitions parameters into three importance-aware regions namely, winning, neighboring, and non-important parameter regions. Each region receives a different adaptation strategy: selective freezing, soft regularization, or unrestricted learning based on its role in the stability-plasticity trade-off. To mitigate classifier bias in class-incremental learning, we employ Quadratic Discriminant Analysis to improve performance on previously learned classes .Experiments on CIFAR-10 and TinyImageNet demonstrate consistent improvement in average incremental accuracy over existing isolation- and regularization-based methods across task- and class-incremental learning. On TinyImageNet, SPARS achieves the best performance with 6% improvement for class-incremental learning while retaining over 60% of parameters for future tasks.
Figures & tables
Figure 1: Overview of the SPARS Framework. The framework consists of parameter importance estimation, Top-k parameter selection, partitioning model parameters into winners (W), neighboring (N), and non-important regions, followed a region-aware parameter adaptation strategy.
Figure 2: Distribution of Fisher importance scores across all model parameters, showing a long-tailed distribution in which only a small subset of parameters has high importance.
Dataset
Train
Test
TIL and CIL Splits
CIFAR-10
50,000
10,000
5 tasks, each with 2 classes
TinyImageNet
100,000
10,000
10 tasks, each with 20 classes
Table 1: Dataset statistics and continual learning task splits used in our experiments.
Figure 3: Per-task Average Accuracy of QDA and LDA across the tasks for CIL on (a) CIFAR-10 and (b) TinyImageNet.
Figure 4: PCA visualization of feature embeddings extracted from the backbone network after tasks 2, 6, and 10 under the CIL setting. Light-colored samples denote earlier feature locations, while darker samples represent their updated positions after subsequent tasks. Arrows visualize the direction and magnitude of feature drift over time.
Top-k
100K
300K
500K
700K
AIA
36.02
36.13
36.14
36.11
AF
15.86
15.48
14.60
14.77
Table 3: AIA and AF across different top- k sizes for CIL on TinyImageNet. Higher AIA and lower AF indicate better performance across all 10 tasks. The best result is achieved at top- k=500k .
Figure 5: Comparison between the number of unique top-(k) parameters across 10 tasks on TinyImageNet under CIL setting, the number of unique neighboring parameters accumulated across tasks, and the overlap among the top-(k) parameters identified across tasks. The blue curve shows the ratio of the combined top-(k) and neighboring parameters to the total number of model parameters.
Figure 6: AIA and AF under different values of λ on TinyImageNet in the CIL setting: λ=0 (no neighbors), λ=2 , λ=100 , and λ=1000 .
Figure 7: Distributions of FIM and CS (log10 scale) for the top- k winner parameters and their neighboring parameters on CIFAR-10 in the CIL setting.
Figure 8: Layer-wise distribution of top-k winner and neighboring parameters across convolutional layers.
Continual learning must balance the learning of new knowledge with the retention of previously learned knowledge to incrementally learn tasks from a data stream without catastrophic forgetting. While leveraging pretrained models has significantly advanced continual learning, existing methods exhibit a scalability bottleneck when trained sequentially on many tasks, suffering from performance degradation due to inter-task interference and loss of plasticity. Inspired by evidence that sparse fine-tuning achieves performance comparable to full fine-tuning, this paper presents a novel sparsity-driven continual learning framework. Our continual learning method, termed CLARE, operates in two stages: it first identifies a sparse, task-critical parameter mask via a sparsity-inducing objective, then performs mask-constrained fine-tuning by only optimizing parameters selected by the mask. This two-stage sparse adapter mechanism enables all tasks to be accumulated within a shared adapter space while reducing destructive interference across tasks. Extensive experiments demonstrate the scalability of CLARE. On the long task-sequence benchmark Omnibenchmark-1k, CLARE outperforms strong baselines in final accuracy by a large margin, e.g, improving EASE by 4.64% and 13.34% after learning 100 tasks, respectively.
Yunxiang Fu, Meng Lou, Zicheng Liao +1
School of Computing and Data Science, The University of Hong Kong · Hong Kong Generative AI Research and Development Center
Class-Incremental Learning (CIL) requires a model to continually learn new classes without forgetting old ones. A common and efficient solution freezes a pre-trained model and employs lightweight adapters, whose parameters are often forced to be orthogonal to prevent inter-task interference. However, we argue that this parameter-constraining method is detrimental to plasticity. To this end, we propose Semantic-Guided Dynamic Sparsification (SGDS), a novel method that proactively guides the activation space by governing the orientation and rank of its subspaces through targeted sparsification. Specifically, SGDS promotes knowledge transfer by encouraging similar classes to share a compact activation subspace, while simultaneously preventing interference by assigning non-overlapping activation subspaces to dissimilar classes. By sculpting class-specific sparse subspaces in the activation space, SGDS effectively mitigates interference without imposing rigid constraints on the parameter space. Extensive experiments on various benchmark datasets demonstrate the state-of-the-art performance of SGDS.
Ruiqi Liu, Boyu Diao, Zijia An +3
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Achieving continual learning (CL) with deep neural networks requires balancing stability and plasticity while enabling knowledge transfer. In this work, we focus on offline learning algorithms under the constraints: (I) no access to training data from prior tasks (II) no access to task-id at inference time. We introduce a novel measure, the relative parameter-importance, which measures the relative importance of each parameter with respect to both the current and past tasks. Parameters with high relative importance are interpreted as more important for maintaining past-task stability and thus heavily regularised, whereas parameters with low relative-importance are allowed to be more freely updated. Unlike existing methods, our approach allows the update of parameters with high past-task importance when they have low relative-importance, thus enabling backward knowledge transfer in addition to tackling the stability-plasticity trade-off. We demonstrate improvements against state-of-the-art CL methods on both class-incremental and domain-incremental learning text classification problems and provide insights for extending our method to text generation problems. Code available at: https://github.com/itsmemala/LACL