Continual learning aims to acquire new knowledge while retaining previously learned knowledge. However, updating shared parameters when learning new tasks often overwrites existing knowledge, causing catastrophic forgetting. Memory replay methods mitigate catastrophic forgetting but rely on previous samples, raising privacy, memory, and scalability concerns. On the other hand, parameter isolation-based methods preserve learned knowledge but may limit knowledge transfer, whereas regularization promotes adaptability but remains susceptible to interference between old and new tasks. To maintain the stability-plasticity trade-off, this paper proposes Stability-Plasticity-Aware Region Selection (SPARS), a replay-free hybrid framework combining global parameter isolation with localized parameter regularization. SPARS estimates parameter importance using either the Fisher Information Matrix (FIM) or a novel Confidence Score (CS) estimator that leverages logit sensitivity and achieves comparable performance to FIM without requiring labelled data. Based on parameter importance, SPARS partitions parameters into three importance-aware regions namely, winning, neighboring, and non-important parameter regions. Each region receives a different adaptation strategy: selective freezing, soft regularization, or unrestricted learning based on its role in the stability-plasticity trade-off. To mitigate classifier bias in class-incremental learning, we employ Quadratic Discriminant Analysis to improve performance on previously learned classes .Experiments on CIFAR-10 and TinyImageNet demonstrate consistent improvement in average incremental accuracy over existing isolation- and regularization-based methods across task- and class-incremental learning. On TinyImageNet, SPARS achieves the best performance with 6% improvement for class-incremental learning while retaining over 60% of parameters for future tasks.
Figures & tables
Figure 1: Overview of the SPARS Framework. The framework consists of parameter importance estimation, Top-k parameter selection, partitioning model parameters into winners (W), neighboring (N), and non-important regions, followed a region-aware parameter adaptation strategy.
Figure 2: Distribution of Fisher importance scores across all model parameters, showing a long-tailed distribution in which only a small subset of parameters has high importance.
Dataset
Train
Test
TIL and CIL Splits
CIFAR-10
50,000
10,000
5 tasks, each with 2 classes
TinyImageNet
100,000
10,000
10 tasks, each with 20 classes
Table 1: Dataset statistics and continual learning task splits used in our experiments.
Figure 3: Per-task Average Accuracy of QDA and LDA across the tasks for CIL on (a) CIFAR-10 and (b) TinyImageNet.
Figure 4: PCA visualization of feature embeddings extracted from the backbone network after tasks 2, 6, and 10 under the CIL setting. Light-colored samples denote earlier feature locations, while darker samples represent their updated positions after subsequent tasks. Arrows visualize the direction and magnitude of feature drift over time.
Top-k
100K
300K
500K
700K
AIA
36.02
36.13
36.14
36.11
AF
15.86
15.48
14.60
14.77
Table 3: AIA and AF across different top- k sizes for CIL on TinyImageNet. Higher AIA and lower AF indicate better performance across all 10 tasks. The best result is achieved at top- k=500k .
Figure 5: Comparison between the number of unique top-(k) parameters across 10 tasks on TinyImageNet under CIL setting, the number of unique neighboring parameters accumulated across tasks, and the overlap among the top-(k) parameters identified across tasks. The blue curve shows the ratio of the combined top-(k) and neighboring parameters to the total number of model parameters.
Figure 6: AIA and AF under different values of λ on TinyImageNet in the CIL setting: λ=0 (no neighbors), λ=2 , λ=100 , and λ=1000 .
Figure 7: Distributions of FIM and CS (log10 scale) for the top- k winner parameters and their neighboring parameters on CIFAR-10 in the CIL setting.
Figure 8: Layer-wise distribution of top-k winner and neighboring parameters across convolutional layers.
Continual learning must balance the learning of new knowledge with the retention of previously learned knowledge to incrementally learn tasks from a data stream without catastrophic forgetting. While leveraging pretrained models has significantly advanced continual learning, existing methods exhibit a scalability bottleneck when trained sequentially on many tasks, suffering from performance degradation due to inter-task interference and loss of plasticity. Inspired by evidence that sparse fine-tuning achieves performance comparable to full fine-tuning, this paper presents a novel sparsity-driven continual learning framework. Our continual learning method, termed CLARE, operates in two stages: it first identifies a sparse, task-critical parameter mask via a sparsity-inducing objective, then performs mask-constrained fine-tuning by only optimizing parameters selected by the mask. This two-stage sparse adapter mechanism enables all tasks to be accumulated within a shared adapter space while reducing destructive interference across tasks. Extensive experiments demonstrate the scalability of CLARE. On the long task-sequence benchmark Omnibenchmark-1k, CLARE outperforms strong baselines in final accuracy by a large margin, e.g, improving EASE by 4.64% and 13.34% after learning 100 tasks, respectively.
Yunxiang Fu, Meng Lou, Zicheng Liao +1
School of Computing and Data Science, The University of Hong Kong · Hong Kong Generative AI Research and Development Center
Achieving continual learning (CL) with deep neural networks requires balancing stability and plasticity while enabling knowledge transfer. In this work, we focus on offline learning algorithms under the constraints: (I) no access to training data from prior tasks (II) no access to task-id at inference time. We introduce a novel measure, the relative parameter-importance, which measures the relative importance of each parameter with respect to both the current and past tasks. Parameters with high relative importance are interpreted as more important for maintaining past-task stability and thus heavily regularised, whereas parameters with low relative-importance are allowed to be more freely updated. Unlike existing methods, our approach allows the update of parameters with high past-task importance when they have low relative-importance, thus enabling backward knowledge transfer in addition to tackling the stability-plasticity trade-off. We demonstrate improvements against state-of-the-art CL methods on both class-incremental and domain-incremental learning text classification problems and provide insights for extending our method to text generation problems. Code available at: https://github.com/itsmemala/LACL
Analytic continual learning has emerged as a strong exemplar-free alternative to gradient-based class-incremental learning because it replaces iterative optimization with closed-form ridge updates. Yet the usual forgetting narrative, centered on stochastic gradient overwriting, does not explain why analytic methods still drift on old classes despite exact recursive solvers. We identify the culprit as spectral interference: the joint ridge classifier for all tasks shares the inverse autocorrelation operator (R+λI)−1, so incoming task samples that load onto old dominant eigendirections dilute the spectrum and perturb old-class logits even when old labels are never revisited. Based on this view, we propose SPARCL, a spectral partitioned analytic continual learner that decomposes the running autocorrelation into a high-energy core and a residual complement, freezes old-class classifier components in the core subspace, and updates only the residual block through recursive least squares with an optional residual random-projection expansion. This yields a simple closed-form update with a provable invariance guarantee for the core contribution of old logits. Across CIFAR-100, CUB-200, ImageNet-R, and ImageNet-A under a frozen ViT-B/16 protocol, SPARCL closes most of the gap from classical analytic learners to strong representation matchers, while remaining complementary to sparse feature-decorrelation approaches such as Fly-CL.