We study logistic regression for continual learning with linear predictors in a high-dimensional statistical setting. Existing theoretical work has largely focused on continual linear regression with variants of L2 regularization, motivated by their analytical simplicity and connection to the kernel regime of neural networks. We first derive the optimal L2 regularization strength for continual linear classification and characterize its generalization performance, recovering behavior analogous to prior results for continual linear regression. Next, we examine the effect of L2 regularization in deeper models using a minimal two-layer diagonal linear network. In the single-task setting, L2 regularization on the weights of such models is known to promote sparsity in the resulting predictor, yielding a rich, sparse feature-learning regime beyond the kernel regime (a similar effect arises in two-layer ReLU networks). However, in continual learning, where L2 regularization is typically centered on previous-task weights, this sparsity bias is no longer preserved. We therefore propose a weighted variant of L2 regularization that extends this beneficial sparsity bias to continual learning. We show theoretically and empirically that, for a sufficiently sparse teacher, our variant converges substantially faster than directly learning a linear predictor with conventional continual L2 or L1 regularization, trading some long-horizon bias for markedly improved performance over moderate task horizons.
Figures & tables
Figure 1: Coordinate-wise comparison of L1 and our sign-weighted L1 regularization.
Figure 2: Qualitative validation of Theorems 1 and 2 . Empirical values are averaged over 100 runs.
Figure 3: Effect of teacher sparsity across models and datasets. sign weighted L1 performs best at high teacher sparsity and remains competitive at moderate sparsity, while L2 becomes preferable as sparsity decreases.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Gaussian
Linear MNIST
ReLU MNIST
Dimension d
500
784
784
Tasks T
30
30
30
Samples/task n
1,000
1,000
1,000
Training pool
50,000
50,000
50,000
Validation
5,000
5,000
2,000
Test
5,000
10,000
8,000
Appendix
Table 1: Key parameters across the three experimental settings.
Figure 4: Temporal L2 does not exploit sparsity of the teacher. Excess risk L(wT)−infwL(w) after T=8 sequential tasks. The dimension d=256 , the per-task sample size n=128 (hence δ=n/d=0.5 ), the horizon T , the noise level ρ=0 and the signal strength ν=∥w⋆∥/d=0.25 are all held fixed. Error bars are one standard error over 3 seeds. The dashed line is the least-squares fit on the log-log scale, with slope +0.005 ( R2=0.11 ).
We study generalization in an overparameterized continual linear regression setting, where a model is trained with L2 (isotropic) regularization across a sequence of tasks. We derive a closed-form expression for the expected generalization loss in the high-dimensional regime that holds for arbitrary linear teachers. We demonstrate that isotropic regularization mitigates label noise under both single-teacher and multiple i.i.d. teacher settings, whereas prior work accommodating multiple teachers either did not employ regularization or used memory-demanding methods. Furthermore, we prove that the optimal fixed regularization strength scales nearly linearly with the number of tasks T, specifically as T/lnT. To our knowledge, this is the first such result in theoretical continual learning. Finally, we validate our theoretical findings through experiments on linear regression and neural networks, illustrating how this scaling law affects generalization and offering a practical recipe for the design of continual learning systems.
Gilad Karpel, Edward Moroshko, Ran Levinstein +3
Department of Data and Decisions Sciences, Technion · School of Engineering, University of Edinburgh · Department of Computer Science, Technion +2
We study continual linear regression in a high-dimensional regime, where a learner observes a sequence of tasks and updates its parameters sequentially. In this setting, we derive an explicit closed-form expression for the generalization loss of a model trained with gradient descent for a fixed number of steps per task, which holds for arbitrary teachers. We prove that the optimal fixed step size decays nearly inversely with the number of tasks, specifically as lnT/T, for a single teacher and, more generally, i.i.d. teachers. Under this scaling, we show that the resulting generalization loss is asymptotically equivalent to that achieved by optimally tuned L2 (isotropic) regularization. Furthermore, we extend our analysis to generalized ridge regression with diagonal regularization matrices, obtaining a closed-form characterization of the generalization loss and identifying necessary and sufficient conditions under which the optimal regularizer is isotropic. Finally, we identify a regime where the optimal step size remains finite even as the number of tasks tends to infinity, thus revealing a non-trivial tradeoff between task variability and implicit regularization, in contrast to the i.i.d. teacher scaling. Our theoretical predictions are corroborated by experiments on both linear models and neural networks, providing practical guidance for the design of continual learning systems.
Gilad Karpel, Daniel Soudry
Department of Data and Decisions Sciences, Technion · Department of Electrical and Computer Engineering, Technion
Continual learning (CL) aims to learn from sequential tasks while mitigating catastrophic forgetting when historical data are no longer accessible. We study continual learning for high-dimensional sparse linear regression, where ℓ1 regularization creates a difficulty absent from continual ridge regression: that recursively reusing an ℓ1-penalized estimator can propagate earlier shrinkage and accumulate regularization bias across tasks. We propose score-preserving continual Lasso (SP-CL), which separates sparse estimation from state transmission by preserving the historical squared-loss score. Under squared loss, its cumulative estimator exactly recovers a full-history Lasso minimizer. When task coefficients coincide, it inherits the full-history sparse rate. For related task-specific coefficients, SP-CL uses a regularized current-task correction that leaves the transmitted state unchanged and attains the offline source-assisted rate under an oracle related-task condition. We also derive Overall/BWT/FWT decompositions that distinguish historical information loss, task conflict, and current-task specialization, and validate the resulting regimes in simulation.