cs.LGOct 14, 2026

Optimal Explicit and Implicit Regularization Strengths in High-Dimensional Continual Linear Regression

Authors: Gilad Karpel, Daniel Soudry

Organizations: Department of Data and Decisions Sciences, Technion · Department of Electrical and Computer Engineering, Technion

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We study continual linear regression in a high-dimensional regime, where a learner observes a sequence of tasks and updates its parameters sequentially. In this setting, we derive an explicit closed-form expression for the generalization loss of a model trained with gradient descent for a fixed number of steps per task, which holds for arbitrary teachers. We prove that the optimal fixed step size decays nearly inversely with the number of tasks, specifically as ln⁡T/T\ln T / T, for a single teacher and, more generally, i.i.d. teachers. Under this scaling, we show that the resulting generalization loss is asymptotically equivalent to that achieved by optimally tuned L2 (isotropic) regularization. Furthermore, we extend our analysis to generalized ridge regression with diagonal regularization matrices, obtaining a closed-form characterization of the generalization loss and identifying necessary and sufficient conditions under which the optimal regularizer is isotropic. Finally, we identify a regime where the optimal step size remains finite even as the number of tasks tends to infinity, thus revealing a non-trivial tradeoff between task variability and implicit regularization, in contrast to the i.i.d. teacher scaling. Our theoretical predictions are corroborated by experiments on both linear models and neural networks, providing practical guidance for the design of continual learning systems.

Figures & tables

Figure 1: The plots provide empirical evidence for the Theorem 2 . Here, we set vz=1v_{z}=1 ; γ^⋆\hat{\gamma}^{\star} denotes the empirically optimal value, while γ⋆\gamma^{\star} is the theoretical prediction by Theorem 2 . We compute the empirical optimal γ^⋆\hat{\gamma}^{\star} , λ^⋆\hat{\lambda}^{\star} with an exhaustive search. Empirical curves are averaged over 50 runs.
Figure 2: The optimal step size remains finite as T→∞T\to\infty . We plot the optimal step size γ⋆\gamma^{\star} of the objective G∞G_{\infty} as a function of the ratio () and the frequency ( ff ), fixing ∥B∥=1\left\|\mathbf{B}\right\|=1 , vz=1v_{z}=1 , vx=1v_{x}=1 , d=100d=100 and N=5N=5 . We compute γ⋆\gamma^{\star} with an exhaustive search.
Figure 3: Optimal step size scaling in linear models and neural networks (NN). We plot the effect of step size on the mean squared error (MSE) after learning TT tasks. Curves are averaged over 5 and 15 random seeds for the linear model and neural network, respectively. The star marks the minimizer of the mean curve; vertical bars indicate the 95% credible interval for the minimizer, and the black line indicates a fitted ln⁡T/T\ln T/T scaling. The empirical MSE is normalized by the maximum MSE across all step sizes for the specific task horizon TT .
Figure 4: Optimal isotropic regularization conditions. We plot the mean MSE with varying regularization strengths and the optimal diagonal regularizer obtained by optimizing the theoretical objective in Theorem 3 . Error bars denote the standard deviation across 50 random seeds.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Continual linear regression with early-stopped GD (i.i.d. teachers).
Figure 6: Continual linear regression with early-stopped GD (sinusoidal teachers).
Figure 7: Continual Generalized L2 Regularization (ascending entries).
Figure 8: Continual Generalized L2 Regularization (descending entries).
Figure 9: Explicit regularization does not replicate implicit regularization in continual learning. We plot the optimal regularization parameter λ⋆\lambda^{\star} as a function of the number of tasks TT for different numbers of gradient steps NN . We compute λ⋆\lambda^{\star} with an exhaustive search.
More in this topic

High-Dimensional Linear Regression

See all 16 papers

Explore similar work

CardsList

  1. On Regularization via Early Stopping for Least Squares Regression

    Jun 6, 2024Rishi Sonthalia, Jackie Lok, Elizaveta RebrovaEarly StoppingLinear Regression

    Department of Mathematics, Boston College · ORFE Department, Princeton University

  2. Convergence of Continual Learning in Homogeneous Deep Networks

    Jun 29, 2026Matan Schliserman, Gon Buzaglo, Itay Evron +1Continual Learning

    Blavatnik School of Computer Science and AI, Tel Aviv University · Princeton University · Department of Electrical and Computing Engineering, Technion

  3. The Sequential Price of Continual Learning

    Aug 30, 2026Zonghuan Xu, Xingjun MaContinual LearningElastic Weight Consolidation

    Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China