Optimal L2 Regularization in High-dimensional Continual Linear Regression

Gilad Karpel; Edward Moroshko; Ran Levinstein; Ron Meir; Daniel Soudry; Itay Evron

cs.LGJan 20, 2026

Optimal L2 Regularization in High-dimensional Continual Linear Regression

Authors: Gilad Karpel, Edward Moroshko, Ran Levinstein, Ron Meir, Daniel Soudry, Itay Evron

Organizations: Department of Data and Decisions Sciences, Technion · School of Engineering, University of Edinburgh · Department of Computer Science, Technion · Department of Electrical and Computing Engineering, Technion ·

Abstract

We study generalization in an overparameterized continual linear regression setting, where a model is trained with L2 (isotropic) regularization across a sequence of tasks. We derive a closed-form expression for the expected generalization loss in the high-dimensional regime that holds for arbitrary linear teachers. We demonstrate that isotropic regularization mitigates label noise under both single-teacher and multiple i.i.d. teacher settings, whereas prior work accommodating multiple teachers either did not employ regularization or used memory-demanding methods. Furthermore, we prove that the optimal fixed regularization strength scales nearly linearly with the number of tasks $T$ , specifically as $T/\ln T$ . To our knowledge, this is the first such result in theoretical continual learning. Finally, we validate our theoretical findings through experiments on linear regression and neural networks, illustrating how this scaling law affects generalization and offering a practical recipe for the design of continual learning systems.

Figures & tables

Panel of Figure 1 — (a) Regularization effects on synthetic data. Interactions between number of tasks ( $T$ ), regularization strength ( $\lambda$ ), and resulting generalization loss ( normalized by a trivial regressor $\mathbf{w}=\mathbf{0}$ ).Here, the label noise and feature variances are $v_{z}=v_{x}=1$ ; Each curve averages over $5$ runs.

Regularization-Based Continual Learning

See all 20 papers

Optimal Explicit and Implicit Regularization Strengths in High-Dimensional Continual Linear Regression
Oct 14, 2026Gilad Karpel, Daniel SoudryHigh-Dimensional Linear Regression Regularization-Based Continual Learning
Department of Data and Decisions Sciences, Technion · Department of Electrical and Computer Engineering, Technion
Score-Preserving Continual Lasso under Task Shift
Oct 23, 2026Jingyi Guo, Runze Li, Lingzhou XueContinual Learning Regularization-Based Continual Learning
Pennsylvania State University
Convergence of Continual Learning in Homogeneous Deep Networks
Jun 29, 2026Matan Schliserman, Gon Buzaglo, Itay Evron +1Continual Learning
Blavatnik School of Computer Science and AI, Tel Aviv University · Princeton University · Department of Electrical and Computing Engineering, Technion

Oct 14, 2026J/K move · Enter open · S save

Optimal Explicit and Implicit Regularization Strengths in High-Dimensional Continual Linear Regression

Gilad Karpel, Daniel Soudry

Department of Data and Decisions Sciences, Technion · Department of Electrical and Computer Engineering, Technion

High-Dimensional Linear Regression Regularization-Based Continual Learning Online Continual Learning Implicit Regularization Statistical Learning Theory Ridge Regression

Open paper AskPDF arXiv

We study continual linear regression in a high-dimensional regime, where a learner observes a sequence of tasks and updates its parameters sequentially. In this setting, we derive an explicit closed-form expression for the generalization loss of a model trained with gradient descent for a fixed number of steps per task, which holds for arbitrary teachers. We prove that the optimal fixed step size decays nearly inversely with the number of tasks, specifically as

\ln T / T

, for a single teacher and, more generally, i.i.d. teachers. Under this scaling, we show that the resulting generalization loss is asymptotically equivalent to that achieved by optimally tuned L2 (isotropic) regularization. Furthermore, we extend our analysis to generalized ridge regression with diagonal regularization matrices, obtaining a closed-form characterization of the generalization loss and identifying necessary and sufficient conditions under which the optimal regularizer is isotropic. Finally, we identify a regime where the optimal step size remains finite even as the number of tasks tends to infinity, thus revealing a non-trivial tradeoff between task variability and implicit regularization, in contrast to the i.i.d. teacher scaling. Our theoretical predictions are corroborated by experiments on both linear models and neural networks, providing practical guidance for the design of continual learning systems.

Abstract

Regularization-Based Continual Learning

Explore similar work

Optimal Explicit and Implicit Regularization Strengths in High-Dimensional Continual Linear Regression

Score-Preserving Continual Lasso under Task Shift

Convergence of Continual Learning in Homogeneous Deep Networks