We study continual linear regression in a high-dimensional regime, where a learner observes a sequence of tasks and updates its parameters sequentially. In this setting, we derive an explicit closed-form expression for the generalization loss of a model trained with gradient descent for a fixed number of steps per task, which holds for arbitrary teachers. We prove that the optimal fixed step size decays nearly inversely with the number of tasks, specifically as lnT/T, for a single teacher and, more generally, i.i.d. teachers. Under this scaling, we show that the resulting generalization loss is asymptotically equivalent to that achieved by optimally tuned L2 (isotropic) regularization. Furthermore, we extend our analysis to generalized ridge regression with diagonal regularization matrices, obtaining a closed-form characterization of the generalization loss and identifying necessary and sufficient conditions under which the optimal regularizer is isotropic. Finally, we identify a regime where the optimal step size remains finite even as the number of tasks tends to infinity, thus revealing a non-trivial tradeoff between task variability and implicit regularization, in contrast to the i.i.d. teacher scaling. Our theoretical predictions are corroborated by experiments on both linear models and neural networks, providing practical guidance for the design of continual learning systems.
Figures & tables
Figure 1: The plots provide empirical evidence for the Theorem 2 . Here, we set vz=1 ; γ^⋆ denotes the empirically optimal value, while γ⋆ is the theoretical prediction by Theorem 2 . We compute the empirical optimal γ^⋆ , λ^⋆ with an exhaustive search. Empirical curves are averaged over 50 runs.
Figure 2: The optimal step size remains finite as T→∞ . We plot the optimal step size γ⋆ of the objective G∞ as a function of the ratio () and the frequency ( f ), fixing ∥B∥=1 , vz=1 , vx=1 , d=100 and N=5 . We compute γ⋆ with an exhaustive search.
Figure 3: Optimal step size scaling in linear models and neural networks (NN). We plot the effect of step size on the mean squared error (MSE) after learning T tasks. Curves are averaged over 5 and 15 random seeds for the linear model and neural network, respectively. The star marks the minimizer of the mean curve; vertical bars indicate the 95% credible interval for the minimizer, and the black line indicates a fitted lnT/T scaling. The empirical MSE is normalized by the maximum MSE across all step sizes for the specific task horizon T .
Figure 4: Optimal isotropic regularization conditions. We plot the mean MSE with varying regularization strengths and the optimal diagonal regularizer obtained by optimizing the theoretical objective in Theorem 3 . Error bars denote the standard deviation across 50 random seeds.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Continual linear regression with early-stopped GD (i.i.d. teachers).
Figure 6: Continual linear regression with early-stopped GD (sinusoidal teachers).
Figure 9: Explicit regularization does not replicate implicit regularization in continual learning. We plot the optimal regularization parameter λ⋆ as a function of the number of tasks T for different numbers of gradient steps N . We compute λ⋆ with an exhaustive search.
A fundamental problem in machine learning is understanding the effect of early stopping on the parameters obtained and the generalization capabilities of the model. Even for linear models, the effect is not fully understood for arbitrary learning rates and data. In this paper, we analyze the dynamics of discrete full batch gradient descent for linear regression. With minimal distributional assumptions, we characterize the trajectory of the parameters and the expected excess risk. Using this characterization, we show that when training with any learning rate schedule and finite time horizon, the early stopped solution is equivalent to the minimum norm solution for a generalized ridge regression problem. We also prove that early stopping is beneficial for generic data with arbitrary spectrum and for a wide variety of learning rate schedules. We provide an estimate for the optimal stopping time and empirically demonstrate the accuracy of our estimate.
Rishi Sonthalia, Jackie Lok, Elizaveta Rebrova
Department of Mathematics, Boston College · ORFE Department, Princeton University
We characterize weakly regularized continual classification in homogeneous models as sequential projections onto task margin sets. This result generalizes prior analyses restricted to either stationary (single-task) deep models or continual linear models. We show that global convergence generally fails, even for simple models linear in data but nonlinear in parameters. Nevertheless, by leveraging results from nonconvex projection theory, we identify regularity properties of homogeneous deep networks that guarantee local linear convergence under random and cyclic task sequences. Finally, we extend our analysis to continual regression, unifying the framework for homogeneous models.
Matan Schliserman, Gon Buzaglo, Itay Evron +1
Blavatnik School of Computer Science and AI, Tel Aviv University · Princeton University · Department of Electrical and Computing Engineering, Technion
Sequential task updates are fundamental to continual learning, but their recency bias can impose a lasting performance cost. We study this cost in an overparameterized linear-regression model with i.i.d. task sampling. We prove that distribution-level forgetting and population loss converge to the same stationary limit. We quantify the additional loss incurred by sequential exact fitting, or the sequential price. In more homogeneous task geometries, it equals the intrinsic loss asymptotically attained by joint training, making the total loss twice as large. We further analyze fixed-strength elastic weight consolidation (EWC) under general task curvatures and characterize its stationary sequential price at every regularization strength. Under strong regularization, the price decays inversely with EWC strength while the mean-square coupling horizon grows proportionally. Experiments on Jester and Rotated MNIST support the predicted sequential price and its reduction by EWC, with quantitative agreement on real-world tasks satisfying the theory's assumptions and qualitative agreement under nonlinear finite-step training.
Zonghuan Xu, Xingjun Ma
Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China