Ridge Regression

Ask about this topic

Overview

Ridge regression studies how shrinking least-squares coefficients with an L2 penalty can reduce prediction error and improve stability when features are correlated or the number of features is large. Work examines how to choose the penalty using cross-validation, SURE, and high-dimensional risk analysis, and how early stopping in gradient descent can act like ridge or other spectral regularization. Kernel ridge regression extends the approach to nonlinear prediction, with Nyström approximations and sketching used to reduce computational and memory costs. A central challenge is balancing variance reduction against bias: the best regularization depends on data geometry, signal and noise, while scalable methods must preserve predictive accuracy.

Latest papers

CardsList

31 papers6 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jun 6, 2024

On Regularization via Early Stopping for Least Squares Regression

A fundamental problem in machine learning is understanding the effect of early stopping on the parameters obtained and the generalization capabilities of the model. Even for linear models, the effect is not fully understood for arbitrary learning rates and data. In this paper, we analyze the dynamics of discrete full batch gradient descent for linear regression. With minimal distributional assumptions, we characterize the trajectory of the parameters and the expected excess risk. Using this characterization, we show that when training with any learning rate schedule and finite time horizon, the early stopped solution is equivalent to the minimum norm solution for a generalized ridge regression problem. We also prove that early stopping is beneficial for generic data with arbitrary spectrum and for a wide variety of learning rate schedules. We provide an estimate for the optimal stopping time and empirically demonstrate the accuracy of our estimate.

Rishi Sonthalia, Jackie Lok, Elizaveta Rebrova

Department of Mathematics, Boston College · ORFE Department, Princeton University

Early StoppingLinear RegressionRidge RegressionGradient Descent Dynamics
AskPDF