Overview

Kernel ridge regression (KRR) fits nonlinear functions in a reproducing-kernel Hilbert space by balancing data fit against a norm penalty, making predictions from similarities between examples. Work studies how kernel spectra, target smoothness, input dimension, and alignment shape learning curves, interpolation, benign overfitting, and generalization. To scale Gram-matrix methods, approaches use Nyström sampling with ridge leverage scores, random features, sketching, and preconditioned iterative solvers; other variants learn or reweight kernels or add unpenalized features. KRR is also used for dynamical systems, treatment-effect estimation, and covariate shift, while a central tension is preserving statistical guarantees and useful inductive bias as kernels and datasets become more complex.

Latest papers

33 papers2 papers in the last four weeks, down 33% on the four weeks before. 0.0% of all new papers.

Oct 2, 2025

Learning Multi-Index Models with Hyper-Kernel Ridge Regression

Deep neural networks excel in high-dimensional problems, outperforming models such as kernel methods, which suffer from the curse of dimensionality. However, the theoretical foundations of this success remain poorly understood. We follow the idea that the compositional structure of the learning task is the key factor determining when deep networks outperform other approaches. Taking a step towards formalizing this idea, we consider a simple compositional model, namely the multi-index model (MIM). In this context, we introduce and study hyper-kernel ridge regression (HKRR), an approach blending neural networks and kernel methods. Our main contribution is a sample complexity result demonstrating that HKRR can adaptively learn MIM, overcoming the curse of dimensionality. Further, we exploit the kernel nature of the estimator to develop ad hoc optimization approaches. Indeed, we contrast alternating minimization and alternating gradient methods both theoretically and numerically. These numerical results complement and reinforce our theoretical findings.
Sep 17, 2025

A Compositional Kernel Model for Feature Learning

We study a compositional variant of kernel ridge regression in which the predictor is applied to a coordinate-wise reweighting of the inputs. Formulated as a variational problem, this model provides a tractable setting for studying feature learning in compositional architectures. From the perspective of variable selection, we show how relevant variables are recovered while noise variables are eliminated. We prove that both global minimizers and stationary points discard noise coordinates when the noise variables are Gaussian distributed. A central finding is that ℓ1\ell_1-type kernels, such as the Laplace kernel, succeed in recovering features contributing to nonlinear effects at stationary points, whereas Gaussian kernels recover only linear ones.
Oct 3, 2024

Highly Adaptive Ridge

In this paper we propose the Highly Adaptive Ridge (HAR): a regression method that achieves an n−1/3n^{-1/3} dimension-free L2\mathscr L_2 convergence rate (up to log factors) in the class of right-continuous functions with square-integrable sectional derivatives. This is a large nonparametric function class that is particularly appropriate for tabular data. For squared error loss, HAR is exactly kernel ridge regression with a specific data-adaptive kernel based on a saturated zero-order tensor-product spline basis expansion, and thus scales reasonably well, computationally. We prove close connections between HAR and kernel ridge regression in Sobolev spaces of dominating mixed derivatives, and highlight practical differences. We use simulation and real data to confirm our theory.