From Cross-Validation to SURE: Asymptotic Risk of Tuned Regularized Estimators

Karun Adusumilli; Maximilian Kasy; Ashia Wilson

math.STMar 20, 2026

From Cross-Validation to SURE: Asymptotic Risk of Tuned Regularized Estimators

Authors: Karun Adusumilli, Maximilian Kasy, Ashia Wilson

Organizations: Department of Economics, University of Pennsylvania. · Department of Economics, University of Oxford. · Department of Electrical Engineering and Computer Science, MIT.

Abstract

We derive the asymptotic risk function of regularized empirical risk minimization (ERM) estimators tuned by $n$ -fold cross-validation (CV). The out-of-sample prediction loss of such estimators converges in distribution to the squared-error loss (risk function) of shrinkage estimators in the normal means model, tuned by Stein's unbiased risk estimate (SURE). This risk function provides a more fine-grained picture of predictive performance than uniform bounds on worst-case regret, which are common in learning theory: it quantifies how risk varies with the true parameter. As key intermediate steps, we show that (i) $n$ -fold CV converges uniformly to SURE, and (ii) while SURE typically has multiple local minima, its global minimum is generically well separated. Well-separation ensures that uniform convergence of CV to SURE translates into convergence of the tuning parameter chosen by CV to that chosen by SURE.

Figures & tables

Panel of Figure 1 — Figure 1: Risk function for JS-shrinkage, dimension 10

Panel of Figure 2 — Figure 2: Examples of multi-modality of $SURE$

Result	Assumptions invoked	Builds on
Lemma 1 (Lipschitz $g^{\lambda}$ )	convexity of $\pi$	—
Lemma 2 (Influence-function approx.)	1, 2, 4.1, 4.2, 5.2	Lemma 1
Lemma 3 (Limiting squared-error loss)	1, 2, 5.3	—
Corollary 2 (Asymptotic distribution)	1, 4.1	Lemmas 1 , 2
Lemma 4 (CV $\to$ SURE)	1, 2, 3, 5.1–5.4	Lemmas 1 , 2 , Cor. 2
Lemma 5 (Tuning-parameter convergence)	1, 3	Lemmas 2 , 4 , Cor. 2

Table 1: Assumptions invoked by each result.

	Loss	Penalty	$p$	$\theta_{0}$	$A$
D1	Linear	Ridge	10	$(1,\dots,1)$	$I$
D2	Linear	Lasso	10	$(3,-3,2,0,\dots,0)$	$I$
D3	Logit	Ridge	10	$(1,\dots,1)$	$I$
D4	Logit	Lasso	10	$(3,-3,2,0,\dots,0)$	$I$
D5	Linear	Ridge	10	$1.5\cdot(1,\dots,1)$	$\mathrm{diag}(40^{(j-1)/9})$

Table 2: Simulation designs.

Design	$\Delta_{\mathrm{IF}}$	$\Delta_{\mathrm{CV}}$	$\Delta_{\mathrm{tune}}$	$R_{n}$	$R$	$\Delta_{R}$
	(Lem. 2)	(Lem. 4)	(Lem. 5)	(Thm. 1)	(limit)	$\|R_{n}-R\|$
D1 Linear / Ridge	0.028	0.208	0.052	3.110	3.084	0.026
D2 Linear / Lasso (sparse)	0.028	0.226	0.111	3.885	3.789	0.096
D3 Logit / Ridge	0.032	0.309	0.055	3.062	3.075	0.013
D4 Logit / Lasso (sparse)	0.038	0.297	0.127	3.731	3.776	0.046
D5 Linear / Ridge (anisotropic)	0.028	0.152	0.036	4.827	4.733	0.094

Table 3: Approximation diagnostics at

n=400

Figure 3: Convergence of the approximations as

n

grows (designs identified in the legend panel, lower right). Top left: the influence-function error

\Delta_{\mathrm{IF}}

(Lemma 2 ),

\propto n^{-1}

. Top right: the cross-validation gap

\Delta_{\mathrm{CV}}

(Lemma 4 ),

\propto n^{-1/2}

. Bottom left: the finite-sample risk

R_{n}

converging to the limit

R

(dotted lines; Theorem 1 ).

Figure 4: Convergence of the risk function (linear–Ridge design). Along the slice

\theta_{0}=t\cdot v

(with

v

a fixed unit vector, so

\|\theta_{0}\|=t

), the finite-sample risk

R_{n}(\theta_{0})

of the CV-tuned estimator approaches the SURE-tuned limit

R(\theta_{0})

; by

n=200

the curves are nearly indistinguishable from the limit.

Explore similar work

A Researcher's Guide to Empirical Risk Minimization
Feb 25, 2026Lars van der LaanEmpirical Risk Minimization Statistical Learning Theory
Department of Statistics, University of Washington
A Simple Approximation to the Distribution of the Ridge Regression Estimator
Aug 3, 2026José Luis Montiel Olea, Ryan Strong, Amilcar Velez +2Ridge Regression Hyperparameter Optimization
Department of Economics at Cornell University
Distribution-free Deviation Bounds and The Role of Domain Knowledge in Learning via Model Selection with Cross-validation Risk Estimation
Mar 15, 2023Diego Marcondes, Cláudia PeixotoModel Selection Statistical Learning Theory

Feb 25, 2026J/K move · Enter open · S save

A Researcher's Guide to Empirical Risk Minimization

Lars van der Laan

Department of Statistics, University of Washington

Empirical Risk Minimization Statistical Learning Theory

Open paper AskPDF arXiv

This guide provides a reference for high-probability regret bounds in empirical risk minimization (ERM). The presentation is modular: we begin with intuition and general proof strategies, then state broadly applicable guarantees under high-level conditions and provide tools for verifying them for specific losses and function classes. We emphasize that many ERM rate derivations can be organized around a three-step recipe -- a basic inequality, a uniform local concentration bound, and a fixed-point argument -- which yields regret bounds in terms of a critical radius, defined via localized Rademacher complexity, under a mild Bernstein-type variance-risk condition. To make these bounds concrete, we upper bound the critical radius using local maximal inequalities and metric-entropy integrals, thereby recovering familiar rates for VC-subgraph, Sobolev/Hölder, and bounded-variation classes. We also study ERM with nuisance components -- including weighted ERM and Neyman-orthogonal losses -- as they arise in causal inference, missing data, and domain adaptation. Following the orthogonal statistical learning framework, we highlight that these problems often admit regret-transfer bounds linking regret under an estimated loss to population regret under the target loss. These bounds typically decompose the regret into (i) statistical error under the estimated loss and (ii) approximation error due to nuisance estimation. Under sample splitting or cross-fitting, the first term can be controlled using standard fixed-loss ERM regret bounds, while the second depends only on nuisance-estimation accuracy. As a novel contribution, we also treat the in-sample regime, in which the nuisances and the ERM are fit on the same data, deriving regret bounds and showing that fast oracle rates remain attainable under suitable smoothness and Donsker-type conditions.

Abstract

Explore similar work

A Researcher's Guide to Empirical Risk Minimization

A Simple Approximation to the Distribution of the Ridge Regression Estimator

Distribution-free Deviation Bounds and The Role of Domain Knowledge in Learning via Model Selection with Cross-validation Risk Estimation