A Transport Geometry of Benign Overfitting

Olaf Yunus Laitinen Imanov; Rana Irem Turhan; Isgandar Jafarli

stat.MLApr 9, 2026

A Transport Geometry of Benign Overfitting

Authors: Olaf Yunus Laitinen Imanov, Rana Irem Turhan, Isgandar Jafarli

Organizations: School of ICT, Metropolia University of Applied Sciences, Myllypurontie 1, 00920, Helsinki, Finland

Abstract

Benign overfitting, the phenomenon that interpolating noisy data can be statistically harmless, is usually characterized through effective-rank conditions on the covariance spectrum. These conditions are fragile: a perturbation of vanishing relative operator and Frobenius norm can turn a benign covariance into one whose minimum-norm interpolator has non-vanishing excess risk. We argue that the natural geometry of benign overfitting is optimal transport of the log-eigenvalues, measured in units of the sample size $n$ . In log-spectral coordinates, the deterministic equivalent of the risk becomes a functional of two measures smoothed by a logistic kernel centred at the interpolation threshold $\logκ$ . We compute its exact first variation: the threshold moves by a sensitivity-weighted average of the log-eigenvalue displacements, and the influence of each eigenvalue decays exponentially with its log-distance to the threshold. We then derive global, dimension-free stability bounds. The variance functional is $8/(27n)$ -Lipschitz in the 1-Wasserstein distance between log-spectra, with a sharp constant, and is controlled more generally by a threshold-localized transport discrepancy; an analogous bound controls the bias. Consequently, any rank- $k$ update of the covariance, of arbitrary norm, changes the variance functional by at most $3k/n$ , and benign overfitting survives every perturbation whose localized transport cost is $o(n)$ . The results extend to ridge regression and to infinite-dimensional trace-class covariances. Matching lower bounds show that the $n$ -scaling is necessary and cannot be replaced by operator norms, Frobenius norms, or dimension-normalized spectral distances. Simulations, one-step feature learning in two-layer networks, and random features of real images support the theory and delineate its limits.

Figures & tables

Figure 1: (a) For

\lambda_{i}=1/i

(

p=2\cdot 10^{4}

n=200

), the exact gradient of the variance functional with respect to the log-eigenvalues ( Theorem 1 ) is concentrated in an

O(1)

window around the interpolation threshold

\log\kappa

; directions far above or far below the threshold are irrelevant. (b) A perturbation with

\|\Sigma_{n}-\Sigma_{n}^{\prime}\|_{\mathrm{op}}<1/n

(here shown without the unit spike of Theorem 4 , which changes

v

by at most

3/n

) turns benign overfitting (

v=1/n

) into tempered overfitting (

v\to\varphi^{-4}

). Lines: deterministic equivalent; markers: simulation of the minimum-norm interpolator (

30

draws per point; markers show

\bar{V}/(\sigma^{2}+\bar{V})

for the simulated variance

\bar{V}

Figure 2: (a) Starting from a benign isotropic covariance (

n=200

p=4000

v=0.05

), we replace

k

eigenvalues by

M\in\{10,10^{3},10^{6}\}

. Lines: DE; markers: simulation (

12

draws). The change in

v

saturates in

M

and stays below the bound

3k/n

of Corollary 3 , although the operator norm of the perturbation grows to

10^{6}

. (b) One giant gradient step on the first layer of a two-layer

\tanh

network (

d=128

N=1024

neurons,

n=256

). The feature covariance changes by up to

115\times

its operator norm, whereas the localized transport certificate

\inf_{c}\Psi_{c}/n

of Theorem 2 remains below

1

and tracks the actual change in

v

Perturbation	rank	rel. op	$\Psi_{0}/n$	$v$ (DE)	$v$ (sim.)
none (original features)	0	$0$	0.000	0.527	0.560 $\pm$ 0.006
flatten top-5 eigenvalues	5	$0.609$	0.000	0.527	0.554 $\pm$ 0.005
flatten top-20 eigenvalues	20	$0.883$	0.002	0.526	0.534 $\pm$ 0.003
flatten top-60 eigenvalues	60	$0.983$	0.055	0.509	0.483 $\pm$ 0.002
amplify top-5 by $10^{3}$	5	$999$	0.000	0.527	0.562 $\pm$ 0.007
amplify top-20 by $10^{3}$	20	$999$	0.002	0.527	0.775 $\pm$ 0.015

Table 1: Random ReLU features (

3000

) of the

1797

digits images, with the population taken as the uniform distribution over all images (

n=300

\log\kappa=-1.56

). Spectral surgeries act linearly on the real features in the population eigenbasis. “rel. op” is

\|\Sigma^{\prime}-\Sigma\|_{\mathrm{op}}/\|\Sigma\|_{\mathrm{op}}

;

\Psi_{0}/n

is the certificate of Theorem 2 (with

c=0

);

v

(sim.) is computed from

60

independent training sets.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Spectrum	$v$	$B$ (DE)	$V$ (DE)	risk (DE)	risk (sim.)
$\lambda_{i}=i^{-1/2}$	0.080	0.2894	0.0217	0.3111	0.3109 $\pm$ 0.0006
$\lambda_{i}=i^{-1}$	0.217	0.0148	0.0692	0.0840	0.0863 $\pm$ 0.0013
$\lambda_{i}=i^{-3/2}$	0.380	0.0008	0.1531	0.1538	0.1541 $\pm$ 0.0025
5 spikes $+$ flat tail	0.079	0.0089	0.0214	0.0303	0.0296 $\pm$ 0.0006

Appendix

Table 2: Deterministic equivalent versus simulation of the minimum-norm interpolator.

Spectrum	design	$\lambda$	risk (DE)	risk (sim.)
$\lambda_{i}=i^{-1}$	Gaussian	0	0.0822	0.0798 $\pm$ 0.0011
$\lambda_{i}=i^{-1}$	Gaussian	0.001	0.0787	0.0804 $\pm$ 0.0011
$\lambda_{i}=i^{-1}$	Gaussian	0.01	0.0584	0.0584 $\pm$ 0.0008
$\lambda_{i}=i^{-1}$	Rademacher	0	0.0822	0.0811 $\pm$ 0.0010
$\lambda_{i}=i^{-1}$	Rademacher	0.001	0.0787	0.0787 $\pm$ 0.0011
$\lambda_{i}=i^{-1}$	Rademacher	0.01	0.0584	0.0585 $\pm$ 0.0008

Appendix

Table 3: Deterministic equivalent (with

\kappa=\kappa_{\lambda}

, Proposition 3 (a)) versus simulation for Gaussian and Rademacher designs, ridgeless and ridge.

$c$	$\lambda_{\max}$	rel. op	$\sup\|N-N^{\prime}\|$	$\Psi/n$	$v$ (DE)	$v$ (sim.)
0	5.3	0.00	0	0.000	0.530	0.524 $\pm$ 0.005
0.5	43.2	8.03	140	0.048	0.524	0.523 $\pm$ 0.005
1	122.7	22.93	255	0.165	0.517	0.523 $\pm$ 0.005
2	266.8	49.91	299	0.338	0.509	0.517 $\pm$ 0.005
4	446.7	83.60	269	0.540	0.502	0.536 $\pm$ 0.005
8	614.9	115.11	230	0.727	0.496	0.652 $\pm$ 0.007

Appendix

Table 4: One-step feature learning. Columns: step size

c=\eta/\sqrt{N}

; top eigenvalue of the feature covariance; relative operator change;

\sup_{t}|N-N^{\prime}|

c=0

; certificate

\inf_{c}\Psi_{c}/n

;

v

by DE and by simulation.

Benign Overfitting

Benign overfitting studies how overparameterized models can fit noisy training data yet still predict accurately on new examples.

See all 21 papers

Benign Overfitting for General Norms and Distributions
Sep 27, 2026Daniel Barzilai, Ohad ShamirBenign Overfitting Statistical Learning Theory
Weizmann Institute of Science · University of Toronto & Vector Institute
Grokking through the Lens of Minimum-Norm Interpolation
Sep 29, 2026Gil Kur, Ileana Rugina, Clémentine Carla Juliette Dominé +1Statistical Learning Theory Implicit Regularization
ETH Zürich · Institute of Science and Technology Austria · Harvard University
Generalization error of min-norm interpolators in transfer learning
Jun 20, 2024Yanke Song, Kenneth Gu, Sohom Bhattacharya +1OOD Generalization Cross-Domain Transfer Learning
Department of Statistics, Harvard University · Department of Statistics, Stanford University · Department of Statistics, University of Florida

Sep 27, 2026J/K move · Enter open · S save

Benign Overfitting for General Norms and Distributions

Daniel Barzilai, Ohad Shamir

Weizmann Institute of Science · University of Toronto & Vector Institute

Benign Overfitting Statistical Learning Theory High-Dimensional Linear Regression

Open paper AskPDF arXiv

Understanding why predictors can generalize despite interpolating noisy training data is a central puzzle in machine learning. Most work on such "benign overfitting" studies minimum-2-norm linear regression, reflecting the inductive bias of gradient descent. However, modern optimizers such as Adam and Muon use non-Euclidean update geometries, favoring solutions associated with other norms. Analyzing regression for non-Euclidean norms is substantially more difficult, with known results essentially limited to Gaussians. In this paper, we develop a method to analyze benign overfitting in linear regression for general norms and general (sub-Gaussian) distributions. As a special case, we prove that minimum-p-norm interpolation with p>1 can benignly overfit even for non-Gaussian distributions, under suitable conditions. Perhaps surprisingly, for the 1-norm, benign overfitting does not hold in general for well-behaved (but non-Gaussian) distributions, showing that existing positive 1-norm results rely crucially on Gaussianity. Our proof analyzes the geometry of the dual optimization problem, using concentration and central limit tools to show it is approximately Euclidean in many high-dimensional cases.

Abstract

Benign Overfitting

Explore similar work

Benign Overfitting for General Norms and Distributions

Grokking through the Lens of Minimum-Norm Interpolation

Generalization error of min-norm interpolators in transfer learning