Double descent for least-squares interpolation on contaminated data: A simulation study

Tino Werner

cs.LGApr 15, 2026

Double descent for least-squares interpolation on contaminated data: A simulation study

Authors: Tino Werner

Organizations: Institute for Mathematics, Carl von Ossietzky University Oldenburg, Carl-von-Ossietzky-Straße 9-11, 26129 Oldenburg, Lower Saxony, Germany

Abstract

Overparametrized models can exhibit an excellent generalization performance, although they should be prone to overfitting according to classical statistical theory. The discovery of the "double descent", indicating that the generalization error decreases after a certain model complexity has been reached, opened a new line of research. Robust statistics considers statistical estimation on contaminated data, which, due to assumptions that do not hold on real data, let data points appear as outliers w.r.t. the assumed "ideal" distribution, potentially severely distorting any classical estimator. We address the question whether a double descent phenomenon can be observed in a linear regression setting with contaminated training data. We compare the performance of the highly non-robust least-squares interpolation estimator with several robust alternatives. It turns out that large overparametrization indeed allows for a double descent phenomenon, resulting in a very good generalization performance of the least-squares interpolator, surpassing that of the robust alternatives.

Figures & tables

$p$	$n_{train}$	$n_{test}$	$\SNR$	$\mu$	$\beta$	Cont.	$r$	$c_{out}$
Eq. 4.1	50	50	$\{0.1,0.5,2,5\}$	0	Normal/uniform	None, $X$ , $Y$	$\{0.1,0.25,0.5\}$	100

Table 1: Basic scenario specifications

Figure 1: Test MSE of minimum $l_{2}$ -norm interpolation when trained on clean training data.

Panel of Figure 2 — Figure 2: Test MSE of minimum $l_{2}$ -norm interpolation when trained on contaminated training data.

Figure 3: Test MSE of Huber-loss interpolation when trained on clean training data.

Panel of Figure 4 — Figure 4: Test MSE of Huber-loss interpolation when trained on contaminated training data.

Figure 5: Test MSE of Tukey-loss interpolation when trained on clean training data.

Panel of Figure 6 — Figure 6: Test MSE of Tukey-loss interpolation when trained on contaminated training data.

Figure 7: Test MSE of SLTS-based interpolation when trained on clean training data.

Figure 8: Test MSE of SLTS when trained on clean training data.

Panel of Figure 9 — Figure 9: Test MSE of SLTS-based interpolation when trained on contaminated training data.

Panel of Figure 10 — Figure 10: Test MSE of SLTS when trained on contaminated training data.

Figure 11: Test MSE of RRBoost-based interpolation when trained on clean training data.

Figure 12: Test MSE of RRBoost when trained on clean training data.

Panel of Figure 13 — Figure 13: Test MSE of RRBoost-based interpolation when trained on contaminated training data.

Panel of Figure 14 — Figure 14: Test MSE of RRBoost when trained on contaminated training data.

Figure 15: Test MSE of minimum $l_{2}$ -norm interpolation when trained on clean training data.

Panel of Figure 16 — Figure 16: Test MSE of minimum $l_{2}$ -norm interpolation when trained on contaminated training data.

Figure 17: Test MSE of Huber-loss interpolation when trained on clean training data.

Panel of Figure 18 — Figure 18: Test MSE of Huber-loss interpolation when trained on contaminated training data.

Figure 19: Test MSE of Tukey-loss interpolation when trained on clean training data.

Panel of Figure 20 — Figure 20: Test MSE of Tukey-loss interpolation when trained on contaminated training data.

Figure 21: Test MSE of SLTS-based interpolation when trained on clean training data.

Figure 22: Test MSE of SLTS when trained on clean training data.

Panel of Figure 23 — Figure 23: Test MSE of SLTS-based interpolation when trained on contaminated training data.

Panel of Figure 24 — Figure 24: Test MSE of SLTS when trained on contaminated training data.

Figure 25: Test MSE of RRBoost-based interpolation when trained on clean training data.

Figure 26: Test MSE of RRBoost when trained on clean training data.

Panel of Figure 27 — Figure 27: Test MSE of RRBoost-based interpolation when trained on contaminated training data.

Panel of Figure 28 — Figure 28: Test MSE of RRBoost when trained on contaminated training data.

Figure 29: Test MSE of minimum $l_{2}$ -norm interpolation when trained on clean training data with $\mu=5$ .

Panel of Figure 30 — Figure 30: Test MSE of minimum $l_{2}$ -norm interpolation when trained on contaminated training data with $\mu=5$ .

Figure 31: Test MSE of Huber-loss interpolation when trained on clean training data.

Panel of Figure 32 — Figure 32: Test MSE of Huber-loss interpolation when trained on contaminated training data.

Figure 33: Test MSE of SLTS-based interpolation when trained on clean training data.

Figure 34: Test MSE of SLTS when trained on clean training data.

Panel of Figure 35 — Figure 35: Test MSE of SLTS-based interpolation when trained on contaminated training data.

Panel of Figure 36 — Figure 36: Test MSE of SLTS when trained on contaminated training data.

Panel of Figure 37 — Figure 37: Test MSE of minimum $l_{2}$ -norm interpolation when trained on $Y$ -contaminated training data.

Panel of Figure 38 — Figure 38: Test MSE of Huber-loss interpolation when trained on $Y$ -contaminated training data.

Panel of Figure 39 — Figure 39: Test MSE of minimum $l_{2}$ -norm interpolation when trained on $Y$ -contaminated training data.

Panel of Figure 40 — Figure 40: Test MSE of Huber-loss interpolation when trained on $Y$ -contaminated training data.

Figure 41: Training MSE of minimum $l_{2}$ -norm interpolation when trained on clean training data.

Panel of Figure 42 — Figure 42: Training MSE of minimum $l_{2}$ -norm interpolation when trained on contaminated training data.

Figure 43: Training MSE of Huber-loss interpolation when trained on clean training data.

Panel of Figure 44 — Figure 44: Training MSE of Huber-loss interpolation when trained on contaminated training data.

Figure 45: Training MSE of Tukey-loss interpolation when trained on clean training data.

Panel of Figure 46 — Figure 46: Training MSE of Tukey-loss interpolation when trained on contaminated training data.

Figure 47: Training MSE of SLTS-based interpolation when trained on clean training data.

Figure 48: Training MSE of SLTS when trained on clean training data.

Panel of Figure 49 — Figure 49: Training MSE of SLTS-based interpolation when trained on contaminated training data.

Panel of Figure 50 — Figure 50: Training MSE of SLTS when trained on contaminated training data.

Figure 51: Training MSE of RRBoost-based interpolation when trained on clean training data.

Figure 52: Training MSE of RRBoost when trained on clean training data.

Panel of Figure 53 — Figure 53: Training MSE of RRBoost-based interpolation when trained on contaminated training data.

Panel of Figure 54 — Figure 54: Training MSE of RRBoost when trained on contaminated training data.

Figure 55: Training MSE of minimum $l_{2}$ -norm interpolation when trained on clean training data.

Panel of Figure 56 — Figure 56: Training MSE of minimum $l_{2}$ -norm interpolation when trained on contaminated training data.

Figure 57: Training MSE of Huber-loss interpolation when trained on clean training data.

Panel of Figure 58 — Figure 58: Training MSE of Huber-loss interpolation when trained on contaminated training data.

Figure 59: Training MSE of Tukey-loss interpolation when trained on clean training data.

Panel of Figure 60 — Figure 60: Training MSE of Tukey-loss interpolation when trained on contaminated training data.

Figure 61: Training MSE of SLTS-based interpolation when trained on clean training data.

Figure 62: Training MSE of SLTS when trained on clean training data.

Panel of Figure 63 — Figure 63: Training MSE of SLTS-based interpolation when trained on contaminated training data.

Panel of Figure 64 — Figure 64: Training MSE of SLTS when trained on contaminated training data.

Figure 65: Training MSE of RRBoost-based interpolation when trained on clean training data.

Figure 66: Training MSE of RRBoost-based interpolation when trained on clean training data.

Panel of Figure 67 — Figure 67: Training MSE of RRBoosting when trained on contaminated training data.

Panel of Figure 68 — Figure 68: Training MSE of RRBoosting when trained on contaminated training data.

Figure 69: Training MSE of minimum $l_{2}$ -norm interpolation when trained on clean training data.

Panel of Figure 70 — Figure 70: Training MSE of minimum $l_{2}$ -norm interpolation when trained on contaminated training data.

Figure 71: Training MSE of Huber-loss interpolation when trained on clean training data.

Panel of Figure 72 — Figure 72: Training MSE of Huber-loss interpolation when trained on contaminated training data.

Figure 73: Training MSE of SLTS-based interpolation when trained on clean training data.

Figure 74: Training MSE of SLTS when trained on clean training data.

Panel of Figure 75 — Figure 75: Training MSE of SLTS-based interpolation when trained on contaminated training data.

Panel of Figure 76 — Figure 76: Training MSE of SLTS when trained on contaminated training data.

Panel of Figure 77 — Figure 77: Training MSE of minimum $l_{2}$ -norm interpolation when trained on $Y$ -contaminated training data.

Panel of Figure 78 — Figure 78: Training MSE of Huber-loss interpolation when trained on $Y$ -contaminated training data.

Panel of Figure 79 — Figure 79: Training MSE of minimum $l_{2}$ -norm interpolation when trained on $Y$ -contaminated training data.

Panel of Figure 80 — Figure 80: Training MSE of Huber-loss interpolation when trained on $Y$ -contaminated training data.

Figure 81: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of minimum $l_{2}$ -norm interpolation when trained on clean training data and the true coefficient vector $\beta$ .

Panel of Figure 82 — Figure 82: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of minimum $l_{2}$ -norm interpolation when trained on contaminated training data and the true coefficient vector $\beta$ .

Figure 83: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of Huber-loss interpolation when trained on clean training data and the true coefficient vector $\beta$ .

Panel of Figure 84 — Figure 84: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of Huber-loss interpolation when trained on contaminated training data and the true coefficient vector $\beta$ .

Figure 85: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of Tukey-loss interpolation when trained on clean training data and the true coefficient vector $\beta$ .

Panel of Figure 86 — Figure 86: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of Tukey-loss interpolation when trained on contaminated training data and the true coefficient vector $\beta$ .

Figure 87: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of SLTS-based interpolation when trained on clean training data and the true coefficient vector $\beta$ .

Figure 88: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of SLTS when trained on clean training data and the true coefficient vector $\beta$ .

Panel of Figure 89 — Figure 89: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of SLTS-based interpolation when trained on contaminated training data and the true coefficient vector $\beta$ .

Panel of Figure 90 — Figure 90: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of SLTS when trained on contaminated training data and the true coefficient vector $\beta$ .

Figure 91: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of RRBoosting-based interpolation when trained on clean training data and the true coefficient vector $\beta$ .

Panel of Figure 92 — Figure 92: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of RRBoosting-based interpolation when trained on contaminated training data and the true coefficient vector $\beta$ .

Figure 93: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of minimum $l_{2}$ -norm interpolation when trained on clean training data and the true coefficient vector $\beta$ .

Panel of Figure 94 — Figure 94: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of minimum $l_{2}$ -norm interpolation when trained on contaminated training data and the true coefficient vector $\beta$ .

Figure 95: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of Huber-norm interpolation when trained on clean training data and the true coefficient vector $\beta$ .

Panel of Figure 96 — Figure 96: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of Huber-norm interpolation when trained on contaminated training data and the true coefficient vector $\beta$ .

Figure 97: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of Tukey-norm interpolation when trained on clean training data and the true coefficient vector $\beta$ .

Panel of Figure 98 — Figure 98: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of Tukey-norm interpolation when trained on contaminated training data and the true coefficient vector $\beta$ .

Figure 99: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of SLTS-based interpolation when trained on clean training data and the true coefficient vector $\beta$ .

Figure 100: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of SLTS when trained on clean training data and the true coefficient vector $\beta$ .

Panel of Figure 101 — Figure 101: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of SLTS-based interpolation when trained on contaminated training data and the true coefficient vector $\beta$ .

Panel of Figure 102 — Figure 102: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of SLTS when trained on contaminated training data and the true coefficient vector $\beta$ .

Figure 103: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of RRBoosting-based interpolation when trained on clean training data and the true coefficient vector $\beta$ .

Panel of Figure 104 — Figure 104: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of RRBoosting-based interpolation when trained on contaminated training data and the true coefficient vector $\beta$ .

Figure 105: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of minimum $l_{2}$ -norm interpolation when trained on clean training data and the true coefficient vector $\beta$ .

Panel of Figure 106 — Figure 106: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of minimum $l_{2}$ -norm interpolation when trained on contaminated training data and the true coefficient vector $\beta$ .

Figure 107: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of Huber-loss interpolation when trained on clean training data and the true coefficient vector $\beta$ .

Panel of Figure 108 — Figure 108: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of Huber-loss interpolation when trained on contaminated training data and the true coefficient vector $\beta$ .

Figure 109: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of SLTS-based interpolation when trained on clean training data and the true coefficient vector $\beta$ .

Figure 110: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of SLTS when trained on clean training data and the true coefficient vector $\beta$ .

Panel of Figure 111 — Figure 111: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of SLTS-based interpolation when trained on contaminated training data and the true coefficient vector $\beta$ .

Panel of Figure 112 — Figure 112: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of SLTS when trained on contaminated training data and the true coefficient vector $\beta$ .

Figure 113: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of RRBoosting-based interpolation when trained on clean training data and the true coefficient vector $\beta$ .

Panel of Figure 114 — Figure 114: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of RRBoosting-based interpolation when trained on contaminated training data and the true coefficient vector $\beta$ .

Panel of Figure 115 — Figure 115: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of minimum $l_{2}$ -norm interpolation when trained on $Y$ -contaminated training data.

Panel of Figure 116 — Figure 116: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of Huber-loss interpolation when trained on $Y$ -contaminated training data.

Panel of Figure 117 — Figure 117: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of minimum $l_{2}$ -norm interpolation when trained on $Y$ -contaminated training data.

Panel of Figure 118 — Figure 118: Differences $||\hat{\beta}-\beta||_{1}/n$ for the estimated coefficient vector $\hat{\beta}$ of Huber-loss interpolation when trained on $Y$ -contaminated training data.

Figure 119: Mean number of iterations of Huber-loss interpolation when trained on clean training data.

Figure 120: Mean number of iterations of Tukey-loss interpolation when trained on clean training data.

Panel of Figure 121 — Figure 121: Mean number of iterations of Huber-loss interpolation when trained on contaminated training data.

Panel of Figure 122 — Figure 122: Mean number of iterations of Tukey-loss interpolation when trained on contaminated training data.

Figure 123: Mean number of iterations of Huber-loss interpolation when trained on clean training data.

Panel of Figure 124 — Figure 124: Mean number of iterations of Huber-loss interpolation when trained on contaminated training data.

Figure 125: Mean number of iterations of Huber-loss interpolation when trained on clean training data.

Figure 126: Mean number of iterations of Tukey-loss interpolation when trained on clean training data.

Panel of Figure 127 — Figure 127: Mean number of iterations of Huber-loss interpolation when trained on contaminated training data.

Panel of Figure 128 — Figure 128: Mean number of iterations of Tukey-loss interpolation when trained on contaminated training data.

Panel of Figure 129 — Figure 129: Mean number of iterations of Huber-loss interpolation when trained on $Y$ -contaminated training data.

Panel of Figure 130 — Figure 130: Mean number of iterations of Huber-loss interpolation when trained on $Y$ -contaminated training data.

Double Descent

Double descent describes non-monotonic test error that can peak near interpolation and fall again as model capacity increases.

See all 20 papers

A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning
Sep 6, 2021Yehuda Dar, Vidya Muthukumar, Richard G. BaraniukDouble Descent Neural Network Generalization
Faculty of Computer and Information Science, Ben-Gurion University · Schools of Electrical and Computer Engineering, and Industrial and Systems Engineering, Georgia Institute of Technology · Electrical and Computer Engineering Department, Rice University
Benign Overfitting for General Norms and Distributions
Sep 27, 2026Daniel Barzilai, Ohad ShamirBenign Overfitting Statistical Learning Theory
Weizmann Institute of Science · University of Toronto & Vector Institute
How abundant are good interpolators?
Jun 4, 2026August Y. Chen, Ahmed El AlaouiLinear Classification Benign Overfitting
Cornell University, Department of Computer Science. Ithaca, NY, USA. · Cornell University, Department of Statistics and Data Science. Ithaca, NY, USA.

Sep 6, 2021J/K move · Enter open · S save

A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning

Yehuda Dar, Vidya Muthukumar, Richard G. Baraniuk

Faculty of Computer and Information Science, Ben-Gurion University · Schools of Electrical and Computer Engineering, and Industrial and Systems Engineering, Georgia Institute of Technology · Electrical and Computer Engineering Department, Rice University

Double Descent Neural Network Generalization Benign Overfitting Statistical Learning Theory Overparameterization

Open paper AskPDF arXiv

The last decade of progress in machine learning (ML), especially the deep learning era, has raised a number of scientific questions that challenge the longstanding dogma of the field. One of the most important riddles was the good empirical generalization of overparameterized models. Overparameterized models are highly complex with respect to the size of the training dataset, which enables them to perfectly fit (i.e., interpolate) even noisy training data. Such interpolation of noisy data is traditionally associated with detrimental overfitting, and yet a wide range of interpolating models -- from simple linear models to deep neural networks -- have been observed to generalize remarkably well on fresh test data. Indeed, the discovery of the double descent phenomenon has revealed that highly overparameterized models can improve over the best underparameterized model in test performance. Understanding learning in this overparameterized regime required new theory and foundational empirical studies, even for the simplest case of the linear model. The underpinnings of this understanding have been laid in foundational analyses of overparameterized linear regression and related statistical learning tasks, mostly published between 2018 and 2022, which resulted in precise analytic characterizations of double descent. This paper provides an overview of the theory of overparameterized ML (henceforth abbreviated as TOPML) by focusing on explaining the most foundational findings through a statistical signal processing perspective. We emphasize the unique aspects that define the TOPML research area as a subfield of modern ML theory and outline interesting open frontiers that remain.

Abstract

Double Descent

Explore similar work

A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning

Benign Overfitting for General Norms and Distributions

How abundant are good interpolators?