Benign Overfitting

Benign overfitting describes the surprising ability of highly overparameterized models to perfectly fit noisy training data while still generalizing well to unseen data, challenging traditional bias-variance trade-off assumptions. Current research focuses on understanding this phenomenon in various model architectures, including linear models, kernel methods, and neural networks (particularly transformers and convolutional networks), often analyzing the impact of training algorithms like gradient descent and implicit regularization. This research is significant because it helps explain the success of modern deep learning and may lead to improved model design and training strategies, ultimately enhancing the reliability and performance of machine learning systems.

Papers

February 10, 2022

Benign-Overfitting in Conditional Average Treatment Effect Prediction with Linear Regression
Masahiro Kato, Masaaki Imaizumi
Causal Inference Linear Regression Conditional Average Treatment Effect Benign Overfitting Causal Prediction

January 27, 2022

The Implicit Bias of Benign Overfitting
Ohad Shamir
Training Data Implicit Bias Benign Overfitting Margin Classifier Linear Predictor

December 31, 2021

Benign Overfitting in Adversarially Robust Linear Classification
Jinghui Chen, Yuan Cao, Quanquan Gu
Adversarial Example Adversarial Training Adversarial Perturbation Benign Overfitting Adversarial Risk Robust Classification

November 9, 2021

Harmless interpolation in regression and classification with structured features
Andrew D. McRae, Santhosh Karnik, Mark A. Davenport, Vidya Muthukumar
Classification Code Novel Regression Interpolation Regime Confidence Bound Benign Overfitting Orthonormal Basis

Benign Overfitting

Papers

Benign-Overfitting in Conditional Average Treatment Effect Prediction with Linear Regression

The Implicit Bias of Benign Overfitting

Benign Overfitting in Adversarially Robust Linear Classification

Harmless interpolation in regression and classification with structured features