Neural Network Generalization

Follow this topic Ask about this topic

The field studies how neural networks move beyond fitting their training examples, including when overparameterized models can memorize data yet still perform well on new examples or shifted distributions. Theoretical approaches use Rademacher complexity, PAC-Bayes bounds, algorithmic stability, kernel and mean-field analyses, and high-dimensional statistics to relate model structure and training to test error. Empirical work examines implicit regularization from optimizers and architectures, representation learning, double descent, grokking, and robustness across settings such as images, language, graphs, and scientific models. A central open tension is that bounds and measures such as flatness do not reliably predict performance in every regime, which depends strongly on data structure, learned features, and distribution shift.

Latest papers

311 papers35 papers in the last four weeks, up 35% on the four weeks before. 0.2% of all new papers.

Universality of Benign Overfitting in Binary Linear Classification
Jan 17, 2025Ichiro Hashimoto, Stanislav Volgushev, Piotr ZwiernikNeural Network Generalization Linear Classification
Department of Statistical Sciences, University of Toronto
Network Dynamics-Based Framework for Understanding Deep Neural Networks
Jan 5, 2025Yuchen Lin, Yong Zhang, Sihan Feng +1Neural Network Generalization Neural Network Training Dynamics
Department of Physics, Xiamen University, Xiamen 361005, China · 2Lanzhou Center for Theoretical Physics, Lanzhou University, Lanzhou 730000, China · 3Shenzhen Angel FOF Management Co., Ltd., Shenzhen 518000, China
Bilateral Sharpness-Aware Minimization for Flatter Minima
Sep 20, 2024Jiaxin Deng, Junbiao Pang, Baochang Zhang +1Sharpness-Aware Minimization Neural Network Generalization
Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China · School of Artificial Intelligence, BeiHang University, Beijing, 100191, China · University of Chinese Academy of Sciences, Chinese Academy of Sciences (CAS), Beijing 100049, China +1
Sharp Generalization for Nonparametric Regression in Interpolation Space by Over-Parameterized Neural Networks Trained with Preconditioned Gradient Descent and Early Stopping
Jul 16, 2024Yingzhen Yang, Ping LiNeural Network Generalization Nonparametric Regression
School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ 85281, USA · VecML Inc., Bellevue, WA 98004, USA
A Generalization Bound for Nearly-Linear Networks
Jul 9, 2024Eugene GolikovNeural Network Generalization Generalization Bounds
Chair of Statistical Field Theory École Polytechnique Fédérale de Lausanne (EPFL)
Instance-Conditioned Adaptation for Large-scale Generalization of Neural Routing Solver
May 3, 2024Changliang Zhou, Xi Lin, Zhenkun Wang +3Neural Network Generalization Intelligent Transportation Systems
Guangdong Provincial Key Laboratory of Fully Actuated System Control Theory and Technology, School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen 518055, China · School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an 710049, China · Huawei Noah’s Ark Lab, Hong Kong SAR, China +1
Branch Scaling Manifests as Implicit Architectural Regularization for Improving Generalization in Overparameterized ResNets
Mar 7, 2024Zixiong Yu, Guhan Chen, Jianfa Lai +2Neural Network Generalization Residual Learning
Huawei Large Model Data Technology Lab, Shenzhen · Tsinghua University, Beijing · Kyoto University, Kyoto
Adversarial Rademacher Complexity of Deep Neural Networks
Nov 27, 2022Jiancong Xiao, Yanbo Fan, Ruoyu Sun +1Neural Network Generalization Neural Network Robustness
The Chinese University of Hong Kong, Shenzhen Shenzhen, China · Nanjing University Suzhou, China
A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks
Oct 28, 2022Zhengdao Chen, Eric Vanden-Eijnden, Joan BrunaNeural Network Generalization Neural Network Optimization
Google Research Mountain View, CA 94043 · Courant Institute, New York University New York, NY 10003
Learning Non-Vacuous Generalization Bounds from Optimization
Jun 9, 2022Chengli Tan, Jiangshe Zhang, Junmin Liu +1Neural Network Generalization Statistical Learning Theory
Northwestern Polytechnical University, Xi’an, China · Xi’an Jiaotong University, Xi’an, China · SGIT AI Lab, State Grid Corporation of China, Xi’an, China
A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning
Sep 6, 2021Yehuda Dar, Vidya Muthukumar, Richard G. BaraniukDouble Descent Neural Network Generalization
Faculty of Computer and Information Science, Ben-Gurion University · Schools of Electrical and Computer Engineering, and Industrial and Systems Engineering, Georgia Institute of Technology · Electrical and Computer Engineering Department, Rice University

Jan 17, 2025J/K move · Enter open · S save

Universality of Benign Overfitting in Binary Linear Classification

Ichiro Hashimoto, Stanislav Volgushev, Piotr Zwiernik

Department of Statistical Sciences, University of Toronto

Neural Network Generalization Linear Classification Benign Overfitting Binary Classification Classification Generalization Bounds

Open paper AskPDF arXiv

The practical success of deep learning has led to the discovery of several surprising phenomena. One of these phenomena, that has spurred intense theoretical research, is ``benign overfitting'': deep neural networks seem to generalize well in the over-parametrized regime even though the networks show a perfect fit to noisy training data. It is now known that benign overfitting also occurs in various classical statistical models. For linear maximum margin classifiers, benign overfitting has been established theoretically in a class of mixture models with very strong assumptions on the covariate distribution. However, even in this simple setting, many questions remain open. For instance, most of the existing literature focuses on the noiseless case where all true class labels are observed without errors, whereas the more interesting noisy case remains poorly understood. We provide a comprehensive study of benign overfitting for linear maximum margin classifiers. We discover a phase transition in test error bounds for the noisy model which was previously unknown and provide some geometric intuition behind it. We further considerably relax the required covariate assumptions in both the noisy and noiseless cases. Our results demonstrate that benign overfitting of maximum margin classifiers holds in a much wider range of scenarios than was previously known and provide new insights into the underlying mechanisms.