Neural Network Generalization

Ask about this topic

Overview

The field studies how neural networks move beyond fitting their training examples, including when overparameterized models can memorize data yet still perform well on new examples or shifted distributions. Theoretical approaches use Rademacher complexity, PAC-Bayes bounds, algorithmic stability, kernel and mean-field analyses, and high-dimensional statistics to relate model structure and training to test error. Empirical work examines implicit regularization from optimizers and architectures, representation learning, double descent, grokking, and robustness across settings such as images, language, graphs, and scientific models. A central open tension is that bounds and measures such as flatness do not reliably predict performance in every regime, which depends strongly on data structure, learned features, and distribution shift.

Latest papers

CardsList

311 papers35 papers in the last four weeks, up 35% on the four weeks before. 0.2% of all new papers.

  1. Benign Overfitting in Adversarial Training for Vision Transformers

    Apr 21, 2026Jiaming Zhang, Meng Ding, Shaopeng Fu +2Adversarial TrainingNeural Network Generalization

    Division of CEMSE, King Abdullah University of Science and Technology · School of Statistics, Renmin University of China · Department of Computer Science and Engineering, State University of New York at Buffalo +1

  2. PAC-Bayes Bounds for Gibbs Posteriors via Singular Learning Theory

    Apr 19, 2026Chenyang Wang, Yun YangNeural Network GeneralizationPAC-Bayesian Generalization Bounds

    Department of Statistics, University of Illinois Urbana-Champaign · Department of Mathematics, University of Maryland, College Park

  3. When Do Early-Exit Networks Generalize? A PAC-Bayesian Theory of Adaptive Depth

    Apr 17, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuNeural Network GeneralizationPAC-Bayesian Generalization Bounds

    The University of Hong Kong, Hong Kong, China · Brain Investing Limited, Hong Kong, China

  4. Stability and Generalization in Looped Transformers

    Apr 16, 2026Asher LabovichNeural Network GeneralizationDynamical Systems

  5. Chaotic CNN for Limited Data Image Classification

    Apr 16, 2026Anusree M, Akhila Henry, Pramod P NairNeural Network GeneralizationConvolutional Neural Networks

    1*Department of Mathematics, Amrita Vishwa Vidyapeetham, Amritapuri, India.

  6. CLion: Efficient Cautious Lion Optimizer with Enhanced Generalization

    Apr 16, 2026Feihu Huang, Guanyi Zhang, Songcan ChenStochastic OptimizationSign-Based Optimization

    College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China

  7. Loop Corrections in Random Feature Models: Training Error and Generalization Gap

    Apr 14, 2026Taeyoung KimRandom Feature MethodsKernel Ridge Regression

    Center for AI and Natural Sciences, Korea Institute for Advanced Study, Seoul 02455, South Korea

  8. VISTA: Validation-Informed Trajectory Adaptation via Self-Distillation

    Apr 13, 2026Eli Corn, Daphna WeinshallNeural Network GeneralizationDeep Learning Optimization

    Department of Computer Science, Hebrew University of Jerusalem, Jerusalem, Israel.

  9. Topology-Aware PAC-Bayesian Generalization Analysis for Graph Neural Networks

    Apr 12, 2026Xinping YiNeural Network GeneralizationPAC-Bayesian Generalization Bounds

    School of Information Science and Engineering, Southeast University, Nanjing, China

  10. Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima

    Apr 10, 2026Huanran Chen, Huaqing Zhang, Xiao Li +3Language Model PretrainingNeural Network Generalization

    Tsinghua University · ByteDance Seed

  11. From Phenomenological Fitting to Endogenous Deduction: A Paradigm Leap via Meta-Principle Physics Architecture

    Apr 9, 2026Helong Hu, HongDan Pan, ShuiQing HuPhysics-Informed MLPhysical Reasoning

    Guangdong Ocean University, No.1 Luqin Road, Yangjiang City, Guangdong Province, China（524088） · China Pacific Insurance (Group) Co., Ltd. Foshan Branch, No.17, Jihua 5th Road, Chancheng District, Foshan City, Guangdong Province, China (528000)

  12. Implicit Regularization and Generalization in Overparameterized Neural Networks

    Apr 8, 2026Zeran JohannsenOverparameterized Neural NetworkNeural Network Generalization

  13. Plateaus, Optima, and Overfitting in Multi-Layer Perceptrons: A Saddle-Saddle-Attractor Scenario

    Apr 2, 2026Alex Alì Maleknia, Yuzuru SatoNeural Network Training DynamicsMultilayer Perceptrons

    Department of Mathematics, Hokkaido University, Kita 12 Nishi 7, Kita-ku, Sapporo, Hokkaido 060-0812, Japan · Present address: IMAG, IROKO, Univ Montpellier, Inria, CNRS · RIES, Hokkaido University, Kita 12 Nishi 7, Kita-ku, Sapporo, Hokkaido 060-0812, Japan

  14. Generalization Bounds for Spectral GNNs via Fourier Domain Analysis

    Apr 1, 2026Vahan A. Martirosyan, Daniele Malitesta, Hugues Talbot +2Generalization BoundsNeural Network Generalization

    Université Paris-Saclay, CentraleSupélec, Inria, France · LTCI, Télécom Paris, Institut Polytechnique de Paris, France

  15. Deep Networks Favor Simple Data

    Apr 1, 2026Weyl Lu, Chenjie Hao, Yubei ChenDensity EstimationOOD Detection

    UC Davis, USA

  16. Grokking From Abstraction to Intelligence

    Mar 31, 2026Junjie Zhang, Zhen Shen, Gang Xiong +1GrokkingNeural Network Generalization

  17. On the Mirage of Long-Range Dependency, with an Application to Integer Multiplication

    Mar 30, 2026Zichao WeiLong-Range Dependency ModelingNeural Network Generalization

    Saarland University

  18. A Compression Perspective on Simplicity Bias

    Mar 26, 2026Tom Marty, Eric Elmoznino, Leo Gagnon +5Neural Network GeneralizationFeature Selection

    Mila – Quebec AI Institute · Université de Montréal

  19. The Rules-and-Facts Model for Simultaneous Generalization and Memorization in Neural Networks

    Mar 26, 2026Gabriele Farné, Fabrizio Boncoraglio, Lenka ZdeborováNeural Network GeneralizationNeural Network Memorization

    Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) CH-1015 Lausanne

  20. A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularization

    Mar 26, 2026Shalima Binta Manir, Anamika Paul RupaGrokkingNeural Network Training Dynamics

    Howard University

  21. Minimal Sufficient Representations for Self-interpretable Deep Neural Networks

    Mar 25, 2026Zhiyao Tan, Liu Li, Huazhen LinNeural Network InterpretabilityRepresentation Learning

    Center of Statistical Research, School of Statistics, Southwestern University of Finance and Economics, Chengdu, China · School of Mathematics and Statistics, Wuhan University, Wuhan, China · New Cornerstone Science Laboratory, Southwestern University of Finance and Economics, Chengdu, China

  22. Are Flat Minima an Illusion?

    Mar 24, 2026Michael Timothy BennettNeural Network GeneralizationFlat Minima

    School of Computing, The Australian National University, Canberra, ACT 2601, Australia

  23. Generalization Limits of In-Context Operator Networks for Higher-Order Partial Differential Equations

    Mar 23, 2026Jamie Mahowald, Tan Bui-ThanhPDE Operator LearningNeural Network Generalization

    Department of Mathematics, University of Texas at Austin, Austin, Texas · Department of Aerospace Engineering and Engineering Mechanics, Oden Institute for Computational Engineering and Sciences, University of Texas at Austin, Texas

  24. Gradient Descent with Projection Finds Over-Parameterized Neural Networks for Learning Low-Degree Polynomials with Nearly Minimax Optimal Rate

    Mar 22, 2026Yingzhen Yang, Ping LiLow-Degree Polynomial MethodNeural Network Generalization

    School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ 85281, USA · VecML Inc., Bellevue, WA 98004, USA

  25. On the Dynamics & Transferability of Latent Generalization during Memorization

    Mar 20, 2026Simran Ketha, Venkatakrishnan RamaswamyNeural Network GeneralizationNeural Network Memorization

    Anuradha & Prashanth Palakurthi Centre for Artificial Intelligence Research, Department of Computer Science & Information Systems, Birla Institute of Technology & Science Pilani, Hyderabad 500078, India.

  26. Rigorous Error Certification for Neural PDE Solvers: From Empirical Residuals to Solution Guarantees

    Mar 19, 2026Amartya Mukherjee, Maxwell Fitzsimmons, David C. Del Rey Fernández +1Neural Network GeneralizationNeural PDE Solvers

    Department of Applied Mathematics, University of Waterloo, Waterloo, ON, Canada

  27. Tula: Optimizing Time, Cost, and Generalization in Distributed Large-Batch Training

    Mar 18, 2026Sahil Tyagi, Feiyi WangDistributed TrainingHyperparameter Optimization

    Oak Ridge National Laboratory, Oak Ridge, Tennessee, USA

  28. Transformers Can Learn Rules They've Never Seen: Proof of Computation Beyond Interpolation

    Mar 17, 2026Andy GrayTransformerNeural Network Generalization

    Kortical

  29. Grokking as a Variance-Limited Phase Transition: Spectral Gating and the Epsilon-Stability Threshold

    Mar 16, 2026Pratyush Acharya, Habish DhakalGrokkingNeural Network Training Dynamics

  30. Controlled Langevin Dynamics for Sampling of Feedforward Neural Networks Trained with Minibatches

    Mar 16, 2026Alessandro Zambon, Francesca Caruso, Riccardo Zecchina +1Stochastic Gradient Langevin DynamicsNeural Network Training Dynamics

    Department of Physics, Università degli Studi di Milano and INFN, via Celoria 16, 20133 Milano, Italy · Department of Computing Sciences and Bocconi Institute for Data Science and Analytics (BIDSA), Bocconi University, 20136 Milano, Italy