Neural Network Generalization

Ask about this topic

Overview

The field studies how neural networks move beyond fitting their training examples, including when overparameterized models can memorize data yet still perform well on new examples or shifted distributions. Theoretical approaches use Rademacher complexity, PAC-Bayes bounds, algorithmic stability, kernel and mean-field analyses, and high-dimensional statistics to relate model structure and training to test error. Empirical work examines implicit regularization from optimizers and architectures, representation learning, double descent, grokking, and robustness across settings such as images, language, graphs, and scientific models. A central open tension is that bounds and measures such as flatness do not reliably predict performance in every regime, which depends strongly on data structure, learned features, and distribution shift.

Latest papers

CardsList

311 papers35 papers in the last four weeks, up 35% on the four weeks before. 0.2% of all new papers.

  1. Understanding the geometry of deep learning with decision boundary volume

    Mar 16, 2026Matthew Burfitt, Jacek Brodzki, Pawel DłotkoDecision Boundary AnalysisNeural Network Generalization

    Beijing Key Laboratory of Topological Statistics and Applications for Complex Systems, Beijing Institute of Mathematical Sciences and Applications (BIMSA) · Mathematical Sciences, University of Southampton · Dioscuri Centre for Topological Data Analysis, Institute of Mathematics, Polish Academy of Sciences

  2. Ternary Gamma Semirings: From Neural Implementation to Categorical Foundations

    Mar 15, 2026Ruoqi SunNeural Network GeneralizationCompositional Representation Learning

    Intelligent Game and Decision Lab(IGDL), No.1 Xianghongqi Road, Haidian District, Beijing, 100091,China.

  3. On Interpolation Formulas Describing Neural Network Generalization

    Mar 14, 2026Jin Guo, Roy Y. He, Jean-Michel MorelNeural Network GeneralizationNeural Tangent Kernel

    Department of Mathematics, City University of Hong Kong, Hong Kong · Division of Industrial Data Science, Lingnan University, Hong Kong

  4. Preventing overfitting in deep learning using differential privacy

    Mar 12, 2026Alizishaan Anwar Hussein KhatriNeural Network GeneralizationDifferential Privacy

    Faculty of the Graduate School of University at Buffalo, State University of New York; Department of Computer Science and Engineering

  5. GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models

    Mar 12, 2026Xiaoyun Liu, Divya Saxena, Jiannong Cao +3LLM PruningStructured Pruning

    Department of Computing, The Hong Kong Polytechnic University, Hong Kong · School of Artificial Intelligence and Data Science, IIT Jodhpur, Jodhpur 342030, India

  6. Upper Generalization Bounds for Neural Oscillators

    Mar 10, 2026Zifeng Huang, Konstantin M. Zuev, Yong Xia +1Generalization BoundsNeural Network Generalization

    Institute for Risk and Reliability, Leibniz University Hannover, Callinstraße 34, Hannover, 30167, Germany · Department of Computing and Mathematical Sciences, California Institute of Technology, Pasadena, California, United States · Joint Research Centre for Marine Infrastructure, Department of Civil and Environmental Engineering, The Hong Kong Polytechnic University, Kowloon, Hong Kong, China +2

  7. An accurate flatness measure to estimate the generalization performance of CNN models

    Mar 9, 2026Rahman Taleghani, Maryam Mohammadi, Francesco MarchettiNeural Network GeneralizationConvolutional Neural Networks

    1*Department of Mathematics ”Tullio Levi-Civita”, University of Padova, Via Trieste 63, Padova, 35121, Veneto, Italy. · Department of Computer Science, Ruhr University Bochum, University street 140, Bochum, 44801, North Rhine-Westphalia, Germany. · Department of Mathematical Sciences and Computer, Kharazmi University, Mofateh Avenue, Tehran, 15719-14911, Tehran, Iran.

  8. Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizers

    Mar 7, 2026Tao Shi, Liangming Chen, Long Jin +1Neural Network OptimizationAdaptive Gradient Methods

    School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China · Helen and John C. Hartmann Department of Electrical and Computer Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA

  9. The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization

    Mar 5, 2026Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung +1GrokkingNeural Network Generalization

    H&K Research Studio, Clevix LLC, Hanoi, Vietnam · Banking Academy of Vietnam, Hanoi, Vietnam

  10. Asymptotic Behavior of Multi--Task Learning: Implicit Regularization and Double Descent Effects

    Mar 5, 2026Ayed M. Alrashdi, Oussama Dhifallah, Houssem SifaouDouble DescentNeural Network Generalization

    Department of Electrical Engineering, College of Engineering, University of Ha’il, P.O. Box 2440, Ha’il, 81441, Saudi Arabia · John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, USA · Department of Electrical and Electronic Engineering, King’s College London – Strand, London WC2R 2LS – London, UK

  11. Does Sparse Connectivity Improve Generalization? Convolutional Networks Below the Edge of Stability

    Mar 5, 2026Tongtong Liang, Esha Singh, Rahul Parhi +2Neural Network GeneralizationGeneralization Bounds

    UC San Diego

  12. Why the Brain Consolidates: Predictive Forgetting for Optimal Generalisation

    Mar 5, 2026Zafeirios Fountas, Adnan Oomerjee, Haitham Bou-Ammar +2Memory ConsolidationNeural Network Generalization

    Huawei Noah’s Ark Lab, London, UK · AI Centre, Department of Computer Science, University College London, London, UK · UCL Institute of Cognitive Neuroscience, University College London, London, UK +1

  13. Biased Generalization in Diffusion Models

    Mar 3, 2026Jerome Garnier-Brun, Luca Biggio, Davide Beltrame +2Neural Network GeneralizationDiffusion Models

    Department of Computing Sciences, Bocconi University, Milan, Italy.

  14. Guiding Sparse Neural Networks with Neurobiological Principles to Elicit Biologically Plausible Representations

    Mar 3, 2026Patrick Inoue, Florian Röhrbein, Andreas KnoblauchBiologically Plausible LearningSparse Neural Networks

    KEIM Institute, Albstadt-Sigmaringen University, Germany · Department of Computer Science, Chemnitz University of Technology, Germany

  15. Joint Training Across Multiple Activation Sparsity Regimes

    Mar 3, 2026Haotian WangNeural Network GeneralizationActivation Sparsity

    Department of Nuclear Medicine and PET-CT Center, the Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, Zhejiang, China

  16. HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalization

    Mar 3, 2026Feihu Huang, Guanyi Zhang, Songcan ChenAdaptive Gradient MethodsNeural Network Generalization

    College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China

  17. Intrinsic Task Symmetry Drives Generalization in Algorithmic Tasks

    Mar 2, 2026Hyeonbin Hwang, Yeachan ParkNeural Network GeneralizationGrokking

    KAIST · Sejong University

  18. Diagnosing Generalization Failures from Representational Geometry Markers

    Mar 2, 2026Chi-Ning Chou, Artem Kirsanov, Yao-Yuan Yang +1Neural Network GeneralizationOOD Generalization

    Flatiron Institute · Harvard University · Google DeepMind

  19. One Operator to Rule Them All? On Boundary-Indexed Operator Families in Neural PDE Solvers

    Mar 2, 2026Lennon J. ShikhmanPDE Operator LearningNeural Operators

    College of Computing, Georgia Institute of Technology · Department of Mathematics and Systems Engineering, Florida Institute of Technology

  20. Random Features for Operator-Valued Kernels: Bridging Kernel Methods and Neural Operators

    Mar 1, 2026Mike Nguyen, Nicole MückeRandom Feature MethodsKernel Methods

    Technical University of Braunschweig

  21. To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters

    Feb 28, 2026Sara Dragutinović, Yedi Zhang, Rajesh RanganathNeural Network GeneralizationMuon Optimizer

    Courant Institute School of Mathematics, Computing, and Data Science, New York University · Gatsby Computational Neuroscience Unit, University College London

  22. Improving Full Waveform Inversion in Large Model Era

    Feb 27, 2026Yinan Feng, Peng Jin, Yuzhe Guo +2Inverse ProblemsOOD Generalization

    School of Data Science and Society University of North Carolina at Chapel Hill Chapel Hill, NC, USA · The Pennsylvania State University State College, PA, USA · Google DeepMind

  23. On Emergences of Non-Classical Statistical Characteristics in Classical Neural Networks

    Feb 27, 2026Hanyu Zhao, Yang Wu, Yuexian HouNeural Network Training DynamicsNeural Network Generalization

    Tianjin University, China

  24. NNiT: Width-Agnostic Neural Network Generation with Structurally Aligned Weight Spaces

    Feb 26, 2026Jiwoo Kim, Swarajh Mehta, Hao-Lun Hsu +3Diffusion TransformerNeural Network Generalization

    Department of Electrical and Computer Engineering, Duke University, NC, USA · Department of Computer Science, Duke University, NC, USA · Computer Science & Artificial Intelligence Laboratory, Massachusetts Institute of Technology, MA, USA

  25. Takeuchi's Information Criteria as Generalization Measures for DNNs Close to NTK Regime

    Feb 26, 2026Hiroki Naganuma, Taiji Suzuki, Rio Yokota +3Neural Network GeneralizationModel Selection

    Department of Computer Science and Operations Research, Université de Montréal, Canada · Mila – Quebec AI Institute, Canada · Department of Mathematical Informatics, The University of Tokyo, Japan +4

  26. The Error of Deep Operator Networks Is the Sum of Its Parts: Branch-Trunk and Mode Error Decompositions

    Feb 25, 2026Alexander Heinlein, Johannes TarazDeep Operator NetworksNeural Operators

    Delft Institute of Applied Mathematics, Delft University of Technology, Delft, 2628CD, The Netherlands

  27. Neural network optimization strategies and the topography of the loss landscape

    Feb 24, 2026Jianneng Yu, Alexandre V. MorozovNeural Network OptimizationStochastic Gradient Descent

    Department of Physics & Astronomy, Rutgers, The State University of New Jersey, 136 Frelinghuysen Rd., Piscataway, NJ 08854, U.S.A.

  28. On the Generalization Behavior of Deep Residual Networks From a Dynamical System Perspective

    Feb 24, 2026Jinshu Huang, Mingfei Sun, Chunlin WuNeural Network GeneralizationGeneralization Bounds

    School of Mathematical Sciences, Nankai University, Tianjin, China

  29. Understanding the Role of Rehearsal Scale in Continual Learning under Varying Model Capacities

    Feb 24, 2026JinLi He, Liang Bai, Xian YangContinual LearningReplay-Based Continual Learning

    Institute of Intelligent Information Processing, Shanxi University, Taiyuan, 030006, China · Alliance Manchester Business School, The University of Manchester, Manchester, M13 9PL, UK

  30. A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning

    Feb 23, 2026Nicolas Anguita, Francesco Locatello, Andrew M. Saxe +4Neural Network GeneralizationRepresentation Learning

    Department of Engineering, University of Cambridge · Institute of Science and Technology, Austria (ISTA) · Gatsby Computational Neuroscience Unit and Sainsbury Wellcome Centre, UCL +1