Neural Network Generalization

Follow this topic Ask about this topic

The field studies how neural networks move beyond fitting their training examples, including when overparameterized models can memorize data yet still perform well on new examples or shifted distributions. Theoretical approaches use Rademacher complexity, PAC-Bayes bounds, algorithmic stability, kernel and mean-field analyses, and high-dimensional statistics to relate model structure and training to test error. Empirical work examines implicit regularization from optimizers and architectures, representation learning, double descent, grokking, and robustness across settings such as images, language, graphs, and scientific models. A central open tension is that bounds and measures such as flatness do not reliably predict performance in every regime, which depends strongly on data structure, learned features, and distribution shift.

Latest papers

311 papers35 papers in the last four weeks, up 35% on the four weeks before. 0.2% of all new papers.

Formalizing the Intrinsic Dimension of Objective Landscapes by a Generalized Notion of Co-dimension
Oct 29, 2026Ernesto Araya, Gitta Kutyniok, Weilin Li +2Intrinsic Dimensionality Loss Landscape Geometry
Department of Mathematics, City University of New York, City College and Graduate Center, New York, USA
Sometimes less is more: Investigating inductive biases of MLPs and Transformers on noisy k-parity features
Oct 27, 2026Alasdair ParenTransformer Multilayer Perceptrons
Chalmers University of Technology
Investigating the Early Signatures of Feature Learning in Deep Linear Networks
Oct 27, 2026Shaunak Bhandarkar, Boris HaninDeep Linear Networks Neural Network Generalization
Princeton University
Mapping the Solution-Set Structure of Grokking on Latin Squares
Oct 26, 2026Zihan Gu, Lulu Zhao, Tianyi Ma +4Grokking Neural Network Generalization
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Alibaba Group +1
Generalization in Learning: Eyring Formula and Predator-prey Model
Oct 25, 2026S. V. Kozyrev, I. A. Lopatin, Alexander PechenNeural Network Generalization Statistical Physics of Learning
Steklov Mathematical Institute of Russian Academy of Sciences, Moscow, Russia · Ivannikov Institute for System Programming of the Russian Academy of Sciences, Moscow, Russia
Lower expected Fisher rank implies better PAC generalization
Oct 23, 2026Marco Miani, Albert Kjøller Jacobsen, Albert Lopez i Serrano +2Neural Network Generalization PAC-Bayesian Generalization Bounds
AWS AI Labs · Technical University of Denmark
Characterizing Cross-Relay Generalization Failure in Machine Learning-Based Power System Protection
Oct 20, 2026Nithin Pradeep Nadayil Kizhakkethil, Julian Oelhaf, Christian Bergler +2Neural Network Generalization Power System Fault Diagnosis
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg · Department of Electrical Engineering, Media and Computer Science, Ostbayerische Technische Hochschule Amberg-Weiden
Neural Expansion: A Unified Mechanism for How Deep Neural Networks Generalize
Oct 16, 2026Chashi Mahiul Islam, Samuel Jacob Chacko, Mao Nishino +2Neural Network Generalization Overparameterized Neural Network
Department of Computer Science, Florida State University · Department of Mathematics, Florida State University · Independent
The Sign Code: The Hidden Binary Nature of Deep Networks
Oct 16, 2026Niclas Göring, Shuofeng Zhang, Roi Holtzman +2Neural Network Generalization Representation Learning
Rudolf Peierls Centre for Theoretical Physics, University of Oxford, Oxford, UK · Kim Jaechul Graduate School of AI, KAIST, Seoul, South Korea
"Deep lifts": Neural networks can generalize even with the dumbest optimizer
Oct 16, 2026Sunil Madhow, Yu-Xiang WangNeural Network Generalization Generalization Bounds
UC San Diego
Flat Loss, Evolving Features: Grokking in Modular Addition
Oct 15, 2026Akash Kumar, Karthik SankaranarayananNeural Network Training Dynamics Neural Network Generalization
University of California, San Diego · Microsoft AI
BridgeAfford: Bridging Query-Target Gap for Generalizable 3D Affordance Grounding
Oct 15, 2026Yijie Su, Yakun Zhu, Yi Bin +53D Affordance Grounding 3D Visual Grounding
Tongji University · University of Electronic Science and Technology of China
Kolmogorov-Arnold Networks: Curse of Dimensionality and Structural Bias
Oct 15, 2026Yunhao Ni, Yuxin Guo, Jie Luo +2Curse of Dimensionality Kolmogorov-Arnold Networks
State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China
Does Compression Imply Generalization? A Minimum Description Length Perspective
Oct 15, 2026Lincen Yang, Jiayang Shi, Nan Pu +4Neural Network Generalization Generalization Bounds
Leiden Institute of Advanced Computer Science, Leiden University, The Netherlands · Centrum Wiskunde & Informatica (CWI), Amsterdam, The Netherlands · Hefei University of Technology, Hefei, China +3
Dimension-Free PAC-Bayes Generalization for Deep Neural Networks via Dirichlet Processes
Oct 15, 2026Sumit VashishthaPAC-Bayesian Generalization Bounds Neural Network Generalization
Independent Researcher
Denoising Score Matching in Diffusion Models: The Effect of Rank Constraints on Generalization
Oct 14, 2026Anand Jerry George, Nicolas MacrisScore-Based Generative Modeling Neural Network Generalization
École Polytechnique Fédérale de Lausanne (EPFL), Lab for Statistical Mechanics of Inference in Large Systems (SMILS), CH-1015 Lausanne, Switzerland
One Size Does Not Fit All: Curvature-Aware Adaptive Perturbation for SAM
Oct 14, 2026Alessia Rossi, Felipe Campelo, Gabriel L. OliveiraCurvature-Aware Optimization Sharpness-Aware Minimization
School of Engineering Mathematics and Technology, University of Bristol, Bristol, Lovelace Building, Ada, Bristol, BS8 1TW, UK
Statistical Vignettes on the Grokking Phenomenon
Oct 13, 2026Baye Malick Gning, Elvis DohmatobGrokking Neural Network Training Dynamics
Concordia University · Mila – Quebec AI Institute
Tiny Recursive Message Passing
Oct 13, 2026Tommaso Salvatori, Matteo Forasassi, Alexis Toumi +1Neural Algorithmic Reasoning Message Passing Neural Networks
Relational Intelligence, Inc · Vienna University of Technology
Compression Through Time: An Interplay of Separability, Stationarity and Capacity
Oct 12, 2026Nikolaus Kopp, Franz Pernkopf, Bernhard GeigerNeural Network Generalization Recurrent Neural Networks
Graz University of Technology, Graz, Austria · Graz University of Technology & Know Center Research GmbH, Graz, Austria
Fisher information and the geometry of memorization in neural networks
Oct 12, 2026Daniel T. Bernstein, Chase Goddard, Luca Di Carlo +2Neural Network Memorization Neural Network Generalization
Princeton University1; Lewis-Sigler Institute · Princeton University2; Joseph Henry Laboratories · University of Sydney3; School of Physics +1
Memorization in image classification: test set behavior and scaling laws
Oct 12, 2026Jeevesh Attri, Florentin GuthNeural Network Memorization Neural Network Training Dynamics
Center for Data Science New York University · Center for Computational Neuroscience Flatiron Institute · Work done while partially affiliated with Center for Data Science, New York University.
Why Heavy-Tailed Weight Spectra Predict Model Quality
Oct 9, 2026Joseph Wilson, Chris van der Heide, Liam Hodgkinson +3Neural Network Generalization Heavy-Tailed Distributions
School of Mathematics and Physics University of Queensland · Dept. of Electrical and Electronic Engineering University of Melbourne · School of Mathematics and Statistics University of Melbourne +3
Toward Joint Optimization of Circuit Depth and Training Data Size in Adaptively Grown Quantum Classifiers
Oct 8, 2026Saeefa Rubaiyet Nowmi, Md Mahmuduzzaman Kamol, Mohammad Saidur RahmanQuantum Circuit Trainability Neural Network Generalization
Old Dominion University Norfolk, VA, USA · University of Texas at El Paso El Paso, TX, USA
HAND: A Biologically-Inspired Activation Function that Improves Generalisation and Sample Efficiency in Image Classification
Oct 8, 2026Michael W. Spratling, Heiko H. SchüttNeural Network Activation Functions Neural Network Generalization
University of Luxembourg, Department of Behavioural and Cognitive Sciences, L-4366 Esch-sur-Alzette, Luxembourg.
Dissecting Representation Structure in Vision Transformers: A Rigorous Architectural Study
Oct 8, 2026Kim-Cuc Nguyen, Ngai-Man CheungVision Transformer Neural Network Generalization
Temasek Laboratories Singapore University of Technology and Design Singapore
Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking
Oct 8, 2026Mohnish HarwaniNeural Network Generalization Flat Minima
Purdue University
Length Generalization Needs Proper Regularization
Oct 3, 2026Pavlo Vasylenko, Matthias Lindemann, André F. T. Martins +1Length Generalization Language Model Generalization
Instituto Superior Técnico, Universidade de Lisboa · Instituto de Telecomunicações · ELLIS Unit Lisbon +1
Generalization of Transformer-Based Neural Quantum States via In-Context Learning
Oct 2, 2026Zhen Qin, Qing Qu, Alfred O. HeroNeural Quantum States Neural Network Generalization
Michigan Institute for Computational Discovery and Engineering, Department of Electrical Engineering and Computer Science and Department of Statistics, University of Michigan, Ann Arbor, MI 48109 USA · Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI 48109 USA
Where Does Jagged Competence Come From?
Oct 2, 2026Ioannis TsiokosNeural Network Generalization Compositional Representation Learning
Automorph Inc., Wilmington, DE, USA

Oct 29, 2026J/K move · Enter open · S save

Formalizing the Intrinsic Dimension of Objective Landscapes by a Generalized Notion of Co-dimension

Ernesto Araya, Gitta Kutyniok, Weilin Li, Johannes Maly+1

Department of Mathematics, City University of New York, City College and Graduate Center, New York, USA

Intrinsic Dimensionality Loss Landscape Geometry Neural Network Generalization

Open paper AskPDF arXiv

In this work, we formalize the concept of intrinsic dimension of objective landscapes introduced in Li et al. (2018), which measures the structural complexity of learning problems by training along randomly generated subspaces. We define the intrinsic dimension of a set

Θ\subset\mathbb{R}^p

as the minimal dimension

d_{\mathrm{int}}(Θ)

such that a random plane of dimension

d_{\mathrm{int}}(Θ)

\mathbb{R}^p

intersects

Θ

with probability one. We show that it is a generalized notion of co-dimension that depends on the global geometry of

Θ

. We study mathematical properties of the intrinsic dimension and develop techniques to estimate the intrinsic dimension of sets

Θ

that arise in learning problems. More specifically, we derive upper and/or lower bounds where

Θ

is the set of optimal parameters in student-teacher settings with shallow and deep ReLU networks, including finite and infinite sample cases, and with and without regularization. We discuss how our theory leads to novel types of generalization bounds, thereby showing that the intrinsic dimension is a measure of learning complexity. Our results connect techniques from seemingly different areas of machine learning research such as classification of symmetries of ReLU networks and stochastic geometry.