Neural Network Generalization

Follow this topic Ask about this topic

Momentum

21 papers in the last four weeks, up 600% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 204

All topics

SORA: Free Second-Order Attacks in Fast Adversarial Training
May 30, 2026Mazdak Teymourian, Ramtin Moslemi, Farzan Rahmani +1Adversarial Training Neural Network Generalization
Department of Computer Engineering, Sharif University of Technology, Tehran, Iran.
A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalization
May 29, 2026Sherin Muckatira, Namrata Shivagunde, Vijeta Deshpande +1Language Model Pretraining Neural Network Generalization
University of Massachusetts Lowell
Inconsistency-Aware Minimization: Improving Generalization with Unlabeled Data
May 29, 2026Hee-Sung Kim, Hyeonseong Kim, Sungyoon LeeNeural Network Generalization Deep Learning Optimization
Department of Computer Science, Hanyang University, Seoul, Korea · Agency for Defense Development, Daejeon, Korea
Graph Neural Networks Are Not Continuous Across Graph Resolutions
May 29, 2026Christian Koke, Yuesong Shen, Abhishek Saroha +4Neural Network Generalization Graph Neural Networks
AITHYRA · Munich Center for Machine Learning · Technical University of Munich, Munich +3
PINNs Failure Modes are Overfitting
May 29, 2026Nigel T. Andersen, Takashi MatsubaraNeural Network Generalization Neural PDE Solvers
Graduate School of Information Science and Technology Hokkaido University Sapporo, Japan · RIKEN Center for Advanced Intelligence Project (AIP) Tokyo, Japan
Quantifying and Optimizing Simplicity via Polynomial Representations
May 28, 2026Tianren Zhang, Xiangxin Li, Minghao Xiao +2Neural Network Generalization
Department of Automation, Tsinghua University, Beijing, China.
Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regime
May 28, 2026Paolo Baglioni, Christian Keup, Vincenzo Zimbardo +4Bayesian Neural Networks Neural Network Generalization
1INFN, Sezione di Milano Bicocca, Piazza della Scienza 3, 20126, Milano, Italy · 2INFN, Gruppo Collegato di Parma, Parco Area delle Scienze 7/A, 43124 Parma, Italy · 3Dipartimento di Scienze Matematiche, Fisiche e Informatiche, Universit`a degli Studi di Parma, Parco Area delle Scienze, 7/A 43124 Parma, Italy +2
Learning to Perturb Hidden Representations for Generalizable Deep Learning
May 28, 2026Hua LiNeural Network Generalization Adversarial Perturbations
Henan University
Do Physics Foundation Models Learn Generalizable Physics? A Bias-Aware Benchmark Across Physical Regimes and Distribution Shifts
May 28, 2026Mengdi Chu, Yang Liu, Ayan Biswas +1Distribution Shift Robustness Neural Network Generalization
The Ohio State University · Los Alamos National Laboratory
Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias
May 27, 2026Mohua Das, Pierfrancesco Beneventano, Shibshankar Dey +2Neural Network Generalization Neural Network Initialization
MIT · Northwestern University
The Hamilton-Jacobi Theory of Deep Learning
May 27, 2026Jose Marie Antonio Miñoza, Erika Fille T. Legara, Christopher P. MonterolaNeural Network Generalization Neural Network Training Dynamics
Center for AI Research PH · Asian Institute of Management
A Training-Time Diagnostic for Generalization via the Log-Alignment Ratio
May 27, 2026Ali Shehper, Ashish VaswaniNeural Network Generalization
Work done at Essential AI.
Dark Quest II: A Wide-Coverage Neural Network Emulator of the Nonlinear Matter Power Spectrum Across Extended Cosmologies
May 27, 2026Satoshi Tanaka, Takahiro Nishimichi, Yosuke KobayashiNeural Surrogate Modeling Neural Network Generalization
Graduate School of Social Data Science, Hitotsubashi University, 2-1, Naka, Kunitachi 186-8601, Tokyo, Japan · Center for Gravitational Physics and Quantum Information, Yukawa Institute for Theoretical Physics, Kyoto University, Kyoto 606-8502, Japan · Department of Astrophysics and Atmospheric Sciences, Faculty of Science, Kyoto Sangyo University, Motoyama, Kamigamo, Kita-ku, Kyoto 603-8555, Japan +3
A Multi-dimensional Framework for Evaluating Generalization in EEG Foundation Models
May 27, 2026Aditya Kommineni, Emily Zhou, Kleanthis Avramidis +2Neural Network Generalization EEG Decoding
Signal Analysis and Interpretation Laboratory, University of Southern California, USA
Law of Neural Interaction: Depth-Width Shape, Interaction Efficiency, and Generalization
May 27, 2026Wenjie Sun, Jinning Yang, Shuai Zhang +1Neural Network Generalization Language Model Scaling Laws
The Chinese University of Hong Kong, Shenzhen · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · New Jersey Institute of Technology
Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent
May 26, 2026Chi-Ning Chou, Oscar Uzdelewicz, Neng-Chun Chiu +2Double Descent Neural Network Generalization
Center for Computational Neuroscience, Flatiron Institute, New York, NY, USA. · Department of Physics, Harvard University, Cambridge, MA, USA. · Kempner Institute, Harvard University, Allston, MA, USA. +1
Learning Sparse Compositional Functions with Norm-Constrained Neural Networks
May 25, 2026Shuo Huang, Lorenzo Fiorito, Lorenzo Rosasco +1Neural Network Generalization Hierarchical Representation Learning
MaLGa, DIMA, Università degli Studi di Genova, Genoa, Italy · CBMM, Massachusetts Institute of Technology, Cambridge, MA, USA
Different Statistical Perspectives for Understanding Generalisation in Graph Neural Networks
May 25, 2026Nil Ayday, Mahalakshmi Sabanayagam, Debarghya GhoshdastidarNeural Network Generalization Graph Neural Networks
Technical University of Munich, School of Computation, Information and Technology, Boltzmannstr. 3, Garching, 85748, Germany. · Australian Institute for Machine Learning, Adelaide University,2026 Adelaide, SA 5000, Australia.
Label-NTK Alignments and A Tighter Convergence Bound in the NTK Regime
May 24, 2026Ruchirinkil Marreddy, Chaoyue LiuNeural Network Generalization Neural Tangent Kernel
Elmore Family School of Electrical and Computer Engineering Purdue University
On the Impact of Class Imbalance on the Learning Dynamics of Deep Neural Networks:An Intuitive Insight
May 24, 2026Ismail B. Mustapha, Shafaatunnur Hasan, Sunday O. Olatunji +1Class-Imbalanced Learning Neural Network Generalization
Faculty of Computing Universiti Teknologi Malaysia Johor, Malaysia · Faculty of Computing Adejkunle Ajasin University Akungba-Akoko, Nigeria
A mathematical theory of balancing relational generalization and memorization
May 21, 2026Luke Cheng, Samuel LipplNeural Network Generalization Logical Reasoning
Center for Theoretical Neuroscience Columbia University New York, NY, 10027
Winner-Take-All bottlenecks enforce disentangled symbolic representations in multi-task learning
May 21, 2026Julian Gutheil, Simon Hitzginger, Robert LegensteinNeural Network Generalization Disentangled Representation Learning
Graz University of Technology · Institute of Machine Learning and Neural Computation Graz University of Technology Graz, Austria
Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective
May 20, 2026David Perera, Victor Moura, Lais Isabelle Alves dos Santos +2Intrinsic Dimensionality Neural Network Generalization
Universidade Federal de Minas Gerais · Queen Mary University of London
TextReg: Mitigating Prompt Distributional Overfitting via Regularized Text-Space Optimization
May 20, 2026Lucheng Fu, Ye Yu, Yiyang Wang +4Neural Network Generalization Large Language Model-Guided Optimization
Georgia Institute of Technology · University of Illinois Urbana-Champaign
A New Framework to Analyse the Distributional Robustness of Deep Neural Networks
May 20, 2026Divij Khaitan, Subhashis BanerjeeDistribution Shift Robustness Neural Network Generalization
Microsoft · Ashoka University
A Sharper Picture of Generalization in Transformers
May 20, 2026Paul Lintilhac, Sair ShaikhNeural Network Generalization PAC-Bayesian Generalization Bounds
Thayer School of Engineering Dartmouth College · Dartmouth College
When Irregularity Helps: A Subclass Analysis of Inductive Bias in Neural Morphology
May 19, 2026Wen ZhangNeural Network Generalization Long-Tail Learning
Axiomatizing Neural Networks via Pursuit of Subspaces
May 19, 2026Mehmet Yamac, Mert Duman, Ugur Akpinar +4Neural Network Generalization Neural Representation Geometry
Tampere University, Faculty of ITC, Finland · Department of Electrical Engineering, Qatar University, Qatar · Donders Institute, Radboud University, The Netherlands
Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing
May 19, 2026Valentina Njaradi, Clémentine Dominé, Rachel Swanson +2Neural Network Generalization Linear Probing
Gatsby Computational Neuroscience Unit · University College London · Institute of Science and Technology Austria +1
MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Models
May 19, 2026Feihu Huang, Yuning Luo, Songcan ChenNeural Network Generalization Deep Learning Optimization
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China · College of Design and Engineering, National University of Singapore, Singapore

May 30, 2026, cs.LGJ/K move · Enter open · S save

SORA: Free Second-Order Attacks in Fast Adversarial Training

Mazdak Teymourian, Ramtin Moslemi, Farzan Rahmani, Mohammad Hossein Rohban

Department of Computer Engineering, Sharif University of Technology, Tehran, Iran.

Adversarial Training Neural Network Generalization Neural Network Robustness Adversarial Robustness

Open paper AskPDF arXiv

Adversarial Training (AT) is a leading defense against adversarial examples but often suffers from Catastrophic Overfitting (CO) in efficient single-step variants, where robustness to multi-step attacks collapses despite high single-step performance. We address this failure mode with two contributions. First, we formalize Epsilon Overfitting (EO), a perspective in which fixed perturbation magnitudes and directions exacerbate CO, and show that introducing perturbation variability significantly improves robust generalization across different architectures and datasets. Second, we propose PertAlign (Perturbation Alignment), a theoretically grounded, computationally negligible metric that predicts CO onset by measuring gradient alignment across attack stages. Leveraging these insights, we introduce SORA, an adaptive step-size AT method that dynamically adjusts perturbations based on loss surface geometry. SORA consistently prevents CO, achieves state-of-the-art robustness and clean accuracy, and generalizes across datasets and architectures using a single fixed set of hyperparameters, which is essential for applicability in fast AT. Extensive experiments on diverse datasets and architectures show that SORA matches or surpasses the robustness of prior methods while delivering higher clean accuracy and superior efficiency. Code is available at https://github.com/SecondOrderAT/SORA.

Momentum

Related topics

Latest papers 204