Two-Layer Neural Networks

Follow this topic Ask about this topic

Recent momentum

+100%

6 papers in the last 28 days · 0.1% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-14

4 new papers

A weekly snapshot of new work published in Two-Layer Neural Networks.

63 papers

Latest in Two-Layer Neural Networks

Open your feed →

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking
Sep 12, 2026Anish KatariaGrokking Over-Parameterized Neural Networks
Learning Orthogonal Multi-Index Models Beyond Small Initialization: Incremental Learning, Competitive Dynamics and Symmetry
Sep 11, 2026Mo Zhou, Weihang Xu, Simon S. Du +1Two-Layer Neural Networks Qualitative Dynamics
Phases in a class of associative memories via hidden neurons
Sep 11, 2026Toshihiro Ota, Masato TakiHopfield Networks Attractors
Teacher Geometry Shapes Learnability in Teacher-Student Networks
Sep 10, 2026Kai J. Sandbrink, Flavio Martinelli, Alexander van Meegen +2Teacher-Student Log-Probability Gap Two-Layer Neural Networks
Critical initialization destabilizes higher input derivatives in wide scalar-input networks
Sep 10, 2026Prashant Singh, Pranav SinghTwo-Layer Neural Networks Mean-Field Limit
A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
Sep 7, 2026Yuqing Wang, Ioannis G. Kevrekidis, Mikhail BelkinGeneralization Bounds Two-Layer Neural Networks
Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws
Aug 13, 2026Liu Ziyin, Yizhou Xu, Tomaso Poggio +1Two-Layer Neural Networks Power-Law Distribution
Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness
Aug 11, 2026Siqiao Mu, Diego KlabjanLipschitz Constant Gradient Descent
Approximation Rates for Metaplectic Neural Networks
Aug 9, 2026Ahmed Abdeljawad, Marcello Carioni, Elena CorderoNeural-Network Approximation Quantum Neural Networks
A Statistical Difference between Single-Layer Learning and Hierarchical Learning in Wide Neural Networks
Jul 26, 2026Sumio WatanabeTwo-Layer Neural Networks Neural Network
Do emulated quantum circuits change what CNNs look at? Performance and explainability comparison in medical image classification
Jul 23, 2026Guillermo Rubiños Rodríguez, Martín Ottavianelli, Mateo Alonso +4Hybrid Quantum Neural Networks Quantum Circuit Simulation
Expressivity of Shallow Neural Networks Over Finite Fields
Jul 19, 2026Maksym Zubkov, Carol Wu, Shiwei Yang +2Two-Layer Neural Networks Chern Numbers
Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations
Jul 15, 2026Chon-Fai Kam, Xavier Cadet, Miloud Bessafi +1Grokking Two-Layer Neural Networks
How the Hessian-Spectrum of Neural Networks Depends on Data
Jul 15, 2026Jasraj Singh, Enea Monzio Compagnoni, Antonio OrvietoCondition Number Two-Layer Neural Networks
Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initialization
Jul 14, 2026Jiajie Zhao, Jianxing Wang, Junjie Yang +2Two-Layer Neural Networks Implicit Bias
Singular perturbations and hierarchical learning in two-layer neural networks
Jul 12, 2026Cédric Gerbelot, Jean-Christophe MourratTwo-Layer Neural Networks Decision Thresholds
A law of robustness for two-layer neural networks with arbitrary weights
Jul 8, 2026Yitzchak ShmaloTwo-Layer Neural Networks Lipschitz Constant
Width-Robust Learnability in Mean-Field Bayesian Neural Networks
Jul 7, 2026Dmitry Vaintrob, Kaarel HänniBayesian Neural Networks Two-Layer Neural Networks
Broken Ergodicity and the Violation of the Fluctuation-Dissipation Theorem Lead to Generalization Beyond Overfitting in Machine Learning
Jul 5, 2026Chan Li, Nigel GoldenfeldMean-Field Limit Two-Layer Neural Networks
Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networks
Jul 2, 2026Matej Benko, Pierre Bousquet, Iwona Chlebicka +1Two-Layer Neural Networks Variational
Multilayer Q-Matrix-Embedded Neural Network for Cognitive Diagnosis (M-QCDNet): Structure-Aware Deep Learning Architecture for Psychometric Interpretability
Jul 1, 2026Yiyao YangCognitive Diagnosis Model Interpretability
GPU Parallelization Strategies for Forward and Backward Propagation in Shallow Neural Networks: A CUDA-Based Comparative Study
Jun 29, 2026Rania Zitouni, Nadine Bousdjira, Sarah Hasnaoui +2Cuda Backpropagation
Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks
Jun 27, 2026Yuto Omae, Kazuki Sakai, Yohei Kakimoto +3Condition Number Gradient Descent
How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks
Jun 26, 2026Julius Girardin, Emanuele Troiani, Yizhou Xu +3Two-Layer Neural Networks Scaling Laws
Convex training of Lipschitz-regularized shallow neural networks
Jun 17, 2026Chao Yin, Antoine Lesage-LandryLipschitz Constant Two-Layer Neural Networks
A Link between Shock-wave Theory and Symmetry-reduced Stochastic Gradient Descent for Artificial Neural Networks
Jun 16, 2026Taiki MiyagawaTwo-Layer Neural Networks Stochastic Gradient Descent
On the Entropy Formula for Real, Complex, and Quaternionic Deep Linear Networks
Jun 15, 2026Luis Contreras, Marco Nahas, Tejas KotwalDeep Network Quaternion
Can Neural Networks Achieve Optimal Computational-statistical Tradeoff? An Analysis on Single-Index Model
Jun 13, 2026Siyu Chen, Beining Wu, Miao Lu +2Optimal Sample Complexity Two-Layer Neural Networks
Beyond Linear and Overcomplete Regimes: A Mean-Field Analysis of Bottleneck Autoencoders
Jun 5, 2026Santanu Das, Ramyak Bilas, Pascal Esser +1Masked Autoencoders Two-Layer Neural Networks
Quantifying Uncertainty In Wide Two-Layer Neural Networks: On The Law Of The Limiting Fluctuation Process
Jun 4, 2026Arnaud Descours, Arnaud Guillin, Geoffrey Lacour +3Two-Layer Neural Networks Central Limit Theorem
GuardNet: Ensemble Strategies of Shallow Neural Networks for Robust Prompt Injection and Jailbreak Detection
Jun 4, 2026Paulo Ricardo Ferreira Neves, Edson Rodrigues da Cruz Filho, Paulo Henrique Eleuterio Falsetti +7Indirect Prompt Injection Jailbreak Detection
Gradient descent at the Edge of Stability: free energy model and kinetic description of the two-layer network
Jun 3, 2026Antonin Chodron de CourcelGradient Descent Two-Layer Neural Networks
Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability
Jun 3, 2026Vincent Bürgin, Daniel Herbst, Ya-Wei Eileen Lin +1Two-Layer Neural Networks Symmetry
Flatness and Generalization: Learning Multi-Index Models with Homogeneous Neural Networks
Jun 3, 2026Harsh Vardhan, Hossein Taheri, Arya MazumdarTwo-Layer Neural Networks Neural Network
A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networks
Jun 3, 2026Tian Ding, Dawei Li, Ruoyu SunTwo-Layer Neural Networks Loss Landscape
Neuron Populations Exhibit Divergent Selectivity with Scale
Jun 2, 2026Amil Dravid, Yasaman Bahri, Alexei A. Efros +1Scaling Laws Power-Law Distribution
Neural Networks Provably Learn Spectral Representations for Group Composition
Jun 2, 2026Jianliang He, Leda Wang, Fengzhuo Zhang +2Implicit Neural Representations Two-Layer Neural Networks
Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
May 29, 2026Tianyu Pang, Vignesh Kothapalli, Shenyang Deng +3Learning Rates Two-Layer Neural Networks
Learning High-Dimensional Parity Functions with Product Networks using Gradient Descent
May 27, 2026Guillaume Larue, Louis-Adrien Dufrène, Quentin Lampin +2Two-Layer Neural Networks Neural Network
T-GINEE: A Tensor-Based Multilayer Graph Representation Learning
May 27, 2026Maolin Wang, Ziting Mai, Xuhui Chen +9Graph Neural Networks Complex Networks
Stochastic Estimation of the Layer-wise Hessian Trace for Monitoring Neural-network Training
May 25, 2026Maxim Bolshim, Alexander KugaevskikhCondition Number Two-Layer Neural Networks
Feature Learning in Wide Neural Networks under $μ$ P: Identifiability and Sparse-Dictionary Decomposition of the Mean-Field LimitFeature Learning in Wide Neural Networks under μP: Identifiability and Sparse-Dictionary Decomposition of the Mean-Field Limit
May 23, 2026Akmal XodarevTwo-Layer Neural Networks Feature Learning
A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification
May 21, 2026Marcel Kühn, Yoon Thelge, Bernd RosenowLearning Rates Scaling Laws
Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
May 21, 2026Margalit Glasgow, Joan BrunaTwo-Layer Neural Networks Wasserstein Gradient Flows
Unveiling Memorization-Generalization Coexistence: A Case Study on Arithmetic Tasks with Label Noise
May 18, 2026Linyu Liu, Pinyan LuOver-Parameterized Neural Networks Memorization
Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descent
May 18, 2026Behrad Moniri, Hamed HassaniTwo-Layer Neural Networks Feature Learning
How does feature learning reshape the function space?
May 18, 2026João Lobo, Bruno Loureiro, Long Tran-Than +1Feature Learning Kernel Method
Rethinking Neural Network Learning Rates: A Stackelberg Perspective
May 15, 2026Sihan Zeng, Sujay Bhatt, Sumitra GaneshLearning Rates Two-Layer Neural Networks
Learning with Shallow Neural Networks on Cluster-Structured Features
May 14, 2026Elisabetta Cornacchia, Laurent MassouliéLow-Dimensional Structure Two-Layer Neural Networks
Scaling Laws from Sequential Feature Recovery: A Solvable Hierarchical Model
May 14, 2026Arie Wortsman-Zurich, Hugo Tabanelli, Yatin Dandi +2Feature Learning Scaling Laws
On the global convergence of gradient descent for wide shallow models with bounded nonlinearities
May 11, 2026Romain Petit, Clarice Poon, Gabriel PeyréTwo-Layer Neural Networks Gradient Descent
Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks
May 11, 2026Minh-Toan Nguyen, Jean BarbierFeature Learning Two-Layer Neural Networks
Criticality and Saturation in Orthogonal Neural Networks
May 7, 2026Max Guillen, Jan E. GerkenTwo-Layer Neural Networks Orthogonality
Communication Dynamics Neural Networks: FFT-Diagonalized Layers for Improved Hessian Conditioning at Reduced Parameter Count
May 4, 2026Lurong PanTwo-Layer Neural Networks Condition Number
Feature Repulsion and Spectral Lock-in: An Empirical Study of Two-Layer Network Grokking
Apr 28, 2026Yongzhong XuFeature Learning Grokking
Quantization robustness from dense representations of sparse functions in high-capacity kernel associative memory
Apr 22, 2026Akira TamamoriHopfield Networks Quantization
Dataset Distillation Efficiently Encodes Low-Dimensional Representations from Gradient-Based Learning of Non-Linear Tasks
Mar 16, 2026Yuri Kinoshita, Naoki Nishikawa, Taro ToyoizumiLow-Dimensional Structure Context Compression
Mean-Field Model for Two-Layer Neural Networks Trained with Consensus-Based Optimization
Nov 26, 2025William De Deyn, Michael Herty, Giovanni SamaeyNeural Network Optimization Two-Layer Neural Networks
The Alexander-Hirschowitz theorem for neurovarieties
Nov 24, 2025A. Massarenti, M. MellaTwo-Layer Neural Networks Identifiability
Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU
Oct 29, 2025Jingzhou LiuTwo-Layer Neural Networks Rectified Linear Unit

Sep 12, 2026 · cs.AIJ/K move · Enter open · S save

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

Anish Kataria

Neural networks trained past memorization frequently undergo a delayed transition to generalization, a phenomenon known as grokking. Despite theoretical progress on \emph{why} this transition occurs, the quantitative structure of \emph{when} it occurs in hyperparameter space remains uncharacterized. We map the memorization-to-generalization boundary across 384 configurations of two-hidden-layer MLPs on modular arithmetic, fitting a power-law scaling relation for generalization onset time:

T_{\mathrm{grok}} \propto H^{-0.27}\, D^{-2.04}\, \eta^{-0.50}\, \lambda^{-0.64}

(

R^2 = 0.732

;

0.821

with interactions). The exponent hierarchy reveals that data complexity (

D^{-2.04}

) is the dominant driver of regime transition, not model capacity (

H^{-0.27}

): doubling data accelerates generalization by

{\sim}4\times

, while doubling width yields only

{\sim}1.2\times

. A sharp phase boundary at weight decay

\lambda \gtrsim 1.0

separates grokking from non-grokking configurations, and weight norm trajectories show monotonic compression during the transition, consistent with implicit regularization selecting low-complexity solutions. These results provide a quantitative foundation for predicting and controlling regime transitions in overparameterized networks.

Grokking Over-Parameterized Neural Networks Two-Layer Neural Networks Memorization Scaling Laws Structural Generalization Exponents Transitions

Open paper Ask about it PDF

Recent momentum

Neighbouring directions

Recent digests

4 new papers

Latest in Two-Layer Neural Networks