Two-Layer Neural Networks

Latest papers

14 papers2 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Silent Alignment in Two-Layer Linear Networks: A High-Dimensional Perspective
Oct 21, 2026Simon Martin, Francis Bach, Giulio BiroliTwo-Layer Neural Networks Neural Network Training Dynamics
DI-ENS, LPENS, SIERRA · DI-ENS, SIERRA · LPENS
A Distribution-Free Separation Between Learned and Fixed Features
Oct 18, 2026Daniel BarzilaiTwo-Layer Neural Networks Statistical Learning Theory
Weizmann Institute of Science
Singular perturbations and hierarchical learning in two-layer neural networks
Jul 12, 2026Cédric Gerbelot, Jean-Christophe MourratMultilayer Perceptrons Neural Network Training Dynamics
Unité de Mathématiques Pures et Appliquées (UMPA), ENS Lyon · CNRS
A law of robustness for two-layer neural networks with arbitrary weights
Jul 8, 2026Yitzchak ShmaloNeural Network Memorization Multilayer Perceptrons
Compressed Computation under $L^4$ Loss is likely Computation in Superposition
Jul 6, 2026Francisco Ferreira da Silva, Stefan HeimersheimReLU Neural Networks Feedforward Neural Networks
Robustness Cannot be Reduced to Regularization: Studying Adversarial Training Beyond the Linear Case
Jun 19, 2026David A. R. Robin, Rafael Pinot, Yann ChevaleyreAdversarial Training Two-Layer Neural Networks
LAMSADE, Dauphine · Universit´e Paris Dauphine PSL Research University · LPSM, Jussieu +1
A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networks
Jun 3, 2026Tian Ding, Dawei Li, Ruoyu SunTwo-Layer Neural Networks Neural Network Optimization
Shenzhen International Center of Industrial and Applied Mathematics, Shenzhen Research Institute of Big Data, China; Shenzhen Loop Area Institute, China; AutoKernel, China. · Shenzhen Loop Area Institute, China · AutoKernel, China +3
Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
May 29, 2026Tianyu Pang, Vignesh Kothapalli, Shenyang Deng +3Deep Learning Optimization Neural Network Training Dynamics
Dartmouth College · Stanford University · Virginia Tech
How does feature learning reshape the function space?
May 18, 2026João Lobo, Bruno Loureiro, Long Tran-Than +1Representation Learning Two-Layer Neural Networks
Department of Computer Science, University of Warwick, United Kingdom · Departement d’Informatique, École Normale Supérieure, PSL & CNRS · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China. Part of work was done at Department of Computer Science, and Centre for Discrete Mathematics and its Applications (DIMAP), University of Warwick, United Kingdom
Dataset Distillation Efficiently Encodes Low-Dimensional Representations from Gradient-Based Learning of Non-Linear Tasks
Mar 16, 2026Yuri Kinoshita, Naoki Nishikawa, Taro ToyoizumiTwo-Layer Neural Networks Dataset Distillation
The University of Tokyo, Tokyo, Japan · Laboratory for Neural Computation and Adaptation, RIKEN Center for Brain Science, Wako, Japan · RIKEN Center for Advanced Intelligence Project, Nihonbashi, Japan
Upper Bounds for Local Learning Coefficients of Three-Layer Neural Networks
Mar 13, 2026Yuki KurumadaniTwo-Layer Neural Networks Singular Learning Theory
The University of Osaka, 1 Chome-3 Machikaneyamacho, Toyonaka, Osaka 560-8531, Japan
On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD
Mar 11, 2026Tongcheng Zhang, Zhanpeng Zhou, Mingze Wang +4Two-Layer Neural Networks Neural Network Training Dynamics
Sch. of Computer Science and Zhiyuan College, Shanghai Jiao Tong University · Peking University · University of Sydney +4
A new Uncertainty Principle in Machine Learning
Feb 24, 2026V. Dolotin, A. MorozovTwo-Layer Neural Networks Neural Network Optimization
MIPT, Dolgoprudny, 141701, Russia · NRC “Kurchatov Institute", 123182, Moscow, Russia · Institute for Information Transmission Problems, Moscow 127994, Russia +1
Understanding Two-Layer Neural Networks with Smooth Activation Functions
Jul 11, 2025Changcun HuangNeural Network Activation Functions Neural Network Approximation Theory
Shuitu Institute of Applied Mathematics, Chongqing 400700, P.R.C

Oct 21, 2026J/K move · Enter open · S save

Silent Alignment in Two-Layer Linear Networks: A High-Dimensional Perspective

Simon Martin, Francis Bach, Giulio Biroli

DI-ENS, LPENS, SIERRA · DI-ENS, SIERRA · LPENS

Two-Layer Neural Networks Neural Network Training Dynamics Gradient Descent Dynamics Shallow Neural Networks High-Dimensional Asymptotics

Open paper AskPDF arXiv

We study gradient flow in a two-layer linear network in the high-dimensional regime where the input, inner and output dimensions diverge proportionally. Under whitened inputs and isotropic imbalance, we combine a closed-form solution of the dynamics with prior random matrix theory results to derive a high-dimensional limit for the dynamics. Our method yields exact asymptotic spectral distributions for the weight matrices at all times and characterizes the evolution of the learned subspace. In the small-initialization limit, we identify a silent alignment phenomenon: the singular directions of the network align on an order-one timescale, while the singular values and empirical loss remain at their initial values. Visible learning occurs later, on a logarithmic timescale, as singular values progressively escape zero and the loss decreases. This phase is characterized by a saddle-to-saddle dynamics, which becomes a continuous process of spectral learning in the high-dimensional limit.