Deep Linear Networks

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 19

All topics

Global Exponential Convergence of Two-Layer Linear Network Training
Oct 7, 2026Stephen Y Zhang, Gabriel PeyréNeural Network Training Dynamics Gradient Descent Dynamics
Flatiron Institute & NYU · CNRS, ENS, PSL Univ.
Learning the identity: a case study of how SGD selects among functional decompositions
Sep 30, 2026Andy Arditi, Weian Xie, David Bau +1Deep Linear Networks Implicit Bias
Northeastern University · Massachusetts Institute of Technology
Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks
Sep 30, 2026Lukas Braun, Erin Grant, Andrew M. SaxeRepresentational Similarity Analysis Neural Representation Geometry
Department of Experimental Psychology, University of Oxford, Oxford, UK · Gatsby Unit & Sainsbury Wellcome Centre, University College London, London, UK
Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks
Sep 28, 2026Alexandre Declèves, Etienne Boursier, Nicolas FlammarionFine-Tuning Deep Linear Networks
TML, EPFL, Lausanne, Switzerland · INRIA, LMO, Université Paris-Saclay, Orsay, France
Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks
Aug 6, 2026Björn Ladewig, Ibrahim Talha Ersoy, Karoline WiesnerStatistical Physics of Learning Representation Learning
Institute of Physics and Astronomy, University of Potsdam
New Complexity-Theoretic Frontiers of Tractability for Neural Network Training
Jul 23, 2026Cornelius Brand, Robert Ganian, Mathis RoctonReLU Neural Networks Neural Network Optimization
Algorithms & Complexity Group Vienna University of Technology Favoritenstraße 9-11, 1040 Vienna, Austria
How the Hessian-Spectrum of Neural Networks Depends on Data
Jul 15, 2026Jasraj Singh, Enea Monzio Compagnoni, Antonio OrvietoNeural Network Optimization Deep Linear Networks
ELLIS Institute, Tübingen, Germany · Universität Basel, Basel, Switzerland · MPI-IS, Tübingen, Germany
Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initialization
Jul 14, 2026Jiajie Zhao, Jianxing Wang, Junjie Yang +2Deep Linear Networks Implicit Bias
School of Mathematical Sciences, Shanghai Jiao Tong University · Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University · School of Artificial Intelligence, Shanghai Jiao Tong University
How are linear representations learned? Exact solutions to the dynamics of abstraction
Jul 9, 2026William W. Yang, Andrew M. Saxe, Peter E. LathamLinear Representation Hypothesis Deep Linear Networks
Gatsby Computational Neuroscience Unit, University College London, London W1T 4JG, United Kingdom. · Sainsbury Wellcome Centre, University College London, London W1T 4JG, United Kingdom.
Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
Jul 8, 2026Yedi Zhang, Peter E. Latham, Leena Chennuru Vankadara +1Learning Rate Scheduling Deep Linear Networks
Gatsby Computational Neuroscience Unit, University College London · Sainsbury Wellcome Centre, University College London
On the Entropy Formula for Real, Complex, and Quaternionic Deep Linear Networks
Jun 15, 2026Luis Contreras, Marco Nahas, Tejas KotwalComplex-Valued Neural Networks Deep Linear Networks
Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
Jun 4, 2026Hu Tan, Kuo Gai, Shihua ZhangReLU Neural Networks Deep Linear Networks
1State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · 3Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China +1
Gradient Descent with Large Step Size Restores Symmetry in Deep Linear Networks with Multi-Pathway
May 29, 2026Hee-Sung Kim, Sungyoon LeeEdge of Stability Symmetry Breaking
Department of Computer Science, Hanyang University, Seoul, Korea.
The Implicit Bias of Depth: From Neural Collapse to Softmax Codes
May 21, 2026Connall Garrod, Jonathan P. Keating, Christos ThrampoulidisDeep Linear Networks Implicit Bias
Mathematical Institute, University of Oxford · Department of Electrical and Computer Engineering, University of British Columbia
Exact Convex Reformulations of Linear Neural Networks via Completely Positive Lifting
May 17, 2026Karthik Prakhya, Alp YurtseverNeural Network Optimization Deep Linear Networks
Umeå University, Sweden
Understanding Sample Efficiency in Predictive Coding
May 12, 2026Gaspard Oliviers, Elene Lominadze, Rafal BogaczBackpropagation Predictive Coding
Nuffield Department of Clinical Neurosciences, University of Oxford, United Kingdom · MRC Centre of Research Excellence in Restorative Neural Dynamics, United Kingdom
Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer
May 8, 2026Clarissa Lauditi, Cengiz Pehlevan, Blake BordelonDeep Linear Networks Neural Network Training Dynamics
John A. Paulson School of Engineering and Applied Sciences, Harvard University · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University · Center for Mathematical Sciences and Applications, Harvard University +1
To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters
Feb 28, 2026Sara Dragutinović, Yedi Zhang, Rajesh RanganathNeural Network Generalization Muon Optimizer
Courant Institute School of Mathematics, Computing, and Data Science, New York University · Gatsby Computational Neuroscience Unit, University College London
Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination
Nov 6, 2023Peng Wang, Xiao Li, Can Yaras +4Hierarchical Representation Learning Deep Linear Networks
University of Michigan · Ohio State University

Oct 7, 2026, cs.LGJ/K move · Enter open · S save

Global Exponential Convergence of Two-Layer Linear Network Training

Stephen Y Zhang, Gabriel Peyré

Flatiron Institute & NYU · CNRS, ENS, PSL Univ.

Neural Network Training Dynamics Gradient Descent Dynamics Deep Linear Networks

Open paper AskPDF arXiv

We prove global exponential (linear) convergence with an explicit rate in the rich scaling for wide two-layer linear networks trained with smooth Polyak-Lojasiewicz predictor losses. Gradient flow in the factors closes exactly in terms of a finite-dimensional Bures flow of the neuron law covariance, in which the predictor dynamics are preconditioned by hidden covariance blocks. Mean-field conservation laws provide uniform spectral lower bounds on the hidden preconditioning blocks when the initial covariance satisfies a spectral support gap condition. This condition encompasses positive definiteness while still allowing for singular initializations. For an initial covariance

Σ_0 = σ^2 \mathrm{Id}

, the loss converges to the global minimum with linear rate at least

4σ^2κ

, where

κ

is the PL constant. We establish stability of this rate under finite-width sampling, as well as global convergence of factor gradient descent for an explicit stepsize interval depending on smoothness, the initial loss, and conserved spectral margins. Our argument extends layerwise to deep linear ResNets, subject to a residual-path bound. In the case of heavy-ball momentum, training dynamics close instead over positions and velocities in terms of a lifted phase covariance. Linear convergence holds under an explicit condition on the energy and damping, specifying a window of admissible dampings. For two-scale white initializations, this interval is nonempty for sufficiently large position scales, with a fixed initial loss gap and velocity covariance. Numerical experiments illustrate the covariance geometry and compare the predicted and observed rates.

Momentum

Related topics

Latest papers 19