Deep Linear Networks

Follow this topic Ask about this topic

Deep linear networks are compositions of linear transformations used to study how depth changes learning, even when the overall input-output function remains linear. Researchers analyze gradient flow and gradient descent, matrix factorization, spectral dynamics, and convex reformulations across settings such as regression, classification, and matrix completion. These models help explain depth-dependent implicit biases, including preferences for low-rank or sparse solutions, as well as feature compression, convergence, and loss-landscape geometry. A central open tension is how reliably these exact results transfer to nonlinear networks, since behavior can depend strongly on initialization, data structure, optimizer, and finite-time dynamics.

Latest papers

30 papers8 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks
Oct 26, 2026Taishi Watanabe, Ryo Karakida, Jun-nosuke TeramaeScaling Laws Neural Network Training Dynamics
Kyoto University · AIST · Institute of Science Tokyo
Width and Depth Limits Commute During Training in Linear Residual Networks
Oct 22, 2026Ruijia Zhang, Soufiane HayouNeural Network Training Dynamics Deep Linear Networks
Department of Applied Mathematics and Statistics, Johns Hopkins University
Convergence of stochastic gradient descent for learning linear neural networks
Oct 20, 2026Gabin Maxime Nguegnang, Bubacarr Bah, Holger Rauhut +1Stochastic Optimization Convergence Stochastic Gradient Descent
A Closed Form Depth-Width Law for Neural Collapse in Deep Linear Networks
Oct 15, 2026Pankhil Gawade, Kirsten Fischer, Guy Wolf +1Neural Representation Geometry Neural Scaling Laws
Institute of AI for Health, Helmholtz Munich · School of Computation, Information and Technology, Technical University of Munich · Université de Montréal +3
The Accuracy Well: Nonmonotone Accuracy Dynamics at Vanishing Initialization
Oct 14, 2026Yaroslav Gusev, Eugene Golikov, Dmitry YarotskyNeural Network Training Dynamics Gradient Descent Dynamics
Applied AI Institute, Moscow, Russia · Steklov Mathematical Institute of Russian Academy of Sciences, Moscow, Russia
Global Exponential Convergence of Two-Layer Linear Network Training
Oct 7, 2026Stephen Y Zhang, Gabriel PeyréNeural Network Training Dynamics Gradient Descent Dynamics
Flatiron Institute & NYU · CNRS, ENS, PSL Univ.
Learning the identity: a case study of how SGD selects among functional decompositions
Sep 30, 2026Andy Arditi, Weian Xie, David Bau +1Deep Linear Networks Implicit Bias
Northeastern University · Massachusetts Institute of Technology
Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks
Sep 30, 2026Lukas Braun, Erin Grant, Andrew M. SaxeRepresentational Similarity Analysis Neural Representation Geometry
Department of Experimental Psychology, University of Oxford, Oxford, UK · Gatsby Unit & Sainsbury Wellcome Centre, University College London, London, UK
Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks
Sep 28, 2026Alexandre Declèves, Etienne Boursier, Nicolas FlammarionFine-Tuning Deep Linear Networks
TML, EPFL, Lausanne, Switzerland · INRIA, LMO, Université Paris-Saclay, Orsay, France
Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks
Aug 6, 2026Björn Ladewig, Ibrahim Talha Ersoy, Karoline WiesnerStatistical Physics of Learning Representation Learning
Institute of Physics and Astronomy, University of Potsdam
New Complexity-Theoretic Frontiers of Tractability for Neural Network Training
Jul 23, 2026Cornelius Brand, Robert Ganian, Mathis RoctonReLU Neural Networks Neural Network Optimization
Algorithms & Complexity Group Vienna University of Technology Favoritenstraße 9-11, 1040 Vienna, Austria
How the Hessian-Spectrum of Neural Networks Depends on Data
Jul 15, 2026Jasraj Singh, Enea Monzio Compagnoni, Antonio OrvietoLoss Landscape Sharpness Neural Network Optimization
ELLIS Institute, Tübingen, Germany · Universität Basel, Basel, Switzerland · MPI-IS, Tübingen, Germany
Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initialization
Jul 14, 2026Jiajie Zhao, Jianxing Wang, Junjie Yang +2Deep Linear Networks Implicit Bias
School of Mathematical Sciences, Shanghai Jiao Tong University · Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University · School of Artificial Intelligence, Shanghai Jiao Tong University
How are linear representations learned? Exact solutions to the dynamics of abstraction
Jul 9, 2026William W. Yang, Andrew M. Saxe, Peter E. LathamLinear Representation Hypothesis Deep Linear Networks
Gatsby Computational Neuroscience Unit, University College London, London W1T 4JG, United Kingdom. · Sainsbury Wellcome Centre, University College London, London W1T 4JG, United Kingdom.
Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
Jul 8, 2026Yedi Zhang, Peter E. Latham, Leena Chennuru Vankadara +1Learning Rate Scheduling Deep Linear Networks
Gatsby Computational Neuroscience Unit, University College London · Sainsbury Wellcome Centre, University College London
On the Entropy Formula for Real, Complex, and Quaternionic Deep Linear Networks
Jun 15, 2026Luis Contreras, Marco Nahas, Tejas KotwalComplex-Valued Neural Networks Deep Linear Networks
Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
Jun 4, 2026Hu Tan, Kuo Gai, Shihua ZhangReLU Neural Networks Deep Linear Networks
1State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · 3Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China +1
Gradient Descent with Large Step Size Restores Symmetry in Deep Linear Networks with Multi-Pathway
May 29, 2026Hee-Sung Kim, Sungyoon LeeEdge of Stability Symmetry Breaking
Department of Computer Science, Hanyang University, Seoul, Korea.
The Implicit Bias of Depth: From Neural Collapse to Softmax Codes
May 21, 2026Connall Garrod, Jonathan P. Keating, Christos ThrampoulidisDeep Linear Networks Implicit Bias
Mathematical Institute, University of Oxford · Department of Electrical and Computer Engineering, University of British Columbia
Exact Convex Reformulations of Linear Neural Networks via Completely Positive Lifting
May 17, 2026Karthik Prakhya, Alp YurtseverNeural Network Optimization Deep Linear Networks
Umeå University, Sweden
Understanding Sample Efficiency in Predictive Coding
May 12, 2026Gaspard Oliviers, Elene Lominadze, Rafal BogaczBackpropagation Predictive Coding
Nuffield Department of Clinical Neurosciences, University of Oxford, United Kingdom · MRC Centre of Research Excellence in Restorative Neural Dynamics, United Kingdom
Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer
May 8, 2026Clarissa Lauditi, Cengiz Pehlevan, Blake BordelonDeep Linear Networks Neural Network Training Dynamics
John A. Paulson School of Engineering and Applied Sciences, Harvard University · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University · Center for Mathematical Sciences and Applications, Harvard University +1
Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networks
Apr 7, 2026Guillaume Corlouer, Avi Semler, Alexander Strang +1Gradient Descent Dynamics Stochastic Gradient Descent
Moirai · University of Oxford · University of California, Berkeley +1
On the Loss Landscape Geometry of Regularized Deep Matrix Factorization: Uniqueness and Sharpness
Mar 28, 2026Anil Kamber, Rahul ParhiLoss Landscape Loss Landscape Geometry
UC San Diego
Minor First, Major Last: A Depth-Induced Implicit Bias of Sharpness-Aware Minimization
Mar 9, 2026Chaewon Moon, Dongkuk Si, Chulhee YunSharpness-Aware Minimization Implicit Bias
Kim Jaechul Graduate School of AI, KAIST · Mobilint, Inc.
Implicit Bias and Loss of Plasticity in Matrix Completion: Depth Promotes Low-Rankness
Mar 5, 2026Baekrok Shin, Chulhee YunImplicit Bias Neural Network Training Dynamics
Kim Jaechul Graduate School of AI, KAIST
To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters
Feb 28, 2026Sara Dragutinović, Yedi Zhang, Rajesh RanganathNeural Network Generalization Muon Optimizer
Courant Institute School of Mathematics, Computing, and Data Science, New York University · Gatsby Computational Neuroscience Unit, University College London
Geodesics in the Deep Linear Network
Sep 18, 2025Alan ChenDeep Linear Networks Riemannian Geometry
The Geometry of the Set of Equivalent Linear Neural Networks
Apr 23, 2024Jonathan Richard Shewchuk, Sagnik BhattacharyaDeep Linear Networks Weight-Space Analysis
Department of Electrical Engineering and Computer Sciences University of California at Berkeley Berkeley, California 94720
Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination
Nov 6, 2023Peng Wang, Xiao Li, Can Yaras +4Hierarchical Representation Learning Deep Linear Networks
University of Michigan · Ohio State University

Oct 26, 2026J/K move · Enter open · S save

Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks

Taishi Watanabe, Ryo Karakida, Jun-nosuke Teramae

Kyoto University · AIST · Institute of Science Tokyo

Scaling Laws Neural Network Training Dynamics Neural Scaling Laws Deep Linear Networks

Open paper AskPDF arXiv

Neural scaling laws describe the predictable power-law dependence of model performance on training time, dataset size, and model scale, with the exponents quantifying how performance changes with resources. Recent theory has shown how scaling exponents are shaped by the data distribution and the structure of the learning task, but whether network depth can improve training-time scaling has remained unsettled. Through a refined analysis of deep linear networks, we clarify that the task spectrum determines when increasing network depth improves training-time scaling. For spectrally difficult tasks, increasing depth improves the asymptotic training-time scaling exponent, whereas for easy tasks, the exponent remains independent of depth, with depth affecting only a logarithmic correction at the boundary between these regimes. Remarkably, we find that a reparameterization of time governed by the predictor norm provides a unified description of the dynamics across task difficulties. Under this reparameterization, the loss follows a common scaling law whose exponent is independent of depth. Numerical experiments on a task derived from benchmark image data and with nonlinear networks exhibit qualitatively consistent task-dependent effects of depth on training-time scaling. Our findings advance the theoretical understanding of how task structure shapes the effects of network architecture on learning and offer a basis for studying scaling behavior across broader classes of neural networks.