Linear Neural Network

Linear neural networks, despite their apparent simplicity, are a crucial area of research in deep learning, serving as a tractable model for investigating fundamental optimization and generalization properties. Current research focuses on understanding the dynamics of gradient descent in these networks, particularly in overparameterized settings and across various architectures including fully-connected and convolutional models, analyzing phenomena like double descent and the impact of regularization techniques such as batch normalization and L2 regularization. These studies provide valuable insights into the behavior of more complex nonlinear networks and contribute to a deeper theoretical understanding of deep learning's success, impacting both algorithm design and the interpretation of model behavior.

Papers

October 12, 2024

September 11, 2024

Convergence of continuous-time stochastic gradient descent with applications to linear deep neural networks
Gabor Lugosi, Eulalia Nualart
Deep Neural Network Financial Application Gradient Descent Stochastic Gradient Descent Early Stage Convergence Linear Neural Network Sufficient Condition

August 25, 2024

Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learning
Nadav Cohen, Noam Razin
Neural Network Deep Learning Strong Generalization Optimization Purpose Statistical Learning Theory Linear Neural Network Lecture Note

July 15, 2024

Deep Learning Evidence for Global Optimality of Gerver's Sofa
Kuangdai Leng, Jia Bi, Jaehoon Cha, Samuel Pinilla, Jeyan Thiyagalingam
Global Optimality Linear Neural Network Deep Learning Inference Tabletop Rearrangement

July 13, 2024

Towards Understanding Epoch-wise Double descent in Two-layer Linear Neural Networks
Amanda Olmin, Fredrik Lindsten
Double Descent Linear Neural Network Diagonal Linear Network Epoch Wise Double Descent

May 30, 2024

GraphAny: A Foundation Model for Node Classification on Any Graph
Jianan Zhao, Hesham Mostafa, Mikhail Galkin, Michael Bronstein, Zhaocheng Zhu, Jian Tang
Graph Drawing Foundation Model Node Classification Graph Structured Data Graph Machine Learning Supervised Baseline Linear Neural Network Inductive Node

April 23, 2024

The Geometry of the Set of Equivalent Linear Neural Networks
Jonathan Richard Shewchuk, Sagnik Bhattacharya
Neural Network Geometric Analysis Linear Neural Network Closing Set Weight Space Tangent Space

February 21, 2024

Asymptotics of Learning with Deep Structured (Random) Features
Dominik Schröder, Daniil Dmitriev, Hugo Cui, Bruno Loureiro
LeArning Abstract Feature Wise Random Participation Linear Neural Network Asymptotic Behavior Covariance Alignment Width Neural Network

December 30, 2023

A Novel Explanation Against Linear Neural Networks
Anish Lakkapragada
Neural Network Long Form Novel Linear Regression Linear Neural Network Nonlinear Function

November 23, 2023

Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation
Markus Gross, Arne P. Raulf, Christoph Räth
Stochastic Gradient Descent Deep Linear Linear Neural Network Mathematical Derivation Flatness Aware Weight Monitoring

September 24, 2023

Geometry of Linear Neural Networks: Equivariance and Invariance under Permutation Groups
Kathlén Kohn, Anna-Laura Sattelberger, Vahid Shahverdi
Geometric Analysis Inter Part Equivariance Equivariant Network Alpha Invariance Linear Neural Network Invariant Network Algebraic Variety Equivariant Function Invariant Function

June 20, 2023

The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks
Yuan Cao, Difan Zou, Yuanzhi Li, Quanquan Gu
Stochastic Gradient Descent Batch Normalization Implicit Bias Deep Linear Linear Model Margin Classifier Linear Neural Network

May 25, 2023

Implicit bias of SGD in $L_{2}$-regularized linear DNNs: One-way jumps from high to low rank
Zihan Wang, Arthur Jacot
Stochastic Gradient Descent Low Rank Implicit Bias Matrix Completion Hidden Layer Deep Linear Linear Neural Network Local Minimum

May 17, 2023

On the ISS Property of the Gradient Flow for Single Hidden-Layer Neural Networks with Linear Activations
Arthur Castello B. de Oliveira, Milad Siami, Eduardo D. Sontag
Native Robustness Gradient Descent Gradient Flow Overparametrization Bound Benign Overfitting Linear Neural Network Layer Neural Network Linear Activation

May 15, 2023

ReLU soothes the NTK condition number and accelerates optimization for wide neural networks
Chaoyue Liu, Like Hui
Optimization Purpose Activation Function Neural Tangent Kernel Wide Neural Network Linear Neural Network Rectified Linear Unit

April 27, 2023

An Algorithm for Computing with Brauer's Group Equivariant Neural Network Layers
Edward Pearce-Crump
Practical Algorithm Digital Computing Linear Neural Network Kronecker Product Orthogonal Basis Symplectic Group Group Equivariant Neural Network Tensor Power

March 16, 2023

Bayesian Generalization Error in Linear Neural Networks with Concept Bottleneck Structure and Multitask Formulation
Naoki Hayashi, Yoshihide Sawada
Multi Task Generalization Performance Generalization Error Concept Bottleneck Model Linear Neural Network Three Layer Neural Network

March 10, 2023

Enhancing the accuracies by performing pooling decisions adjacent to the output layer
Yuval Meir, Yarden Tzach, Ronit D. Gross, Ofek Tevet, Roni Vardi, Ido Kanter
Decision Relevant Information Deep Architecture Linear Neural Network High Accuracy Last Layer Layer Output Pooling Operator

March 7, 2023

Continuous Function Structured in Multilayer Perceptron for Global Optimization
Heeyuen Koh
Multi Layer Natural Gradient Global Optimization Gradient Information Linear Neural Network Continuous Space

Linear Neural Network

Papers

Provable Acceleration of Nesterov's Accelerated Gradient for Rectangular Matrix Factorization and Linear Neural Networks

Training Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow Analysis

Convergence of continuous-time stochastic gradient descent with applications to linear deep neural networks

Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learning

Deep Learning Evidence for Global Optimality of Gerver's Sofa

Towards Understanding Epoch-wise Double descent in Two-layer Linear Neural Networks

GraphAny: A Foundation Model for Node Classification on Any Graph

The Geometry of the Set of Equivalent Linear Neural Networks

Asymptotics of Learning with Deep Structured (Random) Features

A Novel Explanation Against Linear Neural Networks

Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation

Geometry of Linear Neural Networks: Equivariance and Invariance under Permutation Groups

The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks

Implicit bias of SGD in $L_{2}$-regularized linear DNNs: One-way jumps from high to low rank

On the ISS Property of the Gradient Flow for Single Hidden-Layer Neural Networks with Linear Activations

ReLU soothes the NTK condition number and accelerates optimization for wide neural networks

An Algorithm for Computing with Brauer's Group Equivariant Neural Network Layers

Bayesian Generalization Error in Linear Neural Networks with Concept Bottleneck Structure and Multitask Formulation

Enhancing the accuracies by performing pooling decisions adjacent to the output layer

Continuous Function Structured in Multilayer Perceptron for Global Optimization