Deep Linear

Deep linear networks, simplified models of deep neural networks, are used to gain theoretical insights into the learning dynamics and generalization capabilities of their more complex counterparts. Current research focuses on understanding the impact of initialization strategies, optimization algorithms (including gradient descent and predictive coding), and architectural features (like depth and width) on learning dynamics and implicit regularization, often analyzing specific model architectures like linear state space models and convolutional networks. These studies provide crucial theoretical foundations for understanding phenomena like neural collapse and critical learning periods, ultimately informing the design and improvement of more efficient and robust deep learning algorithms for various applications, including image deblurring and matrix completion.

Papers

April 4, 2024

Information-Theoretic Generalization Bounds for Deep Neural Networks
Haiyun He, Christina Lee Yu, Ziv Goldfeld
Deep Neural Network DNN Model Deep Linear Systematic Generalization Information Theoretic Generalization Bound

February 6, 2024

Estimating the Local Learning Coefficient at Scale
Zach Furman, Edmund Lau
Visual Analogue Scale Deep Learning Architecture Deep Linear Singular Learning Learning Coefficient

February 4, 2024

On the Role of Initialization on the Implicit Bias in Deep Linear Networks
Oria Gruber, Haim Avron
Deep Learning Deep Neural Network Integral Role Deep Network Implicit Bias New Initialization Deep Linear

January 29, 2024

Algebraic Complexity and Neurovariety of Linear Convolutional Networks
Vahid Shahverdi
Deep Linear Linear Network Polynomial Complexity One Dimensional Neuronal Diversity

January 9, 2024

Linear Recursive Feature Machines provably recover low-rank matrices
Adityanarayanan Radhakrishnan, Mikhail Belkin, Dmitriy Drusvyatskiy
Feature Learning Auto Encoder Deep Linear Low Rank Matrix Feature Vector Sparse Linear Regression

November 23, 2023

Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation
Markus Gross, Arne P. Raulf, Christoph Räth
Stochastic Gradient Descent Deep Linear Linear Neural Network Mathematical Derivation Flatness Aware Weight Monitoring

November 8, 2023

Efficient Compression of Overparameterized Deep Models through Low-Dimensional Learning Dynamics
Soo Min Kwon, Zekai Zhang, Dogyoon Song, Laura Balzano, Qing Qu
Neural Network Matrix Factorization Learning Dynamic Compression Technique Deep Linear Overparametrization Bound Deep Nonlinear

November 6, 2023

Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination
Peng Wang, Xiao Li, Can Yaras, Zhihui Zhu, Laura Balzano, Wei Hu, Qing Qu
Deep Learning Deep Network Deep Representation Deep Linear Group DIscrimination Feature Compression Deep Layer Deep Nonlinear

November 1, 2023

Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures
Runa Eschenhagen, Alexander Immer, Richard E. Turner, Frank Schneider, Philipp Hennig
Deep Linear Approximate Curvature Modern Neural Network Architecture

October 19, 2023

Training Dynamics of Deep Network Linear Regions
Ahmed Imtiaz Humayun, Randall Balestriero, Richard Baraniuk
Deep Network Generalization Performance Training Dynamic Deep Linear Leaky ReLU

October 9, 2023

On the Convergence of Federated Averaging under Partial Participation for Over-parameterized Neural Networks
Xin Liu, Wei li, Dazhi Zhan, Yu Pan, Xin Ma, Yu Ding, Zhisong Pan
Neural Network Early Stage Convergence Federated Averaging Deep Linear Two Layer ReLU FedAvg Converges Partial Participation

August 23, 2023

Critical Learning Periods Emerge Even in Deep Linear Networks
Michael Kleinman, Alessandro Achille, Stefano Soatto
Neural Network Deep Network Deep Linear Critical Period

July 31, 2023

Stochastic positional embeddings improve masked image modeling
Amir Bar, Florian Bordes, Assaf Shocher, Mahmoud Assran, Pascal Vincent, Nicolas Ballas, Trevor Darrell, Amir Globerson, Yann LeCun
Image Modeling Masked Image Modeling Deep Linear Location Uncertainty Stochastic Positional Embeddings

June 22, 2023

The Inductive Bias of Flatness Regularization for Deep Matrix Factorization
Khashayar Gatmiry, Zhiyuan Li, Ching-Yao Chuang, Sashank Reddi, Tengyu Ma, Stefanie Jegelka
Neural Network Inductive Bias Matrix Factorization Deep Linear Implicit Regularization Effect Flatness Aware

June 20, 2023

The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks
Yuan Cao, Difan Zou, Yuanzhi Li, Quanquan Gu
Stochastic Gradient Descent Batch Normalization Implicit Bias Deep Linear Linear Model Margin Classifier Linear Neural Network

June 1, 2023

May 27, 2023

Knowledge Distillation Performs Partial Variance Reduction
Mher Safaryan, Alexandra Peste, Dan Alistarh
Deep Neural Network Knowledge Distillation Variance Reduction Convex Loss Deep Linear Distillation Loss

May 25, 2023

Implicit bias of SGD in $L_{2}$-regularized linear DNNs: One-way jumps from high to low rank
Zihan Wang, Arthur Jacot
Stochastic Gradient Descent Low Rank Implicit Bias Matrix Completion Hidden Layer Deep Linear Linear Neural Network Local Minimum

May 9, 2023

Robust Implicit Regularization via Weight Normalization
Hung-Hsu Chou, Holger Rauhut, Rachel Ward
Gradient Descent Gradient Flow Implicit Regularization Deep Linear Weight Normalization