Lazy Training

Lazy training describes a phenomenon in neural network training where model parameters remain close to their initial values, effectively behaving like a linear model despite the network's complexity. Current research focuses on understanding the conditions under which lazy training occurs, its relationship to generalization performance (including benign overfitting and adversarial robustness), and its implications for different architectures and training algorithms, such as gradient descent and biologically-plausible alternatives. This research is significant because it provides insights into the dynamics of deep learning, potentially leading to more efficient and robust training methods, and offering theoretical explanations for observed empirical phenomena like grokking.

Papers

April 7, 2024

Demystifying Lazy Training of Neural Networks from a Macroscopic Viewpoint
Yuqing Li, Tao Luo, Qixuan Zhou
Neural Network Gradient Descent Neural Architecture Training Dynamic Gradient Descent Dynamic Lazy Training Sparse Target Macroscopic Change

October 9, 2023

Grokking as the Transition from Lazy to Rich Training Dynamics
Tanishq Kumar, Blake Bordelon, Samuel J. Gershman, Cengiz Pehlevan
Neural Network Strong Generalization Gradient Descent Neural Tangent Kernel Training Dynamic Grokking Phenomenon Lazy Search Late Time Transition Lazy Training

May 30, 2023

Benign Overfitting in Deep Neural Networks under Lazy Training
Zhenyu Zhu, Fanghui Liu, Grigorios G Chrysos, Francesco Locatello, Volkan Cevher
Neural Network Deep Neural Network Neural Tangent Kernel Generalization Error Benign Overfitting Smooth Activation Function Lazy Training

April 26, 2023

Feed-Forward Optimization With Delayed Feedback for Neural Networks
Katharina Flügel, Daniel Coquelin, Marie Weiel, Charlotte Debus, Achim Streit, Markus Götz
Neural Network Back Propagation Delayed Feedback Plausible Learning Improved Training Lazy Training Natural Learning

September 19, 2022

Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficulty
Thomas George, Guillaume Lajoie, Aristide Baratin
Deep Neural Network Deep Network Task Scheduling Spurious Correlation Feature Learning Lazy Search Reverse Graph Linearization Lazy Training Hard Example

September 13, 2022

Skip Training for Multi-Agent Reinforcement Learning Controller for Industrial Wave Energy Converters
Soumyendu Sarkar, Vineet Gundecha, Sahand Ghorbanpour, Alexander Shmakov, Ashwin Ramesh Babu, Alexandre Pichard, Mathieu Cocho
Optimal Controller Lazy Training Wave Energy Converter

June 18, 2022

Adversarial Robustness is at Odds with Lazy Training
Yunjuan Wang, Enayat Ullah, Poorya Mianjy, Raman Arora
Neural Network Adversarial Example Adversarial Robustness Relative Probability Deep Learning Theory Random Neural Network Lazy Training

March 22, 2022

On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimes
Elvis Dohmatob, Alberto Bietti
Neural Network Native Robustness Adversarial Example Adversarial Robustness Two Layer Neural Network Regime Switching Non Robust Lazy Training

February 16, 2022

Quantum Lazy Training
Erfan Abedi, Salman Beigi, Leila Taghavi
Parameterized Quantum Circuit Parameterized Model Quantum Model Sequential Quantum Enhanced Training Lazy Training