Large Learning Rate

Large learning rates in neural network training are a focus of current research, aiming to understand their impact on optimization dynamics and generalization performance. Studies explore how large learning rates affect loss landscapes, influencing training trajectories and the types of solutions found, with investigations spanning various architectures and optimizers including SGD, Adam, and their variants. This research is significant because it challenges conventional wisdom about optimal learning rate strategies and offers potential for more efficient and effective training of large-scale models, particularly in resource-constrained environments.

Papers

October 29, 2024

Where Do Large Learning Rates Lead Us?
Ildus Sadrtdinov, Maxim Kodryan, Eduard Pokonechny, Ekaterina Lobacheva, Dmitry Vetrov
Neural Network Strong Generalization Optimization Purpose Learning Rate Large Learning Rate

October 13, 2024

Dynamic Estimation of Learning Rates Using a Non-Linear Autoregressive Model
Ramin Okhrati
Full Model Learning Rate Non Autoregressive Online Estimation Large Learning Rate

October 7, 2024

Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective
Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, Tengyu Ma
Language Model Learning Rate Loss Landscape Large Learning Rate Decay Function

October 5, 2024

A Learning Rate Path Switching Training Paradigm for Version Updates of Large Language Models
Zhihao Wang, Shiyu Liu, Jianheng Huang, Zheng Wang, Yixuan Liao, Xiaoxin Chen, Junfeng Yao, Jinsong Su
Learning Rate Continual Pre Training Version Identification Large Learning Rate Training Paradigm

June 24, 2024

Adam-mini: Use Fewer Learning Rates To Gain More
Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Yinyu Ye, Zhi-Quan Luo, Ruoyu Sun
Learning Rate Large Learning Rate Adam V2

June 7, 2024

Crafting Heavy-Tails in Weight Matrix Spectrum without Gradient Noise
Vignesh Kothapalli, Tianyu Pang, Shenyang Deng, Zongmin Liu, Yaoqing Yang
Deep Neural Network Two Layer Neural Network Weight Matrix Large Learning Rate Gradient Noise Heavy Tail Full Batch Gradient Descent

May 25, 2024

Does SGD really happen in tiny subspaces?
Minhak Song, Kwangjun Ahn, Chulhee Yun
Deep Neural Network Stochastic Gradient Descent Hessian Matrix Large Learning Rate Active Subspace

February 20, 2024

On the Convergence of Gradient Descent for Large Learning Rates
Alexandru Crăciun, Debarghya Ghoshdastidar
Gradient Descent Stochastic Gradient Descent Core Stability Flat Minimum Large Learning Rate Gradient Descent Trajectory

February 12, 2024

Which Frequencies do CNNs Need? Emergent Bottleneck Structure in Feature Learning
Yuxiao Wen, Arthur Jacot
CNN Model Feature Learning Major Challenge Bottleneck Large Learning Rate Dominant Low Frequency Variational Bottleneck Bottleneck Structure

January 21, 2024

Understanding the Generalization Benefits of Late Learning Rate Decay
Yinuo Ren, Chao Ma, Lexing Ying
Learning Rate Better Generalization Large Learning Rate Optimal Generalization Generalization Benefit

November 19, 2023

Large Learning Rates Improve Generalization: But How Large Are We Talking About?
Ekaterina Lobacheva, Eduard Pockonechnyy, Maxim Kodryan, Dmitry Vetrov
Neural Network Strong Generalization Learning Rate Large Learning Rate

October 26, 2023

September 12, 2023

Exploring Flat Minima for Domain Generalization with Large Learning Rates
Jian Zhang, Lei Qi, Yinghuan Shi, Yang Gao
Domain Generalization Model Generalization Target Domain Flat Minimum Large Learning Rate

July 22, 2023

The instabilities of large learning rate training: a loss landscape view
Lawrence Wang, Stephen Roberts
Gradient Descent Core Stability Hessian Matrix Loss Landscape Large Learning Rate Modern Neural Network

June 29, 2023

Provable Advantage of Curriculum Learning on Parity Targets with Mixed Inputs
Emmanuel Abbe, Elisabetta Cornacchia, Aryo Lotfi
Curriculum Learning Two Layer ReLU Provable Benefit Large Learning Rate Dense Recognition Mixed HASOC 2020 Sparse Symbolic Concept Group Parity

June 7, 2023

Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning
Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan, Mikhail Belkin
Neural Network Strong Generalization Gradient Descent Stochastic Gradient Descent Feature Learning Better Generalization Large Learning Rate Training Loss

May 30, 2023

Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoff
Arthur Jacot
Low Dimensional Representation Large Learning Rate Infinite Depth L2 Regularization Learned Feature Low Dimension Bottleneck Structure

May 28, 2023

Feature-Learning Networks Are Consistent Across Widths At Realistic Scales
Nikhil Vyas, Alexander Atanasov, Blake Bordelon, Depen Morwani, Sabarish Sainathan, Cengiz Pehlevan
Network Programming Wide Neural Network Large Learning Rate Ghanaian Musical Scale First Open Source Width Benchmark

April 6, 2023

Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks
Blake Bordelon, Cengiz Pehlevan
Mean Field Large Learning Rate Dynamic Prediction Infinite Width Dynamical Mean Field Theory Prediction Instability