Learning Rate

Learning rate, a crucial hyperparameter in training neural networks, dictates the step size during optimization. Current research focuses on developing adaptive learning rate schedules, such as warmup-stable-decay and learning rate path switching, to improve training efficiency and generalization, particularly for large language models and other deep learning architectures. These advancements aim to address challenges like finding optimal learning rates across varying model sizes, datasets, and training durations, ultimately leading to faster convergence and better model performance. The impact extends to various applications, from natural language processing and computer vision to scientific computing and reinforcement learning.

Papers

September 21, 2023

State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understanding
Devleena Das, Sonia Chernova, Been Kim
Line by Line Explanation Learning Rate Reward Shaping Sequential Decision Making Concept Based Explanation Human AI Decision Making User Understanding Agent Learning

September 18, 2023

FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data
Hao Sun, Li Shen, Shixiang Chen, Jingwei Sun, Jing Li, Guangzhong Sun, Dacheng Tao
Raw Data Large Scale Learning Rate Linear Speedup FedCF Method

September 16, 2023

Rethinking Learning Rate Tuning in the Era of Large Language Models
Hongpeng Jin, Wenqi Wei, Xuyu Wang, Wenbin Zhang, Yanzhao Wu
Learning Rate New Era LLM Fine Tuning DNN Training Learning Rate Schedule

September 12, 2023

ELRA: Exponential learning rate adaption gradient descent optimization method
Alexander Kleinsorge, Stefan Kupper, Alexander Fauck, Felix Rothe
Gradient Descent Learning Rate Exponential Mechanism Rate Adaptation

August 21, 2023

We Don't Need No Adam, All We Need Is EVE: On The Variance of Dual Learning Rate And Beyond
Afshin Khadangi
Deep Learning Learning Rate Faster Convergence Variance Information Velocity Estimation Dual Learning

August 7, 2023

Optimal Approximation and Learning Rates for Deep Convolutional Neural Networks
Shao-Bo Lin
Deep Convolutional Neural Network Learning Rate Average Approximation Approximation Rate Pooling Operator

August 6, 2023

Learning-Rate-Free Learning: Dissecting D-Adaptation and Probabilistic Line Search
Max McGuinness
Stochastic Gradient Descent Adaptation Concern Learning Rate Optimal Rate Learning Rate Free Stochastic Line Search

July 27, 2023

The Marginal Value of Momentum for Small Learning Rate SGD
Runzhe Wang, Sadhika Malladi, Tianhao Wang, Kaifeng Lyu, Zhiyuan Li
Gradient Descent Learning Rate Residual Momentum Gradient Noise Marginal Probability

July 26, 2023

Function Value Learning: Adaptive Learning Rates Based on the Polyak Stepsize and Function Splitting in ERM
Guillaume Garrigos, Robert M. Gower, Fabian Schaipp
Gradient Descent Stochastic Gradient Descent Learning Rate Empirical Risk Minimization Value Function Step Size Splitting Algorithm Polyak Step Size

July 12, 2023

No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models
Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, Matt J. Kusner
Learning Rate Transformer Based Language Model High Speed Train System Big Gain Pre Trained BERT Training Algorithm

June 26, 2023

Gradient Descent Converges Linearly for Logistic Regression on Separable Data
Kyriakos Axiotis, Maxim Sviridenko
Gradient Descent Learning Rate Logistic Regression Separable Data Learning Guarantee Sparsity Tradeoff

June 22, 2023

Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptation
Xin Yuan, Pedro Savarese, Michael Maire
Neural Network Learning Rate Faster Training Local Optimization Incremental Model Gradient Scaling

June 11, 2023

Parameter-free version of Adaptive Gradient Methods for Strongly-Convex Functions
Deepak Gouda, Hassan Naveed, Salil Kamath
Learning Rate Regret Bound Convex Function Adaptive Gradient Method Strongly Convex Parameter Free General Algorithm

June 9, 2023

Prodigy: An Expeditiously Adaptive Parameter-Free Learner
Konstantin Mishchenko, Aaron Defazio
Adaptation Concern Learning Rate Online Linear Adam V2 Adaptation Method Learning Rate Free

June 1, 2023

May 31, 2023

Mechanic: A Learning Rate Tuner
Ashok Cutkosky, Aaron Defazio, Harsh Mehta
Learning Rate Online Convex Optimization Deep Learning Task Mechanical Engineering Base Algorithm

May 26, 2023

Stability-penalty-adaptive follow-the-regularized-leader: Sparsity, game-dependency, and best-of-both-worlds
Taira Tsuchiya, Shinji Ito, Junya Honda
Multi Armed Bandit Learning Rate Sparsity Increase Optimal Regret Follow the Regularized Leader Bandit Problem Best of Both World Algorithm Dependency Aware Incident

May 25, 2023

ADLER -- An efficient Hessian-based strategy for adaptive learning rate
Dario Balboni, Davide Bacciu
Learning Rate Hessian Matrix Hessian Vector Product Local Approximation Positive Semi Definite

May 24, 2023

Non-Parametric Learning of Stochastic Differential Equations with Non-asymptotic Fast Rates of Convergence
Riccardo Bonalli, Alessandro Rudi
Early Stage Convergence Learning Rate Stochastic Differential Equation Non Asymptotic Fokker Planck Equation Non Parametric Learning Diffusion Coefficient

Learning Rate

Papers

State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understanding

FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data

Rethinking Learning Rate Tuning in the Era of Large Language Models

ELRA: Exponential learning rate adaption gradient descent optimization method

We Don't Need No Adam, All We Need Is EVE: On The Variance of Dual Learning Rate And Beyond

Optimal Approximation and Learning Rates for Deep Convolutional Neural Networks

Learning-Rate-Free Learning: Dissecting D-Adaptation and Probabilistic Line Search

The Marginal Value of Momentum for Small Learning Rate SGD

Function Value Learning: Adaptive Learning Rates Based on the Polyak Stepsize and Function Splitting in ERM

No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models

Gradient Descent Converges Linearly for Logistic Regression on Separable Data

Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptation

Parameter-free version of Adaptive Gradient Methods for Strongly-Convex Functions

Prodigy: An Expeditiously Adaptive Parameter-Free Learner

Chaos persists in large-scale multi-agent learning despite adaptive learning rates

On the Weight Dynamics of Deep Normalized Networks

Mechanic: A Learning Rate Tuner

Stability-penalty-adaptive follow-the-regularized-leader: Sparsity, game-dependency, and best-of-both-worlds

ADLER -- An efficient Hessian-based strategy for adaptive learning rate

Non-Parametric Learning of Stochastic Differential Equations with Non-asymptotic Fast Rates of Convergence