Learning Rate Free

Learning-rate-free optimization aims to eliminate the need for manual tuning of learning rates, a crucial hyperparameter in many machine learning algorithms. Current research focuses on developing adaptive methods, such as modifications of stochastic gradient descent (SGD) and Adam, and applying these techniques across various domains including reinforcement learning and Riemannian manifold optimization. These advancements improve algorithm robustness and user-friendliness by automating a previously laborious and problem-specific process, leading to more efficient and reliable training of machine learning models. The resulting algorithms often demonstrate performance comparable to those with optimally tuned learning rates, offering significant practical benefits.

Papers

August 7, 2024

Learning Rate-Free Reinforcement Learning: A Case for Model Selection with Non-Stationary Objectives
Aida Afshar, Aldo Pacchiano
Reinforcement Learning Learning Rate Case Relevance Reinforcement Learning Algorithm Model Selection Time Varying Objective Learning Rate Free Standard Bandit

June 4, 2024

Learning-Rate-Free Stochastic Optimization over Riemannian Manifolds
Daniel Dodd, Louis Sharrock, Christopher Nemeth
Stochastic Optimization Gradient Based Optimization Riemannian Manifold High Probability Convergence Learning Rate Free

January 6, 2024

Interpreting Adaptive Gradient Methods by Parameter Scaling for Learning-Rate-Free Optimization
Min-Kook Suh, Seung-Woo Seo
Adaptive Gradient Method Learning Rate Free Steepest Descent Parameter Scaling

August 6, 2023

Learning-Rate-Free Learning: Dissecting D-Adaptation and Probabilistic Line Search
Max McGuinness
Stochastic Gradient Descent Adaptation Concern Learning Rate Optimal Rate Learning Rate Free Stochastic Line Search

June 9, 2023

Prodigy: An Expeditiously Adaptive Parameter-Free Learner
Konstantin Mishchenko, Aaron Defazio
Adaptation Concern Learning Rate Online Linear Adam V2 Adaptation Method Learning Rate Free

May 24, 2023

Learning Rate Free Sampling in Constrained Domains
Louis Sharrock, Lester Mackey, Christopher Nemeth
Domain Constraint Particle Based Algorithm Constrained Sampling Learning Rate Free Post Selection

January 18, 2023

Learning-Rate-Free Learning by D-Adaptation
Aaron Defazio, Konstantin Mishchenko
Adaptation Concern Learning Rate Hyperparameter Free Learning Rate Free