Learning Rate Scheduling

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, up 75% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 39

All topics

Last-Iterate Convergence Rate of Normalized Gradient Descent under Hölder Smoothness
Oct 5, 2026Yuki Takezawa, Eduard GorbunovLearning Rate Scheduling Last-Iterate Convergence
Toyota Motor Corporation · MBZUAI
From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes
Sep 30, 2026Yichen Wang, Fanghui Liu, Yudong ChenLanguage Model Pretraining Deep Learning Optimization
Department of Computer Sciences, University of Wisconsin–Madison, USA. · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China.
How Does Local Landscape Geometry Evolve in Language Model Pre-Training?
Sep 30, 2026Zhanpeng Zhou, Yuhan Sun, Bingrui Li +4Language Model Pretraining Learning Rate Scheduling
Shanghai Jiao Tong University · Tsinghua University · Peking University
SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretraining
Sep 28, 2026Qiulin Shang, Binyu Wang, Yongqi Qiao +3Language Model Pretraining Learning Rate Scheduling
Peking University · Nanjing University
Terminal Shrinkage Averaging Reveals a Schedule-Estimator Interaction in LLM Pretraining
Sep 21, 2026Adam Ousherovitch, Yixin WangLanguage Model Pretraining Learning Rate Scheduling
Department of Statistics, University of Michigan
When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay
Sep 8, 2026Hasan Amin, Wei-Kai Chang, Rajiv KhannaLearning Rate Scheduling Weight Decay
Purdue University
The Exact Time-Uniform Rate Frontier for Stochastic Gradient Descent on Smooth Convex Objectives
Sep 8, 2026Ruijie Li, Kang Chen, Tianyu WangLearning Rate Scheduling Last-Iterate Convergence
School of Data Science, Fudan University · Shanghai Center for Mathematical Sciences, Fudan University
BrachistoneLR: A Brachistochrone-Inspired Learning-Rate Schedule and a Controlled Benchmark of Scheduling Policies
Sep 8, 2026Md. Sadekur Rahman Roni, Md. Jalal uddin Chowdhury, Moutusi Dash NimiDeep Learning Optimization Learning Rate Scheduling
Department of Computer Science and Engineering Leading University Sylhet, Bangladesh · DeepNet Research and Development Lab Sylhet 3100, Bangladesh · Department of Electrical and Electronic Engineering Leading UniversitySep Sylhet, Bangladesh
Improved Gradient Descent Lower Bounds Beyond Nesterov
Sep 2, 2026Yuhan Ye, Kaizhao LiuGradient Descent Learning Rate Scheduling
MIT
When Does Online Adaptation Pay on the Edge? A Leakage-Free Evaluation of Warmup, Learning-Rate Selection, and Resource Trade-offs for Time-Series Forecasting
Sep 1, 2026Takumi Fujimoto, Hiroaki NishiMultivariate Time Series Forecasting Learning Rate Scheduling
School of Science for Open and Environmental Systems, Graduate School of Science and Technology, Keio University, Yokohama, Japan
Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss
Aug 28, 2026Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan +4Language Model Pretraining Learning Rate Scheduling
ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology +6
A lower bound for stepsize-based acceleration of gradient descent
Aug 11, 2026Jianhao Ma, Yuxin ChenLearning Rate Scheduling Last-Iterate Convergence
Department of Industrial Engineering, Tsinghua University · Department of Statistics and Data Science, Wharton School, University of Pennsylvania
A Trust-region Framework for Moment Estimation
Jul 26, 2026Oluwasegun A. SomefunLearning Rate Scheduling Trust-Region Optimization
Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided Hölder Regularity
Jul 24, 2026Arzu Ahmadova, Ismail HuseynovLearning Rate Scheduling Nonconvex Optimization
Weierstrass Institute for Applied Analysis and Stochastics Berlin, Germany · Physikalisch-Technische Bundesanstalt (PTB), Germany · Technical University of Berlin, Germany
Hyperball May Not Be a Free Lunch
Jul 24, 2026Yihao Xiao, Jialong Sun, Zitian Gao +5Learning Rate Scheduling Neural Network Optimization
1IQuest Research · 4Shenzhen University of Advanced Technology · 2Peking University +2
Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps
Jul 14, 2026Subham Singh, Ashutosh Mishra, Subha RautLearning Rate Scheduling Gradient Descent
Department of Mathematics and Statistics, Mississippi State University · Independent Researcher
WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training
Jul 12, 2026Jianhao Ma, Yuxin ChenLearning Rate Scheduling
Department of Statistics and Data Science, University of Pennsylvania
Understanding Schedule-Free Methods in Nonconvex Optimization: Rate Guarantees and Escaping Saddles
Jul 10, 2026Jiseok Chae, Donghwan KimLearning Rate Scheduling Optimization Convergence Analysis
Natural Science Research Institute KAIST Daejeon, Republic of Korea · Department of Mathematical Sciences KAIST Daejeon, Republic of Korea
Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures
Jul 9, 2026Hafsa Mateen, Radu Timofte, Dmitry IgnatovLearning Rate Scheduling Neural Network Optimization
Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
Jul 8, 2026Yedi Zhang, Peter E. Latham, Leena Chennuru Vankadara +1Learning Rate Scheduling Deep Linear Networks
Gatsby Computational Neuroscience Unit, University College London · Sainsbury Wellcome Centre, University College London
Curvature-Weighted Gradient Diversity: A Noise Measure for Geometry-Adaptive SGD Schedules
Jun 29, 2026Muhammad Hamza, Ayush GoelLearning Rate Scheduling Stochastic Optimization Convergence
Indian Institute of Technology Kharagpur · Indian Institute of Technology Kharagpur (IIT KGP)
Muown Implicitly Performs Angular Step-size Decay
Jun 22, 2026Florian Hübler, Kai Lion, Antonio Orvieto +1Learning Rate Scheduling Riemannian Optimization
Department of Computer Science · ETH Zurich, Switzerland · ELLIS Institute Tübingen, MPI-IS +1
Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler
Jun 1, 2026Dimitris Oikonomou, Nicolas LoizouSharpness-Aware Minimization Learning Rate Scheduling
Mathematical Institute for Data Science (MINDS), Johns Hopkins University, Baltimore, MD, USA · Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA · Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD, USA
Accelerating Min-Max Optimization via Power-Law Stepsizes
Jun 1, 2026Yue Wu, Weiqiang Zheng, Yang Cai +1Learning Rate Scheduling Last-Iterate Convergence
University of Southern California · Yale University
Agile Online Model Selection: Resolving Adaptation Lag via Safeguarded Large Learning Rates
May 26, 2026Kei Takemura, Ryuta Matsuno, Keita SakumaLearning Rate Scheduling
NEC Corporation Tokyo, Japan
Mapping the Schedule x Bit-Width Boundary in Sub-100M Quantisation-Aware Training
May 25, 2026Christian Brandt ThomassenLLM Quantization Learning Rate Scheduling
Dwarf A/S, Copenhagen, Denmark
Anytime Training with Schedule-Free Spectral Optimization
May 21, 2026Anuj Apte, Pranav Deshpande, Niraj Kumar +2Deep Learning Optimization Learning Rate Scheduling
Global Technology Applied Research, JPMorganChase New York, NY 10001, USA
One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs
May 21, 2026Di He, Songjun Tu, Keyu Wang +2Learning Rate Scheduling Efficient Language Model Training
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Peng Cheng Laboratory · University of Chinese Academy of Sciences +5
Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates
May 19, 2026Parjanya Prajakta Prashant, Jiongli Zhu, Aldan Creo +1Learning Rate Scheduling LLM Fine-Tuning
University of California San Diego
ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models
May 18, 2026Aaron DefazioLanguage Model Pretraining Learning Rate Scheduling
FAIR at Meta Super-Intelligence Labs
Revealing Modular Gradient Noise Imbalance in LLMs: Calibrating Adam via Signal-to-Noise Ratio
May 7, 2026Ziqing Wen, Zhouyang Liu, Jiahuan Wang +4Deep Learning Optimization Learning Rate Scheduling
College of Computer Science and Technology, National University of Defense Technology, Changsha, Hunan, China
Learning Rate Engineering: From Coarse Single Parameter to Layered Evolution
Apr 30, 2026Ming-Hong Yao, Di Wang, Jian Cui +5Deep Learning Optimization Fine-Tuning
School of AI, Jilin International Sciences University, Changchun, China.
Scalable Hyperparameter-Divergent Ensemble Training with Automatic Learning Rate Exploration for Large Models
Apr 27, 2026Hailing Cheng, Tao Huang, Chen Zhu +1Deep Learning Optimization Learning Rate Scheduling
LinkedIn Inc. Mountain View, CA 94043
Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay
Feb 6, 2026Binghui Li, Zilin Wang, Fengling Chen +3Learning Rate Scheduling Kernel Regression
Center for Machine Learning Research, Peking University · School of Mathematical Sciences, Peking University · AI for Science Institute, Beijing
ZENITH: Automated Gradient Norm Informed Stochastic Optimization
Jan 21, 2026Dhrubo SahaDeep Learning Optimization Learning Rate Scheduling
Georgia Institute of Technology
Why Do We Need Warm-up? A Theoretical Perspective
Oct 3, 2025Foivos Alimisis, Rustem Islamov, Aurelien LucchiDeep Learning Optimization Learning Rate Scheduling
University of Basel, Switzerland
Negative Stepsizes Make Gradient-Descent-Ascent Converge
May 2, 2025Henry Shugart, Jason M. AltschulerLearning Rate Scheduling Optimization Convergence Analysis
UPenn
Silver Rate Is (Almost) Optimal for Gradient Descent
Date pendingYuhan Ye, Kaizhao LiuLearning Rate Scheduling Optimization Convergence Analysis
MIT
ExpTest: Loss-Curve Hypothesis Testing for Autonomous Learning-Rate Selection in Deep Neural Networks
Date pendingZan Chaudhry, Naoko MizunoLearning Rate Scheduling Neural Network Training Dynamics
Laboratory of Structural Cell Biology, National Institutes of Health, Bethesda, MD, 20892 USA · Ragon Institute, Cambridge, MA, 02139

Oct 5, 2026, math.OCJ/K move · Enter open · S save

Last-Iterate Convergence Rate of Normalized Gradient Descent under Hölder Smoothness

Yuki Takezawa, Eduard Gorbunov

Toyota Motor Corporation · MBZUAI

Learning Rate Scheduling Last-Iterate Convergence Optimization Convergence Analysis Gradient Descent

Open paper AskPDF arXiv

Normalized gradient descent is a widely studied adaptive optimization method. Most existing analyses focus on the best iterate or a weighted average of the iterates, whereas practical implementations typically return the last iterate. In this paper, we study the last-iterate convergence of normalized gradient descent for convex,

(ν,M_ν)

-Hölder-smooth objectives. For a constant stepsize, we establish an upper bound of

\mathcal{O}\bigl((\log^2(T)/T)^{(1+ν)/2}\bigr)

, which contains a logarithmic overhead relative to the known

\mathcal{O}\bigl(T^{-(1+ν)/2}\bigr)

guarantees for the best and weighted-average iterates. For

ν= 0

, this overhead is known to be unavoidable. We complement this analysis with numerical results based on the performance estimation problem (PEP), investigating the finite-horizon worst-case behavior in the smooth setting and whether the logarithmic overhead reflects an intrinsic limitation of constant-step normalized gradient descent. We then show that a linearly decreasing stepsize yields a last-iterate guarantee of

\mathcal{O}\bigl(T^{-(1+ν)/2}\bigr)

, matching the order of the best-iterate/weighted-average guarantees without requiring knowledge of

ν

and

M_ν

Momentum

Related topics

Latest papers 39