Overview

Learning rate scheduling studies how the size of each optimization update should change over training, since it affects stability, training speed, and the model ultimately learned. Common approaches include warmup followed by cosine or linear decay, step decay and restarts, power-law schedules, and adaptive or schedule-free methods that respond to loss, gradient norms, or other training signals. Researchers evaluate these methods in settings from convex and stochastic optimization theory to image benchmarks and large language model pretraining, where training horizon, batch size, optimizer, and model scale can all matter. A central open tension is whether robust schedules can work across tasks without extensive tuning, or whether the best schedule must adapt to the model, data, noise, and training horizon.

Latest papers

CardsList

65 papers8 papers in the last four weeks, down 27% on the four weeks before. 0.1% of all new papers.

  1. Learning Rate Decay Can Exponentially Accelerate SGD for Global Optimization of Nonconvex Functions

    Nov 9, 2026Guneykan Ozgul, Dylan Herman, Junhyung Lyle Kim +2Nonconvex Stochastic OptimizationGlobal Optimization

  2. Stepsize modulation for cross entropy from local quadratic optimization

    Nov 3, 2026Bruno Bauwens, Viacheslav Krivodub, Bingyu ZhangLearning Rate SchedulingAdaptive Learning Rate

  3. UTA: Universal Training Adaptive Learning Rate Decay

    Oct 26, 2026Anda Tang, Junjie Wang, Xuanlin Yang +2Learning Rate SchedulingDeep Learning Optimization

  4. The Cost of an Unknown Horizon for the Last Iterate of Clipped SGD under Heavy-Tailed Noise

    Oct 24, 2026Yuxing Peng, Zhiqing Tang, Weijia JiaLast-Iterate ConvergenceConvergence Analysis

  5. Data Schedules and Learning Rate Schedules Should Be Designed Together

    Oct 21, 2026Haodong Wen, Kairong Luo, Tingkai Yan +2LLM TrainingContinued Pretraining

  6. MuPACE: Learning to Pace Muon for LoRA Fine-Tuning

    Oct 20, 2026Jan Kobiolka, Christian Frey, Josif GrabockaLearning Rate SchedulingAdaptive Learning Rate

  7. Large Diffusion Models Need More Than EMA

    Oct 16, 2026Chuyu Hu, Huanran Chen, Shitong Shao +1Deep Learning OptimizationDiffusion Models

  8. Asymptotic Risk of Learning-Rate Schedules for Least-Squares Regression with Discretely Distributed Features

    Oct 12, 2026Noé Colomban, Francis Bach, Fabian SchaippLearning Rate SchedulingStochastic Gradient Descent

  9. From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes

    Sep 30, 2026Yichen Wang, Fanghui Liu, Yudong ChenLanguage Model PretrainingDeep Learning Optimization

  10. How Does Local Landscape Geometry Evolve in Language Model Pre-Training?

    Sep 30, 2026Zhanpeng Zhou, Yuhan Sun, Bingrui Li +4Language Model PretrainingLearning Rate Scheduling

  11. Scheduling Recursive Reasoning in Looped Transformers

    Sep 29, 2026Boyuan Wang, Chengyao Yu, Jiaxi Ren +3Adaptive InferenceRecurrent Transformers

  12. SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretraining

    Sep 28, 2026Qiulin Shang, Binyu Wang, Yongqi Qiao +3Language Model PretrainingLearning Rate Scheduling

  13. Linger and Lose: Knowledge Collapse in Low-Bit Language Models

    Sep 26, 2026Prashanna Mani Paudel, Shivanand Venkanna SheshappanavarLow-Precision TrainingLLM Quantization

  14. The Exact Time-Uniform Rate Frontier for Stochastic Gradient Descent on Smooth Convex Objectives

    Sep 8, 2026Ruijie Li, Kang Chen, Tianyu WangLearning Rate SchedulingLast-Iterate Convergence

  15. BrachistoneLR: A Brachistochrone-Inspired Learning-Rate Schedule and a Controlled Benchmark of Scheduling Policies

    Sep 8, 2026Md. Sadekur Rahman Roni, Md. Jalal uddin Chowdhury, Moutusi Dash NimiDeep Learning OptimizationLearning Rate Scheduling

  16. How Evaluation Choices Shape the Measured Benefit of Online Adaptation in Time-Series Forecasting: Warmup, Learning-Rate Selection, and Adaptation-State Memory

    Sep 1, 2026Takumi Fujimoto, Hiroaki NishiMultivariate Time Series ForecastingLearning Rate Scheduling

  17. Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss

    Aug 28, 2026Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan +4Language Model PretrainingLearning Rate Scheduling

  18. A lower bound for stepsize-based acceleration of gradient descent

    Aug 11, 2026Jianhao Ma, Yuxin ChenLearning Rate SchedulingLast-Iterate Convergence