Overview

Learning rate scheduling studies how the size of each optimization update should change over training, since it affects stability, training speed, and the model ultimately learned. Common approaches include warmup followed by cosine or linear decay, step decay and restarts, power-law schedules, and adaptive or schedule-free methods that respond to loss, gradient norms, or other training signals. Researchers evaluate these methods in settings from convex and stochastic optimization theory to image benchmarks and large language model pretraining, where training horizon, batch size, optimizer, and model scale can all matter. A central open tension is whether robust schedules can work across tasks without extensive tuning, or whether the best schedule must adapt to the model, data, noise, and training horizon.

Latest papers

CardsList

65 papers8 papers in the last four weeks, down 27% on the four weeks before. 0.1% of all new papers.

  1. Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay

    Feb 6, 2026Binghui Li, Zilin Wang, Fengling Chen +3Learning Rate SchedulingKernel Regression

  2. Why Do We Need Warm-up? A Theoretical Perspective

    Oct 3, 2025Foivos Alimisis, Rustem Islamov, Aurelien LucchiDeep Learning OptimizationLearning Rate Scheduling

  3. ExpTest: Loss-Curve Hypothesis Testing for Autonomous Learning-Rate Selection in Deep Neural Networks

    Nov 25, 2024Zan Chaudhry, Naoko MizunoLearning Rate SchedulingNeural Network Training Dynamics