Learning Rate Scheduling

Ask about this topic

Overview

Learning rate scheduling studies how the size of each optimization update should change over training, since it affects stability, training speed, and the model ultimately learned. Common approaches include warmup followed by cosine or linear decay, step decay and restarts, power-law schedules, and adaptive or schedule-free methods that respond to loss, gradient norms, or other training signals. Researchers evaluate these methods in settings from convex and stochastic optimization theory to image benchmarks and large language model pretraining, where training horizon, batch size, optimizer, and model scale can all matter. A central open tension is whether robust schedules can work across tasks without extensive tuning, or whether the best schedule must adapt to the model, data, noise, and training horizon.

Latest papers

CardsList

60 papers10 papers in the last four weeks, down 9% on the four weeks before. 0.1% of all new papers.

  1. UTA: Universal Training Adaptive Learning Rate Decay

    Oct 26, 2026Anda Tang, Junjie Wang, Xuanlin Yang +2Learning Rate SchedulingDeep Learning Optimization

    the State Key Lab of General AI School of Intelligence Science and Technology Peking University Beijing, China · the Pazhou Laboratory (Huangpu) · the Institute for Artificial Intelligence

  2. The Cost of an Unknown Horizon for the Last Iterate of Clipped SGD under Heavy-Tailed Noise

    Oct 24, 2026Yuxing Peng, Zhiqing Tang, Weijia JiaLast-Iterate ConvergenceConvergence Analysis

    Faculty of Arts and Sciences, Beijing Normal University, Zhuhai 519087, China · Institute of Artificial Intelligence and Future Networks, Faculty of Arts and Sciences, Beijing Normal University, Zhuhai 519087, China · Guangdong Key Lab of AI and Multi-Modal Data Processing, Beijing Normal-Hong Kong Baptist University, Zhuhai 519087, China

  3. Data Schedules and Learning Rate Schedules Should Be Designed Together

    Oct 21, 2026Haodong Wen, Kairong Luo, Tingkai Yan +2LLM TrainingContinued Pretraining

    Tsinghua University · Peng Cheng Laboratory

  4. MuPACE: Learning to Pace Muon for LoRA Fine-Tuning

    Oct 20, 2026Jan Kobiolka, Christian Frey, Josif GrabockaLearning Rate SchedulingAdaptive Learning Rate

    Department of Computer Science, University of Technology Nuremberg, Nuremberg, Germany

  5. Large Diffusion Models Need More Than EMA

    Oct 16, 2026Chuyu Hu, Huanran Chen, Shitong Shao +1Deep Learning OptimizationDiffusion Models

    Tsinghua University · Shanghai Qi Zhi Institute · The Hong Kong University of Science and Technology

  6. Asymptotic Risk of Learning-Rate Schedules for Least-Squares Regression with Discretely Distributed Features

    Oct 12, 2026Noé Colomban, Francis Bach, Fabian SchaippLearning Rate SchedulingStochastic Gradient Descent

    Inria - École Normale Supérieure PSL Research University

  7. Last-Iterate Convergence Rate of Normalized Gradient Descent under Hölder Smoothness

    Oct 5, 2026Yuki Takezawa, Eduard GorbunovLearning Rate SchedulingLast-Iterate Convergence

    Toyota Motor Corporation · MBZUAI

  8. From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes

    Sep 30, 2026Yichen Wang, Fanghui Liu, Yudong ChenLanguage Model PretrainingDeep Learning Optimization

    Department of Computer Sciences, University of Wisconsin–Madison, USA. · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China.

  9. How Does Local Landscape Geometry Evolve in Language Model Pre-Training?

    Sep 30, 2026Zhanpeng Zhou, Yuhan Sun, Bingrui Li +4Language Model PretrainingLearning Rate Scheduling

    Shanghai Jiao Tong University · Tsinghua University · Peking University

  10. Scheduling Recursive Reasoning in Looped Transformers

    Sep 29, 2026Boyuan Wang, Chengyao Yu, Jiaxi Ren +3Adaptive InferenceRecurrent Transformers

    Southern University of Science and Technology · The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute

  11. SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretraining

    Sep 28, 2026Qiulin Shang, Binyu Wang, Yongqi Qiao +3Language Model PretrainingLearning Rate Scheduling

    Peking University · Nanjing University

  12. Balancing Early Performance Sacrifices with Long-Term Gains: Scaling Learning-Rate Warmup Duration Across Training Horizons

    Sep 27, 2026Kristi Topollai, Anna ChoromanskaLearning Rate SchedulingNeural Network Training Dynamics

    New York University

  13. Linger and Lose: Knowledge Collapse in Low-Bit Language Models

    Sep 26, 2026Prashanna Mani Paudel, Shivanand Venkanna SheshappanavarLow-Precision TrainingLLM Quantization

    Geometric Intelligence Research Lab, Dept. of Electrical Engineering and Computer Science, University of Wyoming, USA.

  14. Conflicting Supervision Moves Commitment, Not Capability: A 12.29σ arrangement effect that is exactly zero under a convention-agnostic score

    Sep 23, 2026Wenhui ChenLearning Rate Scheduling

    University of Macau

  15. Terminal Shrinkage Averaging Reveals a Schedule-Estimator Interaction in LLM Pretraining

    Sep 21, 2026Adam Ousherovitch, Yixin WangLanguage Model PretrainingLearning Rate Scheduling

    Department of Statistics, University of Michigan

  16. Silver Rate Is (Almost) Optimal for Gradient Descent

    Sep 8, 2026Yuhan Ye, Kaizhao LiuLearning Rate SchedulingOptimization Convergence Analysis

    MIT

  17. When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay

    Sep 8, 2026Hasan Amin, Wei-Kai Chang, Rajiv KhannaLearning Rate SchedulingWeight Decay

    Purdue University

  18. The Exact Time-Uniform Rate Frontier for Stochastic Gradient Descent on Smooth Convex Objectives

    Sep 8, 2026Ruijie Li, Kang Chen, Tianyu WangLearning Rate SchedulingLast-Iterate Convergence

    School of Data Science, Fudan University · Shanghai Center for Mathematical Sciences, Fudan University

  19. BrachistoneLR: A Brachistochrone-Inspired Learning-Rate Schedule and a Controlled Benchmark of Scheduling Policies

    Sep 8, 2026Md. Sadekur Rahman Roni, Md. Jalal uddin Chowdhury, Moutusi Dash NimiDeep Learning OptimizationLearning Rate Scheduling

    Department of Computer Science and Engineering Leading University Sylhet, Bangladesh · DeepNet Research and Development Lab Sylhet 3100, Bangladesh · Department of Electrical and Electronic Engineering Leading UniversitySep Sylhet, Bangladesh

  20. Optimizer Memory Schedules for Outscaling the Overtraining Axis

    Sep 4, 2026Katie Everett, Shikai QiuNeural Network OptimizationMuon Optimizer

    MIT CSAIL

  21. Improved Gradient Descent Lower Bounds Beyond Nesterov

    Sep 2, 2026Yuhan Ye, Kaizhao LiuGradient DescentLearning Rate Scheduling

    MIT

  22. When Does Online Adaptation Pay on the Edge? A Leakage-Free Evaluation of Warmup, Learning-Rate Selection, and Resource Trade-offs for Time-Series Forecasting

    Sep 1, 2026Takumi Fujimoto, Hiroaki NishiMultivariate Time Series ForecastingLearning Rate Scheduling

    School of Science for Open and Environmental Systems, Graduate School of Science and Technology, Keio University, Yokohama, Japan

  23. Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss

    Aug 28, 2026Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan +4Language Model PretrainingLearning Rate Scheduling

    ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology +6

  24. Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining

    Aug 25, 2026Zihan Liu, Ruiheng Zheng, Shaobo Zhang +4Neural Network Training DynamicsLanguage Model Pretraining

    Peking University · Ant Group

  25. A lower bound for stepsize-based acceleration of gradient descent

    Aug 11, 2026Jianhao Ma, Yuxin ChenLearning Rate SchedulingLast-Iterate Convergence

    Department of Industrial Engineering, Tsinghua University · Department of Statistics and Data Science, Wharton School, University of Pennsylvania

  26. A Trust-region Framework for Moment Estimation

    Jul 26, 2026Oluwasegun A. SomefunLearning Rate SchedulingTrust-Region Optimization

  27. Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided Hölder Regularity

    Jul 24, 2026Arzu Ahmadova, Ismail HuseynovLearning Rate SchedulingNonconvex Optimization

    Weierstrass Institute for Applied Analysis and Stochastics Berlin, Germany · Physikalisch-Technische Bundesanstalt (PTB), Germany · Technical University of Berlin, Germany

  28. Hyperball May Not Be a Free Lunch

    Jul 24, 2026Yihao Xiao, Jialong Sun, Zitian Gao +5Learning Rate SchedulingNeural Network Optimization

    1IQuest Research · 4Shenzhen University of Advanced Technology · 2Peking University +2

  29. Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps

    Jul 14, 2026Subham Singh, Ashutosh Mishra, Subha RautLearning Rate SchedulingGradient Descent

    Department of Mathematics and Statistics, Mississippi State University · Independent Researcher

  30. WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training

    Jul 12, 2026Jianhao Ma, Yuxin ChenLearning Rate Scheduling

    Department of Statistics and Data Science, University of Pennsylvania