What do near-optimal learning rate schedules look like?

Hiroki Naganuma; Atish Agarwala; Priya Kasimbeg; George E. Dahl

cs.LGMar 11, 2026

What do near-optimal learning rate schedules look like?

Authors: Hiroki Naganuma, Atish Agarwala, Priya Kasimbeg, George E. Dahl

Organizations: Mila · Université de Montréal · Google DeepMind

Abstract

A basic unanswered question in neural network training is: what is the best learning rate schedule shape for a given workload? The choice of learning rate schedule is a key factor in the success or failure of the training process, but beyond having some kind of warmup and decay, there is no consensus on what makes a good schedule shape. To answer this question, we designed a search procedure to find the best shapes within a parameterized schedule family. Our approach factors out the schedule shape from the base learning rate, which otherwise would dominate cross-schedule comparisons. We applied our search procedure to a variety of schedule families on three workloads: linear regression, image classification on CIFAR-10, and small-scale language modeling on Wikitext103. We showed that our search procedure indeed generally found near-optimal schedules. We found that warmup and decay are robust features of good schedules, and that commonly used schedule families are not optimal on these workloads. Finally, we explored how the outputs of our shape search depend on other optimization hyperparameters, and found that weight decay can have a strong effect on the optimal schedule shape. To the best of our knowledge, our results represent the most comprehensive results on near-optimal schedule shapes for deep neural network training, to date.

Figures & tables

Schedule Family	Short Name	Description
Constant	con	Warmup to base LR, followed by a constant LR.
Cosine	cos-std	Warmup to base LR, followed by cosine decay with a fixed exponent = 1.
Generalized Cosine	cos-gen	Warmup to base LR, followed by cosine decay with a parameterized exponent.
Square-root Decay	sqrt	Warmup to base LR, followed by inverse square root decay.
Generalized Rex	rex	Warmup to base LR, followed by decay parameterized through REX ( Chen et al., 2022 ) .
Two-Point Spline	tps	Warmup to base LR, followed by decay parameterized through two spline interpolation points.

Table 1: Learning Rate Schedule (Shape) Families used in our experiments

Panel of Figure 1 — Figure 1: Learning rate schedule families used in our experiments: Constant , Cosine , Generalized Cosine , Generalized Rex , , Square-root Decay , Two-Point Spline , and Two-Point Linear . Markers identify key points such as initial learning rate, warmup completion, intermediate control points, and end of training. Numerical annotations specify parameters particular to each schedule. Peak of can occur in any order compared to control points unlike other schedules.

Panel of Figure 2 — Figure 2: For a linear regression workload, schedules found via random search capture some of the features of the theoretically optimal schedule but fail to match if completely (left). Average losses appear to be better than theoretically optimal; however, when re-evaluating with $1000$ seeds, searched schedules match theoretical prediction and are slightly worse than optimal (right).

Panel of Figure 3 — Figure 3: Training metrics versus base learning rate for best schedules in each family for linear regression (left), CIFAR-10 (middle), and WikiText-103 (right). Each row corresponds to a schedule family, each column to a learning rate, and lighter colors correspond to better performance. Base learning rate is far more important for success than schedule identity, with the exception of the Constant schedule which performs worse in all cases.

Panel of Figure 4 — Figure 4: Best family member does not match optimal schedule (left, blue). We can numerically solve for the best fit curve in the family (left, orange). This obtains very similar performance to the theoretical optimal schedule (right).

Panel of Figure 5 — Figure 5: Near optimal learning rate schedules for CIFAR-10 (left) and WikiText-103 (right). The curves represent the best absolute learning rate schedules for each family found in our search procedure which minimized final train error ( CIFAR-10 ) or train perplexity ( WikiText-103 ). All curves show similar warmup and decay patterns in each workload, including family which does not guaranteed to have those properties. More flexible families perform better than Constant and Cosine .

Panel of Figure 6 — Figure 6: Near optimal learning rate schedules for CIFAR-10 (left) and WikiText-103 (right), selected on best median test error ( CIFAR-10 ) and test perplexity ( WikiText-103 ). Similar trends to selecting on training metrics, with some quantitative differences. CIFAR-10 favors training curves with smaller absolute base learning rate when selecting on test error vs train error. snm and tpl were omitted for clarity.

Panel of Figure 7 — Figure 7: Distribution of best base learning rates for shape search points on CIFAR-10 (left) and WikiText-103 (right). Each curve represents a different schedule family. Support for the distributions is mostly contained inside the search grid, which suggests that the base learning rate range was well-chosen. Optimal base learning rate distributions are similar across families, which the exception of Constant and which are significantly smaller than the others.

Panel of Figure 8 — Figure 8: Top $3$ schedules for CIFAR-10 and WikiText-103 for Cosine , Two-Point Spline , and families. For Cosine , schedules are very similar within workload which suggests that Cosine was searched well. For CIFAR-10 , all Two-Point Spline schedules are similar to each other, while for WikiText-103 all but one Two-Point Spline schedules have similar warmup, base learning rate, and decay. This provides evidence that Two-Point Spline is also generally searched well, but that the WikiText-103 workload admits a wider variety of learning rate schedules with similar final training metrics. schedules show high variability, suggesting they are not well-searched.

Panel of Figure 9 — Figure 9: Coordinate-wise linesearch from optimal schedule shape for CIFAR-10 (top 3 rows) and WikiText-103 (bottom 3 rows). For each coordinate, we vary the coordinate and remeasure the appropriate training metric (blue, 95% CI plotted), with a new base learning rate optimization (best base LR in orange). Green dashed line marks the original value of the coordinate from the search. Varying some coordinates, particularly $y_{2}$ , can significantly improve training metrics suggesting that is not well-optimized.

Panel of Figure 10 — Figure 10: Coordinatewise linesearch for Two-Point Spline schedule. Methodology and labeling are the same as Figure 9 . Gains from linesearch are generally smaller than measurement uncertainty, suggesting Two-Point Spline is well-optimized at the resolution of our experiments — unlike .

Panel of Figure 11 — Figure 11: ECDF (Empirical cumulative distribution functions) of the score reached by random search within each learning rate schedule family. The horizontal axis reports the median training error for CIFAR-10 or the median training perplexity for WikiText-103 , where lower values mark better learning. The vertical axis shows the chance of reaching or beating each score. On CIFAR-10 , the constant schedule and the cosine schedule with exponent 1.0 reduce error with high probability yet stall near 0.05. Square-root Decay schedules move the curve left most sharply. Two-Point Spline , Two-Point Linear and Generalized Cosine trace a similar path. Generalized Rex and attain lower scores at lower CDF probability (alternatively, after more samples). in particular lags because its large parameter space slows the search.

Panel of Figure 12 — Figure 12: Optimal Shape varying $\beta_{1}$ and $\beta_{2}$ | Relative learning-rate schedules that minimise loss after tuning $\beta_{1}$ and $\beta_{2}$ . The top row varies $\beta_{1}$ and the bottom row varies $\beta_{2}$ . Plots on the left use CIFAR-10 , plots on the right use WikiText-103 . Low $\beta_{1}$ values give a short warmup and a brief high-rate plateau, while high $\beta_{1}$ values extend both. On CIFAR-10 , larger $\beta_{2}$ values also extend the warmup, whereas $\beta_{2}$ shows no clear link to schedule shape on WikiText-103 .

$\beta_{1}$ , Selection condition	$\beta_{1}$ , Evaluation condition
	0.8	0.9	0.95	0.975
0.8	$0.07\pm 0.002$	$0.072\pm 0.002$	$0.089\pm 0.001$	$0.123\pm 0.002$
0.9	$0.068\pm 0.002$	$0.067\pm 0.001$	$0.083\pm 0.001$	$0.116\pm 0.002$
0.95	$0.066\pm 0.001$	$0.066\pm 0.001$	$0.082\pm 0.001$	$0.114\pm 0.001$
0.975	$0.068\pm 0.004$	$0.065\pm 0.001$	$0.081\pm 0.001$	$0.11\pm 0.001$

Table 2: Median CIFAR-10 training error with one standard deviation for every pair of

\beta_{1}

values used at schedule selection (rows) and evaluation (columns). A larger

\beta_{1}

in the search phase lowers the error. The lowest error appears when the schedule found at

\beta_{1}

= 0.95 runs with

\beta_{1}

= 0.8. In most rows a lower

\beta_{1}

at evaluation further reduces the error, which suggests that a drop in momentum after a high-momentum search can improve performance.

$\beta_{1}$ , Selection condition	$\beta_{1}$ , Evaluation condition
	0.8	0.9	0.95	0.975
0.8	$28.8\pm 0.96$	$28.2\pm 0.09$	$33.04\pm 0.06$	$46.67\pm 0.59$
0.9	$37.58\pm 1.91$	$27.01\pm 0.92$	$27.11\pm 0.02$	$30.97\pm 0.06$
0.95	$39.27\pm 1.85$	$26.99\pm 1.02$	$26.91\pm 0.04$	$30.51\pm 0.04$
0.975	$40.61\pm 4.09$	$26.92\pm 0.92$	$26.91\pm 0.08$	$30.47\pm 0.07$

Table 3: Median WikiText-103 training perplexity and one standard deviation for every pair of

\beta_{1}

values used in schedule selection (rows) and evaluation (columns). Rows list the

\beta_{1}

that guides the search; columns give the

\beta_{1}

used to run the schedule. A higher

\beta_{1}

in the search stage lowers perplexity, except for selection at

\beta_{1}

= 0.975, which yields larger values. The best result appears when schedules found at

\beta_{1}

= 0.95 or 0.975 run with

\beta_{1}

= 0.95. Evaluation at

\beta_{1}

= 0.8 shows the widest spread, which points to high uncertainty for low-momentum runs.

$\beta_{2}$ , Selection condition	$\beta_{2}$ , Evaluation condition
	0.9	0.99	0.999	0.9995
0.9	$0.061\pm 0.001$	$0.062\pm 0.001$	$0.07\pm 0.001$	$0.072\pm 0.001$
0.99	$0.062\pm 0.001$	$0.062\pm 0.001$	$0.066\pm 0.001$	$0.068\pm 0.001$
0.999	$0.064\pm 0.002$	$0.064\pm 0.001$	$0.072\pm 0.002$	$0.07\pm 0.002$
0.9995	$0.058\pm 0.001$	$0.059\pm 0.001$	$0.066\pm 0.001$	$0.065\pm 0.001$

Table 4: Median CIFAR-10 training error

\pm

SD for every pair of

\beta_{2}

values at schedule selection (rows) and evaluation (columns). Rows list the

\beta_{2}

used to pick the schedule; columns give the value used to run it. Higher

\beta_{2}

at selection tends to cut error, yet the lowest error comes from schedules tuned at

\beta_{2}

= 0.9995 and run at

\beta_{2}

= 0.9.

$\beta_{2}$ , Selection condition	$\beta_{2}$ , Evaluation condition
	0.9	0.99	0.999	0.9995
0.9	$26.23\pm 0.02$	$26.4\pm 0.29$	$26.87\pm 6.03$	$26.77\pm 11.37$
0.99	$26.32\pm 0.01$	$26.29\pm 0.35$	$26.59\pm 0.85$	$26.71\pm 2.26$
0.999	$26.32\pm 0.01$	$26.56\pm 0.38$	$26.74\pm 1.42$	$26.76\pm 1.21$
0.9995	$26.36\pm 0.01$	$26.42\pm 0.27$	$26.75\pm 2.11$	$26.74\pm 2.77$

Table 5: Median WikiText-103 perplexity

\pm

SD for every pair of

\beta_{2}

values used at schedule selection (rows) and evaluation (columns). A higher

\beta_{2}

at selection lowers the median, yet the best result appears when a schedule tuned at

\beta_{2}

= 0.9995 runs with

\beta_{2}

= 0.9. Selection at

\beta_{2}

= 0.9 shows the widest spread, confirming that low

\beta_{2}

during tuning adds noise.

Panel of Figure 13 — Figure 13: Optimal Shape varying Weight Decay | Relative learning rate schedules that yield the lowest loss after tuning weight decay. Left panel shows CIFAR-10 ; right panel shows WikiText-103 . Small decay values lengthen the warmup, while large decay values keep the rate high until the final training steps. Both tasks trace the same pattern.

Panel of Figure 14 — Figure 14: Optimal learning rate shapes for Generalized Cosine (left) and Two-Point Spline (right) for wikitext workload trained with varying training horizon. Performance improves with more steps, which favors a more steady decay. Warmup fraction remains consistent at this scale.

$\lambda_{WD}$ , Selection condition	$\lambda_{WD}$ , Evaluation condition
	0.0	0.001	0.01	0.1
0.0	$0.071\pm 0.001$	$0.127\pm 0.003$	$0.532\pm 0.004$	$0.9\pm 0.0$
0.001	$0.067\pm 0.001$	$0.123\pm 0.002$	$0.477\pm 0.003$	$0.9\pm 0.0$
0.01	$0.101\pm 0.001$	$0.149\pm 0.001$	$0.406\pm 0.004$	$0.9\pm 0.002$
0.1	$0.109\pm 0.001$	$0.164\pm 0.001$	$0.415\pm 0.002$	$0.839\pm 0.004$

Table 6: Median CIFAR-10 training error

\pm

SD for every pair of weight decay values used in schedule selection (rows) and later evaluation (columns). An evaluation decay of 0.01 or 0.1 lifts the error across all rows, while decay 0 or 0.001 yields the lowest values.

$\lambda_{WD}$ , Selection condition	$\lambda_{WD}$ , Evaluation condition
	0.0	0.001	0.01	0.1
0.0	$26.75\pm 1.14$	$29.2\pm 0.78$	$122.39\pm 15.95$	$5336.9\pm 21.36$
0.001	$27.08\pm 0.72$	$27.05\pm 0.35$	$101.69\pm 12.31$	$4169.88\pm 43.03$
0.01	$27.8\pm 0.08$	$27.61\pm 0.06$	$69.48\pm 4.34$	$443.55\pm 21.18$
0.1	$29.16\pm 0.79$	$30.93\pm 3.52$	$165.99\pm 33.78$	$392.74\pm 10.19$

Table 7: Median WikiText-103 training perplexity

\pm

SD for every pair of weight-decay values used in schedule selection (rows) and later evaluation (columns). Lower decay during the search phase gives smaller medians. At evaluation, decay 0 or 0.001 holds perplexity down, whereas 0.01 or 0.1 drives it up.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 15: Progress of schedule descent optimizer on linear regression workload. Each curve represents the state of the schedule at different steps of the optimization of the loss at the final training point

T=1000

. The first step finds a good constant learning rate schedule; after that the algorithm runs standard gradient descent with a fallback if the schedule leads to diverging training. Final loss, and the corresponding schedule seemed to converge after around

300

steps. The schedule is represented by a

T

dimensional vector of discrete learning rates, continuity was not enforced beyond the initial condition.

Figure 16: Training loss curves for linear regression workload. When averaged over

1000

seeds, theory predicts early time and late time behavior well, but shows some systematic deviations at intermediate times where dynamics is near/above edge of stability, and loss curves can become non-monotonic.

Layer	Details	Output Shape
Input	Image (32 $\times$ 32 $\times$ 3)	(32, 32, 3)
Conv2D	32 filters, kernel size = (3, 3), stride = 1	(30, 30, 32)
Activation	ReLU	(30, 30, 32)
Max Pooling	window size = (2, 2), stride = 2	(15, 15, 32)
Conv2D	64 filters, kernel size = (3, 3), stride = 1	(13, 13, 64)
Activation	ReLU	(13, 13, 64)

Appendix

Table 8: Detailed architecture of the CNN model used for CIFAR-10 classification. The model comprises two convolutional blocks, each followed by ReLU activation and max pooling, and two fully connected layers.

Layer Type	Configuration	Output Shape
Input Embedding	Token and position embeddings	(batch, 128, 256)
Transformer Block (repeated $N$ times)
Multi-Head Self-Attention	$h$ heads, each with $d_{k}$ , $d_{v}$	(batch, 128, 256)
LayerNorm + Residual	Pre-norm	(batch, 128, 256)
Feedforward Network	Two linear layers with activation	(batch, 128, 256)
LayerNorm + Residual	Pre-norm	(batch, 128, 256)

Appendix

Table 9: Architecture of the Transformer language model used for WikiText-103 training. The model follows a standard decoder-only Transformer structure with multiple self-attention and feedforward layers.

Schedule Family	Hyperparameter	Search Range
Constant	warmup_steps (fraction)	$[0,0.25]$
Cosine	warmup_steps (fraction)	$[0,0.25]$
	exponent	$1.0$
	alpha	$[0,2.0]$
Square-root Decay	warmup_steps (fraction)	$[0,0.25]$
	alpha	$[0,2.0]$

Appendix

Table 10: Hyperparameter search spaces for all schedule families.

Panel of Figure 17 — Figure 17: Median CIFAR-10 training error under the Two-Point Spline schedule family at four seed counts. Each plot compares the median from 1, 3, 10, or 100 seeds on the horizontal axis with the reference median from 100 seeds on the vertical axis. A single seed produces wide scatter. Three seeds reduce the spread but still drift. Ten seeds lie close to the diagonal and roughly match the reference, showing that ten seeds give a stable estimate.

Panel of Figure 18 — Figure 18: To test our selection procedure, we selected the top $100$ schedules according to the reference measurement ( $100$ seed median, $y$ -axis) and smaller seed data ( $x$ -axis, various number of seeds). When a single seed was used for measurement, we found that $41$ of the $100$ schedules with lowest error according to the reference were not in the top $100$ of the single-seed measurement (left). In comparison, using a $100$ seed measurement, there were $17$ out of the top $100$ references which were not in the top $100$ of the new measurement. We found that using $10$ seeds for the measurement maintained similar false negative rates to the $100$ seed measurement, and captures most of the best performing seeds. This motivated our choice of $10$ seeds for the initial phase of the measurement protocol.

Panel of Figure 19 — Figure 19: Top- $5$ schedules from search for cos-std , cos-gen , and cos-y families. For CIFAR10 non-zero final learning rate closes most, but not all of the gap between cos-std and cos-gen (left). For wikitext, optimal $y$ value is close to $0$ and there is no gain.

Panel of Figure 20 — Figure 20: Top- $5$ schedules from search for tps and tps-y families. Optimal final learning rate values are close to $0$ and there are no appreciable differences between the families.

Figure 21: ECDF (Empirical cumulative distribution) of the best score found by random search in each schedule family. The data is the same as Figure 11 , left panel, except the family was searched with

36000

families (

10\times

more than the rest). The extra samples find schedules with better error than the original experiments, but lags behind the other families (higher error). This supports our claim that the family was not sufficiently searched/optimized relative to the other families.

Learning Rate Scheduling

Learning rate scheduling studies how to vary optimizer step sizes during training to improve stability, convergence, efficiency, and generalization.

See all 58 papers

Sep 8, 2026

BrachistoneLR: A Brachistochrone-Inspired Learning-Rate Schedule and a Controlled Benchmark of Scheduling Policies

The learning-rate schedule is a consequential choice in training deep networks, yet the policies in common use are heuristic, and published comparisons are hard to read, because architecture, dataset, and budget tend to vary alongside the schedule. We study BrachistoneLR, a schedule built by mapping the vertical coordinate of the brachistochrone, the curve of fastest descent under gravity, onto the range between a peak and a floor rate. Expanding the definition shows it to be cosine annealing with the half-period set to E - 1 instead of E, the configuration a standard implementation gives when its period argument is one less than the number of epochs. The rate therefore reaches its floor at the last epoch trained rather than one epoch later, and we show this difference decays as E^-2, making it a short-horizon effect. We then benchmark six schedules over 72 runs on three image classification datasets (MNIST, Fashion-MNIST, CIFAR-10) and four architecture families (fully connected, convolutional, recurrent, residual), fixing the optimizer, data pipeline, and evaluation protocol so that only the schedule varies. Schedules that fall smoothly from peak to floor beat the constant rate and calendar-based decay by margins that grow with task difficulty, reaching 2.5 points of dataset mean on CIFAR-10. Within that leading group, BrachistoneLR, cosine annealing, and warmup-cosine lie within 0.06 accuracy points and 0.17 of a mean rank, which one seed per configuration cannot separate. BrachistoneLR is best on both residual networks and has the highest CIFAR-10 mean, and it sets no milestones, decay factor, warmup length, or restart period. We conclude that the shape of a schedule matters more than its parameterization, that the choice of whether to use a smooth schedule matters more than the choice among them, and that the terminal-rate distinction is worth attention only over short horizons.

Md. Sadekur Rahman Roni, Md. Jalal uddin Chowdhury, Moutusi Dash Nimi

Department of Computer Science and Engineering Leading University Sylhet, Bangladesh · DeepNet Research and Development Lab Sylhet 3100, Bangladesh · Department of Electrical and Electronic Engineering Leading UniversitySep Sylhet, Bangladesh

Deep Learning Optimization Learning Rate Scheduling

Ask PDF

Feb 6, 2026

Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay

We study optimal learning rate (LR) schedules under the functional scaling law (FSL) framework (Li et al., 2025), which decomposes training dynamics into signal learning and noise forgetting. In power-law kernel regression, these two components are governed by a source exponent

s>0

and a capacity exponent

q>1

, respectively, with smaller

s

corresponding to harder tasks. For a fixed training horizon

N

, we characterize the schedules that minimize the final-step loss under a stability constraint and reveal a sharp phase transition. In the easy-task regime

s>1-1/q

, the optimal schedule follows power decay from the beginning of training; in the hard-task regime

s<1-1/q

, it becomes warmup-stable-decay (WSD)-like (Hu et al., 2024), staying at the largest admissible LR for most of training before a final decay. In both regimes, the decay exponent is

2q-1

: task difficulty determines when to decay, while model capacity determines how to decay. Beyond the exact optimum, we study fractional schedules, whose shape is defined over relative training progress. We show that precise tuning of the decay shape is often unnecessary: a broad class of profiles attains the optimal convergence rate, while overly slow terminal decay leads to schedule-induced capacity saturation. Finally, for one-pass SGD in kernel regression, FSL-motivated power-decay schedules achieve optimal last-iterate rates. Experiments support the theoretical predictions and the task-dependent transition between early and delayed decay.

Binghui Li, Zilin Wang, Fengling Chen +3

Center for Machine Learning Research, Peking University · School of Mathematical Sciences, Peking University · AI for Science Institute, Beijing

Learning Rate Scheduling Kernel Regression Stochastic Gradient Descent

Ask PDF

Jul 9, 2026

Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures

Choosing a learning rate scheduling strategy is critical to neural network training, but manual selection is costly and rarely exhaustive. While classical AutoML approaches often treat the scheduler as a secondary hyperparameter, we systematically investigate its impact on classification accuracy across a diverse pool of architectures. We evaluated 30 representative architectures from convolutional and transformer families within the LEMUR neural network dataset. Through automated source-code injection, we applied 25 scheduler configurations across nine PyTorch families, evaluating a total of 3,938 model variants on CIFAR-10. Our best configuration achieved a top-1 accuracy of 86.45%, with 237 variants exceeding 80%. The results show that the choice of scheduler depends heavily on the architecture: CosineAnnealingWarmRestarts and CyclicLR consistently outperform basic decay strategies. The resulting accuracy landscape, contributed to the LEMUR nn-dataset, provides a practical reference for principled scheduler selection.

Hafsa Mateen, Radu Timofte, Dmitry Ignatov

Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany

Learning Rate Scheduling Neural Network Optimization Hyperparameter Optimization

Ask PDF

Abstract

Learning Rate Scheduling

Explore similar work

BrachistoneLR: A Brachistochrone-Inspired Learning-Rate Schedule and a Controlled Benchmark of Scheduling Policies

Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay

Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures