When to restart? Exploring escalating restarts on convergence

Ayush K. Varshney; Šarūnas Girdzijauskas; Konstantinos Vandikas; Aneta Vulgarakis Feljan

cs.LGMar 4, 2026

When to restart? Exploring escalating restarts on convergence

Authors: Ayush K. Varshney, Šarūnas Girdzijauskas, Konstantinos Vandikas, Aneta Vulgarakis Feljan

Organizations: Ericsson Research, Ericsson AB, Sweden · KTH Royal Institute of Technology, Stockholm, Sweden · RISE Research Institutes of Sweden, Stockholm, Sweden

Abstract

Learning rate scheduling plays a critical role in the optimization of deep neural networks, directly influencing convergence speed, stability, and generalization. While existing schedulers such as cosine annealing, cyclical learning rates, and warm restarts have shown promise, they often rely on fixed or periodic triggers that are agnostic to the training dynamics, such as stagnation or convergence behavior. In this work, we propose a simple yet effective strategy, which we call Stochastic Gradient Descent with Escalating Restarts (SGD-ER). It adaptively increases the learning rate upon convergence. Our method monitors training progress and triggers restarts when stagnation is detected, linearly escalating the learning rate to escape sharp local minima and explore flatter regions of the loss landscape. We evaluate SGD-ER across CIFAR-10, CIFAR-100, and TinyImageNet on a range of architectures including ResNet-18/34/50, VGG-16, and DenseNet-101. Compared to standard schedulers, SGD-ER improves test accuracy by 0.5-4.5%, demonstrating the benefit of convergence-aware escalating restarts for better local optima.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison of learning rate schedulers and test accuracy on CIFAR-100 with ResNet-18 architecture with training budget of 2000 epochs. (Left) Learning rate trajectories for our proposed scheduler (ours_exp, in red ) alongside four baselines: SGD with exponential decay, SGD with Warmup-Stable-Decay-Simplified (SGD_WSDS), SGD with Cyclical Learning Rate (SGD_CLR), and SGD with Cosine Annealing and Warm Restarts (SGD_cosA). (Right) Comparison of test accuracy on CIFAR-100 under identical training budgets, results highlight that our approach converges to a better local optima and terminates early when further improvements are not found.

Dataset	SGD_exp	SGD_lin	Adam	CosA	CLR	WSDS	Ours_exp	Ours_lin
CIFAR-10	90.86	91.93	91.34	92.59	92.15	93.05	93.83	93.83
CIFAR-100	68.30	71.00	67.94	71.63	70.44	72.39	74.30	74.30
TinyImageNet	59.09	58.35	54.53	59.46	57.53	59.28	59.71	60.79

Table 1: Test accuracy (in %) results on CIFAR-10, CIFAR-100, and TinyImageNet with ResNet-18. Higher is better.

SGD_exp	SGD_lin	Adam	CosA	CLR	WSDS	Ours_exp	Ours_lin
68.53	62.17	71.27	72.84	72.10	73.59	74.41	74.41

Table 2: Test accuracy (in %) for CIFAR-100 after 2000 epochs with ResNet-18. Higher is better.

Panel of Figure 2 — Figure 2: (Left) Training loss trajectories for our proposed scheduler (ours_exp, in red ) alongside four baselines: SGD with exponential decay, SGD with Warmup-Stable-Decay-Simplified (SGD_WSDS), SGD with Cyclical Learning Rate (SGD_CLR), and SGD with Cosine Annealing and Warm Restarts (SGD_cosA). (Right) Test accuracy on CIFAR-100 under a training budget of 750 epochs; our approach (in red ) finds better local optima.

Approach	Train Loss	Val Loss	Test Loss	Test Acc (%)
CLR	1.60e-05 (±5.82e-07)	0.00488 (±6.88e-05)	0.00496 (±3.10e-05)	70.65 (±0.19)
SGD_exp	1.11e-04 (±5.71e-06)	0.00519 (±1.58e-04)	0.00525 (±9.74e-05)	68.49 (±0.17)
SGD_lin	2.80e-05 (±6.74e-07)	0.00486 (±1.03e-04)	0.00490 (±6.29e-05)	71.08 (±0.08)
Adam	1.58e-04 (±1.22e-04)	0.00588 (±1.44e-04)	0.00590 (±2.02e-04)	64.04 (±3.38)
CosA	1.75e-05 (±1.63e-06)	0.00466 (±1.79e-04)	0.00472 (±1.27e-04)	72.05 (±0.43)
WSDS	1.64e-05 (±1.23e-06)	0.00462 (±7.64e-05)	0.00465 (±8.62e-05)	72.83 (±0.44)

Table 3: Comparison of different learning rate schedulers on CIFAR-100 with ResNet-18 architecture. Values are mean ± std of best utility scores across 3 different seeds.

	CIFAR-10					CIFAR-100
Architecture	SGD_exp	CosA	CLR	WSDS	Ours	SGD_exp	CosA	CLR	WSDS	Ours
ResNet34	90.81	92.73	92.38	92.88	93.82	67.75	72.17	71.04	72.36	74.24
ResNet50	90.16	92.77	91.77	93.13	94.65	65.52	72.10	70.25	73.76	76.77
VGG16	90.27	91.42	90.88	91.44	92.02	65.17	67.35	67.23	68.08	68.56
DenseNet121	86.31	91.81	88.99	92.77	94.19	56.10	71.20	66.61	72.45	76.76

Table 4: Test accuracy (in %) results on CIFAR-10 and CIFAR-100 across architectures with exponential decay.

	ResNet-34					ResNet-50
Dataset	SGD_exp	CosA	CLR	WSDS	Ours	SGD_exp	CosA	CLR	WSDS	Ours
TinyImageNet	58.08	61.12	60.30	62.30	64.53	58.43	62.50	60.62	64.52	64.93

Table 5: Test accuracy (in %) results on TinyImageNet across ResNet architectures with exponential decay.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Comparison of learning rate schedulers and test accuracy on CIFAR-100 with ResNet-18 architecture with training budget of 2000 epochs. (Left) Learning rate trajectories for our proposed scheduler (ours_exp - with exponential decay, ours_lin - with linear decay) alongside these baselines: SGD with exponential decay, SGD with linear decay (SGD_lin), SGD with Warmup-Stable-Decay-Simplified (SGD_WSDS), Adam, SGD with Cyclical Learning Rate (SGD_CLR), and SGD with Cosine Annealing and Warm Restarts (SGD_cosA). (Right) Comparison of test accuracy on CIFAR-100 under identical training budgets, results highlight that our approaches converges to a better solutions.

Panel of Figure 4 — Figure 4: (Left) Training loss trajectories for our proposed scheduler (ours_exp, in red ) alongside four baselines: SGD with exponential decay, SGD with Warmup-Stable-Decay-Simplified (SGD_WSDS), SGD with Cyclical Learning Rate (SGD_CLR), and SGD with Cosine Annealing and Warm Restarts (SGD_cosA). (Right) Test accuracy on CIFAR-10 under a training budget of 500 epochs; our approach (in red ) finds better local optima.

Test Losses
Dataset	SGD_exp	SGD_lin	Adam	CosA	CLR	WSDS	Ours_exp	Ours_lin
CIFAR-10	1.57E-03	1.51E-03	1.54E-03	1.22E-03	1.47E-03	1.19E-03	1.03E-03	1.03E-03
CIFAR-100	5.37E-03	4.96E-03	5.67E-03	4.86E-03	4.96E-03	4.70E-03	4.29E-03	4.29E-03
TinyImageNet	2.65E-02	7.63E-03	9.08E-03	7.05E-03	7.73E-03	7.58E-03	7.39E-03	7.37E-03
Val Losses
CIFAR-10	1.52E-03	1.40E-03	1.49E-03	1.22E-03	1.39E-03	1.22E-03	1.04E-03	1.04E-03

Appendix

Table 6: Loss values (Test, Validation, Train) on CIFAR-10, CIFAR-100, and TinyImageNet with ResNet-18. Lower is better.

Loss type	SGD_exp	SGD_lin	Adam	CosA	CLR	WSDS	Ours_exp	Ours_lin
Test loss	5.36E-03	4.96E-03	6.05E-03	4.73E-03	4.86E-03	4.58E-03	4.30E-03	4.30E-03
Val loss	5.34E-03	5.01E-03	6.14E-03	4.77E-03	4.93E-03	4.62E-03	4.29E-03	4.29E-03
Train loss	1.02E-04	2.51E-05	2.22E-04	1.56E-05	1.29E-05	1.56E-05	2.50E-05	2.50E-05

Appendix

Table 7: Loss metrics for CIFAR-100 after 2000 epochs with ResNet-18. Lower is better.

Test Losses
	CIFAR-10					CIFAR-100
Architecture	SGD_exp	CosA	CLR	WSDS	Ours	SGD_exp	CosA	CLR	WSDS	Ours
ResNet34	1.70E-03	1.37E-03	1.52E-03	1.19E-03	1.09E-03	5.77E-03	4.73E-03	5.19E-03	4.91E-03	4.49E-03
ResNet50	1.67E-03	1.34E-03	1.60E-03	1.33E-03	8.50E-04	6.26E-03	4.71E-03	5.29E-03	4.53E-03	3.97E-03
VGG16	1.78E-03	1.63E-03	1.73E-03	1.64E-03	1.58E-03	6.46E-03	6.12E-03	6.70E-03	6.58E-03	6.52E-03
DenseNet121	2.12E-03	1.33E-03	1.99E-03	1.27E-03	1.03E-03	6.86E-03	5.78E-03	6.67E-03	5.84E-03	3.91E-03

Appendix

Table 8: Loss values (Test, Validation, Train) for CIFAR-10 and CIFAR-100 across architectures. Lower is better.

Panel of Figure 5 — Figure 5: Training loss trajectories for our proposed scheduler (ours_exp, in red ) compared against four baselines: standard SGD with exponential decay, SGD with Warmup - Stable - Decay - Simplified (SGD_WSDS), SGD with Cyclical Learning Rate (SGD_CLR), and SGD with Cosine Annealing and Warm Restarts (SGD_cosA). While the baseline methods largely converge and stop improving with additional training, SGD - ER (ours, in red ) continues to make steady progress, ultimately reaching higher test accuracies and demonstrating its advantage in prolonged - training regimes.

	ResNet-34					ResNet-50
Losses	SGD_exp	CosA	CLR	WSDS	Ours	SGD_exp	CosA	CLR	WSDS	Ours
Test loss	8.35E-03	7.25E-03	7.35E-03	7.21E-03	2.57E-02	8.80E-03	6.62E-03	7.16E-03	6.54E-03	6.39E-03
Val loss	8.32E-03	7.16E-03	7.27E-03	7.17E-03	2.58E-02	8.80E-03	6.62E-03	7.08E-03	6.53E-03	6.35E-03
Train Loss	7.80E-05	1.95E-05	1.81E-05	2.10E-05	1.80E-04	6.63E-05	2.56E-05	2.15E-05	2.58E-05	4.06E-05

Appendix

Table 9: Loss values (Test, Val, and Train) on TinyImageNet across ResNet architectures. Lower is better.

Deep Learning Optimization

Deep learning optimization develops algorithms and training strategies that help neural networks converge, generalize, and use compute efficiently.

See all 206 papers

BrachistoneLR: A Brachistochrone-Inspired Learning-Rate Schedule and a Controlled Benchmark of Scheduling Policies
Sep 8, 2026Md. Sadekur Rahman Roni, Md. Jalal uddin Chowdhury, Moutusi Dash NimiDeep Learning Optimization Learning Rate Scheduling
Department of Computer Science and Engineering Leading University Sylhet, Bangladesh · DeepNet Research and Development Lab Sylhet 3100, Bangladesh · Department of Electrical and Electronic Engineering Leading UniversitySep Sylhet, Bangladesh
WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training
Jul 12, 2026Jianhao Ma, Yuxin ChenLearning Rate Scheduling
Department of Statistics and Data Science, University of Pennsylvania
EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization
May 25, 2026Chung-Yiu Yau, Dawei Li, Athanasios Glentis +3Deep Learning Optimization Gradient Descent
University of Minnesota · 2Amazon AGI · 3The Chinese University of Hong Kong

Sep 8, 2026J/K move · Enter open · S save

BrachistoneLR: A Brachistochrone-Inspired Learning-Rate Schedule and a Controlled Benchmark of Scheduling Policies

Md. Sadekur Rahman Roni, Md. Jalal uddin Chowdhury, Moutusi Dash Nimi

Department of Computer Science and Engineering Leading University Sylhet, Bangladesh · DeepNet Research and Development Lab Sylhet 3100, Bangladesh · Department of Electrical and Electronic Engineering Leading UniversitySep Sylhet, Bangladesh

Deep Learning Optimization Learning Rate Scheduling

Open paper AskPDF arXiv

The learning-rate schedule is a consequential choice in training deep networks, yet the policies in common use are heuristic, and published comparisons are hard to read, because architecture, dataset, and budget tend to vary alongside the schedule. We study BrachistoneLR, a schedule built by mapping the vertical coordinate of the brachistochrone, the curve of fastest descent under gravity, onto the range between a peak and a floor rate. Expanding the definition shows it to be cosine annealing with the half-period set to E - 1 instead of E, the configuration a standard implementation gives when its period argument is one less than the number of epochs. The rate therefore reaches its floor at the last epoch trained rather than one epoch later, and we show this difference decays as E^-2, making it a short-horizon effect. We then benchmark six schedules over 72 runs on three image classification datasets (MNIST, Fashion-MNIST, CIFAR-10) and four architecture families (fully connected, convolutional, recurrent, residual), fixing the optimizer, data pipeline, and evaluation protocol so that only the schedule varies. Schedules that fall smoothly from peak to floor beat the constant rate and calendar-based decay by margins that grow with task difficulty, reaching 2.5 points of dataset mean on CIFAR-10. Within that leading group, BrachistoneLR, cosine annealing, and warmup-cosine lie within 0.06 accuracy points and 0.17 of a mean rank, which one seed per configuration cannot separate. BrachistoneLR is best on both residual networks and has the highest CIFAR-10 mean, and it sets no milestones, decay factor, warmup length, or restart period. We conclude that the shape of a schedule matters more than its parameterization, that the choice of whether to use a smooth schedule matters more than the choice among them, and that the terminal-rate distinction is worth attention only over short horizons.

Abstract

Deep Learning Optimization

Explore similar work

BrachistoneLR: A Brachistochrone-Inspired Learning-Rate Schedule and a Controlled Benchmark of Scheduling Policies

WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training

EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization