Many Optimizers But Only One Training Path: Repeated Resampling for Adaptive Optimizer Selection

Ronald Richman; Mario V. Wüthrich

cs.LGAug 19, 2026

Many Optimizers But Only One Training Path: Repeated Resampling for Adaptive Optimizer Selection

Authors: Ronald Richman, Mario V. Wüthrich

Organizations: insureAI · ETH Zürich

Abstract

An optimizer is usually chosen before training a deep neural network and then kept fixed. Treating optimizer choice as a hyperparameter could boost performance, but it requires several complete training runs and discards all but the winner. Repeated Optimizer Resampling (ROR) instead searches during one evolving run. Every $b$ epochs, each candidate optimizer scouts from the current model weights for $s$ epochs. The best scout continues for the remaining $b-s$ epochs, and that completed segment becomes the new incumbent if it improves the validation objective. This design allows the preferred optimizer to change as training progresses. We compare two variants of ROR on MNIST, Fashion-MNIST, and two motor insurance claim-count models. Nine fixed optimizers and both ROR variants are evaluated with the same ten seeds. One-epoch ROR uses 24% to 35% of the aggregate training needed to identify the best fixed optimizer exhaustively and remains close to that optimizer on all four tasks. These results support short scouting as a practical way to search over optimizers without completing every candidate run.

Figures & tables

Training rule	MNIST			Fashion-MNIST
	Accuracy	Rank	Epoch-eq.	Accuracy	Rank	Epoch-eq.
Adam	98.003 $\pm$ 0.244	7	16.5	88.287 $\pm$ 0.472	13	16.1
AdamW	97.928 $\pm$ 0.213	12	15.0	88.262 $\pm$ 0.454	14	15.8
Lion	97.552 $\pm$ 0.189	16	13.4	88.578 $\pm$ 0.299	9	12.9
Muon	98.095 $\pm$ 0.139	4	15.2	88.431 $\pm$ 0.516	10	13.3
SGD-N	97.787 $\pm$ 0.213	14	18.1	88.084 $\pm$ 0.356	15	17.6

Table 1: Image classification results over ten seeds. Accuracy is the mean test accuracy in percentage points, followed by its sample standard deviation (bigger is better). Ranks use unrounded means across the 16 fitted training rules, with rank 1 best. The full optimizer search row is the aggregate cost of completing all nine fixed runs and is not another fitted model.

Figure 1: Image classification accuracy by seed (bigger is better). Points are individual runs; horizontal bars show means and 95% confidence intervals. ROR uses retained segments of

b=3

epochs.

Architecture	Training rule	Test deviance	Rank	Calibration	Epoch-eq.
MLP	Adam	0.238463 $\pm$ 0.000263	7	0.999	44.3
	AdamW	0.238463 $\pm$ 0.000263	8	0.999	44.2
	Lion	0.238365 $\pm$ 0.000353	2	0.995	31.3
	Muon	0.238652 $\pm$ 0.000178	15	0.999	22.1
	SGD-N	0.249816 $\pm$ 0.007473	16	1.003	58.1
	Nadam	0.238449 $\pm$ 0.000240	6	1.000	47.4

Table 2: Insurance results over ten seeds. Test deviance is the mean Poisson deviance, followed by its sample standard deviation (smaller is better). Calibration is predicted claims divided by observed claims. Ranks use unrounded mean deviance across the 16 fitted training rules within each architecture, with rank 1 best. The full optimizer search row is the aggregate cost of completing all nine fixed runs and is not another fitted model.

Figure 2: Insurance test deviance by seed. Points are individual runs; horizontal bars show means and 95% confidence intervals. The MLP panel is restricted to the range containing the competitive methods; the fixed SGD-Nesterov mean of 0.249816 falls outside that range (smaller is better).

Figure 3: Predictive performance as the scouting period changes from one to three epochs, with retained segment length fixed at

b=3

. Classification accuracy is higher when better; Poisson deviance is lower when better. Error bars show 95% confidence intervals for the mean.

Figure 4: State-preserving ROR schedules (

s=1

b=3

). Image tasks above, insurance models below. Blank cells follow the first rejected tournament. Letters in each cell are defined in the legend.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Policy	Comparison	Metric	Difference (95% CI)	Short eq.	$s=3$ eq.
MNIST	CS-ROR	$s=1$ minus $s=3$	Accuracy (pp)	-0.184 [-0.355, -0.013]	37.4	116.1
	CS-ROR	$s=2$ minus $s=3$	Accuracy (pp)	-0.095 [-0.256, 0.066]	58.9	116.1
	SP-ROR	$s=1$ minus $s=3$	Accuracy (pp)	-0.143 [-0.307, 0.021]	39.6	110.7
	SP-ROR	$s=2$ minus $s=3$	Accuracy (pp)	-0.151 [-0.285, -0.017]	68.4	110.7
Fashion-MNIST	CS-ROR	$s=1$ minus $s=3$	Accuracy (pp)	0.019 [-0.258, 0.296]	53.9	110.7
	CS-ROR	$s=2$ minus $s=3$	Accuracy (pp)	-0.091 [-0.377, 0.195]	81.7	110.7

Appendix

Table 3: Paired short-scout comparisons over ten seeds. Differences are the shorter scout minus

s=3

. Search cost is reported in epoch-equivalents.

Figure 5: Cold-start ROR schedules (

s=1

b=3

). Image tasks above, insurance models below. Every scout, including a repeated winner, starts with newly initialized optimizer state. Letters in each cell are defined in the legend.

Deep Learning Optimization

Deep learning optimization develops algorithms and training strategies that help neural networks converge, generalize, and use compute efficiently.

See all 205 papers

OptiRoulette Optimizer: A New Stochastic Meta-Optimizer for up to 5.3x Faster Convergence
Feb 20, 2026Stamatis MastromichalakisNeural Network Optimization Stochastic Optimization
Independent Researcher, tmnetworks.gr
EvE: An Alternate Optimizer to Adam
Sep 28, 2026Shashank Raj, Kalyanmoy DebDeep Learning Optimization Differential Evolution
Department of Computer Science The University of Alabama Tuscaloosa, AL 35487, USA
AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization
Aug 3, 2026Alok Kumar Pandey, Umang Chaturvedi, Aatish Rana +1Deep Learning Optimization Stochastic Optimization Convergence

Feb 20, 2026J/K move · Enter open · S save

OptiRoulette Optimizer: A New Stochastic Meta-Optimizer for up to 5.3x Faster Convergence

Stamatis Mastromichalakis

Independent Researcher, tmnetworks.gr

Neural Network Optimization Stochastic Optimization Learning to Optimize

Open paper AskPDF arXiv

This paper presents OptiRoulette, a stochastic meta-optimizer that selects update rules during training instead of fixing a single optimizer. The method combines warmup optimizer locking, random sampling from an active optimizer pool, compatibility-aware learning-rate scaling during optimizer transitions, and failure-aware pool replacement. OptiRoulette is implemented as a drop-in, "torch.optim.Optimizer-compatible" component and packaged for pip installation. We report completed 10-seed results on five image-classification suites: CIFAR-100, CIFAR-100-C, SVHN, Tiny ImageNet, and Caltech-256. Against a single-optimizer AdamW baseline, OptiRoulette improves mean test accuracy from 0.6734 to 0.7656 on CIFAR-100 (+9.22 percentage points), 0.2904 to 0.3355 on CIFAR-100-C (+4.52), 0.9667 to 0.9756 on SVHN (+0.89), 0.5669 to 0.6642 on Tiny ImageNet (+9.73), and 0.5946 to 0.6920 on Caltech-256 (+9.74). Its main advantage is convergence reliability at higher targets: it reaches CIFAR-100/CIFAR-100-C 0.75, SVHN 0.96, Tiny ImageNet 0.65, and Caltech-256 0.62 validation accuracy in 10/10 runs, while the AdamW baseline reaches none of these targets within budget. On shared targets, OptiRoulette also reduces time-to-target (e.g., Caltech-256 at 0.59: 25.7 vs 77.0 epochs). Paired-seed deltas are positive on all datasets; CIFAR-100-C test ROC-AUC is the only metric not statistically significant in the current 10-seed study.

Abstract

Deep Learning Optimization

Explore similar work

OptiRoulette Optimizer: A New Stochastic Meta-Optimizer for up to 5.3x Faster Convergence

EvE: An Alternate Optimizer to Adam

AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization