Towards Easier-to-Tune Training Algorithms: New Results from AlgoPerf

Priya Kasimbeg; Ahmed Khaled; Aahlad Chandrabhatta; Yifan Wang; Andy Liu; Niccolò Ajroldi; David Tweedle; Xingyu

cs.LGOct 9, 2026

Towards Easier-to-Tune Training Algorithms: New Results from AlgoPerf

Authors: Priya Kasimbeg, Ahmed Khaled, Aahlad Chandrabhatta, Yifan Wang, Andy Liu, Niccolò Ajroldi, David Tweedle, Xingyu,

Organizations: Alice · Goodfire · University of Tübingen · Google

Abstract

Evaluating deep learning optimizer is challenging because confounding factors such as hyperparameter tuning and optimizer-model co-design often obscure true algorithmic performance. We present results of the updated AlgoPerf V1 benchmark, in which we evaluate a new set of training algorithms on wall-clock time-to-target across nine workloads on fixed hardware. This report focuses on the results of the benchmark's self-tuning ruleset, which requires submissions to use a single configuration across all workloads. Out of the 13 submissions in PyTorch and JAX, Schedule-Free AdamW ranked first. In addition to the wall-clock scores, we also analyze the trade-offs between step efficiency and wall-clock execution, revealing that matrix-preconditioned and two-loop methods achieve good sample efficiency but require more aggressive computational optimizations to be competitive in runtime. Finally, our hyperparameter saliency and retention analyses show that the learning rate and momentum are highly sensitive across domains and algorithms, highlighting the need for methods that are more robust against hyperparameter tuning and can generalize well to new tasks.

Figures & tables

					Validation	Base Runtime
Task	Dataset	Model	Loss	Metric	Target	Budget (s)
Clickthrough rate prediction	Criteo 1TB	DLRMsmall	CE	CE	$0.123735$	$8915$
MRI reconstruction	fastMRI	U-Net	L1	SSIM	$0.723653$	$2745$
Image	ImageNet	ResNet-50	CE	ER	$0.22569$	$49,918$
classification		ViT	CE	ER	$0.22691$	$64,292$
Speech	LibriSpeech	Conformer	CTC	WER	$0.085884$	$43,680$

Table 1: Summary of the nine AlgoPerf base workloads, validation targets, and base runtime budgets. The self-tuning runtime budgets are

1.5\times

these base values.

Algorithm	Brief Description	Citation	Submission Code
Schedule-Free AdamW v2 (PyTorch)	Couples momentum interpolation with dynamic iterate averaging; eliminates LR schedules and maintains batch norm statistics. This version makes use of the batch-norm fixes in AlgoPerf v1 and is as a result more faithful to the published version of Schedule-Free.	( Defazio et al., 2024 )	schedule_free_adamw_v2
AdEMAMix (AdamW equiv.) (PyTorch)	Blends fast and slow gradient EMAs to track short-term fluctuations while retaining long-term directional memory. This submission sets a long warmup for $\alpha$ and $\beta_{3}$ making it effectively equivalent to AdamW.	( Pagliardini et al., 2025 )	ademamix
Schedule-Free AdamW (PyTorch)	Legacy Schedule-Free AdamW implementation submitted to the AlgoPerf competition. This version does not contain batch-norm fixes.	( Defazio et al., 2024 )	schedule_free_adamw
NAdamW (Baseline v0.5) (JAX)	Nesterov-accelerated adaptive gradient optimizer with decoupled weight decay. This version served as the prize qualification baseline for the AlgoPerf v0.5 competition.	( Dozat, 2016 ; Loshchilov and Hutter, 2019 )	nadamw_baselinev05
Schedule-Free AdamW v2 (JAX)	JAX implementation of Schedule-Free AdamW v2 with batch-normalization fix.	( Defazio et al., 2024 )	schedule_free_adamw_jax_v2
Muon (PyTorch)	Matrix-preconditioned optimizer applying quintic Newton-Schulz iterations on 2D or higher-dimensional non-embedding tensors with AdamW for 1D and embedding parameters.	( Jordan et al., 2024b )	muon_torch

Table 2: Summary of evaluated algorithms on the AlgoPerf self-tuning leaderboard.

Figure 1: AlgoPerf self-tuning leaderboard and performance profiles. The leaderboard ranks the submissions according to their AlgoPerf scores, which correspond to the areas under the curves of the performance profiles.

Figure 2: Wall-clock vs. step-based benchmark score per submission.

Figure 3: AlgoPerf score (left) and leaderboard rank (right) as targets are relaxed.

Figure 4: Workload–Hyperparameter Importance Heatmaps Across Optimizers. Normalized 1D Wasserstein importance indices ( $\mathcal{W}_{1}/\text{Span}$ , Equation 1 ) across the AlgoPerf benchmark suite, evaluated at the top $10\%$ best-performing trial threshold ( $k=0.10$ ). Higher values (darker red) denote hyperparameters that have to collapse into a narrow band to achieve top-tier performance, while lower values reflect broad robustness across the search space. Each subplot isolates the active hyperparameters specific to that optimizer.

Figure 5: Spans of learning rate across the top 5% configurations. Disjoint regions indicate fundamental tension for universal tuning. Note that for Muon, the LR indicates the learning rate applied to the paired optimizer and serves as a basis for the Muon LR multiplier.

Figure 6: Distribution of nine-workload mean time-based speedup scores (

\bar{S}

) across hyperparameter sweeps for DiLoCo, NAdamW, Muon, and AdEMAMix (

N=100

per optimizer) under a

2\times

extrapolation horizon on the running-best validation trajectory. Vertical dashed red lines indicate medians; solid black lines mark the single golden configuration.

Tuning Metric	DiLoCo ( $D=4$ )	NAdamW ( $D=3$ )	Muon ( $D=5$ )	AdEMAMix ( $D=7$ )
Golden config score ( $\bar{S}(c^{*})$ )	0.786	0.989	0.927	0.583
Oracle score ( $\bar{S}^{*}$ )	1.279	1.509	1.973	1.403
Retention Rate ( $\mathcal{R}$ )	61.4%	65.5%	47.0%	41.5%
Median Speedup ( $p_{50}$ )	0.427	0.000	0.063	0.000
Failure Rate ( $P(\bar{S}\leq 0.20)$ )	1.0%	70.0%	71.0%	85.0%

Table 3: Hyperparameter sensitivity metrics across

N=100

configurations under extrapolated time-to-target scoring (

S_{w}(c)=T_{\max,w}/\hat{t}_{\text{target}}

, capped at

\hat{t}_{\text{target}}\leq 2\,T_{\max,w}

). Failure rate denotes

\bar{S}(c)\leq 0.20

Figure 7: Retention gap and tuning budget progression across optimizers under a

2\times

extrapolation horizon on the running-best validation trajectory. Left: Golden config score (

\bar{S}(c^{*})

, single configuration across all 9 workloads) versus Oracle score (

\bar{S}^{*}

, per-workload tuning) with retention percentage

\mathcal{R}=\bar{S}(c^{*})/\bar{S}^{*}

. Right: Expected best speedup

\bar{S}

as a function of tuning budget

k\in[1,100]

under universal search (dashed) versus per-workload search (solid); see below.

Figure 8: Training dynamics of Schedule-Free AdamW on the ImageNet ResNet workload. Calibrating Batch Normalization running statistics at

x_{t}

rather than

y_{t}

in Schedule-Free AdamW v2 (PyTorch) resolves the severe covariate shift that stalled validation progress in earlier iterations.

Framework	Implementation	HPs	Score	$\Delta$
JAX	vanilla	JAX	$0.2845$	—
PyTorch	vanilla	JAX	$0.2969$	$+0.012$
PyTorch	sharded	JAX	$0.3137$	$+0.017$
PyTorch	sharded	Torch	$0.4231$	$+0.109$
PyTorch	vanilla	Torch	$0.3827$	$-0.040$

Table 4: Hyperparameter ablations on Muon submissions. In this table we compare the hypothetical scores for Muon (JAX) and Muon (PyTorch) submissions ablations in the overall submission pool. For the Muon (PyTorch) submissions we evaluate both the vanilla and sharded version on a combination of the hyperparameters found for the vanilla (JAX) implementation and the sharded (Torch) implementations. Each row changes one factor (bold) relative to the row above. Rows one and four are the leaderboard entries Muon (JAX) and Muon (PyTorch) ; the final row steps back down to the vanilla implementation.

Figure 9: Empirical speedup across workloads for Muon implementations, contrasting the communication-sharded Muon (PyTorch) against the replicated baseline.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Submission	Criteo 1TB	fastMRI	FineWeb-Edu 10B	ResNet	ViT	Conformer	DeepSpeech	OGBG	WMT
Schedule-Free AdamW v2 (PyTorch)	0.56	0.27	—	—	—	0.85	0.76	0.37	0.43
AdEMAMix (AdamW equiv.) (PyTorch)	0.59	0.37	0.38	—	—	0.75	—	0.37	0.72
Schedule-Free AdamW (PyTorch)	0.77	0.29	—	—	—	—	0.73	0.21	0.83
NAdamW (Baseline v0.5) (JAX)	0.61	0.39	0.43	—	—	—	—	0.24	0.62
Schedule-Free AdamW v2 (JAX)	0.61	0.45	0.32	—	0.88	—	—	0.32	—
Muon (PyTorch)	0.53	—	0.32	—	—	—	—	0.11	0.59

Appendix

Table 5: Time to target as a fraction of each workload’s self-tuning-ruleset time budget (

1.5\times

the external-tuning maximum allowed runtime). — = target not reached.

Submission	Criteo 1TB	fastMRI	FineWeb-Edu 10B	ResNet	ViT	Conformer	DeepSpeech	OGBG	WMT
JAX submissions
Schedule-Free AdamW (JAX)	$1.67$	$1.25$	$0.51$	$1.04$	$1.33$	$1.00$	$0.47$	$1.05$	$1.05$
Schedule-Free AdamW v2 (JAX)	$1.00$	$0.96$	$0.52$	$1.17$	$1.10$	$1.06$	$1.04$	$1.03$	$1.05$
Muon (JAX)	$1.96$	$0.97$	$1.16$	$1.47$	$1.24$	$1.00$	$1.00$	$1.01$	$1.39$
NAdamW (JAX)	$1.01$	$0.73$	$1.00$	$0.99$	$1.00$	$0.97$	$0.91$	$1.01$	$1.01$
NAdamW (Baseline v0.5) (JAX)	$1.00$	$1.00$	$1.00$	$1.00$	$1.00$	$1.00$	$1.00$	$1.00$	$1.00$

Appendix

Table 6: Step execution times, normalized to NAdamW (Baseline v0.5) (JAX) . Each value is the ratio of a submission’s average time per step to the reference’s on that workload, excluding the first evaluation interval to remove compilation time; submissions run at their own batch sizes.

	Wall-clock		Steps
Submission	Score	Rank	Score	Rank	$\Delta$
Schedule-Free AdamW v2 (PyTorch)	0.5679	1	0.5253	1	–
AdEMAMix (AdamW equiv.) (PyTorch)	0.5191	2	0.5146	2	–
Schedule-Free AdamW (PyTorch)	0.4669	3	0.4461	3	–
NAdamW (Baseline v0.5) (JAX)	0.4506	4	0.4422	4	–
Schedule-Free AdamW v2 (JAX)	0.4405	5	0.3945	6	$\downarrow$ 1

Appendix

Table 7: Wall-clock vs. step-based benchmark scores. Both use the same performance-profile scoring; only the notion of training time differs (seconds vs. optimizer steps to target).

\Delta

is the rank change when moving from wall-clock to step-based scoring.

Figure 10: AlgoPerf self-tuning leaderboard and performance profiles (steps to target).

Submission	Criteo 1TB	fastMRI	FineWeb-Edu 10B	ResNet	ViT	Conformer	DeepSpeech	OGBG	WMT
Schedule-Free AdamW v2 (PyTorch)	7,951	12,431	—	—	—	83,184	78,669	28,449	73,222
AdEMAMix (AdamW equiv.) (PyTorch)	8,371	10,206	44,503	—	—	67,706	—	27,247	112,742
Schedule-Free AdamW (PyTorch)	10,346	10,599	—	—	—	—	90,588	15,853	133,517
NAdamW (Baseline v0.5) (JAX)	9,145	6,648	47,039	—	—	—	—	20,879	118,600
Muon (PyTorch)	7,716	—	39,220	—	—	—	—	7,711	97,224
Schedule-Free AdamW v2 (JAX)	9,005	8,750	68,640	—	164,206	—	—	27,457	—

Appendix

Table 8: Median number of optimizer steps to reach the validation target. — = target not reached. Submissions choose their own batch sizes, so step counts compare optimizer updates, not examples seen.

	Standard		10% relaxed
Submission	Score	Rank	Score	Rank	$\Delta$
Schedule-Free AdamW v2 (JAX)	0.4405	5	0.6852	1	$\uparrow$ 4
AdEMAMix (PyTorch)	0.2464	12	0.6246	2	$\uparrow$ 10
Schedule-Free AdamW v2 (PyTorch)	0.5679	1	0.6072	3	$\downarrow$ 2
Muon (PyTorch)	0.4231	6	0.5993	4	$\uparrow$ 2
Schedule-Free AdamW (PyTorch)	0.4669	3	0.5786	5	$\downarrow$ 2

Appendix

Table 9: Standard vs. 10% relaxed-target AlgoPerf self-tuning leaderboard.

\Delta

is the rank change after relaxing targets.

Figure 11: AlgoPerf self-tuning leaderboard and performance profiles with

10\%

relaxed targets.

Submission	Criteo 1TB	fastMRI	FineWeb-Edu 10B	ResNet	ViT	Conformer	DeepSpeech	OGBG	WMT
Schedule-Free AdamW v2 (JAX)	0.03	0.07	0.21	0.11	0.27	—	—	0.13	0.08
AdEMAMix (PyTorch)	0.03	0.09	0.16	—	—	0.67	0.44	0.16	0.14
Schedule-Free AdamW v2 (PyTorch)	0.03	0.06	0.59	—	—	0.61	0.42	0.16	0.06
Muon (PyTorch)	0.03	0.08	0.22	—	—	0.72	—	0.08	0.08
Schedule-Free AdamW (PyTorch)	0.03	0.08	—	—	—	0.72	0.42	0.13	0.08
Schedule-Free AdamW (JAX)	0.03	0.09	0.16	—	0.27	—	—	0.16	0.08

Appendix

Table 10: Time to target with targets relaxed 10%, as a fraction of each workload’s self-tuning-ruleset time budget (

1.5\times

the external-tuning maximum allowed runtime). Bold values mark targets reached only after relaxation; — = target not reached.

Figure 12: Validation metric against wall-clock time for Muon (PyTorch) , Muon (JAX) , and Lion (PyTorch) on the nine base workloads. Lines show the mean over three studies and shaded bands show

\pm 1

standard deviation. The dotted line marks the validation target. Solid lines are PyTorch and dashed lines are JAX implementations. Minimized metrics (loss, WER, perplexity) are on a logarithmic axis. Each curve ends when its longest study stops, either at the target or at the time budget. Muon (JAX) uses different fixed hyperparameters from Muon (PyTorch) . The two differ most on ImageNet ResNet-50 and ImageNet ViT, where Muon (JAX) reaches higher accuracy, and on OGBG GNN and WMT Transformer, where it levels off below the target.

Figure 13: Validation metric against wall-clock time for NAdamW (JAX) , NAdamW (Baseline v0.5) (JAX) , Cautious NAdamW (JAX) , and Single-worker DiLoCo (JAX) on the nine base workloads. All four are JAX implementations. Lines show the mean over three studies and shaded bands show

\pm 1

standard deviation. Minimized metrics (loss, WER, perplexity) are on a logarithmic axis. Single-worker DiLoCo (JAX) trails the NAdamW variants on ImageNet ViT, LibriSpeech Conformer, and OGBG GNN. Cautious NAdamW (JAX) shows a loss spike on Criteo 1TB DLRMsmall and regresses after about four hours on LibriSpeech DeepSpeech.

Figure 14: Validation metric against wall-clock time for the four Schedule-Free AdamW submissions ( Schedule-Free AdamW (PyTorch) , Schedule-Free AdamW v2 (PyTorch) , Schedule-Free AdamW (JAX) , Schedule-Free AdamW v2 (JAX) ) and the two AdEMAMix submissions ( AdEMAMix (AdamW equiv.) (PyTorch) , AdEMAMix (PyTorch) ) on the nine base workloads. Lines show the mean over three studies and shaded bands show

\pm 1

standard deviation. The dotted line marks the validation target. Solid lines are PyTorch and dashed lines are JAX implementations. Minimized metrics (loss, WER, perplexity) are on a logarithmic axis. On ImageNet ResNet-50, the first versions of Schedule-Free AdamW ( Schedule-Free AdamW (PyTorch) , Schedule-Free AdamW (JAX) ) oscillate while both v2 submissions train stably. On ImageNet ViT, the JAX implementations nearly reach the target while the PyTorch submissions level off near 0.65 accuracy.

Optimizer	Tuned Hyperparameter	Sampling Range / Values	Distribution	Fixed Parameters & Architectural Notes
NAdamW ( $D=7$ )	Learning rate ( $\eta$ )	$[10^{-5},\,5\times 10^{-3}]$	Log-uniform	Cosine decay schedule ( cosine_warmup )
	Momentum $1-\beta_{1}$	$[10^{-3},\,0.5]$	Log-uniform	Fixed $\epsilon=10^{-8}$ , bias correction ( debias = True )
	Second moment $1-\beta_{2}$	$[10^{-3},\,0.5]$	Log-uniform	Tied dropout rates ( tie_dropouts = True , when supported)
	Weight decay ( $\lambda$ )	$[10^{-2},\,0.5]$	Log-uniform	l2_decay_factor = None
	LR warmup fraction ( $T_{\text{warmup}}$ )	$\{0.02,\,0.05,\,0.10\}$	Discrete uniform	∗ Tuned only on workloads supporting the regularizer
	Label smoothing ∗	$\{0.0,\,0.2\}$	Discrete uniform

Appendix

Table 11: Hyperparameter search space specifications for the

N=100

quasi-random Halton sweeps evaluated across all nine AlgoPerf workloads.

Figure 15: Comparison of sampled search space vs. optimal hyperparameter distributions. Empirical density (KDE) of hyperparameter values across workloads, comparing all sampled configurations (dashed line, light fill) to the top-10% best-performing trials (solid line, dark fill). Regions where the top-10% density is significantly more concentrated or peaked relative to the sample space indicate favorable hyperparameter settings for that workload.

Figure 16: Comparison of sampled search space vs. optimal hyperparameter distributions. Empirical density (KDE) of hyperparameter values across workloads, comparing all sampled configurations (dashed line, light fill) to the top-10% best-performing trials (solid line, dark fill). Regions where the top-10% density is significantly more concentrated or peaked relative to the sample space indicate favorable hyperparameter settings for that workload.

Figure 17: Comparison of sampled search space vs. optimal hyperparameter distributions. Empirical density (KDE) of hyperparameter values across workloads, comparing all sampled configurations (dashed line, light fill) to the top-10% best-performing trials (solid line, dark fill). Regions where the top-10% density is significantly more concentrated or peaked relative to the sample space indicate favorable hyperparameter settings for that workload.

Figure 18: Comparison of sampled search space vs. optimal hyperparameter distributions. Empirical density (KDE) of hyperparameter values across workloads, comparing all sampled configurations (dashed line, light fill) to the top-10% best-performing trials (solid line, dark fill). Regions where the top-10% density is significantly more concentrated or peaked relative to the sample space indicate favorable hyperparameter settings for that workload.

Figure 19: JAX vs. PyTorch median time per training step. Batch sizes and hyperparameters are held constant across frameworks.

Deep Learning Optimization

Deep learning optimization develops algorithms and training strategies that help neural networks converge, generalize, and use compute efficiently.

See all 204 papers

Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations
Apr 14, 2026Tong Zhang, Jiangning Zhang, Zhucun Xue +9Deep Learning Optimization Adaptive Gradient Methods
Zhejiang University, APRIL Lab · Fudan University · Shanghai Jiaotong University +1
Optimizer Memory Schedules for Outscaling the Overtraining Axis
Sep 4, 2026Katie Everett, Shikai QiuNeural Network Optimization Muon Optimizer
MIT CSAIL
EvE: An Alternate Optimizer to Adam
Sep 28, 2026Shashank Raj, Kalyanmoy DebDeep Learning Optimization Differential Evolution
Department of Computer Science The University of Alabama Tuscaloosa, AL 35487, USA

Apr 14, 2026J/K move · Enter open · S save

Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

Tong Zhang, Jiangning Zhang, Zhucun Xue, Juntao Jiang+8

Zhejiang University, APRIL Lab · Fudan University · Shanghai Jiaotong University+1

Deep Learning Optimization Adaptive Gradient Methods Stochastic Gradient Descent Second-Order Optimization Zeroth-Order Optimization

Open paper AskPDF arXiv

Balancing convergence speed, generalization capability, and computational efficiency remains a core challenge in deep learning optimization. First-order gradient descent methods, epitomized by stochastic gradient descent (SGD) and Adam, serve as the cornerstone of modern training pipelines. However, large-scale model training, stringent differential privacy requirements, and distributed learning paradigms expose critical limitations in these conventional approaches regarding privacy protection and memory efficiency. To mitigate these bottlenecks, researchers explore second-order optimization techniques to surpass first-order performance ceilings, while zeroth-order methods reemerge to alleviate memory constraints inherent to large-scale training. Despite this proliferation of methodologies, the field lacks a cohesive framework that unifies underlying principles and delineates application scenarios for these disparate approaches. In this work, we retrospectively analyze the evolutionary trajectory of deep learning optimization algorithms and present a comprehensive empirical evaluation of mainstream optimizers across diverse model architectures and training scenarios. We distill key emerging trends and fundamental design trade-offs, pinpointing promising directions for future research. By synthesizing theoretical insights with extensive empirical evidence, we provide actionable guidance for designing next-generation highly efficient, robust, and trustworthy optimization methods. The code is available at https://github.com/APRIL-AIGC/Awesome-Optimizer.

Abstract

Deep Learning Optimization

Explore similar work

Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

Optimizer Memory Schedules for Outscaling the Overtraining Axis

EvE: An Alternate Optimizer to Adam