Optimizer Memory Schedules for Outscaling the Overtraining Axis

Katie Everett; Shikai Qiu

cs.LGSep 4, 2026

Optimizer Memory Schedules for Outscaling the Overtraining Axis

Authors: Katie Everett, Shikai Qiu

Organizations: MIT CSAIL

Abstract

We investigate how optimizers scale across the overtraining axis and show that relative optimizer performance and optimal hyperparameters change substantially with training horizon. In particular, we study how matrix-preconditioned methods (Muon and SOAP) and a momentum-scheduled method (ADANA) scale relative to AdamW. We compare these four optimizers across models from 51M to 253M parameters and overtraining (OT) factors from 1x to 256x, sweeping the base learning rate at every setting. The preferred learning rate schedule can reverse across the overtraining axis, the best weight decay coefficient scales approximately as sqrt(OT), and longer horizons generally favor longer fixed memory. ADANA's scaling advantage over AdamW persists after tuning AdamW's fixed memory separately at each horizon. Log-time weight decay and momentum cooldown provide substantial gains for ADANA that compound as training increases. With this treatment, ADANA outscales AdamW with an exponent advantage close to that predicted by DANA theory on power-law random features. Muon and SOAP instead provide roughly constant token-efficiency advantages over AdamW across most of the measured range, although SOAP may gain further at the highest overtraining factors. ADANA begins behind both matrix-preconditioned optimizers but closes the gaps as training increases, surpassing Muon and becoming competitive with SOAP at our highest OT factors. These results establish training horizon as an essential axis for optimizer evaluation and design.

Figures & tables

Figure 1: Optimizer rankings exhibit crossovers as training extends beyond the Chinchilla-optimal horizon. We report the lowest final validation loss after independently sweeping the base learning rate for AdamW, ADANA, Muon, and SOAP on 51M, 124M, and 253M models under uniform weight decay. Muon and SOAP lead at short horizons, whereas ADANA closes its initial gap as the OT factor increases, and SOAP overtakes Muon at the longest measured horizons for the 51M and 124M models. Solid curves are fitted using our shared-

m

variant of the Skaling functional form ( Videau et al., 2026 ) ; see Appendix G .

Figure 2: Across model sizes under uniform weight decay, ADANA improves relative to AdamW as the OT factor increases, whereas Muon and SOAP already have an advantage at short horizons. Muon’s advantage remains roughly constant across OT factors, while SOAP may improve further at the highest OT factors. Top: the vertical axis shows the OT factor AdamW requires to match the indicated optimizer’s loss at fixed model size; values above the equal-OT diagonal favor the indicated optimizer. Bottom: the AdamW-relative token multiplier is the equivalent AdamW OT factor divided by the optimizer’s OT factor, or equivalently the ratio of tokens required by AdamW to those used by the optimizer. Curves compare the fitted optimizer and AdamW losses, while markers invert measured optimizer losses through the fitted AdamW curve. Dotted curve segments and translucent marker fills identify target losses below AdamW’s best measured loss and therefore require extrapolating the AdamW fit.

Figure 3: Longer horizons favor longer optimizer memory, and longer memory requires a smaller learning rate. All panels use the 51M model. Left: the optimal fixed effective memory for AdamW, Muon, and SOAP after independently sweeping the base learning rate at every memory and OT factor. Middle: at

8\times

OT, the optimal base learning rate decreases as effective memory increases; Figure 17 shows the corresponding relationship at every measured OT factor. Right: the within-optimizer validation-loss improvement from replacing fixed

M=50

with the optimal

(\text{memory},\text{base learning rate})

pair at each horizon; positive values favor per-horizon tuning.

Figure 4: After tuning fixed memory separately at every horizon, ADANA still gains on AdamW across OT, while Muon and SOAP retain roughly constant token-efficiency advantages over AdamW. On the 51M model under uniform weight decay, AdamW, Muon, and SOAP select the optimal

(\text{fixed memory},\text{base learning rate})

pair at every OT factor. ADANA uses the fixed scheduled-memory setting

(\kappa,\delta,g_{3})=(0.85,8,8)

and sweeps only the base learning rate. The vertical axis gives the horizon-tuned AdamW OT factor required to attain the same validation loss. Due to the small number of points, we compute equivalent OT by piecewise-linear interpolation of horizon-tuned AdamW loss in log-OT; translucent markers require extrapolation beyond the measured horizon-tuned AdamW loss range.

Figure 5: Log-time weight decay increasingly benefits ADANA, modestly benefits SOAP, has little consistent effect on AdamW, and harms Muon at high OT factors. On the 51M-parameter model, we compare uniform weight decay,

c(f)=8\sqrt{f}

, with log-time weight decay,

a(f)=2\sqrt{f}

, independently sweeping the base learning rate at every optimizer, weight decay schedule, and OT factor. Left: the token multiplier of log-time weight decay relative to the same optimizer under uniform weight decay. Right: AdamW-relative token multipliers for both weight decay schedules, using AdamW with uniform weight decay as the common baseline.

Figure 6: Momentum cooldown selectively benefits ADANA’s log-time memory and combines with log-time weight decay to amplify ADANA’s advantage as the overtraining factor increases, consistently across model sizes. All panels use AdamW with uniform weight decay at the same model size as the token-multiplier baseline. Left: on the 51M model under uniform weight decay, momentum cooldown has little consistent effect on fixed-memory AdamW but substantially improves ADANA at long horizons. Middle: log-time weight decay and momentum cooldown each improve ADANA; combined, they yield the largest gains at long horizons. Right: across all three model sizes, the combined treatment outperforms ADANA with uniform weight decay. The resulting ADANA token multiplier follows an approximately straight line on log–log axes. We therefore use ADANA with log-time weight decay and momentum cooldown in the subsequent comparisons of the strongest treatments.

Figure 7: With log-time weight decay and momentum cooldown, ADANA outscales AdamW with an equivalent OT slope close to the DANA PLRF prediction

2-\kappa

. Left and middle: the AdamW OT factor required to match ADANA’s loss at fixed model size, using AdamW with uniform weight decay and log-time weight decay as baselines, respectively. Across model sizes and both baselines, the fitted log–log slopes range from

1.15

1.20

, close to the PLRF prediction

2-\kappa=1.15

for

\kappa=0.85

. Dotted segments and translucent markers require extrapolating the AdamW fit. Right: relative to AdamW with log-time weight decay, we plot the ADANA token-decay exponent difference implied by each fixed high-token-limit difference under the shared-

m

Skaling fit. The exponent difference remains above zero even where the high-token limit favors AdamW, providing evidence for an improved ADANA token-decay exponent while leaving the high-token-limit difference less well identified.

Figure 8: ADANA versus matrix-preconditioned optimizers under their strongest treatments: ADANA closes the gap to Muon and SOAP as the overtraining factor increases. AdamW and Muon use uniform weight decay, SOAP uses log-time weight decay, and ADANA uses log-time weight decay with momentum cooldown. Left: Muon and SOAP provide roughly constant token-multiplier advantages over AdamW across most of the overtraining axis, although SOAP may gain further at the highest OT factors. This suggests that matrix preconditioning may provide an approximately constant efficiency multiplier over AdamW. ADANA begins behind both matrix-preconditioned optimizers, but has a larger fitted token-decay exponent and closes the gap as the training horizon increases, catching Muon and becoming competitive with SOAP. Dotted segments and translucent markers indicate baseline extrapolation.

Appendix figures & tables49 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Definition
$H$	Number of attention heads
$D$	Model dimension
$D_{\mathrm{head}}$	Dimension of each attention head
$N$	Number of Transformer blocks
$F$	Feed-forward intermediate dimension
$L$	Sequence length in tokens

Appendix

Table 1: Notation used for the architecture, training budget, and nominal compute.

Component	Choice
Block normalization	Pre-RMSNorm; learned scale, no bias; $\epsilon=10^{-6}$
Attention	Multi-head attention with $D_{\mathrm{head}}=64$
Q/K normalization	QK-RMSNorm before RoPE; learned scale, no bias; $\epsilon=10^{-6}$
Positional encoding	RoPE with base $10{,}000$
Activation	SwiGLU
Tokenizer	GPT-2 with vocabulary padded to $V=50{,}304$

Appendix

Table 2: Model architecture.

Model	$D$	$N$	$F$	$P_{\rm nonemb}$	$P$	$P_{\rm train}$	$S_{1\times}$
51M	512	6	2,048	25,173,248	50,921,472	76,684,544	1,942
124M	768	9	3,072	84,950,400	123,568,128	162,217,344	4,713
253M	1,024	12	4,096	201,353,728	252,837,888	304,376,320	9,645
458M	1,280	15	5,120	393,257,600	457,605,120	522,035,840	17,456
757M	1,536	18	6,144	679,536,384	756,744,192	834,070,272	28,867
1.17B	1,792	21	7,168	1,079,064,448	1,169,129,472	1,259,353,984	44,598

Appendix

Table 3: Model sizes. Model names use rounded nominal parameter counts

P=16ND^{2}+DV

. Non-embedding and exact trainable parameter counts include learned normalization scales.

Parameter	Standard deviation
Token embedding	$0.02$
Query, key, and value projections	$D^{-1/2}$
Attention output projection	$(2ND)^{-1/2}$
Feed-forward input projection	$D^{-1/2}$
SwiGLU gate projection	$D^{-1/2}$
Feed-forward output projection	$(2NF)^{-1/2}$

Appendix

Table 4: Initialization standard deviations.

Control	Default
Sequence length	$L=2{,}048$ tokens
Global batch size	$B=256$ sequences
Gradient clipping	Global $\ell_{2}$ norm of $1.0$ , applied after gradient accumulation
Learning rate warmup	Linear from $0.01\eta$ to $\eta$ over $W=\max\{100,\min\{\lfloor S/50\rfloor,5{,}000\}\}$ updates

Appendix

Table 5: Common training controls.

Parameter class	AdamW	Muon	SOAP	ADANA	Learning rate group	Weight decay
Token embedding	Adam	Adam	SOAP	ADANA	$\gamma_{e}$	Yes
Attention and feed-forward matrices	Adam	Muon	SOAP	ADANA	$\gamma_{h}$	Yes
Readout matrix	Adam	Adam	SOAP	ADANA	$\gamma_{r}$	Yes
Learned normalization scales	Adam	Adam	Adam	ADANA	$\gamma_{e}$	No

Appendix

Table 6: Parameter routing for the four primary optimizers. The weight decay column gives the common independent weight decay mask.

Figure 9: Linear and cosine learning rate decay schedules after warmup. Both schedules use the same linear warmup before decaying from the peak learning rate to zero.

Figure 10: Cosine decay vs linear decay learning rate schedule advantage across OT factor and coefficient of the uniform WD schedule on the 51M parameter model. Each cell reports $10^{3}(L_{\mathrm{linear}}-L_{\mathrm{cosine}})$ after selecting the best base learning rate separately for each schedule at the indicated weight decay coefficient; positive values favor cosine decay to zero. The row labeled “best” selects the best weight decay coefficient for each OT factor separately for each schedule. Momentum hyperparameters are fixed to AdamW $(\beta_{1}=0.9,\beta_{2}=0.95)$ , ADANA $(g_{3}=8,\kappa=0.85)$ , Muon $(\beta=0.95)$ , and SOAP $(\beta_{1}=0.95,\beta_{2}=0.99,\beta_{\mathrm{Sh}}=0.95)$ .

Figure 11: High overtraining factors favor cosine decay over linear decay on the 51M parameter model for all optimizers. Each point reports

10^{3}(L_{\mathrm{linear}}-L_{\mathrm{cosine}})

after selecting the best base learning rate and weight decay coefficient pair for each schedule; positive values favor cosine decay to zero. Momentum hyperparameters are fixed as in Figure 10 .

Figure 12: The apparent advantage of the log-time WD schedule is substantially diminished when the learning rate decays to zero rather than to 10% of its peak. We compare the uniform and log-time WD schedules with the same base learning rate and matched total parameter shrinkage from weight decay alone on the 51M parameter model at

8\times

OT. Markers show validation evaluations at selected training steps, and lines linearly interpolate between them. With a 10% endpoint, the log-time WD schedule reduces final validation loss by 32.3 millipoints for AdamW and 16.9 millipoints for ADANA; with a zero endpoint, the corresponding differences are 1.3 and 0.3 millipoints.

Figure 13: The relative performance of the uniform and log-time WD schedules depends on both the optimizer and OT factor. On the 51M parameter model with cosine decay to zero and fixed optimizer-specific momentum, each point shows

10^{3}(L_{\mathrm{log}}-L_{\mathrm{uniform}})

after selecting the best

(\text{base learning rate},\text{WD coefficient})

pair for each WD schedule. Positive values favor the uniform WD schedule, while negative values favor the log-time WD schedule.

Figure 14: The best WD coefficient changes with OT factor. Each curve shows coefficient regret at one OT factor for a fixed optimizer and WD schedule on the 51M parameter model. The regret of coefficient

c

10^{3}\!\left[L(c)-\min_{c^{\prime}}L(c^{\prime})\right]

, where

L(c)

is the best final validation loss over the base learning rate sweep at coefficient

c

. The movement of the minima across curves shows that a coefficient selected at one training horizon need not transfer to another. The learning rate schedule uses cosine decay to zero and optimizer-specific momentum is held fixed. Upward triangles at the upper boundary represent regrets greater than

25

millipoints.

Figure 15: The best WD coefficients are consistent with square-root scaling across OT factor. For each optimizer, WD schedule, and OT factor on the 51M parameter model, points show the WD coefficient from the best

(\text{base learning rate},\text{WD coefficient})

pair in the two-dimensional grid. The solid black curves show the separate power-law fits in Equation 53 , while the dashed curves show the adopted square-root rules. The learning rate schedule uses cosine decay to zero and optimizer-specific momentum is held fixed.

Figure 16: The low-regret region shifts toward longer fixed memory as the training horizon increases. For each optimizer, effective memory, and OT factor, we independently sweep the base learning rate and report regret relative to the best memory for that optimizer and horizon. Black outlines mark the minimizing memory. Diagonal slashes indicate points whose learning rate sweeps were incomplete.

Figure 17: The best base learning rate generally decreases as fixed memory grows, with optimizer- and horizon-dependent deviations. Each panel fixes the OT factor and compares AdamW, Muon, and SOAP across effective memory. At each memory, the plotted value is the base learning rate attaining the lowest final validation loss in an independent learning rate sweep. Open circles indicate points whose learning rate sweeps were incomplete.

Figure 18: Per-horizon memory tuning improves fixed-memory optimizers. For the 51M model, circles show each optimizer with fixed effective memory

M=50

, after independently sweeping the base learning rate at every OT factor. Triangles show the lowest measured loss after jointly sweeping fixed memory and base learning rate at each horizon. We convert these loss values to token multipliers relative to AdamW with

M=50

, corresponding to

\beta_{2}=0.98

, by inverting the fixed-

M=50

AdamW curve from the shared-

m

Skaling fit. Because scaling fits are not well identified from this small set of horizon-tuned points, we connect adjacent points with piecewise-linear segments rather than fitted curves. Per-horizon memory tuning provides modest gains for AdamW but substantially strengthens Muon and SOAP at the longest measured horizons.

Optimizer	Prescribed optimizer-specific settings
AdamW	$\beta_{1}=0.9,\ \beta_{2}=0.98$
Muon	$\beta=0.98$ ; Adam route $\beta_{1}=0.9,\ \beta_{2}=0.95$
SOAP	$\beta_{1}=0.95,\ \beta_{2}=0.98,\ \beta_{\mathrm{Sh}}=0.95,\ K=10$
ADANA	$g_{2}=1,\ g_{3}=8,\ \kappa=0.85,\ \delta=8$

Appendix

Table 7: Optimizer-specific settings used in the primary comparison. The settings are held fixed across model sizes and OT factors; ADANA’s memory changes within training according to its prescribed log-time rule. The base learning rate is swept independently at every coordinate.

Functional form	Parameters	RMSE (mpt)	AICc	BIC
Chinchilla additive	17	14.349	-600.54	-571.47
Parameter–token interaction	21	14.850	-580.77	-548.94
Additive OT, shared model exponent	18	3.272	-821.82	-791.86
Additive OT, optimizer-specific model exponents	21	3.113	-818.24	-786.41
Skaling, optimizer-specific model exponents	21	3.079	-819.90	-788.07
Shared- $m$ Skaling	18	3.251	-822.78	-792.83

Appendix

Table 8: In-sample fit and information criteria for the candidate scaling forms on the 76 uniform weight decay loss values.

Figure 19: The separable Chinchilla model and the parameter–token interaction model leave systematic residuals. Residuals are fitted minus observed validation loss in millipoints. The Chinchilla-additive and parameter–token interaction models retain a pronounced U-shaped structure, whereas the five additive-OT and Skaling candidates remain substantially closer to zero across the observed loss range.

Functional form	Hold out OT	Hold out longest	Hold out two longest	Joint frontier	Hold out model
Additive OT, shared model exponent	5.36	5.89	6.00	6.92	36.30
Additive OT, optimizer-specific model exponents	5.64	6.06	5.99	7.65	13.98
Skaling, optimizer-specific model exponents	5.94	5.99	5.49	7.70	184.49
Shared- $m$ Skaling	5.73	5.86	5.53	6.95	59.45
Skaling, shared inner exponent	5.97	6.11	5.61	7.32	59.68

Appendix

Table 9: Pooled RMSE in millipoints for the structured holdout tests for the five better-fitting models.

				Additive OT		Skaling
Test	Held-out group	Train $n$	Held-out $n$	Shared $\alpha$	Optimizer-specific $\alpha_{o}$	Optimizer-specific $m_{o}$	Shared $m$	Shared $\alpha$
Hold out OT	$1\times$	64	12	9.49	10.35	11.37	10.75	11.32
	$2\times$	64	12	3.51	3.69	4.12	3.97	4.21
	$4\times$	64	12	5.37	5.46	5.09	5.02	4.99
	$8\times$	64	12	3.80	3.51	3.45	3.82	4.03
	$16\times$	68	8	2.99	2.76	2.60	2.80	2.96

Appendix

Table 10: Per-fold support and RMSE in millipoints for each structured holdout test for the five better-fitting models.

Figure 20: Residuals of the selected shared-

m

Skaling fit remain small across model sizes and OT values. Residuals are fitted minus observed validation loss in millipoints, shown separately for the 51M, 124M, and 253M models. Lines connect adjacent OT factors for the same optimizer. The overall RMSE is

3.251

millipoints.

Figure 21: Uniform-WD optimizers exhibit distinct outscaling behavior relative to AdamW. Columns compare ADANA, Muon, and SOAP with AdamW. The top row plots the AdamW OT factor required to match each optimizer at the OT factor on the horizontal axis; values above the equal-OT diagonal favor the optimizer named first. The middle row reports the corresponding AdamW-relative token multiplier. Curves are fit-to-fit comparisons under the shared-

m

Skaling fit, while markers invert measured loss values through the fitted AdamW curve. Dotted curve segments and translucent marker fills require extrapolating AdamW beyond its largest measured token budget at that model size. The bottom row profiles the tradeoff between the 51M high-token limit difference and token-decay exponent difference under four functional forms; symbols mark the unconstrained optima and the displayed curves satisfy the 95% profile threshold. Positive values on either profile axis favor the optimizer named first. ADANA’s token multiplier rises strongly with OT, Muon’s is comparatively flat, and SOAP’s turns upward at the longest horizons.

Figure 22: Rankings among the non-Adam uniform-WD treatments also change with training horizon. Columns compare SOAP with Muon, ADANA with Muon, and ADANA with SOAP. The top, middle, and bottom rows respectively show equivalent OT, baseline-relative token multipliers, and floor–decay profiles using the conventions of Figure 21 . SOAP overtakes Muon only at the longest measured horizons, while ADANA gains on both matrix-preconditioned treatments and reaches or exceeds them at some of the highest measured model–horizon settings. The profiles show that these ranking changes need not have the same asymptotic explanation: SOAP’s late advantage over Muon is compatible with a lower fitted high-token limit despite a worse token-decay exponent, whereas ADANA’s comparisons retain a tradeoff between the two quantities.

Figure 23: Log-time WD and momentum cooldown make ADANA increasingly competitive with the strongest measured versions of the three other optimizers. Columns compare ADANA using log-time WD and momentum cooldown with AdamW using log-time WD, Muon using uniform WD, and SOAP using log-time WD. The top, middle, and bottom rows respectively show equivalent OT, baseline-relative token multipliers, and floor–decay profiles using the same conventions as Figure 21 . Strong ADANA’s relative token efficiency increases with OT in all three comparisons. Across the profiled functional forms, the comparisons favor an improved ADANA token-decay exponent, while the corresponding high-token limit differences remain less well identified. Open circles indicate points whose learning rate sweeps were incomplete; the fits use only the filled points.

Figure 24: With log-time WD and momentum cooldown, ADANA outscaling is close to the PLRF prediction

2-\kappa

across model sizes. We plot equivalent OT scaling across model sizes: for each ADANA OT factor, the vertical axis gives the AdamW OT factor required to attain the same fitted validation loss. Lines compare uniform-WD ADANA with ADANA using log-time WD and momentum cooldown; markers instead invert measured ADANA loss values through the fitted AdamW curve. Translucent markers and dotted curve segments require extrapolating AdamW beyond its largest measured token budget. The displayed fit values are log–log slopes over each curve’s plotted range. The dashed reference has slope

1

, while the dotted reference has the PLRF-predicted slope

2-\kappa=1.15

; both pass through

(1,1)

. The combined-treatment slopes are

1.20

1.16

, and

1.15

for the 51M, 124M, and 253M models, respectively, whereas the uniform-WD slopes are

1.15

1.06

, and

1.03

Figure 25: At fixed token count, ADANA loses efficiency at smaller batch sizes than AdamW in the measured

8\times

-OT slice. We train the 51M parameter model for

8\times

OT while holding the total number of training tokens fixed and varying the global batch size from 32 to 1,024. At each batch size, we jointly sweep the base learning rate and either AdamW

\beta_{2}

or ADANA

g_{3}

; AdamW

\beta_{1}=0.9

and ADANA

\kappa=0.85,\delta=8

remain fixed. Left: the best final validation loss relative to

B=32

. Middle: the selected AdamW second-moment memory

M_{2}=1/(1-\beta_{2})

; the

M_{2}\propto 1/B

guide corresponds to approximately constant token memory

BM_{2}

, consistent with Marek et al. (2025) . Right: the selected ADANA

g_{3}

. The lower panels show the base learning rate selected jointly with each optimizer hyperparameter. This fixed-token slice does not by itself establish a general batch scaling law for ADANA.

Figure 26: AdamW and LaProp remain close at matched

\beta_{1}

. We compare

\beta_{1}\in\{.90,.95\}

at fixed

\beta_{2}=.98

, independently sweeping the base learning rate for every optimizer–

\beta_{1}

–OT coordinate. Left: best validation loss. Right: token multiplier relative to AdamW with

\beta_{1}=.90

and

\beta_{2}=.98

Figure 27: Preconditioner order under uniform weight decay across model sizes. The top row reports final validation loss for ADANA and LADANA with log-time denominator memory. The bottom row converts the same loss values to token multipliers relative to uniform-WD AdamW at each model size. The 51M panels additionally show the fixed-

v

LADANA ablation. Open circles indicate points whose learning rate sweeps were incomplete.

Figure 28: Preconditioner order under log-time weight decay. Final validation loss (left) and token multipliers relative to uniform-WD AdamW (right) for ADANA and LADANA with log-time denominator memory on the 51M model. Both treatments use the log-time weight-decay coefficient rule from Appendix E.4 .

Figure 29: Mechanics of momentum cooldown. We illustrate the exact responsive-memory rule used in our experiments for a representative $h8$ ADANA run at $128\times$ OT with cosine learning rate decay. Left: log-time momentum normally uses an increasingly long timescale. We define the learning rate decay timescale as $\eta_{t}/|\eta_{t+1}-\eta_{t}|$ and set the cooled momentum timescale to the minimum of this quantity and the original log-time momentum timescale. Right: the resulting median and 90th-percentile ages of the momentum kernel, defined as the shortest recent history containing 50% and 90% of its mass. Cooldown leaves the kernel unchanged earlier in training but makes it increasingly concentrated on recent gradients during terminal decay.

Optimizer	Cooldown target	Best base LR	Validation loss	Gain vs. standard (mpt)
AdamW	None	$2^{-7.0}$	3.430405	0.00
AdamW	First moment	$2^{-7.0}$	3.428106	$+2.30$
AdamW	Second moment	–	Unstable	–
AdamW	First and second moments	$2^{-7.0}$	3.428112	$+2.29$
ADANA	None	$2^{-12.5}$	3.432476	0.00
ADANA	Memory window	$2^{-12.5}$	3.421004	$+11.47$

Appendix

Table 11: Momentum cooldown treatment selection at

8\times

OT. Each reported minimum comes from an independent base learning rate sweep, and gains are measured relative to the same optimizer without cooldown.

Optimizer	Fixed momentum treatment	$c$ at $1\times$	$c$ at $8\times$	$c$ at $32\times$
AdamW	$\beta_{1}=0.9,\ \beta_{2}=0.95$	$\{2,4,8,16\}$	$\{4,8,16,32\}$	$\{16,32,64,128\}$
ADANA	$g_{3}=8,\ \kappa=0.85$	$\{4,8,16\}$	$\{8,16,32,64\}$	$\{16,32,64,128\}$
Muon	$\beta=0.95$	$\{2,4,8,16\}$	$\{8,16,32,64\}$	$\{16,32,64,128\}$
SOAP	$\beta_{1}=0.95,\ \beta_{2}=0.99,\ \beta_{\mathrm{Sh}}=0.95$	$\{2,4,8,16\}$	$\{4,8,16,32,64\}$	$\{16,32,64,128\}$

Appendix

Table 12: Fixed momentum treatments and coefficients of the uniform WD schedule for the linear-to-zero and cosine-to-zero comparison on the 51M parameter model. At every listed coefficient, each schedule receives a separate base learning rate sweep.

Figure 30: Base learning rate sweeps at each uniform WD coefficient on the 51M parameter model with linear decay to zero. Each colored curve holds the WD coefficient fixed while varying the base learning rate; stars mark the minimum final validation loss within each sweep. Momentum hyperparameters are AdamW

(\beta_{1}=0.9,\beta_{2}=0.95)

, ADANA

(g_{3}=8,\kappa=0.85)

, Muon

(\beta=0.95)

, and SOAP

(\beta_{1}=0.95,\beta_{2}=0.99,\beta_{\mathrm{Sh}}=0.95)

. All 48 displayed sweeps satisfy the one-millipoint interiority criterion.

Figure 31: Base learning rate sweeps at each uniform WD coefficient on the 51M parameter model with cosine decay to zero. Each colored curve holds the WD coefficient fixed while varying the base learning rate; stars mark the minimum final validation loss within each sweep. Solid lines with filled markers denote sweeps that satisfy the one-millipoint interiority criterion. Momentum hyperparameters are AdamW

(\beta_{1}=0.9,\beta_{2}=0.95)

, ADANA

(g_{3}=8,\kappa=0.85)

, Muon

(\beta=0.95)

, and SOAP

(\beta_{1}=0.95,\beta_{2}=0.99,\beta_{\mathrm{Sh}}=0.95)

. These sweeps provide the uniform WD grid used in the learning rate schedule comparison. All 48 displayed sweeps satisfy the one-millipoint interiority criterion.

Uniform WD
		WD coefficient $c$
Optimizer	Fixed momentum treatment	$1\times$ OT	$8\times$ OT	$32\times$ OT	$64\times$ OT
AdamW	$\beta_{1}=0.9,\ \beta_{2}=0.98$	$4,8,16$	$8,16,32$	$16,32,64,128$	$16,32,64,128$
ADANA	$g_{3}=8,\ \kappa=0.85$	$4,8,16$	$8,16,32,64$	$16,32,64,128$	$16,64,128,256$
Muon	$\beta=0.98$	$4,8,16$	$8,16,32,64$	$16,32,64,128$	$16,32,64,128,256$
SOAP	$\beta_{1}=0.95,\ \beta_{2}=0.98,\ \beta_{\mathrm{Sh}}=0.95$	$4,8,16$	$8,16,16\sqrt{2},32$	$16,32,32\sqrt{2},64$	$8,32,64,128$

Appendix

Table 13: WD coefficient grids for the uniform and log-time WD sweep figures. Each OT cell lists the coefficients

c

included in the corresponding base learning rate sweeps.

Figure 32: Base learning rate sweeps at each uniform WD coefficient on the 51M parameter model with cosine decay to zero. Columns are optimizers and rows are OT factors. Each colored curve holds

c

fixed; circles are measured learning rates and black-edged stars mark the minimum within each sweep. All 60 sweeps satisfy the interiority criterion in Section B.5 . Facets use independent vertical ranges to expose the local learning rate basin; upward triangles retain observed losses above a facet’s displayed range.

Figure 33: Base learning rate sweeps at each log-time WD coefficient on the 51M parameter model with cosine decay to zero. Columns are optimizers and rows are OT factors. Each colored curve holds

c

fixed; circles are measured learning rates and black-edged stars mark the minimum within each sweep. We use the

1\times

-anchored offset

\tau=0.1S_{1\times}

. All 77 sweeps satisfy the interiority criterion in Section B.5 . Facets use independent vertical ranges, and upward triangles retain observed losses above a facet’s displayed range.

Figure 34: Complete uniform-WD learning rate sweeps across model sizes and OT factors. Columns are optimizers and rows are model sizes; colors identify OT factor. The treatment uses cosine decay to zero and uniform WD with

c(f)=8\sqrt{f}

. Circles are measured learning rates and black-edged stars mark the selected minimum within each sweep. All 76 displayed sweeps satisfy the one-millipoint interiority criterion.

Figure 35: Complete log-time-WD learning rate sweeps across model sizes and OT factors. Columns are optimizers and rows are model sizes; colors identify OT factor. The treatment uses cosine decay to zero and the

1\times

-anchored log-time WD coefficient

a(f)=2\sqrt{f}

. Circles are measured learning rates and black-edged stars mark the selected minimum within each sweep. All 72 displayed sweeps satisfy the one-millipoint interiority criterion.

Figure 36: Complete fixed-memory learning rate sweeps from

1\times

through

8\times

OT. Columns are optimizers, rows are OT factors, and colors identify effective memory. Black-edged stars mark the minimum of each learning rate sweep. Solid curves with filled markers denote sweeps satisfying the one-millipoint interiority criterion; dashed curves with hollow markers denote sweeps whose within-one-millipoint set reaches a tested boundary.

Figure 37: Complete fixed-memory learning rate sweeps from

16\times

through

128\times

OT. The layout and visual encoding match Figure 36 . Open circles indicate points whose learning-rate sweeps were incomplete. The sweep for SOAP at

128\times

OT with

M=2560

is incomplete.

Figure 38: Complete learning rate sweeps for the fixed-memory preconditioner-order comparison. Panels compare AdamW and LaProp at

\beta_{1}\in\{.90,.95\}

, fixed

\beta_{2}=.98

, and OT factors

1

through

64

. Each curve is an independent base learning rate sweep. All 28 displayed sweeps satisfy the one-millipoint interiority criterion.

Figure 39: Uniform-WD learning rate sweeps for preconditioner order across model sizes. Rows are the 51M, 124M, and 253M models; columns are ADANA, LADANA with log-time denominator memory, and the fixed-

v

LADANA ablation (available only for the 51M model). Thirty-seven of 39 displayed sweeps satisfy the one-millipoint interiority criterion. Open circles indicate points whose learning-rate sweeps were incomplete. The incomplete sweeps are log-

v

LADANA at 51M/

128\times

and 124M/

32\times

. ADANA is retained once as the visual reference for the two preconditioner-order treatments.

Figure 40: Log-time-WD learning rate sweeps for h8 preconditioner order. Panels compare ADANA and LADANA with log-time denominator memory across OT factors

1

through

128

. Open circles indicate points whose learning rate sweeps were incomplete.

Figure 41: Complete uniform-WD learning rate sweeps for the momentum cooldown interventions. Panels show the momentum-cooldown variants of AdamW and ADANA across OT factors

1

through

128

. Corresponding standard-treatment sweeps appear in the primary-treatment and preconditioner-order sweeps and are not repeated here. All 16 displayed sweeps satisfy the one-millipoint interiority criterion.

Figure 42: Large-model uniform-WD learning rate sweeps for ADANA with and without momentum cooldown. Rows are the 124M and 253M parameter models, columns are the standard and momentum-cooldown treatments, and colors identify OT factor.

Figure 43: Complete log-time-WD learning rate sweeps for the momentum cooldown intervention. The panel shows ADANA with momentum cooldown across OT factors

1

through

128

; the corresponding standard ADANA sweeps appear in the log-time-WD preconditioner sweep grid and are not repeated here. All eight displayed sweeps satisfy the one-millipoint interiority criterion.

Figure 44: Learning rate sweeps for cooldown-component selection at

8\times

OT. Each panel overlays the independently swept component treatments for one optimizer. AdamW first-moment and both-moment cooldown are nearly coincident. The three second-moment-only AdamW observations have losses from 15.2 to 42.9 and are shown as capped upward triangles; they are instability outcomes rather than ordinary open minima. All other displayed selected minima satisfy the one-millipoint learning rate interiority rule.

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 321 papers

May 11, 2026

Optimistic Dual Averaging Unifies Modern Optimizers

We introduce SODA, a generalization of Optimistic Dual Averaging, which provides a common perspective on state-of-the-art optimizers like Muon, Lion, AdEMAMix and NAdam, showing that they can all be viewed as optimistic instances of this framework. Based on this framing, we propose a practical SODA wrapper for any base optimizer that eliminates weight decay tuning through a theoretically-grounded

1/k

decay schedule. Empirical results across various scales and training horizons show that SODA consistently improves performance without any additional hyperparameter tuning.

Thomas Pethick, Wanyun Xie, Roman Machacek +1

Independent Researcher · EPFL (LIONS) · University of Bern

Stochastic Optimization Deep Learning Optimization

Ask PDF

Aug 28, 2026

Blog: Survey of Optimizers

Neural-network optimization in 2025-2026 is no longer well described as a succession of new Adam variants. The design space has expanded from coordinates to matrices and layers, from fixed training horizons to policies over time, and from mathematical update rules to state representations that must survive sharding and low-precision computation. This survey organizes recent optimizers and training optimization methods along four largely independent axes: temporal estimation, update geometry, horizon management, and representation and systems. It connects the spectral normalization of Muon, the historical matrix statistics of Shampoo and SOAP, adaptive and hybrid matrix methods, memory-efficient optimizers, schedule-free training, small-batch corrections, and quantized optimizer states. The central empirical conclusion is deliberately non-triumphal: matrix-aware methods represent a genuine advance, but there is no context-independent replacement for AdamW. Rankings change with model scale, data-to-parameter ratio, batch size, schedule, parameter partition, tuning budget, and whether the target metric is tokens, FLOPs, wall-clock time, or memory. The practical consequence is a compositional view of optimizer design and a stricter protocol for evaluating optimizer claims.

Ruoran Xu

Neural Network Optimization Memory-Efficient Optimization

Ask PDF

May 21, 2026

Anytime Training with Schedule-Free Spectral Optimization

Standard neural network training relies on learning-rate schedules tied to a fixed horizon, leading to strong path dependence and costly re-tuning as data availability changes. Schedule-Free (SF) methods address this by removing explicit schedules, yet SF-AdamW, the current state-of-the-art anytime optimizer, consistently underperforms well-tuned AdamW baselines. We propose SF-NorMuon, a schedule-free spectral optimizer that closes this gap: with a single hyperparameter configuration, SF-NorMuon matches or exceeds tuned AdamW on 125M and 772M parameter language models across

1

8\times

Chinchilla horizons. On the theoretical side, we prove a stationarity guarantee for schedule-free spectral dynamics and identify weight decay at the fast iterate as essential for long-horizon stability. SF-NorMuon enables practitioners to obtain high-quality checkpoints at any point during training without committing to a horizon in advance. By closing the performance gap with tuned baselines, SF-NorMuon makes horizon-free optimization more practical, taking a step towards truly open-ended, continual learning.

Anuj Apte, Pranav Deshpande, Niraj Kumar +2

Global Technology Applied Research, JPMorganChase New York, NY 10001, USA

Deep Learning Optimization Learning Rate Scheduling Neural Network Training Dynamics Spectral Optimization

Ask PDF

Abstract

Neural Network Optimization

Explore similar work

Optimistic Dual Averaging Unifies Modern Optimizers

Blog: Survey of Optimizers

Anytime Training with Schedule-Free Spectral Optimization