Optimizer-dependent training dynamics converge to the same one-third optimal data scaling

Hyunseok Lee; Mihir Basil; Yizhou Liu; Jeff Gore

cs.LGSep 29, 2026

Optimizer-dependent training dynamics converge to the same one-third optimal data scaling

Authors: Hyunseok Lee, Mihir Basil, Yizhou Liu, Jeff Gore

Organizations: Department of Physics, Massachusetts Institute of Technology, Cambridge, MA · Department of Physics, University of Pennsylvania, Philadelphia, PA

Abstract

Neural scaling, in which loss falls as a power law with training, is central to large language models, and one recent proposal is that a $1/3$ exponent emerges from learning peaked distributions. That account describes SGD, but models in practice are trained with adaptive optimizers. Here we separate two exponents the $1/3$ account does not distinguish: how fast the loss falls with training steps along a single run, and how fast the optimally tuned loss falls with dataset size $D$ . We show that the first, a dynamic exponent, is optimizer-specific while the second, an optimal data exponent, converges to $1/3$ across optimizers. In an online teacher-student model we decompose the loss into norm growth (radial) and alignment toward the teacher direction (tangential), each decaying as a power law with dynamic exponents $α_{r}$ and $α_{t}$ . Under SGD, both are close to $1/3$ , so the data exponent is also $1/3$ across different learning rates. Under Adam the two separate: $α_{r} \simeq 0.48$ but $α_{t} \simeq 0.08$ . Since the total loss is minimized when these two parts are balanced, the optimal learning rate is optimizer-dependent: $D$ -independent for SGD but falls with $D$ for Adam. Yet tuned to that optimum, the loss returns to $D^{-1/3}$ for both. A stochastic-dynamics analysis explains why: the optimizers can trade decay speed between the two channels, but they all fall on a single dynamic exponent relation, $2α_{r}+ α_{t} = 1$ , which fixes the optimal data exponent at $1/3$ . Across seven optimizers, including Muon, the measured exponents are consistent with this relation, and the optimal-loss envelopes agree with $D^{-1/3}$ across them. The optimizer sets how fast a model learns per step; tuned optimally, it changes the prefactor but not the rate at which loss falls per sample.

Figures & tables

Figure 1: Optimizers produce qualitatively different dynamics on the same task. Online teacher–student softmax classification with

n=128

classes and input dimension

m=32

, trained with SGD (top row, batch size

B=256

) and Adam (bottom row, batch size

B=64

). For each optimizer

\eta

and

B

are chosen so that the tangential loss stays positive and the norm growth remains drift-dominated (Appendix G ); the exponents do not depend on batch size for either optimizer (Figure 7 ), so the comparison is not affected by the choice. Color denotes learning rate

\eta

; the two rows use different ranges, since the useful learning rates differ by roughly two decades between the optimizers. (a) Training dynamics of SGD follows

1/3

time scaling. Left: test loss

L

follows a single clean power law across the full range, with fitted exponent

0.34\pm 0.01

. Right: weight norm

\beta=\lVert W_{S}\rVert_{F}

is consistent with

\beta\sim t^{1/3}

(fitted

0.33\pm 0.01

). (b) Training dynamics of Adam deviates from

1/3

time scaling. Left: test loss curves bend over an extended transient and no single exponent describes the trajectory. Right: weight norm expands faster than

t^{1/3}

throughout (fitted

0.44\pm 0.04

Figure 2: The loss splits into a radial and a tangential component with distinct power-law dynamics and trade-offs between them. (a) Geometry of the decomposition. The student $W_{S}$ is compared to the rescaled teacher $W_{R}=\beta\widehat{W}_{T}$ , which carries the student’s current norm but the teacher’s direction. The radial loss $L_{r}=\mathrm{CE}(p,p_{R})$ measures scale mismatch and the tangential loss $L_{t}=L-L_{r}$ measures misalignment, which the restoring Hessian reduces and stochastic injection from mini-batch noise sustains. (b, c) $L_{r}$ and $L_{t}$ against training step, for SGD (top) and Adam (bottom), colored by learning rate over the same ranges as in Figure 1 . Both decay as power laws whose exponents are independent of the learning rate: $\alpha_{r}=0.35\pm 0.01$ and $\alpha_{t}=0.33\pm 0.02$ for SGD, $\alpha_{r}=0.48\pm 0.03$ and $\alpha_{t}=0.08\pm 0.01$ for Adam.

Figure 3: Optimal learning rate scales differently with $B$ and $D$ for SGD and Adam, but the optimal loss follows one-third data scaling for both. Learning-rate sweeps over $8$ batch sizes $B\in[16,2048]$ and $16$ learning rates, spanning $[10^{-3},10]$ for Adam and $[10^{-1},10^{3}]$ for SGD, with $D=Bt$ the number of online samples processed. In each panel the upper row is SGD and the lower row Adam. Slopes $s_{B}$ and $s_{D}$ quoted in (a) and (b) are from the joint log-log plane fit $\log_{10}\eta^{\ast}=s_{D}\log_{10}D+s_{B}\log_{10}B+c$ described in Appendix H . (a) $\eta^{\ast}$ against $D$ , one curve per batch size (grayscale). SGD’s optimum is nearly flat ( $s_{D}=-0.04\pm 0.01$ ), while Adam’s decreases ( $s_{D}=-0.35\pm 0.01$ ), close to $-1/3$ and to the recently reported “magic exponent” $0.32$ ( Bjorck et al., 2024 ) . (b) $\eta^{\ast}$ against batch size, one curve per data size (color). SGD follows $\eta^{\ast}\propto B$ ( $s_{B}=1.01\pm 0.01$ ), while Adam gives $s_{B}=0.76\pm 0.01$ , steeper than the square-root rule of common practice and than the value reported for LLM training ( Ren et al., 2026 ) . (c) Test loss along the optimal- $\eta$ envelope against $D$ , with individual runs colored by training step and shaded by batch size. Both optimizers give the same optimal data scaling, $L^{\ast}(D)\sim D^{-1/3}$ , with fitted exponents $\alpha_{D}=0.341\pm 0.001$ for SGD and $0.341\pm 0.003$ for Adam. The quoted exponent is the pooled estimate across batch sizes of Appendix H .4. The $\eta^{\ast}$ traces in (a) are visibly noisy at large $D$ while the envelope in (c) is clean; Figure 5 shows the underlying sweeps and the flatness of the basin.

Figure 4: A sum rule for training exponents holds across optimizers, and fixes the optimal data-scaling exponent at

1/3

. (a) Measured radial and tangential time-scaling exponents,

L_{r}\sim t^{-\alpha_{r}}

and

L_{t}\sim t^{-\alpha_{t}}

, for SGD, the PowerAdam family

A=V^{-a}

a\in\{0.125,0.25,0.375,0.5\ (\text{Adam})\}

, Muon, and SignGD. Crosses give fit uncertainties. The dashed line is

2\alpha_{r}+\alpha_{t}=1

, a parameter-free prediction with no fitted quantities; the black point marks the

a=0

prediction

(1/3,1/3)

. Adam sits at

\alpha_{r}\simeq 0.48

, above the nominal

2/5

, corresponding to an effective

q\simeq 0.9

; Appendix G rules out two alternative explanations. (b) Optimal data-scaling exponent

\alpha_{D}

, with

L^{\ast}(D)\sim D^{-\alpha_{D}}

. Six of the seven lie within 2.5% of

1/3

, slightly above it by the finite-norm excess (Appendix H .4); SignGD approaches it from below over this window, its larger bar reflecting a longer transient rather than a different exponent (Appendix H .4). This follows from (a): minimizing over

z

gives

\alpha_{D}=(2\alpha_{r}+\alpha_{t})/3

, which equals

1/3

exactly on the line. Error bars are the spread across four lower fitting cutoffs (Appendix H ).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	general	SGD ( $q=0$ )	Adam (nominal)	Adam (measured)
$\alpha_{r}$	$1/(3-q)$	$1/3$	$2/5$	$0.48\pm 0.03$
$\alpha_{t}$	$(1-q)/(3-q)$	$1/3$	$1/5$	$0.08\pm 0.01$
$s_{D}$	$-q/3$	$0$	$-1/6$	$-0.35\pm 0.01$
$s_{B}$	$1-p$	$1$	$1/2$	$0.76\pm 0.01$
$\alpha_{D}$	$1/3$	$1/3$	$1/3$	$0.341\pm 0.003$

Appendix

Table 1: Predicted and measured exponents. Every nominal entry for Adam is displaced by the measurement, yet

\alpha_{D}

is unchanged: the measured

\alpha_{r}

corresponds to an effective

q\simeq 0.92

by inversion, and the measured

s_{B}

to an effective

p\simeq 0.24

, a separation discussed in Section 5 . The final row is measured for all seven optimizers in Figure 4 b and Table 4 .

	$\nu=0$	$\nu=1$	general $\nu$
sum rule	$2\alpha_{r}+\alpha_{t}=1$	$\alpha_{r}+\alpha_{t}=1$	$\tfrac{2}{\nu+1}\alpha_{r}+\alpha_{t}=1$
$\alpha_{D}$	$1/3$	$1/2$	$(\nu+1)/(\nu+3)$
$L_{r}^{\ast}/L_{t}^{\ast}$	$2$	$1$	$2/(\nu+1)$

Appendix

Table 2: Effect of the margin density on the sum rule and the optimum. The exponents shift with

\nu

, but none of these quantities depends on the optimizer.

optimizer	learning-rate range
SGD	$[10^{-1},\ 10^{3}]$
PowerAdam( $0.125$ )	$[10^{-1},\ 10^{3}]$
PowerAdam( $0.25$ )	$[10^{-2},\ 10^{2}]$
PowerAdam( $0.375$ )	$[10^{-2},\ 10^{1}]$
Adam $=$ PowerAdam( $0.5$ )	$[10^{-3},\ 10^{1}]$
Muon	$[10^{-2.5},\ 10^{1}]$

Appendix

Table 3: Learning-rate sweep ranges, each with

16

logarithmically spaced values. The ranges shift downward as the preconditioner power increases, spanning six decades across the family; this is the unquantified form of the optimizer-dependence measured in Figure 3 .

Figure 5: The loss basin is broad and quadratic in

\log\eta

. Total loss against learning rate for Adam, one point per swept learning rate, with grey level denoting batch size at a fixed training step. Red markers are the parabola vertices used as

\eta^{\ast}

; they fall between grid points because they are interpolated. Left: the full swept range. Right: the same data near the minimum, on a linear vertical axis. Two things are visible. The minima are interior to the swept range at every batch size, so the estimator never has to extrapolate to a grid edge. And the basin is flat: the loss changes little over a factor of several in

\eta

. Since the loss is quadratic in

\log\eta

near its minimum, the location of the optimum is fixed only at second order while its value is fixed at first, which is why the

\eta^{\ast}

traces in Figure 3 a are noisy at large

D

while the envelope in Figure 3 c is clean.

Optimizer	$10^{5}$	$3\times 10^{5}$	$10^{6}$	$3\times 10^{6}$	$\alpha_{D}$
	$D_{\min}$
SGD	0.343	0.342	0.340	0.341	$0.341\pm 0.001$
PowerAdam(0.125)	0.334	0.332	0.332	0.333	$0.333\pm 0.001$
PowerAdam(0.25)	0.335	0.334	0.331	0.333	$0.333\pm 0.002$
PowerAdam(0.375)	0.337	0.336	0.336	0.339	$0.337\pm 0.001$
Adam	0.339	0.339	0.342	0.347	$0.341\pm 0.003$

Appendix

Table 4: Envelope exponents against the lower fitting cutoff.

\alpha_{D}

fitted over

D\geq D_{\min}

for four choices of

D_{\min}

, with the quoted value the mean across the four and the uncertainty their spread. The envelope is built from checkpoints with

t\geq 10^{3}

, the same window used for the dynamic exponents

\alpha_{r}

and

\alpha_{t}

, so that all four exponents are fitted over matched data. The envelope is defined as the minimum of the loss over hyperparameters at each budget, as in Figure 3 c, with its exponent estimated by a pooled fit with a single slope in

\log D

and one free level per batch size (Appendix H .4). Six of the seven lie within 2.5% of

1/3

, with a scatter across optimizers of

0.004

; SignGD approaches

1/3

from below over this window, its per-batch-size envelope slopes, fitted separately, rising from

0.06

B=16

0.33

B=2048

, so the batch sizes that reach furthest into the asymptotic regime already give the family value. We stop at

D_{\min}=3\times 10^{6}

because the number of batch sizes reaching a given budget falls as

D

grows: since each covers

D\in[10^{3}B,10^{4}B]

, fewer than three contribute above

5\times 10^{6}

and only one above

10^{7}

, so later cutoffs measure the largest batch sizes rather than a minimum over hyperparameters.

Figure 6: The radial loss is a function of the weight norm alone.

L_{r}

against

\beta

for SGD (left) and Adam (right), colored by learning rate over the ranges of Figure 1 . Curves at different learning rates collapse onto a single master curve approaching

L_{r}\propto\beta^{-1}

(dashed). This tests Eq. ( 5 ) directly, without comparing two separately fitted exponents.

Figure 7: The channel exponents do not depend on batch size. $L_{r}$ (left) and $L_{t}$ (right) against training step for all eight batch sizes at fixed learning rate, SGD above and Adam below, with the exponents of Figure 2 shown as dashed guides. Batch size shifts the curves without changing their slopes, which together with Figure 2 establishes that $\alpha_{r}$ and $\alpha_{t}$ depend on neither hyperparameter. Larger batches lower $L_{t}$ , while $L_{r}$ is nearly batch-independent for SGD: at fixed step $L_{r}$ depends on batch size only through $B^{p}$ , and $p=0$ for SGD. At a fixed sample budget $D=Bt$ the comparison changes sign, since a larger batch is less noisy and lowers $L_{t}$ but buys fewer optimization steps and so leaves $L_{r}$ higher. The erratic early-time behavior at small batch size is $L_{t}<0$ , the finite-norm transient described in Appendix B .

Figure 8: A dynamic exponent and a hyperparameter exponent are not independent. Measured

s_{D}

against

\alpha_{r}

for all seven optimizers, with the prediction of Eq. ( 32 ) as a dashed curve and the SGD value

(1/3,0)

as a black point. The two axes come from independent measurements,

\alpha_{r}

from the loss curves and

s_{D}

from the plane fit, so this test does not involve

\alpha_{t}

. The curve terminates at

(1/2,-1/3)

, the limit

\alpha_{t}\to 0

q\to 1

, beyond which the tangential channel no longer decays.

Figure 9: The two optimal-learning-rate exponents across optimizers. Measured

s_{D}

against

s_{B}

, with the SGD prediction

(1,0)

as a black point and the values reported by Ren et al. (2026) for their hypersphere optimizer as a star. Since

s_{D}=-q/3

and

s_{B}=1-p

, the horizontal axis measures the preconditioner’s batch exponent and the vertical axis its scale exponent.

Figure 10: The two channels reach a fixed ratio at the optimum. $L_{t}/L$ against $z/z^{\ast}$ for each of the seven optimizers, with $z=\eta B^{p-1}$ and $z^{\ast}(D)=10^{c}D^{s_{D}}$ taken from the plane fit of Appendix H . Each panel contains all $16$ learning rates and all $8$ batch sizes at five training times ( $t\approx 300$ , $1000$ , $3000$ , $5000$ , $9000$ ), and they collapse onto a single curve: $L_{t}/L$ depends on the hyperparameters only through $z/z^{\ast}$ . Every optimizer passes through $1/3$ at $z=z^{\ast}$ (dashed lines), as Theorem 2 predicts with no fitted quantity beyond the plane fit that sets $z^{\ast}$ . The collapse follows $R/(1+R)$ with $R=\tfrac{1}{2}(z/z^{\ast})^{3\alpha_{r}}$ ; deviations appear only for $L_{t}/L\lesssim 0.1$ , where $L_{t}$ is a small difference of two larger quantities and the finite-norm transient of Appendix B can drive it negative.

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 516 papers

May 28, 2026

On the Optimizer Dependence of Neural Scaling Laws

The scaling exponent

α

in neural scaling laws

L(N) \propto N^{-α}

is commonly treated as a fixed constant set by architecture and data. We present evidence that

α

depends systematically on the optimizer. In controlled random-feature regression experiments -- the canonical theoretical framework for neural scaling -- we measure

α

across five optimizer variants and six spectral conditions. Preconditioned optimizers consistently yield steeper scaling (larger

α

), with the

α

-shift increasing across most of the tested spectral range, peaking near

s = 1.5

, and remaining large at

s = 2.0

. At

s \approx 1.0

(characteristic of natural language), the full natural gradient achieves

α\approx 0.31

versus

α\approx 0.12

for gradient descent -- a

2.6\times

larger fitted exponent that, within the random-feature model, compounds with each model-size doubling. Whether and how this exponent shift transfers to large-scale LLM training -- where recent evidence suggests the advantage may attenuate with scale -- remains an important open question. Our results imply that scaling-law forecasts should account for optimizer choice, and we provide a spectral diagnostic predicting when advanced optimizers will pay off.

Vansh Ramani, Shourya Vir Jain

Department of Computer Science and Engineering, Indian Institute of Technology Delhi · Department of Electrical Engineering, Indian Institute of Technology Delhi

Natural Gradient Methods Neural Network Optimization Random Feature Methods Neural Scaling Laws

Ask PDF

Jun 28, 2026

On the Nonlinearity of Learning Rate Scaling for LLM Training

Learning-rate transfer can reduce the cost of training large language models: instead of sweeping learning rates at target scale, practitioners extrapolate from smaller runs. Existing approaches often assume that the optimal learning rate follows a log-linear scaling law in data scale and model size. We carefully examine and evaluate this scaling law. In our empirical study of GPT-2--style models from 22M to 707M parameters trained on 5B to 100B tokens, the optimal learning rate develops upward curvature at larger scales, leading to inaccurate extrapolation. We find that this curvature largely disappears when learning rates are replaced by effective learning rate (the step size in normalized weight space), and when data

D

extrapolation is used instead of model size

N

extrapolation. Next, we explain nonlinearity in scaling: weight-norm converges to equilibrium slower when optimal learning is small, requiring a larger step size to reduce the transient phase. Experiments with AdamH, which directly controls the effective learning rate, further support this explanation.

Zaiwen Yang, Huaqing Zhang, Jing Xu +1

Tsinghua University · Tsinghua University, Shanghai Qi Zhi Institute

Language Model Scaling Laws Hyperparameter Transfer Neural Network Training Dynamics LLM Training

Ask PDF

Oct 5, 2026

A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling

A recurring design principle in modern optimizers is to decouple update magnitude from the raw gradient norm, yet its consequences for learning-curve and resource scaling remain unclear. We isolate this mechanism by studying normalized SGD in a random-feature model with power-law teacher and data covariance. Fixed-norm updates induce an effective learning rate that grows as gradients shrink. We derive a dynamical mean-field theory (DMFT) describing the joint dependence of the loss on training time, model width and batch size. Normalization initially accelerates SGD, mapping the power-law exponent

r_{\rm SGD}<1

2r_{\rm SGD}/(1-r_{\rm SGD})

, with exponential convergence at

r_{\rm SGD}=1

and formal finite-time convergence for

r_{\rm SGD}>1

. At finite step size, however, the same feedback ultimately breaks the acceleration and leads to marginal stability. The late-time theory yields width-limited, edge-of-stochastic-stability (EoSS), and deterministic edge-of-stability (EoS) regimes. These phases determine when larger batches or wider models reduce serial training time at comparable compute. We quantify in which of these phases increased batch size or width can compensate the excess compute use per step by fewer optimization steps to target loss. Linearized ResNet experiments on CIFAR-5M support the predicted acceleration, breakdown, and resource-scaling trends. Together, these results connect normalization-induced acceleration, EoS effects, and width--batch allocation within a solvable theory.

Itay Lavie, Clarissa Lauditi, Cengiz Pehlevan

John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA · Center for Brain Science, Harvard University, Cambridge, MA · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, Cambridge, MA

Edge of Stability Neural Scaling Laws Stochastic Gradient Descent

Ask PDF

Abstract

Neural Network Training Dynamics

Explore similar work

On the Optimizer Dependence of Neural Scaling Laws

On the Nonlinearity of Learning Rate Scaling for LLM Training

A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling