cs.LGSep 29, 2026
SaveOptimizer-dependent training dynamics converge to the same one-third optimal data scaling
Organizations: Department of Physics, Massachusetts Institute of Technology, Cambridge, MA · Department of Physics, University of Pennsylvania, Philadelphia, PA
Abstract
Neural scaling, in which loss falls as a power law with training, is central to large language models, and one recent proposal is that a exponent emerges from learning peaked distributions. That account describes SGD, but models in practice are trained with adaptive optimizers. Here we separate two exponents the account does not distinguish: how fast the loss falls with training steps along a single run, and how fast the optimally tuned loss falls with dataset size . We show that the first, a dynamic exponent, is optimizer-specific while the second, an optimal data exponent, converges to across optimizers. In an online teacher-student model we decompose the loss into norm growth (radial) and alignment toward the teacher direction (tangential), each decaying as a power law with dynamic exponents and . Under SGD, both are close to , so the data exponent is also across different learning rates. Under Adam the two separate: but . Since the total loss is minimized when these two parts are balanced, the optimal learning rate is optimizer-dependent: -independent for SGD but falls with for Adam. Yet tuned to that optimum, the loss returns to for both. A stochastic-dynamics analysis explains why: the optimizers can trade decay speed between the two channels, but they all fall on a single dynamic exponent relation, , which fixes the optimal data exponent at . Across seven optimizers, including Muon, the measured exponents are consistent with this relation, and the optimal-loss envelopes agree with across them. The optimizer sets how fast a model learns per step; tuned optimally, it changes the prefactor but not the rate at which loss falls per sample.
Figures & tables
Figure 1: Optimizers produce qualitatively different dynamics on the same task. Online teacher–student softmax classification with classes and input dimension , trained with SGD (top row, batch size ) and Adam (bottom row, batch size ). For each optimizer and are chosen so that the tangential loss stays positive and the norm growth remains drift-dominated (Appendix G ); the exponents do not depend on batch size for either optimizer (Figure 7 ), so the comparison is not affected by the choice. Color denotes learning rate ; the two rows use different ranges, since the useful learning rates differ by roughly two decades between the optimizers. (a) Training dynamics of SGD follows time scaling. Left: test loss follows a single clean power law across the full range, with fitted exponent . Right: weight norm is consistent with (fitted ). (b) Training dynamics of Adam deviates from time scaling. Left: test loss curves bend over an extended transient and no single exponent describes the trajectory. Right: weight norm expands faster than throughout (fitted ).
Figure 2: The loss splits into a radial and a tangential component with distinct power-law dynamics and trade-offs between them. (a) Geometry of the decomposition. The student is compared to the rescaled teacher , which carries the student’s current norm but the teacher’s direction. The radial loss measures scale mismatch and the tangential loss measures misalignment, which the restoring Hessian reduces and stochastic injection from mini-batch noise sustains. (b, c) and against training step, for SGD (top) and Adam (bottom), colored by learning rate over the same ranges as in Figure 1 . Both decay as power laws whose exponents are independent of the learning rate: and for SGD, and for Adam.
Figure 3: Optimal learning rate scales differently with and for SGD and Adam, but the optimal loss follows one-third data scaling for both. Learning-rate sweeps over batch sizes and learning rates, spanning for Adam and for SGD, with the number of online samples processed. In each panel the upper row is SGD and the lower row Adam. Slopes and quoted in (a) and (b) are from the joint log-log plane fit described in Appendix H . (a) against , one curve per batch size (grayscale). SGD’s optimum is nearly flat ( ), while Adam’s decreases ( ), close to and to the recently reported “magic exponent” ( Bjorck et al., 2024 ) . (b) against batch size, one curve per data size (color). SGD follows ( ), while Adam gives , steeper than the square-root rule of common practice and than the value reported for LLM training ( Ren et al., 2026 ) . (c) Test loss along the optimal- envelope against , with individual runs colored by training step and shaded by batch size. Both optimizers give the same optimal data scaling, , with fitted exponents for SGD and for Adam. The quoted exponent is the pooled estimate across batch sizes of Appendix H .4. The traces in (a) are visibly noisy at large while the envelope in (c) is clean; Figure 5 shows the underlying sweeps and the flatness of the basin.
Figure 4: A sum rule for training exponents holds across optimizers, and fixes the optimal data-scaling exponent at . (a) Measured radial and tangential time-scaling exponents, and , for SGD, the PowerAdam family at , Muon, and SignGD. Crosses give fit uncertainties. The dashed line is , a parameter-free prediction with no fitted quantities; the black point marks the prediction . Adam sits at , above the nominal , corresponding to an effective ; Appendix G rules out two alternative explanations. (b) Optimal data-scaling exponent , with . Six of the seven lie within 2.5% of , slightly above it by the finite-norm excess (Appendix H .4); SignGD approaches it from below over this window, its larger bar reflecting a longer transient rather than a different exponent (Appendix H .4). This follows from (a): minimizing over gives , which equals exactly on the line. Error bars are the spread across four lower fitting cutoffs (Appendix H ).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| general | SGD ( ) | Adam (nominal) | Adam (measured) | |
|---|---|---|---|---|
Table 1: Predicted and measured exponents. Every nominal entry for Adam is displaced by the measurement, yet is unchanged: the measured corresponds to an effective by inversion, and the measured to an effective , a separation discussed in Section 5 . The final row is measured for all seven optimizers in Figure 4 b and Table 4 .
| general | |||
|---|---|---|---|
| sum rule | |||
Table 2: Effect of the margin density on the sum rule and the optimum. The exponents shift with , but none of these quantities depends on the optimizer.
| optimizer | learning-rate range |
|---|---|
| SGD | |
| PowerAdam( ) | |
| PowerAdam( ) | |
| PowerAdam( ) | |
| Adam PowerAdam( ) | |
| Muon |
Table 3: Learning-rate sweep ranges, each with logarithmically spaced values. The ranges shift downward as the preconditioner power increases, spanning six decades across the family; this is the unquantified form of the optimizer-dependence measured in Figure 3 .
Figure 5: The loss basin is broad and quadratic in . Total loss against learning rate for Adam, one point per swept learning rate, with grey level denoting batch size at a fixed training step. Red markers are the parabola vertices used as ; they fall between grid points because they are interpolated. Left: the full swept range. Right: the same data near the minimum, on a linear vertical axis. Two things are visible. The minima are interior to the swept range at every batch size, so the estimator never has to extrapolate to a grid edge. And the basin is flat: the loss changes little over a factor of several in . Since the loss is quadratic in near its minimum, the location of the optimum is fixed only at second order while its value is fixed at first, which is why the traces in Figure 3 a are noisy at large while the envelope in Figure 3 c is clean.
| Optimizer | |||||
|---|---|---|---|---|---|
| SGD | 0.343 | 0.342 | 0.340 | 0.341 | |
| PowerAdam(0.125) | 0.334 | 0.332 | 0.332 | 0.333 | |
| PowerAdam(0.25) | 0.335 | 0.334 | 0.331 | 0.333 | |
| PowerAdam(0.375) | 0.337 | 0.336 | 0.336 | 0.339 | |
| Adam | 0.339 | 0.339 | 0.342 | 0.347 | |
Table 4: Envelope exponents against the lower fitting cutoff. fitted over for four choices of , with the quoted value the mean across the four and the uncertainty their spread. The envelope is built from checkpoints with , the same window used for the dynamic exponents and , so that all four exponents are fitted over matched data. The envelope is defined as the minimum of the loss over hyperparameters at each budget, as in Figure 3 c, with its exponent estimated by a pooled fit with a single slope in and one free level per batch size (Appendix H .4). Six of the seven lie within 2.5% of , with a scatter across optimizers of ; SignGD approaches from below over this window, its per-batch-size envelope slopes, fitted separately, rising from at to at , so the batch sizes that reach furthest into the asymptotic regime already give the family value. We stop at because the number of batch sizes reaching a given budget falls as grows: since each covers , fewer than three contribute above and only one above , so later cutoffs measure the largest batch sizes rather than a minimum over hyperparameters.
Figure 6: The radial loss is a function of the weight norm alone. against for SGD (left) and Adam (right), colored by learning rate over the ranges of Figure 1 . Curves at different learning rates collapse onto a single master curve approaching (dashed). This tests Eq. ( 5 ) directly, without comparing two separately fitted exponents.
Figure 7: The channel exponents do not depend on batch size. (left) and (right) against training step for all eight batch sizes at fixed learning rate, SGD above and Adam below, with the exponents of Figure 2 shown as dashed guides. Batch size shifts the curves without changing their slopes, which together with Figure 2 establishes that and depend on neither hyperparameter. Larger batches lower , while is nearly batch-independent for SGD: at fixed step depends on batch size only through , and for SGD. At a fixed sample budget the comparison changes sign, since a larger batch is less noisy and lowers but buys fewer optimization steps and so leaves higher. The erratic early-time behavior at small batch size is , the finite-norm transient described in Appendix B .
Figure 8: A dynamic exponent and a hyperparameter exponent are not independent. Measured against for all seven optimizers, with the prediction of Eq. ( 32 ) as a dashed curve and the SGD value as a black point. The two axes come from independent measurements, from the loss curves and from the plane fit, so this test does not involve . The curve terminates at , the limit or , beyond which the tangential channel no longer decays.
Figure 9: The two optimal-learning-rate exponents across optimizers. Measured against , with the SGD prediction as a black point and the values reported by Ren et al. (2026) for their hypersphere optimizer as a star. Since and , the horizontal axis measures the preconditioner’s batch exponent and the vertical axis its scale exponent.
Figure 10: The two channels reach a fixed ratio at the optimum. against for each of the seven optimizers, with and taken from the plane fit of Appendix H . Each panel contains all learning rates and all batch sizes at five training times ( , , , , ), and they collapse onto a single curve: depends on the hyperparameters only through . Every optimizer passes through at (dashed lines), as Theorem 2 predicts with no fitted quantity beyond the plane fit that sets . The collapse follows with ; deviations appear only for , where is a small difference of two larger quantities and the finite-norm transient of Appendix B can drive it negative.
More in this topic
Neural Network Training Dynamics
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
See all 516 papers