stat.MLOct 31, 2026

Adaptive Optimizers Can Get a Policy Ranking Wrong: Momentum, Dependent Data, and a Two-Run Remedy

Authors: Sunyoung An, Xiaoming Huo

Organizations: H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology

Abstract

Operators choosing among decision rules, such as when a battery charges or where an idle taxi goes, rank them by estimated long-run value. A standard estimator, linear temporal-difference (TD) learning, fits values from one stream of correlated observations. Large fits often use an adaptive optimizer, such as adaptive moment estimation (Adam), with fixed averaging rates. We show that these rates can change what is estimated, not just how fast: the same observations drive the updates and the optimizer's step-size scaling, moving the fit away from the intended TD value. The shift differs across rules, so it need not cancel in a comparison, and more data only makes a wrong ranking look more certain. Our remedy refits the same data at a second averaging rate that halves the effective gain, the number to which the leading shift is proportional. Twice the new fit minus the original removes that shift. In New York taxi and battery models calibrated to real data, common optimizer settings leave the selected rule unchanged but order close rules wrongly and misstate its value. Our remedy corrects nearly every wrong ordering, with one extra fit when the shift is moderate and two when it is larger.

Figures & tables

Explore similar work

Nov 7, 2026

Breaking Precision Walls in First-, Second-, and Zeroth-Order Optimizers: One Constant, Set in Advance

Under power-law signal decay across directions, an optimizer can fit strong directions and miss weak ones while average loss hides it. A precision wall measures the steepest decay where the weakest direction recovers within 5% in a fixed budget. On low-rank matrix sensing and quadratics one constant sets the wall: once it dominates the denominator, Adam or grafted Shampoo is momentum descent at step η/cη/c, setting the wall; in Shampoo, matching grafting's constant to damping restores the wall after three hypotheses failed. In zeroth-order Adam, ε\varepsilon acts against ε∗\varepsilon^\ast from a mean-square spectral model; below ε∗\varepsilon^\ast error hits a noise floor. Set from true curvature or spectrum on new shapes, the rule raised the wall above the same-step default by a forecast amount: on two sensing shapes by 1.96,1.151.96, 1.15 (Adam) and 1.59,1.741.59, 1.74 (grafted Shampoo), and on two Hadamard-rotated quadratics by 1.12,0.731.12, 0.73 (zeroth-order Adam, longer budget), rising on all seeds; fixed-step momentum at that step reaches those walls. A cosine-scheduled default (small constant giving weak modes large steps) is higher on one sensing shape, matched on new seeds of the other, and lower with the rule's constant. A default step retuned per budget catches up on rotated quadratics and population logistic regression, fails at 1 query/step on two diagonal quadratics (catching up at 16), and exceeds the rule on digits logistic regression (full-batch and minibatches, where gradient noise sets the η=0.01,0.1η= 0.01, 0.1 default walls). On logistic regression the rule did not raise the first-order wall: same-step default Adam was higher (its wall on 4 new shapes at the forecast stability limit), while in zeroth order on the population loss setting ε\varepsilon raised the wall by 0.98,0.880.98, 0.88.
Sep 29, 2026

Optimizer-dependent training dynamics converge to the same one-third optimal data scaling

Neural scaling, in which loss falls as a power law with training, is central to large language models, and one recent proposal is that a 1/31/3 exponent emerges from learning peaked distributions. That account describes SGD, but models in practice are trained with adaptive optimizers. Here we separate two exponents the 1/31/3 account does not distinguish: how fast the loss falls with training steps along a single run, and how fast the optimally tuned loss falls with dataset size DD. We show that the first, a dynamic exponent, is optimizer-specific while the second, an optimal data exponent, converges to 1/31/3 across optimizers. In an online teacher-student model we decompose the loss into norm growth (radial) and alignment toward the teacher direction (tangential), each decaying as a power law with dynamic exponents αrα_{r} and αtα_{t}. Under SGD, both are close to 1/31/3, so the data exponent is also 1/31/3 across different learning rates. Under Adam the two separate: αr≃0.48α_{r} \simeq 0.48 but αt≃0.08α_{t} \simeq 0.08. Since the total loss is minimized when these two parts are balanced, the optimal learning rate is optimizer-dependent: DD-independent for SGD but falls with DD for Adam. Yet tuned to that optimum, the loss returns to D−1/3D^{-1/3} for both. A stochastic-dynamics analysis explains why: the optimizers can trade decay speed between the two channels, but they all fall on a single dynamic exponent relation, 2αr+αt=12α_{r}+ α_{t} = 1, which fixes the optimal data exponent at 1/31/3. Across seven optimizers, including Muon, the measured exponents are consistent with this relation, and the optimal-loss envelopes agree with D−1/3D^{-1/3} across them. The optimizer sets how fast a model learns per step; tuned optimally, it changes the prefactor but not the rate at which loss falls per sample.
Jun 28, 2026

Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noise

Shuffle order can be a larger source of fine-tuning noise than a memoryless analysis predicts: fixed-clock optimizer memory makes local equal-multiset contrasts first order in the learning rate rather than second order, and the resulting order channel can be large enough for a single seed to flip a close A/B comparison. We isolate this mechanism and derive a fit-free way to size the noise it produces. For a memoryless optimizer, reordering an equal multiset has no first-order endpoint term; the leading local contrast is the O(η2)O(η^2) gradient bracket. Fixed-clock optimizers such as AdamW are different. Their moment buffers, preconditioner state, and de-biasing counters advance with the step index rather than with the learning-rate-scaled time τ=ηkτ=ηk, so the same gradient can receive a position-dependent endpoint weight. For any fixed finite measurement window, a lifted-state expansion gives an O(η)O(η) equal-multiset contrast whenever the first-order replay coefficient is nonzero, while regular and clock-matched controls remain O(η2)O(η^2); a bare fixed-ββ momentum buffer is already enough. A bitwise-deterministic replay from one warmed optimizer state isolates the mechanism, giving order-variance slopes 1.83 for AdamW, 2.00 for fixed-ββ momentum, and 4.00 for SGD; matching the memory clock to ττ restores the regular exponent. For AdamW with a frozen preconditioner, the same impulse-weight kernel gives a closed-form asymptotic order-variance floor after the local potentials are measured, with no fitted coefficients. The result is local to the measurement window (independent reshuffling can average the channel across windows), but it yields order-noise error bars, positional attribution weights, and a seed-budget criterion for fine-tuning comparisons.