stat.MLOct 31, 2026

Adaptive Optimizers Can Get a Policy Ranking Wrong: Momentum, Dependent Data, and a Two-Run Remedy

Authors: Sunyoung An, Xiaoming Huo

Organizations: H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology

Abstract

Operators choosing among decision rules, such as when a battery charges or where an idle taxi goes, rank them by estimated long-run value. A standard estimator, linear temporal-difference (TD) learning, fits values from one stream of correlated observations. Large fits often use an adaptive optimizer, such as adaptive moment estimation (Adam), with fixed averaging rates. We show that these rates can change what is estimated, not just how fast: the same observations drive the updates and the optimizer's step-size scaling, moving the fit away from the intended TD value. The shift differs across rules, so it need not cancel in a comparison, and more data only makes a wrong ranking look more certain. Our remedy refits the same data at a second averaging rate that halves the effective gain, the number to which the leading shift is proportional. Twice the new fit minus the original removes that shift. In New York taxi and battery models calibrated to real data, common optimizer settings leave the selected rule unchanged but order close rules wrongly and misstate its value. Our remedy corrects nearly every wrong ordering, with one extra fit when the shift is moderate and two when it is larger.

Figures & tables

Explore similar work

CardsList
  1. Optimizer-dependent training dynamics converge to the same one-third optimal data scaling

    Sep 29, 2026Hyunseok Lee, Mihir Basil, Yizhou Liu +1Neural Network Training DynamicsNeural Scaling Laws