cs.LGNov 7, 2026

Breaking Precision Walls in First-, Second-, and Zeroth-Order Optimizers: One Constant, Set in Advance

Authors: Lngel Kyo

Organizations: A2O Labs

Abstract

Under power-law signal decay across directions, an optimizer can fit strong directions and miss weak ones while average loss hides it. A precision wall measures the steepest decay where the weakest direction recovers within 5% in a fixed budget. On low-rank matrix sensing and quadratics one constant sets the wall: once it dominates the denominator, Adam or grafted Shampoo is momentum descent at step η/cη/c, setting the wall; in Shampoo, matching grafting's constant to damping restores the wall after three hypotheses failed. In zeroth-order Adam, ε\varepsilon acts against ε∗\varepsilon^\ast from a mean-square spectral model; below ε∗\varepsilon^\ast error hits a noise floor. Set from true curvature or spectrum on new shapes, the rule raised the wall above the same-step default by a forecast amount: on two sensing shapes by 1.96,1.151.96, 1.15 (Adam) and 1.59,1.741.59, 1.74 (grafted Shampoo), and on two Hadamard-rotated quadratics by 1.12,0.731.12, 0.73 (zeroth-order Adam, longer budget), rising on all seeds; fixed-step momentum at that step reaches those walls. A cosine-scheduled default (small constant giving weak modes large steps) is higher on one sensing shape, matched on new seeds of the other, and lower with the rule's constant. A default step retuned per budget catches up on rotated quadratics and population logistic regression, fails at 1 query/step on two diagonal quadratics (catching up at 16), and exceeds the rule on digits logistic regression (full-batch and minibatches, where gradient noise sets the η=0.01,0.1η= 0.01, 0.1 default walls). On logistic regression the rule did not raise the first-order wall: same-step default Adam was higher (its wall on 4 new shapes at the forecast stability limit), while in zeroth order on the population loss setting ε\varepsilon raised the wall by 0.98,0.880.98, 0.88.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. The Loss Does Not See the Basis, but Adam Does

    Aug 5, 2026Devender SinghLow-Rank Matrix DecompositionMatrix Optimization