cs.LGAug 16, 2026

Second-Moment Memory in Coordinatewise Adam

Authors: Jeonseong Kim

Abstract

Adam retains a moving average of past squared gradients in its denominator, but the optimization cost of this memory is not well understood. We show that second-moment memory can itself suppress progress toward the optimum even under finite-variance stochastic gradients. For a simple two-point oracle, the expected positive normalized update is O(M2−1/2)O(M_2^{-1/2}) after an initialization transient, where M2=(1−β2)−1M_2=(1-β_2)^{-1} is the second-moment memory length. We convert this directional bound, under the stated memory and stepsize scaling, into an average-stationarity lower bound of the same order on a smooth convex problem with normalized gap, smoothness, and variance. Long second-moment memory can slow optimization even when the gradient noise has finite variance.

Explore similar work

CardsList
  1. Adam under Generalized Smoothness with Second-Moment-Type Stochastic Gradients

    Sep 29, 2026Ruinan Jin, Difei Cheng, Ling Chen +3Convergence AnalysisStochastic Optimization Convergence

  2. Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization

    Date pendingSharan Sahu, Abir Sarkar, Cameron J. Hogan +1Stochastic OptimizationStochastic Optimization Convergence

  3. Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimization

    May 13, 2026Yaxin Yu, Long Chen, Minfu FengStochastic OptimizationMomentum Methods