cs.LGNov 6, 2026

Coordinatewise Epsilon for Adam under Heavy-Tailed Noise

Authors: Jeonseong Kim, Chenglin Fan

Organizations: Yonsei University Seoul, Korea · Seoul National University Seoul, Korea

Abstract

Adam estimates second moments coordinatewise, but typically adds the same scalar ε\varepsilon to every denominator. We study how this shared ε\varepsilon affects convergence when noise and curvature vary across coordinates. Under coordinatewise finite-pp noise moments, p∈(1,2]p\in(1,2], choosing εi\varepsilon_i from the noise and curvature bounds for each coordinate attains the heavy-tailed ℓ1\ell_1-stationarity benchmark O(T−p−13p−2)O(T^{-\frac{p-1}{3p-2}}), matching the known optimal time exponent for smooth nonconvex stochastic optimization. For 1<p<21<p<2, Adam with a shared εt\varepsilon_t admits a lower bound of order Ω(T−p−12p)Ω(T^{-\frac{p-1}{2p}}). This lower bound holds for every predictable scalar choice of ηtη_t, β1,tβ_{1,t}, β2,tβ_{2,t}, and εt\varepsilon_t, precluding a dimension-uniform benchmark rate for this class.

Explore similar work

CardsList
  1. Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization

    May 5, 2026Sharan Sahu, Abir Sarkar, Cameron J. Hogan +1Stochastic OptimizationStochastic Optimization Convergence