cs.LGNov 2, 2026

On the Convergence of Vanilla Adam under Heavy-Tailed Noise and Index Sampling

Authors: Kanata Oowada, Hideaki Iiduka

Organizations: Meiji University

Abstract

Understanding how stochastic gradient noise affects the convergence of Adaptive Moment Estimation (Adam) is an important problem in stochastic optimization, particularly when the noise exhibits heavy-tailed behavior. In this paper, we investigate the convergence of vanilla Adam in both the expected risk and the empirical risk minimization, focusing on the effects of heavy-tailed noise and its relationship to index sampling. For the expected risk minimization, we establish convergence guarantees under Hölder smoothness and bounded p\mathfrak{p}-th moment assumptions, allowing stochastic gradient noise with infinite variance. Our analysis provides explicit upper bounds on the expected squared gradient norm and shows that the iteration-dependent convergence term decays more slowly as p\mathfrak{p} decreases. For the empirical risk minimization, we establish asymptotic and non-asymptotic convergence guarantees and demonstrate that heavy-tailed population noise can deteriorate convergence bounds through dataset-dependent constants as the dataset size increases. Numerical experiments support our theoretical findings on the effects of heavy-tailed noise.

Figures & tables

Explore similar work

CardsList
  1. Coordinatewise Epsilon for Adam under Heavy-Tailed Noise

    Nov 6, 2026Jeonseong Kim, Chenglin FanStochastic Optimization ConvergenceAdaptive Gradient Methods