On the Convergence of Vanilla Adam under Heavy-Tailed Noise and Index Sampling
Organizations: Meiji University
Abstract
Understanding how stochastic gradient noise affects the convergence of Adaptive Moment Estimation (Adam) is an important problem in stochastic optimization, particularly when the noise exhibits heavy-tailed behavior. In this paper, we investigate the convergence of vanilla Adam in both the expected risk and the empirical risk minimization, focusing on the effects of heavy-tailed noise and its relationship to index sampling. For the expected risk minimization, we establish convergence guarantees under Hölder smoothness and bounded -th moment assumptions, allowing stochastic gradient noise with infinite variance. Our analysis provides explicit upper bounds on the expected squared gradient norm and shows that the iteration-dependent convergence term decays more slowly as decreases. For the empirical risk minimization, we establish asymptotic and non-asymptotic convergence guarantees and demonstrate that heavy-tailed population noise can deteriorate convergence bounds through dataset-dependent constants as the dataset size increases. Numerical experiments support our theoretical findings on the effects of heavy-tailed noise.
Figures & tables
| Reference | Vanilla Adam | Noise Asm. | Smoothness | LR | Beta | Rate |
| Défossez et al. (2022) | No | A.s. bounded SG | Lipschitz | |||
| Liang et al. (2025) | No | A.s. bounded SG | Lipschitz | , | ||
| Zou et al. (2019) | No | Bounded 2nd moment of SG | Lipschitz | , | ||
| Wang et al. (2023) | No | Bounded variance | Lipschitz | |||
| Jin et al. (2025) | No | ABC condition | Lipschitz | , | ||
| Theorem 2 | Yes | A.s. bounded SG | Hölder | , |
Stochastic Optimization Convergence
Stochastic optimization convergence studies how noisy, sampled, or delayed updates approach solutions and how fast they do so.
See all 228 papers