stat.MLNov 14, 2026

High-dimensional limit theorems for adaptive variants of SGD I: Ballistic limits

Authors: Aukosh Jagannath, Varnan Sarangian

Organizations: Department of Statistics and Actuarial Science, Department of Applied Math Science, and Cheriton School of Computer Science, University of Waterloo, Canada · Department of Statistics and Actuarial Science, University of Waterloo, Canada

Abstract

We develop the high-dimensional effective dynamics for adaptive, momentum-based variants of Stochastic Gradient Descent (SGD). These effective dynamics are given by a Markov process. They key observation is that the non-Markovianity in the algorithm due to the exponential moving average and preconditioning can be treated using the method of averaging. Our approach applies to a broad class of algorithms including to Adaptive momentum estimation (ADAM), Momentum orthogonalized by Newton-Schultz (MUON), and its variants. It also applies to a broad class of learning tasks, such at tensor PCA, Gaussian projective models, teacher-student networks, and a multi-layer neural network trained on the XOR problem using a hybrid of MUON and ADAM as commonly done in practice. We demonstrate how to this technique can give precise, quantitative comparisons between algorithms. In particular, we find interesting phenomena, such as the cross-over phenomenon where online SGD has better generalization than ADAM early in training until the learning curves cross and ADAM dominates. Furthermore, we prove a version of the Edge of Stability phenomenon. In particular, we introduce an analogue of the infinity Laplacian that sharply characterizes the Edge of Stability.

Figures & tables

Explore similar work

CardsList
  1. Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization

    May 5, 2026Sharan Sahu, Abir Sarkar, Cameron J. Hogan +1Stochastic OptimizationStochastic Optimization Convergence

  2. Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails

    Mar 3, 2026Ruinan Jin, Yingbin Liang, Shaofeng ZouStochastic Optimization ConvergenceAdaptive Gradient Methods

  3. Stochastic Gradient Descent with Momentum is Algorithmically Stable

    May 27, 2026Yunwen Lei, Zimeng Wang, Xiaoming YuanMomentum MethodsAlgorithmic Stability