cs.LGJul 24, 2026

Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided Hölder Regularity

Authors: Arzu Ahmadova, Ismail Huseynov

Organizations: Weierstrass Institute for Applied Analysis and Stochastics Berlin, Germany · Physikalisch-Technische Bundesanstalt (PTB), Germany · Technical University of Berlin, Germany

Abstract

We study adaptive gradient descent for continuously differentiable, possibly nonconvex objectives under one-sided Hölder regularity. Unlike classical Hölder- or Lipschitz-gradient assumptions, which control the full gradient variation, our condition bounds only the directional term appearing in the descent inequality. This can allow less conservative step sizes when large gradient changes are orthogonal to, or favorable along, the update direction. We propose an adaptive scalar-step method based on an estimate of positive one-sided Hölder curvature, combined with a simple sufficient-decrease safeguard. For nonconvex objectives on a convex region containing the accepted update segments, we prove an explicit best-iterate stationarity bound with a rate determined by the Hölder exponent. Unlike predetermined diminishing step-size schemes, the method adapts to the local descent geometry. We evaluate the approach on two full-batch benchmarks designed to separate directional curvature from full gradient variation. On a binary classification problem, the method achieves the lowest final cross-entropy, objective value, and gradient norm, together with the largest classification margin among the compared scalar gradient methods. On a nonconvex Hölder regression problem, it attains the lowest final objective gap and gradient norm. These results indicate that one-sided Hölder curvature is an effective adaptive step-size signal when full-gradient variation is inflated by directions that do not hinder descent.

Explore similar work

CardsList
Jun 4, 2026cs.LG

Flatland: The Adventures of Gradient Descent with Large Step Sizes

The training of neural networks often entails objective functions that are not globally LL-smooth. For these functions, it is both theoretically and practically difficult to reply to the question: what is the largest possible step size that ensures the convergence of gradient descent (GD)? We address this longstanding open question in deep learning by providing a unifying definition of "large" step sizes that requires only local Lipschitz (or even Hölder) continuity of the gradient. We design first-order adaptive methods that provably yield large step sizes and show that they operate at the edge of stability (EoS) right from the start of the training. In particular, the loss decreases nonmonotonically and the product between the step size and sharpness, i.e., the largest eigenvalue of the Hessian, stays above the EoS threshold of 2 throughout training. Using our method, we are also able to minimize the sharpness all the way down to its global minimum. Contrary to expectation, we find that encountering globally-flat regions too early in the training may both slow down convergence and jeopardize the generalization ability of the network. Exploiting a self-stabilization argument, we allow GD to enter slightly sharper valleys and turn unsuccessful training runs into very successful ones.
Leonardo Galli, Curtis Fox, Wiebke Bartolomaeus +2
Gradient DescentLarge Step SizesZeroth-Order Optimization
AskPDF
Sep 14, 2026cs.LG

Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise and H\"{o}lder Smoothness

Classical convergence guarantees for stochastic gradient methods typically assume Lipschitz-smooth objectives and finite-variance gradient noise, both frequently violated in practice. In contrast, we study nonconvex stochastic optimization under the joint relaxation of these assumptions: objectives with (L,s)(L,s)-H"older continuous gradients, s∈(0,1]s\in(0,1], and gradient noise satisfying only a bounded α\alpha-th moment condition for α∈(1,2]\alpha\in(1,2]. We establish three convergence results. Firstly, that standard SGD converges at rate O(T−s/(1+s))O(T^{-s/(1+s)}) whenever α≥1+s\alpha\ge1+s, extending the classical nonconvex SGD rate to heavy-tailed noise and H"older smoothness simultaneously. Secondly, we analyze δ\delta-regularized gradient clipping (δ\delta-GClip), a provable trainer of wide and deep nets, and establish a stationarity rate of O(T−2s(α−1)/[(1+s)(2α−1)])O(T^{-2s(\alpha-1)/[(1+s)(2\alpha-1)]}) under the same condition. Thirdly, we analyze standard gradient clipping (G-Clip) and show that it recovers the above rate for α≥1+s\alpha\ge1+s while in the very heavy-tailed regime α<1+s\alpha<1+s, it has a convergence rate O(T−2s(α−1)/[(α−1)+s(2α−1)])O(T^{-2s(\alpha-1)/[(\alpha-1)+s(2\alpha-1)]}) --- the first convergence guarantee in this regime for any stochastic gradient based method.
Misbah Uz Zaman, Anirbit Mukherjee
Stochastic Gradient DescentLipschitz ConstantConvergence
AskPDF
May 28, 2026cs.LG

Convergence of Steepest Descent and Adam under Non-Uniform Smoothness

Recent work has analyzed the convergence of first-order methods under non-uniform smoothness assumptions that better model the loss landscape in machine learning tasks. We generalize this assumption to objectives whose curvature is an affine function of the objective value. This property is satisfied by a broad class of problems, including logistic regression, generalized linear models with a logistic link function, softmax policy gradient in reinforcement learning, and a class of neural networks. Under this assumption and gradient domination conditions, we establish a general convergence rate for the steepest descent method, and deterministic, diagonal variants of RMSProp and Adam. Our results imply that for logistic regression on separable data and the softmax policy gradient objective, sign GD converges linearly and is provably faster than GD. Furthermore, we show that for a class of two-layer neural networks on separable data, RMSProp and Adam can converge at a linear rate with a constant step-size and momentum parameter. Finally, we present a lower bound demonstrating that, under our assumption, RMSProp and Adam are provably faster than AdaGrad, AMSGrad, gradient descent, and heavy-ball momentum.
Sharan Vaswani, Yifan Sun, Reza Babanezhad
Gradient DescentAdamSmoothingZeroth-Order OptimizationConvergencePolicy Gradient
AskPDF