cs.LGJul 4, 2026

Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adam

Authors: Ashmitha RJörg Frochte

Organizations: Department of Artificial Intelligence and Data Science, Sri Ramakrishna Engineering College, Anna University, Coimbatore 641022, Tamil Nadu, India · Interdisciplinary Institute for Applied AI and Data Science Ruhr (AKIS), Department of Electrical Engineering and Computer Science, Bochum University of Applied Sciences, Am Hochschulcampus 1, 44801 Bochum, Germany

Abstract

The local sharpness of the loss, the top Hessian eigenvalue λ1λ_1, determines the largest stable gradient step, but measuring it normally requires Lanczos or Hessian-vector iterations. We observe that a single Armijo backtracking line search already carries this information at the cost of a few forward passes: the accepted step αα brackets the \emph{directional} curvature q=gHg/g2q = g^\top H g/\|g\|^2 within the multiplicative band set by the backtracking factor. Across CIFAR-10, Fashion-MNIST and Imagenette, logα\logα tracks logλ1\logλ_1 at Pearson 0.91-0.91 to 0.95-0.95, giving a low-cost online Edge-of-Stability reading. Used once at initialisation, this measurement yields a learning-rate cap (a safeguard, not a faster optimiser) that makes Adam robust to a too-large initial learning rate across more than three orders of magnitude (10310^{-3} to 3.03.0), at about one percent overhead, and it is a no-op when the chosen rate is already safe. One probe is enough: periodic in-training probing adds no robust benefit. The raw-gradient probe exposes the mechanism but needs a safety factor calibrated to the architecture by a one-minute divergence sweep. Probing along Adam's own update direction removes this calibration: a single fixed safety factor κ=2κ= 2 avoids divergence on all nine architectures we test and across the full learning-rate grids of all four benchmarks, and the recipe transfers to AdamW unchanged.

Explore similar work

CardsList
  1. Why Muon Outperforms Adam: A Curvature Perspective

    Jun 3, 2026Shuche Wang, Fengzhuo Zhang, Jiaxiang Li +2AdamMuon