cs.LGJul 16, 2026

Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers

Authors: Akhilesh Gogikar

Organizations: 1Independent Researcher.

Abstract

Interpreting optimizers as gradient-flow discretizations has motivated applying higher-order Runge-Kutta (RK) integrators to neural networks. We build a representative Adam variant (Bogacki-Shampine 3(2) RK pair, FSAL reuse, local-error step control) and evaluate it under a strict compute-matched protocol giving every method the same gradient-evaluation budget - an accounting this literature rarely enforces. Under it the RK variant loses to plain Adam on training loss in both minibatch and full-batch (RK's best-case) training. Instrumenting it shows the "adaptivity" is illusory: normalized error stays far below tolerance, the step size pins at its growth cap from step one (98-100 percent of steps), and no rtol x hmax x h0 setting makes it act; tolerances spanning 100x give bit-identical trajectories. The method is exactly fixed-step Adam with an averaged gradient at 3-4x cost. Repairing it (true reject branch; error on the applied map) reverses the full-batch result - about 40x lower training loss than tuned Adam - and a fixed-step control isolates adaptivity (an emergent warmup-and-growth schedule) as the mechanism. But the gain is fragile to the initial step size and does not reach test accuracy. A pre-registered follow-up rules out the obvious explanations: deeper minimization does not overfit, and an explicit temperature knob only hurts - leaving a trajectory effect, the controller selecting a minimum generalizing 1.3-3.4 points below first-order descent at equal depth. An n=10 study confirms one secondary effect: gradient averaging is a genuine implicit regularizer, beating lr-matched Adam and AdamW on 10/10 seeds - yet RMSprop and NAdam match or beat it at a third the per-step cost. Higher-order adaptive integration buys deeper deterministic minimization and a small regularization effect, but nothing a cheaper, well-tuned first-order baseline does not already provide.

Explore similar work

CardsList
Aug 31, 2026cs.LG

Convergence rates for the RMSprop optimizer with full control of the hyperparameters

Popular adaptive stochastic gradient descent (SGD) methods to train artificial intelligence (AI) systems include the RMSprop, the Adam, and the AdamW optimizers, where the adaptivity parts in Adam and AdamW basically just coincide with RMSprop. Such adaptive methods involve several hyperparameters including the regularization parameter εε (which ensures that one does not divide by 0 and is often chosen to be very close to zero such as 10−810^{-8} in PyTorch by default) and the second moment decay parameter ββ (which is often chosen to be very close to 11 such as 0.99 (RMSprop) and 0.999 (Adam and AdamW) in PyTorch by default). Despite the high relevance of such methods, it remains an open research problem to provide error estimates for such methods with the error constants being not exploding but uniformly bounded with the respect to the hyperparameters, even in the situation of convex stochastic optimization problems. It is the key contribution of this work to essentially solve this problem for RMSprop. Specifically, we bound the expectation of the stopped evaluation of the objective function at the RMSprop process from above by the sum of an initialization term that decays exponentially in the training time, a stochastic approximation remainder of order γnγ_n, and a memory error of order (1−β)2( 1 - β)^2 with the error constants being uniformly controlled over all admissible choices of the step sizes, the second moment decay parameter ββ and the regularization parameter ε∈[0,1]ε\in[0,1] (also covering ε=0ε=0). Our non-asymptotic error estimates hold not just for all sufficiently large n but hold for every gradient step n=1,2,3,...n=1,2,3,... with all error constants being explicitly specified. The key innovative new feature in the proof of our analysis are suitable inverse moment bounds for the second moment process in RMSprop.
Steffen Dereich, Arnulf Jentzen
AdamStochastic Smooth Convex OptimizationGradient DescentHyperparameter OptimizationRegularization Techniques
AskPDF
Jun 20, 2024math.OC

Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses

The standard stochastic gradient descent (SGD) optimization method, as well as adaptive methods such as the Adam optimizer fail to converge if the learning rates do not converge to zero (particularly, in the situation of constant learning rates). In practice, human-tuned deterministic learning rate schedules or small constant learning rates are often used, and implementations in machine learning frameworks like Tensorflow and Pytorch typically employ constant learning rates. We propose a learning-rate-adaptive approach for SGD methods, adjusting the learning rate based on empirical estimates for the objective function values. Specifically, we propose a learning-rate-adaptive variant of the Adam optimizer and implement it for several machine learning problems, including deep learning methods for partial differential equations such as deep Kolmogorov methods, physics-informed neural networks, and deep Ritz methods. We refer to https://github.com/deeplearningmethods/adaptive-learning-rate for the Python source codes for the numerical simulations in this work. Our results show that the proposed adaptive Adam variant achieves faster reductions of the objective function value compared to Adam with default learning rates. For certain quadratic minimization problems, we rigorously prove that an adaptive SGD variant converges to the global minimizer. This proof uses properties of invariant measures of the SGD dynamics and a generalized convergence analysis for SGD with random predictable learning rates which we develop in this work.
Steffen Dereich, Arnulf Jentzen, Adrian Riekert
Stochastic Gradient DescentAdamLearning RatesGradient DescentPartial Differential Equations
AskPDF
Date pendingstat.ML

Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization

We provide a theoretical analysis of Adam under non-stationary stochastic objectives, separating two regimes: Euclidean tracking under adaptive strong monotonicity of the Adam-preconditioned mean-gradient operator, and high-probability projected stationarity guarantees under general LL-smooth objectives. In the tracking regime, we derive finite-time expected and high-probability bounds that decompose sharply into four components: initialization, objective drift, a first-moment tracking error governed by β1\beta_1, and a preconditioner perturbation governed by β2\beta_2. We characterize the burn-in time required for the transient terms to decay to the asymptotic tracking bound under constant and step-decay schedules. We also prove a high-probability bound on the average projected stationarity gap for Adam under distribution shift. Across both analyses, our bounds reveal a noise--drift tradeoff: in noise-dominated regimes, first-moment averaging and adaptive preconditioning can yield favorable upper guarantees, whereas in drift-dominated regimes, stale first-moment information and preconditioner perturbations can enlarge Adam's tracking guarantee, potentially allowing vanilla SGD to attain a smaller tracking error. Our explicit (β1,β2,ϵ)(\beta_1,\beta_2,\epsilon)-dependent bounds identify mechanisms through which adaptive step-sizing can help or hurt under nonstationarity and provide theoretical explanations consistent with Adam's empirical instability and stabilization under distribution shift.
Sharan Sahu, Abir Sarkar, Cameron J. Hogan +1
AdamStationarityStochastic Gradient DescentSpectral PreconditioningGradient StalenessMonotonicity+1
AskPDF