Loss Landscape Features That Make Adam Stall: Definitions, Estimators, and the Preconditioned Hessian View

Rodion Podorozhny

cs.LGAug 23, 2026

Loss Landscape Features That Make Adam Stall: Definitions, Estimators, and the Preconditioned Hessian View

Authors: Rodion Podorozhny

Organizations: Texas State University

Abstract

Across implicit-neural-representation (INR) architectures and analytic benchmarks we observe that a thoroughly tuned Adam (especially its learning rate (lr), e.g. in a hyperparameter sweep from $lr = 0.05$ to $10^{-8}$ ) can potentially reach a very low loss even on ill-conditioned loss landscape or converge at a plateau far above the loss attained by second-order methods. This report defines the measured metrics that help determine if Adam can mitigate the ill-conditioning on a given loss landscape. We provide the indicators by which each outcome is determined, that are: the condition number of the Hessian and of the Adam-preconditioned Hessian $D^{-1/2}HD^{-1/2}$ (with the derivation from Adam's update rule), the diagonal mass $ρ$ that distinguishes axis-aligned from cross-coupled ill-conditioning, the negative spectral mass estimated by stochastic Lanczos quadrature, and the gradient energy fractions over curvature bands, including the flat fraction that indicates the Adam stall. A worked out $2\times 2$ example and an illustration show the reasons why a diagonal preconditioning by Adam can remove axis-aligned ill-conditioning by rescaling and why it cannot do the same if the ill-conditioning is cross coupled. In addition, we present a case study of FINER image fitting architecture that goes over the whole loss landscape analysis framework: the fitting architecture description, reasons due to which its landscape stalls Adam at saddles, the measured PSNR values through our tuned baselines to the $120$ -- $134$ ,dB results of the blockwise second order methods, the error maps behind those numbers, and description of the benefits such image fitting accuracy gives in practice.

Figures & tables

Figure 1: Diagonal preconditioning on the two $2\times 2$ examples of Section 3.1 . Contours of the local quadratic model before (left) and after (right) the rescaling $\varphi=D^{1/2}\theta$ with $D=\operatorname{diag}(H)$ . (a) When the stiff direction is a coordinate axis, per-coordinate rescaling equalizes the curvature scales and the contours become circles: Adam behaves as if the problem were well conditioned. (b) When stiff and soft directions are mixtures of the coordinates, the diagonal of $H$ is uniform, the rescaling is a multiple of the identity, and the contours, and $\kappa$ , are unchanged: Adam faces the raw ill-conditioning, and by van der Sluis’ theorem no other diagonal would do better than a modest factor.

Benchmark	$\kappa_{\mathrm{raw}}$	$\kappa_{\mathrm{adam}}$	reduction	$\rho$	neg. mass	flat frac	outcome
35k multi-freq.	4.9e2	3.1e1	16 $\times$	.04	.92	.000	Adam at precision floor
SIREN 2D	1.6e3	9.9e1	16 $\times$	.09	.09	.000 a	comparable
WIRE 2D	1.2e5	8.4e2	143 $\times$	.65	.00	.000	comparable
FINER 2D	5.4e4	6.1e1	887 $\times$	.98	.19 b	.000	ARC $+59.5$ dB
SDF-SIREN	8.9e3	2.4e1	372 $\times$	.06	.67	.000 c	comparable
SDF-FINER	1.0e3	6.9	146 $\times$	.58	.68	.001	Adam error 5.5 $\times$

Table 1: Landscape fingerprints along the tuned Adam trajectory. Reduction

=\kappa_{\mathrm{raw}}/\kappa_{\mathrm{adam}}

. Architectures: SIREN [ 11 ] , WIRE [ 12 ] , FINER [ 10 ] ; the 35k multi-frequency benchmark is a SIREN-family network with six octave-spaced target frequencies [ 16 ] ; the SDF rows fit signed distance functions (Thai statue) under IGR supervision [ 15 ] with SIREN and FINER backbones. The two analytic rows are decided by final loss (the benchmarks have no PSNR): the first-order arms terminate on the saddle plateau at any tested tuning while cubic-regularized arms pass it by orders of magnitude. Notes: a

.111

mid-run; b

.81

at initialization; c

.689

transient mid-run.

Figure 2: FINER image 0 at native $240^{2}$ resolution: target, reconstruction, and $\log_{10}$ absolute-error map for extended-budget converged Adam ( $78.2$ dB at $1{,}000{,}001$ iterations, top) and converged ARC- $\varphi_{1}$ ( $133.5$ dB at $2{,}824$ sweeps, bottom). The error-map color scale is shared across both rows and spans $10^{-1}$ to $10^{-8}$ . The reconstructions are visually indistinguishable from the target and from each other, both are far above the $\sim$ 50 dB perceptual threshold. The error maps differ by nearly three orders of magnitude and in structure : Adam’s residual is a smooth $\sim$ $10^{-4}$ field organized in broad regions separated by dark zero-crossing curves, a low-frequency bias field, while the ARC- $\varphi_{1}$ residual is a band-uniform $\sim$ $2\cdot 10^{-7}$ speckle with no spatial organization.

Loss Landscape

See all 22 papers

Sep 16, 2026

Beyond Quadratic Loss: The Stability Phase Diagram of Adam

Loss spikes are recurrent instabilities in neural-network training and can arise from multiple mechanisms. For Adam in particular, macroscopic loss spikes have been linked to optimizer dynamics, yet how its two momentum timescales govern them remains unclear. We investigate this dependence by mapping training dynamics across the

(β_1,β_2)

plane. Across a range of model--task settings, an approximately linear boundary,

1-β_2=C(1-β_1)

, separates spiky from non-spiky dynamics, whereas a one-dimensional quadratic loss produces approximately cubic slope. A one-dimensional superquadratic loss

L(x)\propto|x|^n

recovers the near-linear scaling and links the boundary coefficient to the effective loss exponent

n

. We further show that confident cross-entropy losses develop a core--wall landscape comprising a narrow quadratic core followed by a steep wall, which produces effective superquadratic behavior at the scale of an optimizer update. Together, these results connect Adam loss spikes to both the mismatch between momentum timescales and finite-scale superquadratic loss geometry beyond the Hessian.

Gaoxiang Tang, Huanran Chen, Ziming Liu

IIIS, Tsinghua University · College AI, Tsinghua University · Shanghai Qizhi Institute +1

Adaptive Gradient Methods Neural Network Training Dynamics

Ask PDF

Aug 17, 2026

On the Principles Behind Neural Network Optimizers

Reliable optimization is central to neural network (NN) training, yet Adam, the default optimizer for modern LLMs, rests on a fragile foundation. This thesis develops a principled grounding for Adam and motivates new designs. First, we revisit Adam's divergence--convergence debate and show the existence of a problem-dependent phase transition: with properly chosen, batch-size-dependent hyperparameters, Adam converges, whereas under small-

β_2

regimes it can diverge. Second, we investigate why Adam substantially outperforms SGD on Transformers through Hessian structure. We find that the Hessian evolves toward a near-block-diagonal form along training, accompanied by strong block heterogeneity. We prove that this structure makes Adam's diagonal preconditioner effective. We further show that this special Hessian structure originates from consecutive multiplications of large matrix variables, and we provide a rigorous analysis based on random matrix theory. Finally, these insights motivate Adam-mini, a new optimizer that reduces Adam's memory footprint by 50% while preserving its performance. Our results also have broader implications beyond Adam: they reveal new local structures in matrix-based nonconvex problems, and also help understand and improve recent NN optimizers, such as Muon.

Yushun Zhang

School of Data Science, The Chinese University of Hong Kong, Shenzhen, China

Neural Network Optimization Adaptive Gradient Methods

Ask PDF

Mar 2, 2026

Never Saddle for Reparameterized Steepest Descent as Mirror Flow

How does the choice of optimization algorithm shape a model's ability to learn features? To address this question for steepest descent methods --including sign descent, which is closely related to Adam --we introduce steepest mirror flows as a unifying theoretical framework. This framework reveals how optimization geometry governs learning dynamics, implicit bias, and sparsity and it provides two explanations for why Adam and AdamW often outperform SGD in fine-tuning. Focusing on diagonal linear networks and deep diagonal linear reparameterizations (a simplified proxy for attention), we show that steeper descent facilitates both saddle-point escape and feature learning. In contrast, gradient descent requires unrealistically large learning rates to escape saddles, an uncommon regime in fine-tuning. Empirically, we confirm that saddle-point escape is a central challenge in fine-tuning. Furthermore, we demonstrate that decoupled weight decay, as in AdamW, stabilizes feature learning by enforcing novel balance equations. Together, these results highlight two mechanisms how steepest descent can aid modern optimization.

Tom Jacobs, Chao Zhou, Rebekka Burkholz

CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

Neural Network Training Dynamics Gradient Descent Dynamics Adaptive Gradient Methods Gradient Descent Mirror Descent Representation Learning

Ask PDF

Abstract

Loss Landscape

Explore similar work

Beyond Quadratic Loss: The Stability Phase Diagram of Adam

On the Principles Behind Neural Network Optimizers

Never Saddle for Reparameterized Steepest Descent as Mirror Flow