In nonconvex Burer--Monteiro low-rank matrix sensing of U∗∈RV×D under power-law covariance Σ=diag(j−s)j=1V, the precision-wall approximation predicts that Adam stalls when the smallest non-gauge Hessian eigenvalue drops below its denominator floor, giving s∗(V,D)=[log(1/εeff)−logV]/log(VD)+δAdam. If εeff were set by ε0, lowering ε0 from 10−10 to 10−12 would raise s∗ by 0.95–1.33. Instead, across six values of ε0 two decades apart, 10−2 to 10−12, the boundary follows the ε0 slope down to 10−6, bends near 10−8, and is flat below 10−8, while warm-started L-BFGS and Gauss--Newton reach machine precision on stalled instances. We trace this saturation to a self-generated dynamical floor εdyn∝α: high-curvature modes oscillate persistently near the adaptive edge of stability with amplitude ∝α(1−β1)/(1+β1) and leak gradient fluctuations of scale κα into rare modes via measurement coupling, pinning v^V≈κα≫ε0. Fitting a single threshold-crossing wall parameter εdyn=4.76×10−9 (with the detection lag frozen at its companion calibration) predicts the saturated failure boundary for D≥4 across five (V,D) scales to RMS 0.098 (breaking down at D=2), while shrinking ε0 below 10−8 is ineffective and lowering the step size α reduces the floor directly.
Figures & tables
Figure 1: (a) Empirical failure onset s∗ at 60k steps vs. Adam’s hyperparameter ε0 across six values of ε0 two decades apart ( 10−2 to 10−12 ) in fp64 ( n=5 seeds, evaluated with the seed-mean linear-interpolation onset estimator of § 2 ): (V,D)=(8,4) across all 6 values ( s∗=2.01,3.47,4.68,5.339,5.414,5.407 ) and (16,8) at 10−4,10−6,10−10,10−12 ( s∗=2.242,3.228,3.947,3.923 ), shown with the ε0 -law reference slopes ( +1.33 and +0.95 per two decades, anchored at 10−4 with extrapolations 2.14 at (8,4),10−2 and 3.191 at (16,8),10−6 ) and saturated plateaus. Filled markers are the original sweeps; the two open markers ( (8,4) at 10−2 ; (16,8) at 10−6 , on a grid extended beyond the original s≤3.0 ) are re-runs with the released code, which evaluates the Adam step in a different floating-point order (§ 3.1 ; Appendix B ); this panel shows that s∗ tracks the ε0 slope from 10−2 to 10−6 , bends near 10−8 , and saturates below 10−8 . (b) Endpoint relative error rel-dM at 60k steps vs. spectral exponent s at (V,D)=(8,4) in re-runs with the released code for ε0=10−10 vs. 10−12 across matched problem instances ( n=5 seeds), showing that the two curves superimpose at threshold crossing rather than shifting by the predicted +1.33 .
T
s∗(10−10)
s∗(10−12)
Δs∗(12−10)
LOO of Δs∗
s∗(10−12) ( n=20 )
5-of-20 95% (SD)
60k
5.62
5.61
−0.007
[−0.02,+0.00]
5.60
[5.37,5.99] (0.14)
200k
5.82
5.69
−0.129
[−0.28,+0.02]
5.76
[5.55,6.14] (0.18)
600k
6.12
6.12
+0.009
[−0.12,+0.11]
5.56
[5.07,6.30] (0.43)
Table 1: Multi-budget test of ε0 -insensitivity at (V,D)=(8,4) . While the absolute onset s∗(T) creeps upward with step budget T , the matched-seed difference Δs∗(10−12−10−10) remains near zero at every budget (against the two-decade ε0 -law prediction of +1.33 ). Columns 2–5 use seeds 0–9 ( n=10 ); the (8,4) row of Table 2 uses seeds 0–4 and gives 5.407 in the original runs and 5.423 in the re-runs, both inside the 60k 5-of-20 range below, so the two tables differ by seed set, not by estimator. Columns 6–7 report the n=20 onset and the 95% range across all 15504 five-seed subsets at ε0=10−12 (each subset onset uses the same seed-mean linear-interpolation estimator as every other onset in this paper, § 2 ; at (8,4) , the 5-of-20 subset SD is 0.14 at 60k and 0.43 at 600k with 547 censored subsets dropped at 600k , Appendix B , item 2).
Figure 2: (a) Verification of the amplitude law ( 7 ) across (ε0,α,λ) spanning 10000× in ε0 and 100× in α : rescaled time-averaged RMS amplitude ARMS/A∗ on the burst-and-relaxation attractor collapses onto the empirical tracking curve 1.08(1−38/u) across β2∈{0.99,0.999} (within 2.3% , vs. 10.5% against the unscaled exact period-2 orbit 1−38/u ; worst cross-leg spread 0.66% at matched u ; Observation 4.2 ). (b) Per-step log-envelope slope of the flip amplitude vs. u=αλ/ε0 (stable below, unstable/escaping above the stability threshold uc2=38 ).
V
D
VD
m
pred. splateau∗
s∗(10−12)
re-run s∗(10−12) ( Δ )
κ(10−6)
domain
8
4
32
48
5.41
5.407
5.423 ( +0.016 )
2.77
D≥4
16
4
64
96
4.42
4.391
4.515 ( +0.124 )
0.51
D≥4
16
8
128
192
3.86
3.923
3.889 ( −0.034 )
0.53
D≥4
32
4
128
192
3.72
3.571
3.650 ( +0.079 )
0.10
D≥4
32
8
256
384
3.31
3.459
3.239 ( −0.220 )
0.047
D≥4
16
2
32
48
5.21
6.315
6.316 ( +0.001 )
0.018
D=2 ( Δ=+1.10 )
Table 2: Multi-scale validation of the saturated plateau prediction ( 11 ) at ε0=10−12 ( n=5 seeds, 60k steps; each onset is the seed-mean linear-interpolation onset of § 2 over seeds 0–4, and at (8,4) the 5-of-20 subset SD is 0.14 at 60k and 0.43 at 600k , Appendix B , item 2). Here δAdam=0.482 is frozen from the companion ε0=10−8 calibration and εdyn=4.76×10−9 is fitted across the five D≥4 rows of the original runs (RMS 0.098 ). Column 7 reports the re-run onsets (§ 3.1 ) and their shifts Δ from Column 6, which give εdyn=4.68×10−9 (ratio 0.98 to the original fit) and D≥4 RMS 0.062 . Column 8 reports the post-stall 5-seed geometric mean κ=v^V/α ( ×10−6 ), measured in re-runs at s≈s∗(10−12)+0.5 .
(V,D)
probed s
κ(10−6)
ratio to (8,4)
Δpred
Δmeas
(8,4)
6.0
2.77
1.00
+0.00
+0.00
(16,4)
5.0
0.51
0.18
+0.41
-0.03
(16,8)
4.5
0.53
0.19
+0.34
+0.06
(32,4)
4.0
0.10
0.037
+0.68
-0.15
(32,8)
4.0
0.047
0.017
+0.74
+0.15
(16,2)
7.0
0.018
0.0066
+1.45
+1.10
Table 3: Matched-offset test of whether εdyn tracks per-cell κα ( ε0=10−12 , n=5 seeds). κ is the 5-seed geometric mean of v^V/α at the probed s (the grid point nearest s∗(10−12)+0.5 ); Δpred is the onset shift that per-cell κα would cause, and Δmeas is the measured residual from the constant- εdyn prediction in Table 2 .
Figure 3: Left: Fast-mode sign-alternation duty cycle D(t)=Pr(rel-dM>0.05) in 100k -step windows across t∈[0,2×106] before and after a 10× learning-rate drop at step 106 ( (V,D)=(8,4) , s=6.0 , ε0=10−12 , n=3 seeds); reducing α to 10−3 quenches the (1M,2M] chatter duty cycle to 0.000 (vs. 0.172 on constant α=10−2 ). Right: Recovery error rel-dM(t) for constant α=10−2 vs. 10× step-size decay ( α:10−2→10−3 ) up to 2×106 steps across three seeds; equal-budget constant- α1 controls show a non-monotone trade-off across the two measured exponents ( 1.365× advantage for late decay at s=6.0 , vs. 1.81× disadvantage at s=5.5 ).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Hypothesis
Protocols
Outcome and Key Quantitative Finding
H1
Underflow / ULP
P1, P30, P42, P44
Refuted. fp64 sweep at (8,4) gives onsets 2.01 ( 10−2 , P44), 3.47 ( 10−4 ), 4.68 ( 10−6 ), 5.339 ( 10−8 ), 5.414 ( 10−10 ), and 5.407 ( 10−12 , de-censored shift 0.045 ; log-minus-linear max shift +0.112 ); at (16,8) , onsets are 2.242 ( 10−4 ), 3.228 ( 10−6 , P42), 3.947 ( 10−10 ), and 3.923 ( 10−12 ); 10−10→10−12 move is −0.007 ( +0.000 under log interpolation) at (8,4) and −0.024 at (16,8) . ULP roundoff SNR exceeds 108 .
H2
Landscape traps
P2
Refuted. Warm-started from Adam’s 12 stalled 60k checkpoints ( rel-dM∈[0.05,0.22] ), L-BFGS converges on 11/12 ( rel-dM≤5.3×10−10 , 7.9k–96k steps) and damped Gauss–Newton on 12/12 .
H3
Finite step budget
P3, P27–P29, P34, P37
Refuted as an ε0 explanation; power-law creep documented. On the current path ( n=10 ), Δs∗(10−12−10−10) is −0.007 ( 60k ), −0.129 ( 200k ), and +0.009 ( 600k ). Across 20 seeds ( 15504 , 15504 , 15504 5-seed subsets, 0 , 0 , 547 censored subsets dropped), 5-seed subset SDs are 0.14 , 0.18 , 0.43 (95% widths 0.625 , 0.590 , 1.235 ).
H4
1-D flip orbit
P5, P9, P10, P12, P14, P15, P41
Entry point only; two 1-D limits disclosed. Exact 1-D orbit A(u)=38α(1−38/u) for u>38 , tracked by ARMS≈1.08A∗(1−38/u) within 2.3% (cross-leg collapse 0.66% in P41). However, 1-D λstall=38ε0/α∝ε0 and ∣μ∣max=1.10>1 , so 1-D alone cannot sustain an ε0 -independent floor.
H5
Cross-mode leak
P4, P6, P7, P16, P17, P32–P40, P43, P45
Confirmed with domain boundaries. At (8,4) , v^V/α=5.65×10−6 ( v^V=5.65×10−8 ). Fitted εdyn=4.76×10−9 matches D≥4 onsets with RMS 0.098 ( 0.062 on reruns); D=2 breaks down ( +1.10 ). Six-cell κ probe (P43) confirms WALL-EQUIVALENT-SCALAR ( 4/4 ). Verified across P16, P32, P33, P35, P36, P39, P40, and P43.
Appendix
Table 4: Summary ledger of hypotheses H1–H5 and experimental protocols P1–P45.
Main-text statement
Values
§ 3 , budget test, n=5 comparison at 60k
Δs∗(10−12−10−10)=+0.004 , LOO [−0.0001,+0.0045] ; onsets 5.419 vs. 5.423
§ 3 , 600k onset under log-interpolation
5.986 , with 2 LOO folds above 6.0
§ 4.2 , walls from an unscaled κα
4.70 at (8,4) under the companion lag δAdam=0.482 ; under the lag calibrated on the ε0 -dominated rungs, δbranch=1.35=0.482+0.87 (where εeff/ε0≈21 ), 5.57 at (8,4) and 4.22 at (16,8) , with residuals +0.16 and +0.30
§ 5.1 , (δ,εdyn) ridge
εdyn from 4.76×10−9 to 10.29×10−9 over that δ range; free two-parameter fit δ=0.67 , εdyn=10.8×10−9 , RMS 0.093 ; at δ=1.0 , εdyn=45×10−9 , RMS 0.107 ; at δ=1.35 , RMS 0.146
§ 5.1 , 5-of-20 subsets at (8,4)
SD 0.43 at 600k ; the 95% subset range at 60k is 3.6× the LOO band width
§ 5.2 , item 2, β1 slope
without the β1=0.5 arm: 0.84 (that arm carries 55% of the leverage and has per-arm log-ratio spread 0.34 dex vs. median 0.14 dex)
Appendix
Table 5: Dispersion intervals and alternative fits behind main-text statements.
In low-rank matrix sensing of a rank-D factor U∗∈RV×D, replacing isotropic Gaussian designs with power-law measurement covariance Σ=diag(j−s)j=1V leaves the Burer--Monteiro landscape free of spurious local minima, yet division-normalised first-order optimizers stall above a critical exponent s∗. We trace this failure to a numerical precision wall. At the ground truth, the smallest non-gauge expected Hessian eigenvalue scales at leading order as V−(s+1)D−s. When this curvature drops below the effective denominator floor εO of optimizer O, normalisation noise overwhelms rare-direction gradients, yielding the leading-order boundary sO∗(V,D)=[log(1/εO)−logV]/log(VD)+δO, where δO is a metric-dependent detection lag. With δAdam calibrated on four configurations and frozen, this formula predicts Adam's onset within 0.45 across all 16 cells of a held-out grid (V∈{64,128,256,512}×D∈{4,8,16,32}), separates fixed-VD pairs, and distinguishes denominator-precision walls from conditioning and sign-quantisation walls. The approximation fails at D=2, reflects a finite-D spectral crossover, and saturates below ε0=10−8 at a dynamical floor.
Gradient descent on a factored model W=UV⊤ is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not. We trace the difference to the gauge symmetry of the loss, its invariance under (U,V)↦(UQ,VQ). Gradient flow's low-rank mechanism is available to an optimizer only if that optimizer is gauge-equivariant, a condition necessary for the transfer but not sufficient for low-rank recovery. Gradient descent, momentum, "shared-scalar" Adam, Muon, and Shampoo satisfy it. Adam, RMSProp, and the other coordinate-wise methods do not. A structure theorem characterizes the memoryless equivariant rules as exactly the Gram-determined left preconditioners, and a transfer theorem carries gradient flow's pathwise properties to common-scalar flows. We then sort nine update rules on underdetermined matrix sensing by recovery error against the planted ground truth. A one-parameter family from coordinate-wise to shared-scalar preconditioning restores the bias monotonically, isolating anisotropy as the cause. A "spectral schedule" reconciles two opposing reports about Muon: equal-rate updates recover exactly low-rank targets but lose their edge as the spectral tail grows. In transformers, Adam separates two gauge-equivalent initializations at the first step, where the equivariant optimizers stay at float precision, and ends with the per-head invariants WQ⊤WK 56% apart in relative Frobenius distance, a gap no per-head rotation can close. On two hyperspectral datasets at matched training loss, gradient descent cuts held-out error by 43-44% at the lowest sampling density, and at lower effective rank. Basis choice is therefore not a tuning detail but a decision about which interpolant the optimizer selects.
Devender Singh
Department of Computer Science Memorial University of Newfoundland
Across implicit-neural-representation (INR) architectures and analytic benchmarks we observe that a thoroughly tuned Adam (especially its learning rate (lr), e.g. in a hyperparameter sweep from lr=0.05 to 10−8) can potentially reach a very low loss even on ill-conditioned loss landscape or converge at a plateau far above the loss attained by second-order methods. This report defines the measured metrics that help determine if Adam can mitigate the ill-conditioning on a given loss landscape. We provide the indicators by which each outcome is determined, that are: the condition number of the Hessian and of the Adam-preconditioned Hessian D−1/2HD−1/2 (with the derivation from Adam's update rule), the diagonal mass ρ that distinguishes axis-aligned from cross-coupled ill-conditioning, the negative spectral mass estimated by stochastic Lanczos quadrature, and the gradient energy fractions over curvature bands, including the flat fraction that indicates the Adam stall. A worked out 2×2 example and an illustration show the reasons why a diagonal preconditioning by Adam can remove axis-aligned ill-conditioning by rescaling and why it cannot do the same if the ill-conditioning is cross coupled. In addition, we present a case study of FINER image fitting architecture that goes over the whole loss landscape analysis framework: the fitting architecture description, reasons due to which its landscape stalls Adam at saddles, the measured PSNR values through our tuned baselines to the 120--134,dB results of the blockwise second order methods, the error maps behind those numbers, and description of the benefits such image fitting accuracy gives in practice.