A Self-Generated Floor on Adam's $\varepsilon$: Evidence from Power-Law Matrix Sensing

Lngel Kyo

cs.LGNov 1, 2026

A Self-Generated Floor on Adam's $\varepsilon$ : Evidence from Power-Law Matrix Sensing

Authors: Lngel Kyo

Organizations: A2O Labs

Abstract

In nonconvex Burer--Monteiro low-rank matrix sensing of $U^\ast \in \mathbb{R}^{V \times D}$ under power-law covariance $Σ= \mathrm{diag}(j^{-s})_{j=1}^V$ , the precision-wall approximation predicts that Adam stalls when the smallest non-gauge Hessian eigenvalue drops below its denominator floor, giving $s^\ast(V,D) = [\log(1/\varepsilon_{\mathrm{eff}}) - \log V]/\log(VD) + δ_{\mathrm{Adam}}$ . If $\varepsilon_{\mathrm{eff}}$ were set by $\varepsilon_0$ , lowering $\varepsilon_0$ from $10^{-10}$ to $10^{-12}$ would raise $s^\ast$ by $0.95\text{--}1.33$ . Instead, across six values of $\varepsilon_0$ two decades apart, $10^{-2}$ to $10^{-12}$ , the boundary follows the $\varepsilon_0$ slope down to $10^{-6}$ , bends near $10^{-8}$ , and is flat below $10^{-8}$ , while warm-started L-BFGS and Gauss--Newton reach machine precision on stalled instances. We trace this saturation to a self-generated dynamical floor $\varepsilon_{\mathrm{dyn}} \propto α$ : high-curvature modes oscillate persistently near the adaptive edge of stability with amplitude $\propto α(1-β_1)/(1+β_1)$ and leak gradient fluctuations of scale $κα$ into rare modes via measurement coupling, pinning $\sqrt{\hat{v}_V} \approx κα\gg \varepsilon_0$ . Fitting a single threshold-crossing wall parameter $\varepsilon_{\mathrm{dyn}} = 4.76\times 10^{-9}$ (with the detection lag frozen at its companion calibration) predicts the saturated failure boundary for $D \ge 4$ across five $(V,D)$ scales to RMS $0.098$ (breaking down at $D = 2$ ), while shrinking $\varepsilon_0$ below $10^{-8}$ is ineffective and lowering the step size $α$ reduces the floor directly.

Figures & tables

Figure 1: (a) Empirical failure onset

s^{\ast}

60\text{k}

steps vs. Adam’s hyperparameter

\varepsilon_{0}

across six values of

\varepsilon_{0}

two decades apart (

10^{-2}

10^{-12}

) in

\mathrm{fp64}

(

n=5

seeds, evaluated with the seed-mean linear-interpolation onset estimator of § 2 ):

(V,D)=(8,4)

across all

6

values (

s^{\ast}=2.01,3.47,4.68,5.339,5.414,5.407

) and

(16,8)

10^{-4},10^{-6},10^{-10},10^{-12}

(

s^{\ast}=2.242,3.228,3.947,3.923

), shown with the

\varepsilon_{0}

-law reference slopes (

+1.33

and

+0.95

per two decades, anchored at

10^{-4}

with extrapolations

2.14

(8,4),10^{-2}

and

3.191

(16,8),10^{-6}

) and saturated plateaus. Filled markers are the original sweeps; the two open markers (

(8,4)

10^{-2}

;

(16,8)

10^{-6}

, on a grid extended beyond the original

s\leq 3.0

) are re-runs with the released code, which evaluates the Adam step in a different floating-point order (§ 3.1 ; Appendix B ); this panel shows that

s^{\ast}

tracks the

\varepsilon_{0}

slope from

10^{-2}

10^{-6}

, bends near

10^{-8}

, and saturates below

10^{-8}

. (b) Endpoint relative error

\mathrm{rel}\text{-}d_{M}

60\text{k}

steps vs. spectral exponent

s

(V,D)=(8,4)

in re-runs with the released code for

\varepsilon_{0}=10^{-10}

vs.

10^{-12}

across matched problem instances (

n=5

seeds), showing that the two curves superimpose at threshold crossing rather than shifting by the predicted

+1.33

$T$	$s^{\ast}(10^{-10})$	$s^{\ast}(10^{-12})$	$\Delta s^{\ast}(12{-}10)$	LOO of $\Delta s^{\ast}$	$s^{\ast}(10^{-12})$ ( $n{=}20$ )	5-of-20 95% (SD)
$60\text{k}$	5.62	5.61	$-0.007$	$[-0.02,\,+0.00]$	5.60	$[5.37,\,5.99]$ (0.14)
$200\text{k}$	5.82	5.69	$-0.129$	$[-0.28,\,+0.02]$	5.76	$[5.55,\,6.14]$ (0.18)
$600\text{k}$	6.12	6.12	$+0.009$	$[-0.12,\,+0.11]$	5.56	$[5.07,\,6.30]$ (0.43)

Table 1: Multi-budget test of

\varepsilon_{0}

-insensitivity at

(V,D)=(8,4)

. While the absolute onset

s^{\ast}(T)

creeps upward with step budget

T

, the matched-seed difference

\Delta s^{\ast}(10^{-12}-10^{-10})

remains near zero at every budget (against the two-decade

\varepsilon_{0}

-law prediction of

+1.33

). Columns 2–5 use seeds 0–9 (

n=10

); the

(8,4)

row of Table 2 uses seeds 0–4 and gives

5.407

in the original runs and

5.423

in the re-runs, both inside the

60\text{k}

5-of-20 range below, so the two tables differ by seed set, not by estimator. Columns 6–7 report the

n=20

onset and the 95% range across all

15504

five-seed subsets at

\varepsilon_{0}=10^{-12}

(each subset onset uses the same seed-mean linear-interpolation estimator as every other onset in this paper, § 2 ; at

(8,4)

, the 5-of-20 subset SD is

0.14

60\text{k}

and

0.43

600\text{k}

with

547

censored subsets dropped at

600\text{k}

, Appendix B , item 2).

Figure 2: (a) Verification of the amplitude law ( 7 ) across

(\varepsilon_{0},\alpha,\lambda)

spanning

10000\times

\varepsilon_{0}

and

100\times

\alpha

: rescaled time-averaged RMS amplitude

A_{\mathrm{RMS}}/A^{\ast}

on the burst-and-relaxation attractor collapses onto the empirical tracking curve

1.08\,(1-38/u)

across

\beta_{2}\in\{0.99,0.999\}

(within

2.3\%

, vs.

10.5\%

against the unscaled exact period-2 orbit

1-38/u

; worst cross-leg spread

0.66\%

at matched

u

; Observation 4.2 ). (b) Per-step log-envelope slope of the flip amplitude vs.

u=\alpha\lambda/\varepsilon_{0}

(stable below, unstable/escaping above the stability threshold

u_{c2}=38

$V$	$D$	$VD$	$m$	pred. $s^{\ast}_{\mathrm{plateau}}$	$s^{\ast}(10^{-12})$	re-run $s^{\ast}(10^{-12})$ ( $\Delta$ )	$\kappa\ (10^{-6})$	domain
8	4	32	48	5.41	5.407	5.423 ( $+0.016$ )	2.77	$D\geq 4$
16	4	64	96	4.42	4.391	4.515 ( $+0.124$ )	0.51	$D\geq 4$
16	8	128	192	3.86	3.923	3.889 ( $-0.034$ )	0.53	$D\geq 4$
32	4	128	192	3.72	3.571	3.650 ( $+0.079$ )	0.10	$D\geq 4$
32	8	256	384	3.31	3.459	3.239 ( $-0.220$ )	0.047	$D\geq 4$
16	2	32	48	5.21	6.315	6.316 ( $+0.001$ )	0.018	$D=2$ ( $\Delta=+1.10$ )

Table 2: Multi-scale validation of the saturated plateau prediction ( 11 ) at

\varepsilon_{0}=10^{-12}

(

n=5

seeds,

60\text{k}

steps; each onset is the seed-mean linear-interpolation onset of § 2 over seeds 0–4, and at

(8,4)

the 5-of-20 subset SD is

0.14

60\text{k}

and

0.43

600\text{k}

, Appendix B , item 2). Here

\delta_{\mathrm{Adam}}=0.482

is frozen from the companion

\varepsilon_{0}=10^{-8}

calibration and

\varepsilon_{\mathrm{dyn}}=4.76\times 10^{-9}

is fitted across the five

D\geq 4

rows of the original runs (RMS

0.098

). Column 7 reports the re-run onsets (§ 3.1 ) and their shifts

\Delta

from Column 6, which give

\varepsilon_{\mathrm{dyn}}=4.68\times 10^{-9}

(ratio

0.98

to the original fit) and

D\geq 4

RMS

0.062

. Column 8 reports the post-stall 5-seed geometric mean

\kappa=\sqrt{\hat{v}_{V}}/\alpha

(

\times 10^{-6}

), measured in re-runs at

s\approx s^{\ast}(10^{-12})+0.5

$(V,D)$	probed $s$	$\kappa\ (10^{-6})$	ratio to $(8,4)$	$\Delta_{\mathrm{pred}}$	$\Delta_{\mathrm{meas}}$
$(8,4)$	6.0	2.77	1.00	+0.00	+0.00
$(16,4)$	5.0	0.51	0.18	+0.41	-0.03
$(16,8)$	4.5	0.53	0.19	+0.34	+0.06
$(32,4)$	4.0	0.10	0.037	+0.68	-0.15
$(32,8)$	4.0	0.047	0.017	+0.74	+0.15
$(16,2)$	7.0	0.018	0.0066	+1.45	+1.10

Table 3: Matched-offset test of whether

\varepsilon_{\mathrm{dyn}}

tracks per-cell

\kappa\alpha

(

\varepsilon_{0}=10^{-12}

n=5

seeds).

\kappa

is the 5-seed geometric mean of

\sqrt{\hat{v}_{V}}/\alpha

at the probed

s

(the grid point nearest

s^{\ast}(10^{-12})+0.5

);

\Delta_{\mathrm{pred}}

is the onset shift that per-cell

\kappa\alpha

would cause, and

\Delta_{\mathrm{meas}}

is the measured residual from the constant-

\varepsilon_{\mathrm{dyn}}

prediction in Table 2 .

Figure 3: Left: Fast-mode sign-alternation duty cycle

D(t)=\Pr(\mathrm{rel}\text{-}d_{M}>0.05)

100\text{k}

-step windows across

t\in[0,2\times 10^{6}]

before and after a

10\times

learning-rate drop at step

10^{6}

(

(V,D)=(8,4)

s=6.0

\varepsilon_{0}=10^{-12}

n=3

seeds); reducing

\alpha

10^{-3}

quenches the

(1\text{M},2\text{M}]

chatter duty cycle to

0.000

(vs.

0.172

on constant

\alpha=10^{-2}

). Right: Recovery error

\mathrm{rel}\text{-}d_{M}(t)

for constant

\alpha=10^{-2}

vs.

10\times

step-size decay (

\alpha:10^{-2}\to 10^{-3}

) up to

2\times 10^{6}

steps across three seeds; equal-budget constant-

\alpha_{1}

controls show a non-monotone trade-off across the two measured exponents (

1.365\times

advantage for late decay at

s=6.0

, vs.

1.81\times

disadvantage at

s=5.5

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Hypothesis	Protocols	Outcome and Key Quantitative Finding
H1	Underflow / ULP	P1, P30, P42, P44	Refuted. $\mathrm{fp64}$ sweep at $(8,4)$ gives onsets $2.01$ ( $10^{-2}$ , P44), $3.47$ ( $10^{-4}$ ), $4.68$ ( $10^{-6}$ ), $5.339$ ( $10^{-8}$ ), $5.414$ ( $10^{-10}$ ), and $5.407$ ( $10^{-12}$ , de-censored shift $0.045$ ; log-minus-linear max shift $+0.112$ ); at $(16,8)$ , onsets are $2.242$ ( $10^{-4}$ ), $3.228$ ( $10^{-6}$ , P42), $3.947$ ( $10^{-10}$ ), and $3.923$ ( $10^{-12}$ ); $10^{-10}\to 10^{-12}$ move is $-0.007$ ( $+0.000$ under log interpolation) at $(8,4)$ and $-0.024$ at $(16,8)$ . ULP roundoff SNR exceeds $10^{8}$ .
H2	Landscape traps	P2	Refuted. Warm-started from Adam’s $12$ stalled $60\text{k}$ checkpoints ( $\mathrm{rel}\text{-}d_{M}\in[0.05,0.22]$ ), L-BFGS converges on $11/12$ ( $\mathrm{rel}\text{-}d_{M}\leq 5.3\times 10^{-10}$ , $7.9\text{k}\text{--}96\text{k}$ steps) and damped Gauss–Newton on $12/12$ .
H3	Finite step budget	P3, P27–P29, P34, P37	Refuted as an $\varepsilon_{0}$ explanation; power-law creep documented. On the current path ( $n=10$ ), $\Delta s^{\ast}(10^{-12}-10^{-10})$ is $-0.007$ ( $60\text{k}$ ), $-0.129$ ( $200\text{k}$ ), and $+0.009$ ( $600\text{k}$ ). Across $20$ seeds ( $15504$ , $15504$ , $15504$ 5-seed subsets, $0$ , $0$ , $547$ censored subsets dropped), 5-seed subset SDs are $0.14$ , $0.18$ , $0.43$ (95% widths $0.625$ , $0.590$ , $1.235$ ).
H4	1-D flip orbit	P5, P9, P10, P12, P14, P15, P41	Entry point only; two 1-D limits disclosed. Exact 1-D orbit $A(u)=\frac{\alpha}{38}(1-38/u)$ for $u>38$ , tracked by $A_{\mathrm{RMS}}\approx 1.08A^{\ast}(1-38/u)$ within $2.3\%$ (cross-leg collapse $0.66\%$ in P41). However, 1-D $\lambda_{\mathrm{stall}}=38\varepsilon_{0}/\alpha\propto\varepsilon_{0}$ and $\|\mu\|_{\max}=1.10>1$ , so 1-D alone cannot sustain an $\varepsilon_{0}$ -independent floor.
H5	Cross-mode leak	P4, P6, P7, P16, P17, P32–P40, P43, P45	Confirmed with domain boundaries. At $(8,4)$ , $\sqrt{\hat{v}_{V}}/\alpha=5.65\times 10^{-6}$ ( $\sqrt{\hat{v}_{V}}=5.65\times 10^{-8}$ ). Fitted $\varepsilon_{\mathrm{dyn}}=4.76\times 10^{-9}$ matches $D\geq 4$ onsets with RMS $0.098$ ( $0.062$ on reruns); $D=2$ breaks down ( $+1.10$ ). Six-cell $\kappa$ probe (P43) confirms WALL-EQUIVALENT-SCALAR ( $4/4$ ). Verified across P16, P32, P33, P35, P36, P39, P40, and P43.

Appendix

Table 4: Summary ledger of hypotheses H1–H5 and experimental protocols P1–P45.

Main-text statement	Values
§ 3 , budget test, $n=5$ comparison at $60\text{k}$	$\Delta s^{\ast}(10^{-12}-10^{-10})=+0.004$ , LOO $[-0.0001,+0.0045]$ ; onsets $5.419$ vs. $5.423$
§ 3 , $600\text{k}$ onset under log-interpolation	$5.986$ , with $2$ LOO folds above $6.0$
§ 4.2 , walls from an unscaled $\kappa\alpha$	$4.70$ at $(8,4)$ under the companion lag $\delta_{\mathrm{Adam}}=0.482$ ; under the lag calibrated on the $\varepsilon_{0}$ -dominated rungs, $\delta_{\mathrm{branch}}=1.35=0.482+0.87$ (where $\varepsilon_{\mathrm{eff}}/\varepsilon_{0}\approx 21$ ), $5.57$ at $(8,4)$ and $4.22$ at $(16,8)$ , with residuals $+0.16$ and $+0.30$
§ 5.1 , $(\delta,\varepsilon_{\mathrm{dyn}})$ ridge	$\varepsilon_{\mathrm{dyn}}$ from $4.76\times 10^{-9}$ to $10.29\times 10^{-9}$ over that $\delta$ range; free two-parameter fit $\delta=0.67$ , $\varepsilon_{\mathrm{dyn}}=10.8\times 10^{-9}$ , RMS $0.093$ ; at $\delta=1.0$ , $\varepsilon_{\mathrm{dyn}}=45\times 10^{-9}$ , RMS $0.107$ ; at $\delta=1.35$ , RMS $0.146$
§ 5.1 , 5-of-20 subsets at $(8,4)$	SD $0.43$ at $600\text{k}$ ; the 95% subset range at $60\text{k}$ is $3.6\times$ the LOO band width
§ 5.2 , item 2, $\beta_{1}$ slope	without the $\beta_{1}=0.5$ arm: $0.84$ (that arm carries $55\%$ of the leverage and has per-arm log-ratio spread $0.34$ dex vs. median $0.14$ dex)

Appendix

Table 5: Dispersion intervals and alternative fits behind main-text statements.

Adaptive Gradient Methods

Adaptive Gradient Methods adjust updates using gradient history, curvature, or noise estimates to improve optimization across diverse problems.

See all 169 papers

The Precision Wall: An Optimizer-Dependent Boundary for Optimization under Power-Law Measurement Spectra
Oct 29, 2026Lngel Kyo
A2O Labs
The Loss Does Not See the Basis, but Adam Does
Aug 5, 2026Devender SinghLow-Rank Matrix Decomposition Matrix Optimization
Department of Computer Science Memorial University of Newfoundland
Loss Landscape Features That Make Adam Stall: Definitions, Estimators, and the Preconditioned Hessian View
Aug 23, 2026Rodion PodorozhnyLoss Landscape Adaptive Gradient Methods
Texas State University

Oct 29, 2026J/K move · Enter open · S save

The Precision Wall: An Optimizer-Dependent Boundary for Optimization under Power-Law Measurement Spectra

Lngel Kyo

A2O Labs

Open paper AskPDF arXiv

In low-rank matrix sensing of a rank-

D

factor

U^\ast \in \mathbb{R}^{V \times D}

, replacing isotropic Gaussian designs with power-law measurement covariance

Σ= \mathrm{diag}(j^{-s})_{j=1}^V

leaves the Burer--Monteiro landscape free of spurious local minima, yet division-normalised first-order optimizers stall above a critical exponent

s^\ast

. We trace this failure to a numerical precision wall. At the ground truth, the smallest non-gauge expected Hessian eigenvalue scales at leading order as

V^{-(s+1)}D^{-s}

. When this curvature drops below the effective denominator floor

ε_O

of optimizer

O

, normalisation noise overwhelms rare-direction gradients, yielding the leading-order boundary

s^\ast_O(V, D) = [\log(1/ε_O) - \log V]/\log(VD) + δ_O

, where

δ_O

is a metric-dependent detection lag. With

δ_{\mathrm{Adam}}

calibrated on four configurations and frozen, this formula predicts Adam's onset within

0.45

across all 16 cells of a held-out grid (

V \in \{64,128,256,512\} \times D \in \{4,8,16,32\}

), separates fixed-

VD

pairs, and distinguishes denominator-precision walls from conditioning and sign-quantisation walls. The approximation fails at

D = 2

, reflects a finite-

D

spectral crossover, and saturates below

\varepsilon_0 = 10^{-8}

at a dynamical floor.

Abstract

Adaptive Gradient Methods

Explore similar work

The Precision Wall: An Optimizer-Dependent Boundary for Optimization under Power-Law Measurement Spectra

The Loss Does Not See the Basis, but Adam Does

Loss Landscape Features That Make Adam Stall: Definitions, Estimators, and the Preconditioned Hessian View