Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence

Rodion Podorozhny

cs.LGAug 22, 2026

Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence

Authors: Rodion Podorozhny

Organizations: Texas State University

Abstract

Overparameterized networks often have ill-conditioned, saddle rich loss landscapes. They are usually trained with first-order methods that struggle on those landscapes, while many second-order methods stay infeasible at scale. Some scalable cubic methods replace the Hessian by a computationally cheap approximation and drop the exact block curvature. This paper introduces ARC-block, a blockwise adaptive regularization with cubics (ARC) optimizer. For each parameter tensor it minimizes a cubic model over the true block Hessian, with a per-block adaptive cubic constant and a monotone guard on the full loss. Large tensors are handled matrix-free in a Lanczos-built Krylov subspace. ARC-block's CubicKrylov subsolver's descent is monotone with a per-block cubic decrease bound. For exact steps, and for Krylov steps that meet the Cartis--Gould--Toint residual test, we provide an $\mathcal{O}(\sqrt{B}\varepsilon^{-3/2})$ sweep bound above a cross-block coupling floor. Another subsolver, ARC- $\varphi_1$ , uses a trust-controlled exponential relaxation step. Among the cubic Newton methods evaluated here, these are the only ones whose steps stay exact on every parameter block (within the Krylov subspace), including on a $91.4$ M-parameter implicit neural representation (INR). Run to convergence on FINER 2D image-fitting INR network, ARC- $\varphi_1$ reaches $133.5$ dB peak signal-to-noise ratio (PSNR), with no post peak collapse. Tuned Adam peaks at $78.2$ dB at $2{,}629$ s (step ${\sim}625$ k, about $44$ minutes), further steps degrade its PSNR. On the same benchmark, SOAP, L-BFGS, and the scalable diagonal cubic method AdaCubic stay at or below Adam's $78.2$ dB peak. Over the benchmark's 20 images, run to a plateau rule, ARC- $\varphi_1$ 's per-image PSNR has median $132.4$ and mean $131.5\pm3.3$ dB against tuned Adam's $72.8\pm4.5$ dB.

Figures & tables

Figure 1: FINER on ESM image 0 in equal wall clock time of $2$ h, $\log_{10}$ absolute error on a shared scale. ARC- $\varphi_{1}$ at Lanczos depth $L{=}25$ reaches $125.5$ dB in $8{,}547$ sweeps. Tuned Adam converges to $78.2$ dB ( $10^{6}$ steps, $4{,}214$ s). AdaCubic ( Tsingalis et al., 2026 ) with its authors’ universal hyperparameters reaches $46.4$ dB at $6{,}328$ s and then oscillates (Appendix I.5 ). SOAP ( Vyas et al., 2024 ) converges at $77.65$ dB and L-BFGS stays at $7.3$ dB at the initialization saddle (Table 6 ). At its tuned depth $L{=}163$ , ARC- $\varphi_{1}$ converges to $133.5$ dB ( $2{,}824$ sweeps, $17{,}173$ s). The faint bright filament along the warm/cold front boundary in the Adam panel is signal dependent error.

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: The DSO relaxation function. Left, $R_{s}(\lambda)=P_{s}(\lambda)/s$ for $s=3,4,5$ on the normalized spectrum. Each curve has $R_{s}(0)=1$ , oscillates with small amplitude on the interior of $[0,1]$ , returns to $(-1)^{s-1}$ at $\lambda=1$ , and grows without bound for $\lambda<0$ . Right, the sinc function $\varphi(x)=\sin(x)/x$ that governs $R_{s}$ near both ends of the spectrum through $x=2s\sqrt{\lambda}$ . Its first minimum is $\varphi(x_{1})\approx-0.217$ at $x_{1}\approx 4.49$ , and $|\varphi(x)|\leq 0.22$ for all $x\geq x_{\mathrm{cr}}\approx 2.55$ (shaded). Reaching $\lambda_{\min}=1/\kappa$ with $2L\sqrt{\lambda_{\min}}\geq x_{\mathrm{cr}}$ gives the degree rule $L\geq(x_{\mathrm{cr}}/2)\sqrt{\kappa}\approx 1.28\sqrt{\kappa}$ of Remark C.1 .

Step	Build	Solve	Cost
Cheby-ON (DSO step)	full $\mathbf{H}_{b}$ every step ( $n_{b}$ gevals)	degree- $L$ recurrence, $\mathcal{O}(Ln_{b}^{2})$	$n_{b}+1$
Cheby-OFF (Newton)	full $\mathbf{H}_{b}$ every step	linalg.solve , $\mathcal{O}(n_{b}^{3})$	$n_{b}+1$
Block-CN-Lazy (eigh)	full $\mathbf{H}_{b}$ every $p_{b}$ steps	cached eigh + secular, $\mathcal{O}(n_{b}^{2})$	$n_{b}/p_{b}+1$
CubicKrylov	none	$L{+}1$ HVPs + tridiagonal secular	$\mathbf{L+3}$

Appendix

Table 1: Per-step, per-block cost of the step variants, block size

n_{b}

, Lanczos depth

L

(bounded by the Chebyshev degree analysis), laziness

p_{b}

. The gradient equivalent count follows the ViSIR ablation experiment. One gradient, one HVP, and one function evaluation each cost one, and the CubicKrylov HVPs are

L

for the basis and one for the model decrease.

Large block step	best PSNR (dB)	gevals	wall (s)	notes
Damped Newton (DSO-ablation fallback)	20.73	26.8k	11	reference
CubicKrylov, fixed $M{=}6L_{H}$ , stale grad	25.91	16.1k	14	non-monotone, no phase-B safeguard
+ adaptive $M$ , joint accept-or-reject	25.12	28.7k	16	319 rejections; $M$ drifts to $596$
+ two-phase accept-or-reject (B only)	23.13	31.9k	16	stale gradient, 875 rejections
+ fresh gradient in phase B	31.66	17.7k	15	104 rejections; every $M_{b}$ at floor
Reg. Newton lazy, $m{=}n$ ( Doikov et al., 2023 )	31.69	30.2k	457	1 Hessian build, 15k steps

Appendix

Table 2: Nano A/B comparison of large block step variants and full Hessian references (single image experiment, image #0, seed 42). Mechanism rows run 100 steps. The reference rows below the second rule are full Hessian methods granted twice the CubicKrylov oracle budget, 35.3k gevals. Rows exceeding it do so because a single dense

15\mathrm{k}\times 15\mathrm{k}

Hessian build, 15.2k gevals, is atomic. The three mechanism rows correspond to the design decisions (a)–(c) and the fresh-gradient rule of Section 4 .

Method	PSNR (dB)	final loss	LPIPS	wall (s) †
CubicKrylov, degree $L{=}10$ in Krylov subspace on large blocks	51.65	$\mathbf{6.7\times 10^{-6}}$	$\mathbf{0.000}$	148
Cheby-ON $n{=}5$ (Chebyshev-2)	41.75	$2.0\times 10^{-4}$	0.001	80
Cheby-OFF $n{=}5$ (damped Newton)	22.81	$8.3\times 10^{-3}$	0.322	67
Block-CN-Lazy, damped Newton on large blocks	22.25	$8.9\times 10^{-3}$	0.426	76
SOAP	20.33	$1.3\times 10^{-2}$	0.374	5.7
Adam	15.65	0.142	1.321	1.3

Appendix

Table 3: Full ViSIR single image ablation (91,383,042 parameters, ESM image #1, 100 steps). † Wall clock is the 100-step time on a single idle RTX 6000 (Blackwell). The Chebyshev-ON and Chebyshev-OFF rows are an implementation of the DSO step following Appendix C and of its damped Newton counterpart under the same experimental setup. LPIPS for Adam, Muon+AdamW, and SOAP and the final loss, LPIPS, and wall clock entries of the damped Newton large block row were measured in a second run under the identical setup (image #1, seed 42, 100 steps, idle GPU). The PSNRs of the two runs agree (Adam and SOAP exactly, Muon+AdamW 6.43 against 5.73 dB and damped Newton large blocks 21.98 against 22.25 dB, within run-to-run variation).

Task	$\kappa_{\mathrm{raw}}$	$\kappa_{\mathrm{Adam}}$	red.	$\rho$	neg. mass	flat frac	outcome
35k spectral	4.9e2	3.1e1	16 $\times$	.04	.92	.000	Adam at precision floor
SIREN 2D	1.6e3	9.9e1	16 $\times$	.09	.09	.000	parity
WIRE 2D	1.2e5	8.4e2	143 $\times$	.65	.00	.000	parity
FINER 2D	5.4e4	6.1e1	887 $\times$	.98	.19 (.81 init)	.000	ARC $+59.5$ dB
SDF-SIREN	8.9e3	2.4e1	372 $\times$	.06	.67	.000	parity; CubicKrylov best test
SDF-FINER	1.0e3	6.9	146 $\times$	.58	.68	.001	Adam chamfer $4.6$ – $5.5\times$ worse

Appendix

Table 4: Landscape fingerprints along the Adam trajectory (late trajectory values, initialization in parentheses).

\dagger

marks an indefinite spectrum, where (

\lambda_{\max}\approx-\lambda_{\min}

) makes the raw ratio uninformative.

\kappa_{\mathrm{Adam}}

is the residual conditioning. Outcomes compare plateau-converged tuned first-order methods (Adam, SOAP) with the blockwise cubic methods. INR rows use 240

\times

240 image regression and Thai-statue SDF fitting with 3

\times

256 coordinate MLPs. The FINER 2D row is ESM image 0. Over all 20 ESM images the step-

10^{5}

values span

\kappa_{\mathrm{raw}}=1.5\times 10^{4}

–

5.1\times 10^{5}

\kappa_{\mathrm{Adam}}=4.6

–

2.6\times 10^{3}

(median

1.2\times 10^{2}

\rho=0.95

–

0.99

, flat fraction

\leq 0.001

, and the initial negative mass spans

0.57

–

0.89

(median

0.81

optimizer	Low	Mid-Low	Mid-High	High	$H/L$
ARC- $\varphi_{1}$	$\mathbf{2.41{\times}10^{-2}}$	$\mathbf{2.21{\times}10^{-2}}$	$\mathbf{2.19{\times}10^{-2}}$	$\mathbf{2.18{\times}10^{-2}}$	1.10
ARC-block (cubic)	$3.94{\times}10^{-2}$	$3.57{\times}10^{-2}$	$3.52{\times}10^{-2}$	$3.49{\times}10^{-2}$	1.07
Adam (source paper experimental setup)	$6.61{\times}10^{-2}$	$6.46{\times}10^{-2}$	$6.46{\times}10^{-2}$	$6.48{\times}10^{-2}$	0.81
Adam (untuned)	$2.1{\times}10^{1}$	$1.8{\times}10^{1}$	$1.8{\times}10^{1}$	$1.7{\times}10^{1}$	0.37

Appendix

Table 5: FINER, 500 steps, mean residual FFT power per band over the 60 grid cells.

H/L

= median high/low band ratio. Values near 1 indicate a band uniform residual.

Optimizer	$421$ s full-budget run	$4{,}207$ s full-budget run
Adam (tuned)	$67.7$ dB	$78.2$ dB
SOAP (Adam’s lr)	$\approx 72.9$ dB	$\approx 77.3$ dB
L-BFGS ( $m{=}10,20,50$ )	$7.3$ dB	$7.3$ dB
ARC- $\varphi_{1}$ ( $L_{\max}{=}163$ )	$58.3$ dB	$\mathbf{95.6}$ dB
AdaCubic (authors’ settings)	$41.3$ dB	$46.1$ dB

Appendix

Table 6: Iso-time comparison at Adam’s two full-budget runs, FINER ESM image 0. SOAP values are linearly interpolated between its logged checkpoints. AdaCubic values are the best PSNR reached within each wall clock allowance on its own

2

h run. All others are direct readings.

image	best	sweep	s	best	step	s	final	best	step	final
	ARC- $\varphi_{1}$			Adam				AdaCubic
0	132.64	1994	10582	78.16	615983	2444	63.18	46.36	374124	38.11
1	135.61	1980	10327	77.19	975354	3890	67.61	46.69	417803	42.64
2	116.08	2000	11002	70.73	985679	3926	63.87	46.93	377292	35.68
3	132.58	1821	10321	71.47	998731	3986	64.61	46.65	466877	39.27
4	108.73	2000	10934	67.47	943498	3762	57.26	46.34	404278	20.94

Appendix

Table 7: 20 ESM images, seed 0. Best PSNR (dB) with the sweep or step and the wall time (s) at which it was reached. “final” is the value at the end of the run. ARC-

\varphi_{1}

ran with

L_{\max}=163

for

2{,}000

sweeps, Adam for

10^{6}

steps, and AdaCubic for

2

h. Medians are over the 20 images.

\ast

marks image 19, whose Adam run shared its GPU with an unrelated job for its whole duration (

194

against

{\sim}256

steps/s on the other images). Its step count and PSNR are unaffected, but its wall time is not comparable.

image	at 2k	continued	sweep	s	h	cap	Adam	gain
0	132.64	133.49	2425	12703	4.10	12 h	78.16	55.3
2	116.08	131.80	4001	26042	7.93	12 h	70.73	61.1
4	108.73	128.83	4048	23556	6.72	12 h	67.47	61.4
8	84.34	128.50	12007	71662	19.97	48 h	78.40	50.1
9	78.65	119.69	13398	74650	20.76	48 h	77.60	42.1
10	101.86	129.81	5050	27433	8.21	12 h	63.44	66.4

Appendix

Table 8: Continuation of the eleven ARC-

\varphi_{1}

runs whose best value was still moving at

2{,}000

sweeps, same configuration from initialization, stopped by the plateau rule (less than

0.05

dB gained over

400

sweeps, after at least

4

h). “at 2k” is the

2{,}000

-sweep value of Table 7 . “sweep” and “s” locate the continued best. “h” is the run’s wall time. “cap” is the wall clock limit under which the run plateaued. “gain” is the paired margin over Adam’s best on the same image. Images 8, 9, and 16 first ran under the

12

h cap, which stopped them, still rising, at

117.2

107.6

, and

123.5

dB. Their rows give the reruns under the

48

h cap.

lr	$0.05$	$0.03$	$0.01$	$3{\times}10^{-3}$	$10^{-3}$	$3{\times}10^{-4}$	$10^{-4}$
best total MSE	$2.1{\times}10^{-3}$	$0.587$	$0.591$	$5.3{\times}10^{-6}$	$5.3{\times}10^{-7}$	$3.0{\times}10^{-13}$	$6.3{\times}10^{-14}$
steps	5000 ep.	60k	60k	60k	60k	60k	76k
wall (s)	—	163	168	170	169	164	212

Appendix

Table 9: Adam on the 35k SIREN-like benchmark hyperparameter sweep

benchmark (sweeps)	initial loss	ARC- $\varphi_{1}$	ARC-block (cubic)
Rosenbrock–Ackley (300)	$1.98{\times}10^{4}$	$\mathbf{3.4{\times}10^{-2}}$ (69.4k)	$1.37$ (21.4k)
Chebyshev–Rosenbrock (300)	$1.30{\times}10^{4}$	$\mathbf{7.8{\times}10^{-3}}$ (69.8k)	$0.73$ (21.4k)
multi-saddle (300)	$7.51{\times}10^{3}$	$\mathbf{2.2{\times}10^{-3}}$ (67.0k)	$2.5{\times}10^{-2}$ (29.4k)
mf-SIREN 35k (40)	$4.61$	$\mathbf{4.1{\times}10^{-10}}$ (18.6k)	$1.3{\times}10^{-4}$ (9.6k)

Appendix

Table 10: Small scale benchmarks, terminal loss after the stated sweeps (CPU, matched budgets, gradient equivalents in parentheses). ARC-

\varphi_{1}

uses

\sigma_{0}{=}1

c_{h}{=}3

L{=}15

. The reference is ARC-block (CubicKrylov step) with

\sigma_{0}{=}1

L_{\max}{=}15

Method	Curvature in the step	Demonstrated scale	Exact cubic model step on true curvature
AdaCubic ( Tsingalis et al., 2026 )	Hutchinson diagonal	deep learning training; FINER/ESM here, $46.4$ dB at $2$ h (Appendix I.5 )	no: diagonal surrogate
Block-Kronecker Fisher ( Martins Gomes, 2025 )	per-layer Kronecker factors	deep learning training	no: factored surrogate
Full Gauss–Newton study ( Abreu et al., 2026 )	full or layerwise Gauss–Newton	150M parameter transformers	outside the cubic Newton family; supports per-tensor blocking
Quasi-Newton ARC ( Forristal et al., 2022 ; Ranganath et al., 2025 )	limited-memory low-rank	deep network training	exact solve, but on a surrogate Hessian
True-Hessian subspace ARC ( Bellavia et al., 2025 ; Al Daas and Gould, 2025 ; Dussault et al., 2023 ; Cartis et al., 2025 ; Tansley and Cartis, 2025 )	exact Hessian (subspace or shifted solves)	standard test sets, moderate dimension	not demonstrated at network scale
Stochastic, sketched, and multilevel cubic Newton ( Pasechnyuk-Vilensky et al., 2025 ; Scheinberg and Xie, 2023 ; Chen et al., 2026 ; Higuchi et al., 2025 ; Tsipinakis et al., 2026 )	sampled, sketched, or compressed	small to moderate	subspace or compression replaces the exact model

Appendix

Table 11: Curvature model, demonstrated scale, and cubic model exactness of the second-order families reviewed in this section. Exactness is meant in the sense of Lemma A.4 , that the step minimizes the cubic model over the true (block) Hessian, accessed through Hessian–vector products, within the constructed subspace.

Inner solver	Methods	Computation per solve	Working memory
generalized Lanczos trust region (GLTR) tridiagonalization + secular solve	Cartis et al. (2011a) ; Gould et al. (1999) ; Carmon and Duchi (2018)	$L$ HVPs + $\mathcal{O}(L)$ secular iteration	basis of $L$ $n$ -vectors, or a second Lanczos pass with $\mathcal{O}(1)$ vectors
Multi-shift Lanczos conjugate gradient (CG) over a shift grid	Dussault et al. (2023)	one Krylov process, $L$ HVPs shared by all shifts	$\mathcal{O}(1)$ $n$ -vectors per shift; no stored basis
Reused polynomial/rational Krylov subspace	Bellavia et al. (2025)	basis built once, reused across outer iterations; rational variant needs shifted direct solves	stored basis; factors for the rational variant
Extended-Krylov solution-family basis	Al Daas and Gould (2025)	one matrix factorization + basis extension; reduced solve serves every regularization weight	factorization + basis
Sketched reduced cubic model	Cartis et al. (2025) ; Tansley and Cartis (2025) ; Higuchi et al. (2025)	$\ell$ projection products + dense reduced solve $\mathcal{O}(\ell^{3})$	$\ell\times n$ sketch or basis
CG / minimal-residual (MINRES) iteration on a shifted quadratic	Zhou et al. (2025) ; Zeng et al. (2026a) ; Zeng et al. (2026b)	one HVP per iteration, short recurrence; no cubic model	$\mathcal{O}(1)$ $n$ -vectors

Appendix

Table 12: Inner subproblem solvers of the methods in Table 11 , by computation per subproblem solve and working memory beyond the iterate and gradient. In the notation,

n

is the problem (or block,

n_{b}

) dimension,

L

the number of Krylov steps,

\ell

the sketch or subspace dimension,

m

the quasi-Newton memory. Computation excludes the gradient evaluation itself.

Subsolver	$L_{\max}$	best PSNR (dB)	final PSNR (dB)	peak GB	gevals
Lanczos basis	4	74.1	43.3	9.6	$2.8{\times}10^{6}$
Lanczos basis	10	78.6	67.1	9.6	$2.8{\times}10^{6}$
Lanczos basis	21	69.4	63.4	9.6	$2.8{\times}10^{6}$
Lanczos basis (full-budget run)	15	64.4	—	${\sim}11$	—
Chebyshev recurrence (full-budget run)	free	64.2	—	7.0–8.7	—

Appendix

Table 13: Subsolver comparison at equal solver memory on the 91.4M parameter ViSIR (single image, seed 0). The time-limited rows use 2 h wall clock and an fp32 HVP oracle, and the full-budget run rows are the recorded 150-sweep runs of Section 6.1 . Both best and final PSNR are quoted because the bounded runs oscillate late. Run-to-run variation of this single image fit spans several dB (Appendix V ).

Scale	Subsolver	fp32 (dB)	bf16 (dB)	$\Delta$	residual (med/max)
nano	Lanczos basis	36.4	30.5	$-5.8$	$1.8{\times}10^{-5}$ / $0.17$
nano	recurrence	35.5	26.5	$-9.0$	$0.75$ / $0.98$
91.4M	Lanczos basis	64.4	63.5	$-0.9$	—
91.4M	recurrence	64.2	59.7	$-4.5$	—

Appendix

Table 14: bf16 HVP-oracle robustness at matched configurations. Nano runs use the 6.53M-geval budget and a 2,400 s limit, 91.4M runs a single image and a 2 h limit (basis at

L{=}15

). The fp32 references are the per-block ARC and CubicCheby-DSO v2r rows of Table 32 (nano) and the full-budget run runs of Table 13 (91.4M).

Arch	method	PSNR	$H/L$	steps	gevals	conv
FINER	Adam	65.2	0.22	112k	1.1e5	20/20
FINER	SOAP	75.1	0.05	200k	2.0e5	20/20
FINER	ARC-block	124.7	1.22	15k	3.7e6	2/5 $+$ 3
FINER	CubicKrylov	86.8	1.32	35k	4.3e6	5/5
SIREN	Adam	70.7	0.93	150k	1.5e5	20/20
SIREN	SOAP	71.4	0.05	218k	2.2e5	20/20

Appendix

Table 15: 2D image fitting at convergence (mean best PSNR, dB). First-order experimental setups use 20 images and second-order experimental setups images 0–4 (ARC-block on SIREN 0–1, WIRE 0–1 for all setups). Restricting the first-order means to the same images changes them by less than

0.4

dB. conv

a/b

means the run reached the plateau criterion on

a

b

images, and

+N

that

N

runs ended at the time limit while still improving (their PSNR is a lower bound).

Backbone	method	chamfer-L1	train dB	test dB	steps	conv
FINER	Adam	0.0842	76.1	27.3	100k	yes
FINER	SOAP	0.0153	82.1	34.5	165k	yes
FINER	ARC-block	0.0177	131.0	34.0	9.1k	yes
FINER	CubicKrylov	0.0185	106.8	32.5	11.9k	yes
SIREN	Adam	0.0092	92.7	47.7	178k	yes
SIREN	SOAP	0.0089	80.8	47.1	282k	yes

Appendix

Table 16: SDF fitting, Thai statue, IGR supervision, 100k training samples (one run per row, seed 0). conv = plateau criterion reached (“limit” means that time limit reached while still improving).

experiment	method	SNR (dB)	held out MSE	gevals	rejection
production, span	Adam, 1,500 steps	32.98	—	1.5e3	—
production, span	ARC-block, 30 sweeps, $\sigma_{0}=300$	16.19	—	3.2e3	27–42 %
production, span	ARC- $\varphi_{1}$ , 30 sweeps, $c_{h}=0.1$	12.76	—	5.0e3	26–30 %
production, span	identity	19.50	—	—	—
depth, one window	Adam, best at step 4,700	27.87	9.3e-5	4.7e3	—
depth, one window	ARC-block, 1,500 sweeps	28.28	9.7e-5	2.2e5	24 %

Appendix

Table 17: Declipping with the tuned FINER, Adam against the blockwise cubic methods on the same windows and loss. Span SNR is over the union of five fitted windows per clip on two clips (

-6

dB clipping); identity is the clipped input. The depth rows are one window with 557 clipped samples (identity

10.64

dB). Gevals count gradients, Hessian–vector products, and function evaluations. Rejection is the share of rejected trial steps.

variant	$\kappa_{\mathrm{raw}}$	$\kappa_{\mathrm{Adam}}$	red.	$\rho$	$\rho_{\mathrm{block}}$	flat frac	hinge/MSE	SNR	diverged
tuned FINER ( $\omega_{0}$ 0.12, lr 1.7e-4, $W$ 4096)	4.0e4	1.2e4	3.3 $\times$	2e-4	.71	.92	102 / 2.7	29.5	0/6
tuned SIREN ( $\omega_{0}$ 3.3, lr 3e-4, $W$ 2048)	6.4e4	5.8e4	1.1 $\times$	1e-4	.47	.37	188 / 4.4	30.0	2/6
FINER with SIREN first layer	4.4e4	3.9e4	1.1 $\times$	1e-4	.59	.90	159 / 2.7	26.3	0/6
SIREN at lr 1.7e-4	4.6e4	2.2e4	2.1 $\times$	2e-4	.59	.89	188 / 1.6	23.0	0/6
SIREN at $W$ 4096	5.7e4	3.5e4	1.7 $\times$	3e-4	.71	.97	126 / 2.2	31.7	1/6
FINER at lr 3e-4	4.9e4	1.7e4	2.8 $\times$	1e-3	.73	.90	102 / 2.6	29.8	0/6

Appendix

Table 18: Fingerprint of the declipping landscape along tuned Adam, median values over six windows at step

1{,}500

(diverged windows are omitted from the medians and listed in the last column). “red.” is

\kappa_{\mathrm{raw}}/\kappa_{\mathrm{Adam}}

\rho

is the diagonal mass and

\rho_{\mathrm{block}}

the block mass over the ten parameter tensors. “hinge/MSE” is the ratio of the largest eigenvalues of the two loss terms at initialization and at step

1{,}500

. The negative mass is

0.003

–

0.037

at initialization, where

\lambda_{\max}\approx-\lambda_{\min}

makes

\kappa_{\mathrm{raw}}

uninformative, and

0

from step

300

on. SNR is the clipped sample SNR at step

1{,}500

. The first block of rows is the

-10

dB clips, the second the

-6

dB clips.

benchmark ( $n$ , $f_{0}$ )	ARC- $\varphi_{1}$	pan-cg	pan-dense	Adam	CG breakdowns	mean CG its
Rosenbrock–Ackley ( $881$ , $1.9{\times}10^{4}$ )	$\mathbf{2.3{\times}10^{-2}}$	$1.0{\times}10^{4}$	$5.4{\times}10^{3}$	$1.2{\times}10^{4}$	$336/500$	$3.8$
multi-saddle ( $881$ , $6.7{\times}10^{3}$ )	$\mathbf{2.0{\times}10^{-3}}$	$3.1{\times}10^{3}$	$8.9{\times}10^{2}$	$3.9{\times}10^{3}$	$52/500$	$4.0$
35k SIREN ( $35{,}543$ , $4.6$ )	$3.3{\times}10^{-10}$	$\mathbf{8.1{\times}10^{-13}}$	–	$1.4{\times}10^{-4}$	$0/500$	$78.8$

Appendix

Table 19: Primal accelerated Newton (PAMM) against ARC-

\varphi_{1}

on the small benchmarks,

500

steps, best loss,

L_{2}R

tuned per row. pan-dense uses an exact factorized solve and is a reference, and pan-cg is the Hessian-free variant. ARC-

\varphi_{1}

uses

\sigma_{0}{=}1

L_{\max}{=}15

c_{h}{=}3

. Adam tuned on a log grid.

experiment	arm	SIREN	wall	rej. / bd.	FINER	wall	rej. / bd.
$6{\times}10^{4}$ gevals	ARC-block -PAMM, $k_{\mathrm{cg}}{=}100$	$\mathbf{50.61}$	$252$ s	$30/25$	$48.29$	$272$ s	$138/45$
	ARC-block -PAMM, $k_{\mathrm{cg}}{=}30$	$49.02$	$271$ s	$16/16$	$49.69$	$294$ s	$222/35$
	ARC-block , $L{=}25$	$49.26$	$1{,}724$ s	$30/$ –	$57.06$	$842$ s	$138/$ –
	ARC- $\varphi_{1}$ , $L{=}25$	$48.48$	$410$ s	$16/$ –	$\mathbf{57.19}$	$438$ s	$70/$ –
$500$ steps, tuned	ARC-block -PAMM, $k_{\mathrm{cg}}{=}100$ , $\sigma_{0}{=}10$	$\mathbf{56.57}$	$700$ s		$72.05$	$755$ s
	ARC-block , $L{=}50$ (tuned $\sigma_{0}$ )	$54.32$	$574$ s		$\mathbf{78.67}$	$636$ s

Appendix

Table 20: ARC-block -PAMM against the Lanczos cubic and

\varphi_{1}

subsolvers in the same outer scheme, image 0 (rows at equal oracle calls) and mean of images 0 and 7 (rows at the same number of steps), best PSNR in dB.

k_{\mathrm{cg}}

is the conjugate gradient iteration limit, rej. the rejected trials, and bd. the CG breakdowns. Wall clock on a shared GPU, so gevals and steps are the exact axes.

	cyclic	greedy-pad, $N$	greedy-pad, $60{,}000$	greedy-pad, $20{,}000$	greedy-tensor, $\left\\|g_{b}\right\\|_{2}$
FINER	$\mathbf{64.79}$	$42.12$	( $38.84$ )	$39.03$	( $41.83$ )
SIREN	$50.14$	$\mathbf{52.30}$	$51.88$	( $47.95$ )	$50.92$

Appendix

Table 21: Block selection inside the ARC-block outer scheme with the ARC-

\varphi_{1}

subsolver (

L{=}25

c_{h}{=}1

\sigma_{0}{=}0.3

500

-step confirmation, mean best PSNR (dB) over images 0 and 7. Screen values (

120

steps, image 0) in parentheses where no confirmation was run. greedy-pad block dimension in parameters, where

n{=}198{,}915

is the whole vector, i.e. no blocking.

Stop rule	$L_{\max}$	PSNR (dB)	gevals	rejected	certified	at cap	bound met	slope
gradient form	25	60.77	$1.03{\times}10^{5}$	3.4%	10%	47%	84%	$-0.92$
step form	25	60.27	$1.42{\times}10^{5}$	38.7%	34%	66%	94%	$-1.02$
gradient form	50	73.16	$1.53{\times}10^{5}$	12.5%	10%	29%	84%	$-1.27$
step form	50	71.40	$2.08{\times}10^{5}$	46.9%	58%	42%	94%	$-1.36$
gradient form	163	82.65	$2.87{\times}10^{5}$	8.9%	10%	19%	84%	$-1.68$
step form	163	75.89	$3.76{\times}10^{5}$	48.1%	68%	32%	92%	$-1.41$

Appendix

Table 22: CubicKrylov stopping rules on ESM image 0,

500

sweeps. “Certified” is the fraction of accepted trials meeting ( 19 ) at

\kappa_{\theta}=0.1

. “at cap” is the fraction of accepted trials that reached

L_{\max}

without meeting the active stopping rule. “bound met” is the fraction whose realized decrease reached

\eta_{1}\tfrac{M_{b}}{12}\left\|s_{b}\right\|^{3}

outright (the rest were accepted inside the guard tolerance

t

, where the bound is below fp32 loss resolution). The slope is the least-squares exponent of the running minimum

\left\|\nabla f\right\|

against sweep count over the last three quarters of the run. Theorem B.10 guarantees

-2/3

Iterate	$\beta$	$\delta_{F}$	$\delta_{\mathrm{spec}}$	$L_{H}$	$\left\\|\nabla f\right\\|$	$\hat{\delta}$
init	$0.165$	$2.11{\times}10^{3}$	$135$	$6.7$	$5.62$	—
sweep $50$	$0.842$	$1.13{\times}10^{3}$	$617$	$20$	$1.16{\times}10^{-3}$	$0.076$
sweep $250$	$0.785$	$1.45{\times}10^{3}$	$634$	$37$	$2.63{\times}10^{-4}$	$0.130$

Appendix

Table 23: Tensor-block coupling on ESM image 0 along the

L_{\max}=25

gradient-form run.

\beta=\left\|\mathrm{blkdiag}\,\nabla^{2}f\right\|_{F}^{2}/\left\|\nabla^{2}f\right\|_{F}^{2}

and

\delta_{F}=\left\|\nabla^{2}f\right\|_{F}\sqrt{1-\beta}

from

16

Hutchinson probes.

\delta_{\mathrm{spec}}=\max_{b}\sigma_{\max}(P_{b}\nabla^{2}f\,P_{\neq b})

L_{H}

is the max HVP secant.

\hat{\delta}=\left\|\nabla f\right\|/(\sqrt{B}\,R)

is computed from the same sweep’s

(R,\left\|\nabla f\right\|)

Horizon	$c_{h}$	PSNR (dB)	gevals	rejected	PSNR at $1.33{\times}10^{5}$ gevals
trust $c_{h}/\sigma_{b}$	1.0	64.78	$1.32{\times}10^{5}$	2%	64.78
regularized $c_{h}/\sqrt{\sigma_{b}\left\\|g_{b}\right\\|}$	1.7	66.61	$2.37{\times}10^{5}$	48%	61.13
regularized	3.0	66.69	$2.34{\times}10^{5}$	47%	62.12
regularized	5.5	66.19	$2.46{\times}10^{5}$	50%	61.02

Appendix

Table 24: Trust against gradient-regularized horizon for ARC-

\varphi_{1}

, FINER,

500

sweeps,

L=25

\sigma_{0}=0.3

, mean of images 0 and 7. The last column shows each run’s best PSNR at the trust run’s geval total (

1.33{\times}10^{5}

Method	$L_{\max}$	PSNR (dB)	gevals	rejected
$\varphi_{1}$ trust	25	64.78	$1.32{\times}10^{5}$	$2\%$
CubicKrylov, gradient form	25	60.77	$1.03{\times}10^{5}$	$3.4\%$
CubicKrylov, step form	25	60.27	$1.42{\times}10^{5}$	$38.7\%$
$\varphi_{1}$ trust	163	$90.65$	—	$1$ – $2\%$
CubicKrylov, gradient form	163	$82.65$	$2.87{\times}10^{5}$	$8.9\%$
CubicKrylov, step form	163	$75.89$ ( $72.1$ )	$3.76{\times}10^{5}$	$48.1\%$

Appendix

Table 25: FINER reject rates and PSNR for trust

\varphi_{1}

against CubicKrylov.

L=25

\varphi_{1}

is the mean of images 0 and 7 from Table 24 . Cubic rows and

L=163

PSNR are image 0 from Table 22 . The

72.1

dB figure is the step form at the gradient form’s

2.87{\times}10^{5}

gevals. Trust

\varphi_{1}

L_{\max}=163

is the

500

-sweep confirmation of Section 6.3 . Its

1

–

2\%

reject range is the recorded image-0 runs at the same

(c_{h},\sigma_{0},L)

$L_{\max}$	PSNR (dB)	sweeps	gevals	rejected	$\left\\|\nabla f\right\\|_{\min}$
$25$	$85.67$	$2{,}000$	$5.27{\times}10^{5}$	$1.58\%$	$1.11{\times}10^{-5}$
$163$	$132.64$	$2{,}000$	$3.03{\times}10^{6}$	$0.70\%$	$1.85{\times}10^{-7}$

Appendix

Table 26: Trust ARC-

\varphi_{1}

on ESM image 0,

2{,}000

sweeps. Best PSNR for

L_{\max}=163

is at sweep

1{,}994

. Rejected is the fraction of block trials. Median trial horizon

h_{b}

10^{8}

on both runs (

\sigma_{b}

\sigma_{\min}=10^{-8}

$L_{\max}$	series	all	early	mid	late
$25$	$\left\\|\nabla f\right\\|_{\min}$	$-1.28$ ( $0.98$ )	$-1.04$ ( $1.00$ )	$-1.62$ ( $1.00$ )	$-1.80$ ( $1.00$ )
$25$	$f_{\min}$	$-2.41$ ( $0.98$ )	$-2.06$ ( $1.00$ )	$-3.19$ ( $0.99$ )	$-3.63$ ( $1.00$ )
$163$	$\left\\|\nabla f\right\\|_{\min}$	$-2.42$ ( $0.97$ )	$-2.05$ ( $0.98$ )	$-3.12$ ( $0.97$ )	$-2.27$ ( $0.90$ )
$163$	$f_{\min}$	$-5.18$ ( $0.93$ )	$-4.01$ ( $0.98$ )	$-8.24$ ( $0.99$ )	$-8.92$ ( $0.99$ )

Appendix

Table 27: OLS slope of

\log\left\|\nabla f\right\|_{\min}

(and

\log f_{\min}

) against

\log k

on ESM image 0. The windows are all

k\in[3,2000]

, early

[101,700]

, mid

[501,2000]

, late

[1001,2000]

R^{2}

in parentheses.

Method	PSNR (dB)	steps in $T$
CubicKrylov	67.28	100
Cheby-2 $n{=}5$	52.46	187
Cheby-1 $n{=}3$	33.84	151
Cheby-2 $n{=}3$	25.69	201
Cheby-1 $n{=}5$	25.19	141
Newton $n{=}3$	24.26	243

Appendix

Table 28: Iso-time comparison at

T=148

s per method (

91.4

M ViSIR, image #1). “steps” is the number of steps in time T

Panel of Figure 3 — Figure 3: Iso-time training curves (PSNR and MSE loss vs. wall clock time) at the $T=148$ s budget.

Method	gevals/step	PSNR at ${\leq}36{,}836$ gevals
CubicKrylov	368	67.3 ‡
SOAP	1	20.6
Adam	1	20.4
Cheby-ON $n{=}5$ (Chebyshev-2)	8049	19.1
Newton $n{=}5$ (damped)	8049	13.8
Muon+AdamW	1	11.3

Appendix

Table 29: Best PSNR reached within a matched oracle budget of

36{,}836

gradient equivalents (CubicKrylov’s 100-step cost) on

91.4

M ViSIR. ‡ See Table 28 .

Figure 4: PSNR vs. cumulative gradient equivalents (log scale). At matched oracle budget CubicKrylov (named CubicCheby in the plot) dominates the full Hessian Chebyshev steps by ${\sim}48$ dB (they exhaust the budget in ${\sim}5$ steps), because its matrix-free Krylov step is ${\sim}22\times$ cheaper per step.

	aggregate over 60 runs				paired vs. CubicKrylov
Method	PSNR (dB)	median	wall (s)	gevals	$\Delta$ PSNR	win%	$p_{W}$
CubicKrylov (nano)	$\mathbf{32.8\pm 1.4}$	32.9	21	$1.8{\times}10^{4}$	—	—	—
Reg. Newton lazy, $m{=}n$ ( Doikov et al., 2023 )	$32.4\pm 1.2$	32.5	298	$3.0{\times}10^{4}$	$+0.3$	85%	$3.9{\times}10^{-4}$
CN-Lazy, $m{=}n$ ( Doikov et al., 2023 )	$32.1\pm 1.1$	32.1	305	$3.0{\times}10^{4}$	$+0.7$	95%	$3.8{\times}10^{-6}$
Spectral-precond. GD $\tau{=}10$ ( Doikov et al., 2024 )	$27.7\pm 3.3$	28.5	116	$3.6{\times}10^{4}$	$+5.1$	100%	$1.9{\times}10^{-6}$
SSCN $\tau{=}64$ ( Zhao et al., 2025 )	$24.4\pm 1.2$	24.4	126	$3.6{\times}10^{4}$	$+8.4$	100%	$1.9{\times}10^{-6}$

Appendix

Table 30: ESM-20-style nano benchmark (20 images

\times

3 seeds, ViSIR-Nano, 15,235 parameters). Best PSNR per run, aggregated over the 60 runs per method, with mean wall clock and total gradient equivalents per run. The right part is the paired per-image comparison against CubicKrylov (mean over seeds), with the mean PSNR advantage of CubicKrylov, its per-image win rate, and the Wilcoxon signed-rank

p

-value (

N{=}20

images).

Method	mean (s)	std (s)	min (s)	max (s)	CV
CubicKrylov	20.5	8.7	14.5	60.4	43%
Reg. Newton lazy, $m{=}n$	298.3	35.6	274.9	424.3	12%
CN-Lazy, $m{=}n$	304.9	27.6	284.8	390.1	9%
Spectral-precond. GD $\tau{=}10$	115.7	5.9	107.4	136.8	5%
SSCN $\tau{=}64$	125.7	7.1	114.1	144.8	6%
FD-CNM, $m{=}50$	123.9	22.5	106.3	232.5	18%

Appendix

Table 31: Per-run wall clock statistics of the multi-image ViSIR-Nano benchmark (Table 30 ), with mean, standard deviation, extremes, and coefficient of variation over the 60 runs per method.

Method	best PSNR (dB)	steps	gevals used	limited by
ARC, global Krylov ( Cartis et al., 2011a ; Cartis et al., 2011b )	37.21	16,391	$4.6{\times}10^{5}$	wall clock
CubicKrylov (nano)	36.64	7,334	$5.8{\times}10^{6}$	wall clock
ARC, per-block $\sigma_{b}$ , reduced-operator acceptance ( Doikov and Nesterov, 2025 )	36.49	2,790 sweeps	$1.4{\times}10^{6}$	wall clock
ARC, per-block $\sigma_{b}$ (Appendix M )	36.37	1,795 sweeps	$1.2{\times}10^{6}$	wall clock
Reg. Newton lazy, $m{=}n$ ( Doikov et al., 2023 )	36.36	120,000	$2.4{\times}10^{5}$	wall clock
CN-Lazy, $m{=}n$ ( Doikov et al., 2023 )	36.18	115,000	$2.4{\times}10^{5}$	wall clock

Appendix

Table 32: Equal-budget comparison, every method granted DSO-Adaptive’s plateau budget of

6.53{\times}10^{6}

gradient equivalents (

{\sim}185\times

the benchmark rule) and a 2,400 s wall clock limit, on ESM image #0, seed 42. “Limited by” is the stopping constraint.

Second-Order Optimization

Second-order optimization uses curvature information, such as Hessians or their approximations, to choose efficient steps toward better solutions.

See all 90 papers

Apr 10, 2026

AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learning

A novel regularization technique, AdaCubic, is proposed that adapts the weight of the cubic term. The heart of AdaCubic is an auxiliary optimization problem with cubic constraints that dynamically adjusts the weight of the cubic term in Newton's cubic regularized method. We use Hutchinson's method to approximate the Hessian matrix, thereby reducing computational cost. We demonstrate that AdaCubic inherits the cubically regularized Newton method's local convergence guarantees. Our experiments in Computer Vision, Natural Language Processing, and Signal Processing tasks demonstrate that AdaCubic outperforms or competes with several widely used optimizers. Unlike other adaptive algorithms that require hyperparameter fine-tuning, AdaCubic is evaluated with a fixed set of hyperparameters, rendering it a highly attractive optimizer in settings where fine-tuning is infeasible. This makes AdaCubic an attractive option for researchers and practitioners alike. To our knowledge, AdaCubic is the first optimizer to leverage cubic regularization in scalable deep learning applications.

Ioannis Tsingalis, Constantine Kotropoulos, Corentin Briat

Aristotle University of Thessaloniki, Greece · University of Applied Sciences Northwestern Switzerland, Muttenz, Switzerland

Deep Learning Optimization

Ask PDF

Feb 23, 2023

Unified Convergence Theory of Stochastic and Variance-Reduced Cubic Newton Methods

We study stochastic Cubic Newton methods for solving general possibly non-convex minimization problems. We propose a new framework, which we call the helper framework, that provides a unified view of the stochastic and variance-reduced second-order algorithms equipped with global complexity guarantees. It can also be applied to learning with auxiliary information. Our helper framework offers the algorithm designer high flexibility for constructing and analyzing the stochastic Cubic Newton methods, allowing arbitrary size batches, and the use of noisy and possibly biased estimates of the gradients and Hessians, incorporating both the variance reduction and the lazy Hessian updates. We recover the best-known complexities for the stochastic and variance-reduced Cubic Newton, under weak assumptions on the noise. A direct consequence of our theory is the new lazy stochastic second-order method, which significantly improves the arithmetic complexity for large dimension problems. We also establish complexity bounds for the classes of gradient-dominated objectives, that include convex and strongly convex problems. For Auxiliary Learning, we show that using a helper (auxiliary function) can outperform training alone if a given similarity measure is small.

El Mahdi Chayti, Martin Jaggi, Nikita Doikov

Machine Learning and Optimization Laboratory (MLO), EPFL

Second-Order Optimization Stochastic Optimization Convergence Nonconvex Stochastic Optimization Variance Reduction

Ask PDF

Jul 16, 2026

Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers

Interpreting optimizers as gradient-flow discretizations has motivated applying higher-order Runge-Kutta (RK) integrators to neural networks. We build a representative Adam variant (Bogacki-Shampine 3(2) RK pair, FSAL reuse, local-error step control) and evaluate it under a strict compute-matched protocol giving every method the same gradient-evaluation budget - an accounting this literature rarely enforces. Under it the RK variant loses to plain Adam on training loss in both minibatch and full-batch (RK's best-case) training. Instrumenting it shows the "adaptivity" is illusory: normalized error stays far below tolerance, the step size pins at its growth cap from step one (98-100 percent of steps), and no rtol x hmax x h0 setting makes it act; tolerances spanning 100x give bit-identical trajectories. The method is exactly fixed-step Adam with an averaged gradient at 3-4x cost. Repairing it (true reject branch; error on the applied map) reverses the full-batch result - about 40x lower training loss than tuned Adam - and a fixed-step control isolates adaptivity (an emergent warmup-and-growth schedule) as the mechanism. But the gain is fragile to the initial step size and does not reach test accuracy. A pre-registered follow-up rules out the obvious explanations: deeper minimization does not overfit, and an explicit temperature knob only hurts - leaving a trajectory effect, the controller selecting a minimum generalizing 1.3-3.4 points below first-order descent at equal depth. An n=10 study confirms one secondary effect: gradient averaging is a genuine implicit regularizer, beating lr-matched Adam and AdamW on 10/10 seeds - yet RMSprop and NAdam match or beat it at a third the per-step cost. Higher-order adaptive integration buys deeper deterministic minimization and a small regularization effect, but nothing a cheaper, well-tuned first-order baseline does not already provide.

Akhilesh Gogikar

1Independent Researcher.

Neural Network Optimization Adaptive Gradient Methods Neural Network Training Dynamics Implicit Regularization

Ask PDF

Abstract

Second-Order Optimization

Explore similar work

AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learning

Unified Convergence Theory of Stochastic and Variance-Reduced Cubic Newton Methods

Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers