AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learning

Ioannis Tsingalis; Constantine Kotropoulos; Corentin Briat

cs.LGApr 10, 2026

AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learning

Authors: Ioannis Tsingalis, Constantine Kotropoulos, Corentin Briat

Organizations: Aristotle University of Thessaloniki, Greece · University of Applied Sciences Northwestern Switzerland, Muttenz, Switzerland

Abstract

A novel regularization technique, AdaCubic, is proposed that adapts the weight of the cubic term. The heart of AdaCubic is an auxiliary optimization problem with cubic constraints that dynamically adjusts the weight of the cubic term in Newton's cubic regularized method. We use Hutchinson's method to approximate the Hessian matrix, thereby reducing computational cost. We demonstrate that AdaCubic inherits the cubically regularized Newton method's local convergence guarantees. Our experiments in Computer Vision, Natural Language Processing, and Signal Processing tasks demonstrate that AdaCubic outperforms or competes with several widely used optimizers. Unlike other adaptive algorithms that require hyperparameter fine-tuning, AdaCubic is evaluated with a fixed set of hyperparameters, rendering it a highly attractive optimizer in settings where fine-tuning is infeasible. This makes AdaCubic an attractive option for researchers and practitioners alike. To our knowledge, AdaCubic is the first optimizer to leverage cubic regularization in scalable deep learning applications.

Figures & tables

\rho_{k}=\frac{F(\mathbf{x}_{k})-F(\mathbf{x}_{k}+\mathbf{s}_{k+1})}{F(\mathbf{x}_{k})-\mathfrak{m}_{\nu_{k+1}}(\mathbf{s}_{k+1})}

Algorithm 1 AdaCubic algorithm

	AdaCubic Hyperparameters
Hyperparameter	$\eta_{1}$	$\eta_{2}$	$\alpha_{1}$	$\alpha_{2}$	$\kappa_{\text{easy}}$	$\epsilon_{m}$
Assigned Value	$0.05$	$0.75$	$2.5$	$0.25$	$0.01$	$10^{-6}$

Table 1: Universal AdaCubic hyperparameter settings. All hyperparameters in Algorithm 2 are fixed across benchmarks.

\epsilon_{m}

denotes a numerical safeguard used in Algorithm 1 .

Task	Dataset	Model	Batch	Epochs	Optimizers
CV	CIFAR-10	ResNet20 / ResNet32	256	500	SGD , Adam , AdaHessian , AdaCubic
CV	CIFAR-100	ResNet18	256	200	SGD , Adam , AdaHessian , AdaCubic
NLU	SST-2 , QNLI , RTE , WNLI	SqueezeBERT	32	15	SGD , AdaHessian , AdaCubic
	MRPC , QQP	SqueezeBERT	32	15	SGD , AdaHessian , AdaCubic
	STS-B , MNLI	SqueezeBERT	32	15	SGD , AdaHessian , AdaCubic
LM	WikiText-2	RoBERTa / BERT / DistilBERT	8	6	SGD , AdaHessian , AdaCubic

Table 2: Summary of model architectures, training settings, and optimizers used in all experiments.

Optimizer	Task	Dataset(s)	Initial LR	LR Schedule / Tuning
SGD	CV	CIFAR-10 / CIFAR-100	$0.1$	Step decay (tuned)
Adam	CV	CIFAR-10 / CIFAR-100	$10^{-3}$	Step decay (tuned)
AdaHessian	CV	CIFAR-10 / CIFAR-100	$0.15$	Step decay (tuned)
AdaCubic	CV	CIFAR-10 / CIFAR-100	no LR	Universal parameters
SGD	NLU	SST-2 , QNLI , RTE , WNLI	$2{\times}10^{-2}$	Tuned
SGD	NLU	STS-B	$2{\times}10^{-3}$	Tuned

Table 3: Summary of LRs used in all experiments. For the CV benchmark, LRs are decayed by a factor of

10

at epochs

80

and

120

on CIFAR-10 , and by a factor of

20

at epochs

60

120

, and

160

on CIFAR-100 . For NLU, LM, and CMI benchmarks, no LR decay is applied. For CMI, LR is decayed by a factor of

10

at epochs

80

and

120

. AdaCubic is used with a fixed universal parameter set and does not require LR tuning.

	CIFAR-10		CIFAR-100
	ResNet20	ResNet32	ResNet18
SGD	88.52 $\pm$ 0.24	89.02 $\pm$ 0.20	72.62 $\pm$ 0.002
Adam	90.26 $\pm$ 0.19	91.24 $\pm$ 0.20	72.04 $\pm$ 0.13
AdaHessian	91.64 $\pm$ 0.46	93.15 $\pm$ 0.12	72.45 $\pm$ 0.16
AdaHessian	-	-	72.59 $\pm$ 0.271
AdaCubic	91.49 $\pm$ 0.46	92.65 $\pm$ 0.19	71.81 $\pm$ 0.003

Table 4: Accuracy (%) and std of the accuracy measures for ResNet18/20/32 models on CIFAR-10 and CIFAR-100 datasets.

\Delta

reports the gap between the strongest competing optimizer and AdaCubic .

Panel of Figure 1 — Figure 1: Training loss curve of ResNet20 (top) and ResNet32 (bottom) on CIFAR-10 for Adam , AdaHessian , and AdaCubic optimizers.

Optimizer	RoBERTa	BERT	DistilBERT
SGD	3.547	13.380	6.118
AdaCubic	3.756	5.759	6.565
AdaHessian	4.374	16.151	6.822

Table 6: Perplexity achieved by SGD , AdaHessian , and AdaCubic on wikitext-2 dataset.

Optimizer	RoBERTa	BERT	DistilBERT
SGD	4.345	17.344	8.299
AdaCubic	5.145	14.170	7.334
AdaHessian	7.582	20.851	10.182

Table 7: Perplexity using SGD , AdaHessian , and AdaCubic on the PTB dataset.

Panel of Figure 2 — Figure 2: Perplexity vs. epochs for RoBERTa , BERT , and DistilBERT models on wikitext-2 dataset.

Panel of Figure 3 — Figure 3: Perplexity vs. epochs for RoBERTa , BERT , and DistilBERT models on PTB dataset.

	AdaCubic			Adam
	Native	WhatsApp	YouTube	Native	WhatsApp	YouTube
Fold 0	97.40	96.10	94.59	96.10	93.50	91.9
Fold 1	93.51	93.51	93.24	94.80	90.90	93.24
Fold 2	94.81	92.22	94.59	90.90	88.31	95.94
Fold 3	93.42	93.43	91.89	93.42	94.73	82.43
Fold 4	94.73	88.16	93.24	94.73	88.15	95.94

Table 8: CMI accuracy (%) using ResNet18 .

Figure 4: Cumulative time vs. epochs for SGD , AdaHessian , and AdaCubic for ResNet20 on CIFAR-10 .

Panel of Figure 5 — Figure 5: Comparison of SGD , AdaHessian , and AdaCubic on ResNet20 and CIFAR-10 . Training loss vs. cumulative time over epochs (Left). Training loss vs. epochs (Right).

	SGD		AdaHessian		AdaCubic
	Epoch	Time	Epoch	Time	Epoch	Time
Result	83	35.16	81	61.85	55	42.40

Table 9: Execution time in minutes required to reach a target loss threshold when ResNet20 is trained on CIFAR-10 .

Optimizer	Order	Sensitivity	Extra Backward Pass	Time Cost	Memory Footprint
SGD	First	High	No	$d$	$d$
Adam	First	High	No	$d$	$3d$
AdaHessian	Second	Medium	Yes	$\pazocal{S}\,d$	$4d$
AdaCubic	Second	Low	Yes	$\pazocal{S}\,d$	$2d$

Table 10: Comparison of optimization methods used in the experimental evaluation. Recall that

d

denotes the number of model parameters and

\pazocal{S}

the number of random vectors used in the diagonal Hessian approximation.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Logical connection between key lemmata, theorems, and corollaries throughout Sections 2 , 3 and 4 and Appendices B.1 , B.2 , B.3 , B.4 , B.5 , B.6 , B.7 , B.8 , B.9 , B.10 , B.11 , B.12 and B.13 .

Deep Learning Optimization

Deep learning optimization develops algorithms and training strategies that help neural networks converge, generalize, and use compute efficiently.

See all 205 papers

Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence
Aug 22, 2026Rodion PodorozhnySecond-Order Optimization Neural Network Optimization
Texas State University
Optimistic Dual Averaging Unifies Modern Optimizers
May 11, 2026Thomas Pethick, Wanyun Xie, Roman Machacek +1Stochastic Optimization Deep Learning Optimization
Independent Researcher · EPFL (LIONS) · University of Bern
An Improved Adaptive PID Optimizer with Enhanced Convergence and Stability for Deep Learning
May 21, 2026Saurabh Saini, Kapil Ahuja, Thomas Wick +1Deep Learning Optimization Stochastic Optimization Convergence
Department of Computer Science & Engineering, Indian Institute of Technology Indore, India. · Leibniz Universität Hannover, Institut für Angewandte Mathematik, Germany. · National Remote Sensing Centre, Indian Space Research Organisation, India.

Aug 22, 2026J/K move · Enter open · S save

Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence

Rodion Podorozhny

Texas State University

Second-Order Optimization Neural Network Optimization

Open paper AskPDF arXiv

Overparameterized networks often have ill-conditioned, saddle rich loss landscapes. They are usually trained with first-order methods that struggle on those landscapes, while many second-order methods stay infeasible at scale. Some scalable cubic methods replace the Hessian by a computationally cheap approximation and drop the exact block curvature. This paper introduces ARC-block, a blockwise adaptive regularization with cubics (ARC) optimizer. For each parameter tensor it minimizes a cubic model over the true block Hessian, with a per-block adaptive cubic constant and a monotone guard on the full loss. Large tensors are handled matrix-free in a Lanczos-built Krylov subspace. ARC-block's CubicKrylov subsolver's descent is monotone with a per-block cubic decrease bound. For exact steps, and for Krylov steps that meet the Cartis--Gould--Toint residual test, we provide an

\mathcal{O}(\sqrt{B}\varepsilon^{-3/2})

sweep bound above a cross-block coupling floor. Another subsolver, ARC-

\varphi_1

, uses a trust-controlled exponential relaxation step. Among the cubic Newton methods evaluated here, these are the only ones whose steps stay exact on every parameter block (within the Krylov subspace), including on a

91.4

M-parameter implicit neural representation (INR). Run to convergence on FINER 2D image-fitting INR network, ARC-

\varphi_1

reaches

133.5

dB peak signal-to-noise ratio (PSNR), with no post peak collapse. Tuned Adam peaks at

78.2

dB at

2{,}629

s (step

{\sim}625

k, about

44

minutes), further steps degrade its PSNR. On the same benchmark, SOAP, L-BFGS, and the scalable diagonal cubic method AdaCubic stay at or below Adam's

78.2

dB peak. Over the benchmark's 20 images, run to a plateau rule, ARC-

\varphi_1

's per-image PSNR has median

132.4

and mean

131.5\pm3.3

dB against tuned Adam's

72.8\pm4.5

dB.

Abstract

Deep Learning Optimization

Explore similar work

Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence

Optimistic Dual Averaging Unifies Modern Optimizers

An Improved Adaptive PID Optimizer with Enhanced Convergence and Stability for Deep Learning