Dynamic Momentum Recalibration in Online Gradient Learning

Zhipeng Yao; Rui Yu; Guisong Chang; Ying Li; Yu Zhang; Dazhou Li

cs.LGMar 6, 2026

Dynamic Momentum Recalibration in Online Gradient Learning

Authors: Zhipeng Yao, Rui Yu, Guisong Chang, Ying Li, Yu Zhang, Dazhou Li

Organizations: Shenyang University of Chemical Technology · University of Louisville · Northeastern University

Abstract

Stochastic Gradient Descent (SGD) and its momentum variants form the backbone of deep learning optimization, yet the underlying dynamics of their gradient behavior remain insufficiently understood. In this work, we reinterpret gradient updates through the lens of signal processing and reveal that fixed momentum coefficients inherently distort the balance between bias and variance, leading to skewed or suboptimal parameter updates. To address this, we propose SGDF (SGD with Filter), an optimizer inspired by the principles of Optimal Linear Filtering. SGDF computes an online, time-varying gain to dynamically refine gradient estimation by minimizing the mean-squared error, thereby achieving an optimal trade-off between noise suppression and signal preservation. Furthermore, our approach could extend to other optimizers, showcasing its broad applicability to optimization frameworks. Extensive experiments across diverse architectures and benchmarks demonstrate SGDF surpasses conventional momentum methods and achieves performance on par with or surpassing state-of-the-art optimizers.

Figures & tables

Method	Bias Bound	Variance Bound	Limit as $\beta\to 1$
SGD ( $\beta=0$ )	$0$ (Assuming $\alpha\to 0$ )	$\sigma^{2}+2V^{2}$	N/A
EMA ( $u=1-\beta$ )	$\left(\frac{\alpha LG}{1-\beta}+\frac{\alpha\sigma L}{\sqrt{2(1-\beta)}}\right)^{2}$	$(1-\beta)\sigma^{2}+2V^{2}$	Bias $\to\infty$ , Var $\to 2V^{2}$
CM ( $u=1$ )	$\left(\frac{\alpha LG}{(1-\beta)^{3}}+\frac{\alpha L\sigma}{\sqrt{2}(1-\beta)^{2.5}}+\frac{\beta G}{1-\beta}\right)^{2}$	$\frac{\sigma^{2}}{1-\beta}+\frac{2V^{2}}{(1-\beta)^{2}}$	Bias $\to\infty$ , Var $\to\infty$

Table 1: Bias and variance bounds for different momentum estimators. As

\beta\to 1

, static estimators suffer from either diverging bias or diverging variance, highlighting the fundamental trade-off.

Panel of Figure 1 — Figure 1: Test accuracy ([ $\mu\pm\sigma$ ]) on CIFAR.

Method	SGDF	SGD	AdaBelief	PAdam	AdaBound	Yogi	MSVAG	Adam	RAdam	AdamW
Top-1	$\textbf{70.51}_{\pm 0.05}$	70.23 †	70.08 ∗	70.07 †	68.13 †	68.23 †	65.99 ∗	63.79 † (66.54 ‡ )	67.62 ‡	67.93 †
Top-5	$\textbf{89.69}_{\pm 0.16}$	89.40 †	-	89.47 †	88.55 †	88.59 †	-	85.61 †	-	88.47 †

Table 2: Top-1, 5 accuracy (%) of ResNet18 on ImageNet. ∗ † ‡ is reported in [ 89 , 8 , 49 ] .

Model	VGG11	VGG13	ResNet34	ResNet50	DenseNet121	DenseNet161
SGD 1 1 1 Results from PyTorch official pre-trained models .	70.37	71.58	73.31	76.13	74.43	77.13
SGDF	$\textbf{71.34}_{\pm 0.21}$	$\textbf{72.74}_{\pm 0.07}$	$\textbf{74.07}_{\pm 0.21}$	$\textbf{76.72}_{\pm 0.09}$	$\textbf{75.75}_{\pm 0.09}$	$\textbf{78.34}_{\pm 0.08}$

Table 3: Comparison of top-1 accuracy (%) across different model variants and optimizers on the ImageNet classification.

Model	Method	CIFAR-10	CIFAR-100	Oxford-IIIT-Pets	Oxford Flowers-102	Food101	ImageNet
ViT-B/32	SGD	$98.71_{\pm 0.03}$	$90.62_{\pm 0.07}$	$89.71_{\pm 0.32}$	$96.79_{\pm 0.29}$	$88.56_{\pm 0.05}$	$81.42_{\pm 0.04}$
ViT-B/32	SGDF	$\textbf{98.74}_{\pm 0.10}$	$\textbf{91.44}_{\pm 0.13}$	$\textbf{92.68}_{\pm 0.04}$	$\textbf{97.17}_{\pm 0.47}$	$\textbf{89.35}_{\pm 0.09}$	$\textbf{81.52}_{\pm 0.02}$
ViT-L/32	SGD	$98.73_{\pm 0.05}$	$91.30_{\pm 0.17}$	$85.21_{\pm 0.39}$	$96.52_{\pm 0.15}$	$89.13_{\pm 0.20}$	$81.28_{\pm 0.04}$
ViT-L/32	SGDF	$\textbf{98.83}_{\pm 0.04}$	$\textbf{92.20}_{\pm 0.14}$	$\textbf{91.96}_{\pm 0.18}$	$\textbf{96.79}_{\pm 0.12}$	$\textbf{90.04}_{\pm 0.08}$	$\textbf{81.38}_{\pm 0.01}$

Table 5: Post-training in ViT. We report Top-1 accuracy (%).

Figure 2: Convergence comparison between Sign SGDF and Adam.

Panel of Figure 3 — Figure 3: Histogram of Top 50 Hessian Eigenvalues. Lower values indicate better performance on the test dataset.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Optimizer	Learning Rate	$\beta_{1}$	$\beta_{2}$	Epochs	Schedule	Weight Decay	Batch Size	$\varepsilon$
SGDF	0.5	0.9	0.999	200	StepLR	0.0005	128	1e-8
SGD	0.1	0.9	-	200	StepLR	0.0005	128	-
Adam	0.001	0.9	0.999	200	StepLR	0.0005	128	1e-8
RAdam	0.001	0.9	0.999	200	StepLR	0.0005	128	1e-8
AdamW	0.001	0.9	0.999	200	StepLR	0.01	128	1e-8
MSVAG	0.1	0.9	0.999	200	StepLR	0.0005	128	1e-8

Appendix

Table 8: Hyperparameters used for CIFAR-10 and CIFAR-100 datasets.

Algorithm	CIFAR-10			CIFAR-100
Algorithm	VGG11	ResNet34	DenseNet121	VGG11	ResNet34	DenseNet121
SGDF	$\textbf{91.61}_{\pm 0.21}$	$\textbf{95.33}_{\pm 0.19}$	$\textbf{95.74}_{\pm 0.06}$	$\textbf{68.12}_{\pm 0.15}$	$\textbf{77.69}_{\pm 0.64}$	$\textbf{80.17}_{\pm 0.19}$
SGD	$89.83_{\pm 0.05}$	$94.62_{\pm 0.07}$	$94.52_{\pm 0.03}$	$63.48_{\pm 0.39}$	$76.88_{\pm 0.12}$	$78.77_{\pm 0.27}$
Adam	$88.12_{\pm 0.10}$	$94.30_{\pm 0.06}$	$94.37_{\pm 0.17}$	$56.27_{\pm 0.32}$	$72.81_{\pm 0.45}$	$74.67_{\pm 0.45}$
AdamW	$88.59_{\pm 0.20}$	$94.42_{\pm 0.00}$	$94.61_{\pm 0.06}$	$58.09_{\pm 0.69}$	$72.74_{\pm 0.45}$	$74.96_{\pm 0.10}$
RAdam	$90.47_{\pm 0.34}$	$93.41_{\pm 0.21}$	$93.75_{\pm 0.04}$	$60.20_{\pm 0.37}$	$74.08_{\pm 0.35}$	$75.82_{\pm 0.28}$

Appendix

Table 9: Test Accuracies for CIFAR-10 and CIFAR-100 across different models and algorithms.

Panel of Figure 4 — Figure 4: Training (top row) and test (bottom row) accuracy of CNNs on CIFAR-10 dataset. We report confidence interval ([ $\mu\pm\sigma$ ]) of 3 independent runs.

Panel of Figure 5 — Figure 5: Training (top row) and test (bottom row) accuracy of CNNs on CIFAR-100 dataset. We report confidence interval ([ $\mu\pm\sigma$ ]) of 3 independent runs.

Optimizer	Learning Rate	$\beta_{1}$	$\beta_{2}$	Epochs	Schedule	Weight Decay	Batch Size	$\varepsilon$
SGDF	0.5	0.5	0.999	100/90	Cosine	0.0001	256	1e-8
SGD	0.1	0.9	-	100/90	Cosine	0.0001	256	-

Appendix

Table 10: Hyperparameters used for ImageNet.

Panel of Figure 6 — Figure 6: Training and test accuracy (top-1) of ResNet18 on ImageNet.

Optimizer	Learning Rate	$\beta_{1}$	$\beta_{2}$	Epochs	Schedule	Weight Decay	Batch Size	$\varepsilon$
SGDF	0.01	0.9	0.999	4	StepLR	0.0001	2	1e-8
SGD	0.01	0.9	-	4	StepLR	0.0001	2	-
Adam	0.0001	0.9	0.999	4	StepLR	0.0001	2	1e-8
AdamW	0.0001	0.9	0.999	4	StepLR	0.0001	2	1e-8
RAdam	0.0001	0.9	0.999	4	StepLR	0.0001	2	1e-8

Appendix

Table 11: Hyperparameters for object detection on PASCAL VOC using Faster-RCNN+FPN with different optimizers.

Panel of Figure 7 — Figure 7: Detection examples using Faster-RCNN + FPN trained on PASCAL VOC.

Method	SGDF	Adam	RMSProp	RAdam	Fromage	Yogi	AdaBound	SGD	MSVAG
FID	$88.7\pm 4.9$	$\textbf{78.6}\pm 4.8$	$109.2\pm 14.5$	$93.4\pm 8.3$	$101.5\pm 28.9$	$138.7\pm 21.2$	$119.8\pm 24.6$	$250.3\pm 30.1$	$239.7\pm 5.2$

Appendix

Table 12: FID score of WGAN-GP.

Optimizer	Learning Rate	$\beta_{1}$	$\beta_{2}$	Epochs	Batch Size	$\varepsilon$
SGDF	0.01	0.5	0.999	100	64	1e-8
SGD	0.01	0.5	-	100	64	-
Adam	0.0002	0.5	0.999	100	64	1e-8
AdamW	0.0002	0.5	0.999	100	64	1e-8
Fromage	0.01	0.5	0.999	100	64	1e-8
RMSProp	0.0002	0.5	0.999	100	64	1e-8

Appendix

Table 13: Hyperparameters for Image Generation Tasks.

Panel of Figure 8 — Figure 8: Training (top row) and test (bottom row) perplexity on Penn-TreeBank dataset, lower is better.

Optimizer	Learning Rate	$\beta_{1}$	$\beta_{2}$	Epochs	Schedule	Weight Decay	Batch Size	$\varepsilon$
SGDF	60	0.9	0.999	200	StepLR	1.2e-6	20	1e-8
SGD	30	0.9	-	200	StepLR	1.2e-6	20	-
Adam	0.001	0.9	0.999	200	StepLR	1.2e-6	20	1e-8
RAdam	0.001	0.9	0.999	200	StepLR	1.2e-6	20	1e-8
AdamW	0.001	0.9	0.999	200	StepLR	1.2e-6	20	1e-8
MSVAG	30	0.9	0.999	200	StepLR	1.2e-6	20	1e-8

Appendix

Table 14: Hyperparameters used for LSTM.

Optimizer	Learning Rate	$\beta_{1}$	$\beta_{2}$	Epochs	Schedule	Weight Decay	Batch Size	$\varepsilon$	Resolution
SGDF	0.5	0.9	0.999	10	Cosine	0	512	1e-8	384
SGD	0.03	0.9	-	10	Cosine	0	512	-	384

Appendix

Table 15: Hyperparameters used for fine-tuning ViT.

Panel of Figure 9 — Figure 9: Histogram of Top 50 Hessian Eigenvalues.

Panel of Figure 10 — Figure 10: Visualization of loss landscape. Adam converges to sharp minima.

Optimizer	Per-Parameter FLOPs	Operation Breakdown
SGD	$\approx$ 2 ops/param (+2 ops) (+1 op)	$\{\,1\times,\;1+\}$
SGDM	$\approx$ 4 ops/param (+2 ops) (+1 op)	$\{\,2\times,\;2+\}$
Adam	16 ops/param $\rightarrow$ 14 ops/param (+2 ops) (+1 op)	$\{\,7\times,\;5+,\;3\div,\;1\sqrt{\;}\}\;\rightarrow\;\{\,7\times,\;4+,\;2\div,\;1\sqrt{\;}\}$
SGDF	22 ops/param $\rightarrow$ 20 ops/param (+2 ops) (+1 op)	$\{\,10\times,\;6+,\;2-,\;2\div,\;1\sqrt{\;}\}\;\rightarrow\;\{\,10\times,\;6+,\;1-,\;1\div,\;1\sqrt{\;}\}$

Appendix

Table 16: Arithmetic cost and operation breakdown of optimizers. Red: Coupled Weight Decay , Green: Decoupled Weight Decay .

Model	SGDM (h)	Adam (h)	SGDF (h)
VGG13	45.71	45.90	$47.78_{{\color[rgb]{0.21,0.49,0.74}\downarrow 0.63}}$
ResNet101	35.11	35.32	$39.77_{{\color[rgb]{0.21,0.49,0.74}\downarrow 1.59}}$
DenseNet161	57.68	58.03	$64.81_{{\color[rgb]{0.21,0.49,0.74}\downarrow 1.99}}$

Appendix

Table 17: Total training time (h) for 100 epochs with batch size 256 on FP16 AMP in

224^{2}

Pixel ImageNet.

Panel of Figure 11 — Figure 11: SGDF with or without the correction factor. The curve shows the accuracy of the test.

Panel of Figure 13 — Figure 13: Test accuracy of CNNs on CIFAR-100 dataset. We train vanilla Adam and Adam combined with Optimal Linear Filter.

Deep Learning Optimization

Deep learning optimization develops algorithms and training strategies that help neural networks converge, generalize, and use compute efficiently.

See all 205 papers

Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization
Aug 13, 2026Zhixin Ren, Yao Lyu, Congrong Li +2Deep Learning Optimization Momentum Methods
Department of Mathmatical Science, Tsinghua University · College of AI, Tsinghua University · School of Vehicle and Mobility, Tsinghua University
Muon Under Gradient Noise and the Limits of Orthogonalization Near Optima
Sep 26, 2026Xiaohui XieMuon Optimizer Gradient Descent Dynamics
Department of Computer Science University of California, Irvine
Hybrid Momentum Methods for Optimizing Deep Learning Models
Oct 20, 2026Haifeng Xu, Yijin Ren, Chenyu Zhou +3Deep Learning Optimization Momentum Methods
Shanghai University of Finance and Economics · Columbia University · Shanghai Jiao Tong University +2

Aug 13, 2026J/K move · Enter open · S save

Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization

Zhixin Ren, Yao Lyu, Congrong Li, Liping Zhang+1

Department of Mathmatical Science, Tsinghua University · College of AI, Tsinghua University · School of Vehicle and Mobility, Tsinghua University

Deep Learning Optimization Momentum Methods Adaptive Gradient Methods Accelerated Gradient Methods

Open paper AskPDF arXiv

Momentum-based optimizers are widely used in modern deep learning, yet the relations among momentum recursion, update geometry, and acceleration remain only partially understood. We develop an

\textbf{A}

DMM-

\textbf{I}

nspired

\textbf{M}

omentum (AIM) framework based on residual-penalty variable splitting, which interprets momentum as a multiplier-like correction driven by the splitting residual. AIM recovers the exponential moving average of gradients from an ADMM-style multiplier update and separates two mechanisms that are usually intertwined in practical optimizers: the residual penalty determines the update geometry, whereas the approximation of the objective-related subproblem determines the acceleration form. Building on AIM, we propose

\textbf{R}

elativistic

\textbf{A}

daptive gradient

\textbf{D}

escent with

\textbf{A}

ccelerated

\textbf{R}

esidual (RADAR), which combines relativistic adaptive geometry, decoupled residual correction, and second-order momentum filtering to improve the update direction and momentum estimation. We establish stochastic convergence through a variance-perturbed Lyapunov drift analysis. Experiments on supervised vision learning, language modeling, and reinforcement learning show that RADAR achieves consistent improvements over strong adaptive optimizer baselines.

Abstract

Deep Learning Optimization

Explore similar work

Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization

Muon Under Gradient Noise and the Limits of Orthogonalization Near Optima

Hybrid Momentum Methods for Optimizing Deep Learning Models