Moment-Guided Optimizers for Noisy Deep Learning: MGO-SC as a Simple Consistency Gate for Adam-Style Updates

Alicem Koyun

cs.LGOct 27, 2026

Moment-Guided Optimizers for Noisy Deep Learning: MGO-SC as a Simple Consistency Gate for Adam-Style Updates

Authors: Alicem Koyun

Organizations: Master of Science in Computer Science, Westcliff University, Irvine, USA

Abstract

Adaptive optimizers are widely used in deep learning, yet their behavior can degrade under heavy stochasticity and label noise where the sign and magnitude of gradients can be unstable. We study MGO-SC (Moment-Guided Optimizer with Sign-Consistency), a lightweight modification of Adam-style methods that derives a coordinate-wise consistency gate from the magnitude of a sign exponential moving average (EMA): coordinates whose gradient sign is stable are trusted, while coordinates whose sign fluctuates are down-weighted. We show that the naive form of this gate, which can only shrink updates, loses effective step size and underperforms when the learning rate is matched to the baseline. We therefore introduce a mean-preserving variant that reallocates trust across coordinates without changing the net per-tensor step magnitude. On CIFAR-10 with ResNet-18, under a matched learning rate and decoupled weight decay, mean-preserving MGO-SC consistently improves over its AdamW baseline across clean, symmetric, and asymmetric label noise (e.g., +1.4 points at 40% symmetric noise; paired bootstrap 95% CI excluding zero; Wilcoxon p<0.01), while adding no second-order information or per-sample filtering. We report matched-seed effect sizes, nonparametric bootstrap confidence intervals, and paired significance tests, and we release code and raw logs.

Figures & tables

Noise	Optimizer	Test Acc (%)
$\eta=0.0$	Adam ( $L_{2}$ )	$83.03\pm 0.41$
	AdamW	$82.06\pm 0.21$
	AMSGrad	$82.86\pm 0.28$
	SGD+momentum	$86.61\pm 0.10$
	AdamW + GCE	$80.46\pm 0.15$
	MGO-SC (shrink)	$79.05\pm 0.17$

TABLE I: Main results on CIFAR-10 / ResNet-18 under symmetric label noise, at a matched learning rate (

3\times 10^{-4}

for all Adam-family methods; SGD at

0.1

). Test accuracy (%) at best-validation checkpoint, mean

\pm

std over

n

seeds. MGO-SC (mean-preserving) is the proposed method; its controlled baseline is AdamW.

Fig. 1: Noise $=0.4$ ( $n=8$ ): distribution-aware paired comparison of mean-preserving MGO-SC vs. AdamW at a matched learning rate. The violin/box summarizes per-seed test accuracy at best validation; lines connect matched seeds.

$\gamma$	Shrink gate (%)	Mean-preserving (%)
$0.0$ (=AdamW)	$73.49\pm 0.54$	$73.49\pm 0.54$
$0.3$	$71.35\pm 0.51$	$74.64\pm 0.41$
$0.5$	$70.06\pm 0.25$	$\mathbf{74.97\pm 0.29}$
$0.7$	$68.69\pm 0.58$	$75.04\pm 0.42$

TABLE II: Gate ablation on CIFAR-10 / ResNet-18 at

\eta=0.4

, fixed learning rate

3\times 10^{-4}

n=8

. Test accuracy (%) at best validation.

\gamma=0

disables the gate and equals AdamW. The shrink gate degrades monotonically with

\gamma

; the mean-preserving gate improves with

\gamma

, isolating the benefit as trust reallocation rather than step-size reduction.

Optimizer	Test Acc (%)
Adam ( $L_{2}$ )	$76.00\pm 0.48$
AdamW	$73.70\pm 0.86$
AdamW + GCE	$71.94\pm 0.42$
MGO-SC (shrink)	$69.89\pm 0.90$
MGO-SC (mean-pres.)	$\mathbf{74.40\pm 0.51}$

TABLE III: Asymmetric (class-dependent) label noise at

\eta=0.4

on CIFAR-10 / ResNet-18, matched learning rate,

n=5

. Test accuracy (%) at best validation.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 2: CIFAR-10 / ResNet-18 test accuracy under 40% label noise (matched LR). Error bars indicate $\pm$ 1 std across seeds.

Fig. 3: Paired seed-wise comparison (same seeds for both optimizers). This view highlights whether an improvement is consistent across runs.

Fig. 4: Bootstrap intervals (nonparametric) for the mean test-accuracy delta between mean-preserving MGO-SC and AdamW under 40% label noise ( $n=8$ ).

Adaptive Gradient Methods

Adaptive Gradient Methods adjust updates using gradient history, curvature, or noise estimates to improve optimization across diverse problems.

See all 161 papers

Directional Consistency as a Complementary Optimization Signal: The GONO Framework
May 7, 2026Victor Daniel GeraDeep Learning Optimization Stochastic Optimization Convergence
Department of Artificial Intelligence Anurag University, Hyderabad, India
Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum
Feb 19, 2026Minxin Zhang, Yuxuan Liu, Hayden SchaefferDeep Learning Optimization Adaptive Gradient Methods
Department of Mathematics, University of California, Los Angeles, Los Angeles, CA 90095, USA
Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?
Jun 22, 2026Dingzhi Yu, Hongyi Tao, Yuanyu Wan +2Stochastic Optimization Heavy-Tailed Noise
1Nanjing University · 2Zhejiang University · 3Fudan University

May 7, 2026J/K move · Enter open · S save

Directional Consistency as a Complementary Optimization Signal: The GONO Framework

Victor Daniel Gera

Department of Artificial Intelligence Anurag University, Hyderabad, India

Deep Learning Optimization Stochastic Optimization Convergence Adaptive Gradient Methods Gradient Alignment

Open paper AskPDF arXiv

We identify and formalize an underexplored phenomenon in deep learning optimization: directional alignment and loss convergence can be decoupled. An optimizer can exhibit near-perfect directional consistency (cc_t -> 1, measured via consecutive gradient cosine similarity) while the loss remains high or decreases slowly. This observation reveals that existing optimizers such as Adam, SGD, and RMSprop lack explicit mechanisms to exploit temporal consistency in gradient directions, relying instead on magnitude-based signals that fail to distinguish plateaus, saddle points, and genuine convergence. Motivated by this, we introduce GONO (Gradient-Oriented Norm-Adaptive Optimizer), which adapts Adam's momentum coefficient beta_1 based on cc_t: amplifying momentum under directional consistency and suppressing it during oscillation. We prove GONO matches Adam's O(1/sqrt(T)) convergence rate and reduces exactly to Adam when the signal is uninformative. Empirically, cc_t achieves oscillation detection with F1=1.00 (vs. 0.45 for gradient norm), and GONO remains competitive with AdamW on MNIST (98.15%), CIFAR-10 (43.14%), and ResNet-18 (75.44%), establishing directional alignment as a theoretically grounded, practically actionable optimization signal. Code: https://github.com/victordaniel/gono-optimizer

Abstract

Adaptive Gradient Methods

Explore similar work

Directional Consistency as a Complementary Optimization Signal: The GONO Framework

Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum

Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?