cs.LGOct 21, 2025

MAGDrop: Momentum Adaptive Gradient Dropout With Provable Generalization Guarantees

Authors: Adeel Safder, Bum Jun Kim

Organizations: Quaid-i-Azam University, Islamabad

Abstract

Standard dropout applies a fixed, uniform noise rate to all units, ignoring available gradient information. We propose momentum adaptive gradient dropout (MAGDrop), a Bayesian linear layer that adapts per-unit weight noise by tracking an exponential moving average of gradient magnitudes. Units with large recent gradient magnitudes receive less noise to preserve learned features; units with below-average gradient magnitudes receive more noise to promote exploration. This adaptation transforms the classifier head into a data-dependent Bayesian predictor whose mean predictor has no additional inference cost. Our primary contribution is a certificate-aware low-rank posterior, MAGDrop-Subspace, that addresses the tension between strong accuracy and tight bounds from probably approximately correct (PAC)-Bayes theory. A full-head stochastic posterior incurs a Kullback--Leibler (KL) divergence of approximately 8,625 nats. By confining the posterior to an rr-dimensional subspace constructed from an independent prior-construction sample S0S_0 and certifying on an independent sample SS with S0⊥SS_0\perp S, MAGDrop-Subspace reduces KL by approximately a factor of 800 while keeping accuracy close to the baseline. On CIFAR-10 with a 13-layer convolutional neural network and r=50r=50, MAGDrop-Subspace achieves a PAC-Bayes Gibbs-risk bound of 0.1882 at 90% confidence using binary KL inversion, compared with 0.4439 for the full-head baseline. Mean-predictor accuracy is 82.31%, compared with 83.74% for the baseline. Complementary accuracy experiments on ImageNet-1K show that MAGDrop reaches top-1 accuracy similar to the strong baseline: 78.57% compared with 78.51% on the base Vision Transformer with 16×1616\times16 patches, and 82.14% compared with 82.23% on ConvNeXt-Base.

Figures & tables

Explore similar work

Jun 3, 2026

DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum

Differentially private stochastic gradient descent (DP-SGD) has become the standard framework for privacy-preserving machine learning, yet its reliance on a fixed gradient clipping threshold to limit sensitivity remains a significant practical limitation. Adaptive clipping algorithms such as AdaClip shift and scale the gradient prior to clipping and adding noise so that the clipped gradient yields a more informative descent direction. The shift and scaling parameters are selected adaptively based on the empirical mean and variance. However, in existing adaptive clipping algorithms, these empirical estimates have not been also used for momentum to accelerate training itself. On the other hand, DP-Adam is an algorithm that exploits Adam-like momentum updates based on the gradient mean and variance to accelerate training, but does not exploit these estimates for adaptive clipping. In this work, we propose Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum (DP-MacAdam), a novel algorithm that combines these two approaches so as to use the same mean and variance estimates for both clipping and momentum. We perform an analysis showing that DP-MacAdam estimates the gradient variances in a bias-free manner. In addition, we empirically evaluate the privacy and accuracy of DP-MacAdam, demonstrating that it achieves improved model utility compared to DP-SGD, AdaClip, and DP-Adam baselines, without requiring manual tuning of the clipping threshold.
May 4, 2026

Robust and Fast Training via Per-Sample Clipping

We propose a robust gradient estimator based on per-sample gradient clipping and analyze its properties both theoretically and empirically. We show that the resulting method, per-sample clipped SGD (PS-Clip-SGD), achieves optimal in-expectation convergence rates for non-convex optimization problems under heavy-tailed gradient noise. Moreover, we establish high-probability convergence guarantees that match the in-expectation rates up to polylogarithmic factors in the failure probability. We complement our theoretical results with multiple numerical experiments. In particular, we demonstrate that PS-Clip-SGD outperforms both vanilla SGD with momentum and standard gradient clipping when training AlexNet on the CIFAR-100 dataset, even after accounting for the additional computational time caused by per-sample clipping. We also empirically show that, in the presence of gradient accumulation, applying clipping at the mini-batch level can improve training performance while incurring virtually no additional computational cost. This finding is particularly interesting, as it contradicts the common practice of applying clipping only after all accumulation steps have been completed.
Oct 27, 2026

Moment-Guided Optimizers for Noisy Deep Learning: MGO-SC as a Simple Consistency Gate for Adam-Style Updates

Adaptive optimizers are widely used in deep learning, yet their behavior can degrade under heavy stochasticity and label noise where the sign and magnitude of gradients can be unstable. We study MGO-SC (Moment-Guided Optimizer with Sign-Consistency), a lightweight modification of Adam-style methods that derives a coordinate-wise consistency gate from the magnitude of a sign exponential moving average (EMA): coordinates whose gradient sign is stable are trusted, while coordinates whose sign fluctuates are down-weighted. We show that the naive form of this gate, which can only shrink updates, loses effective step size and underperforms when the learning rate is matched to the baseline. We therefore introduce a mean-preserving variant that reallocates trust across coordinates without changing the net per-tensor step magnitude. On CIFAR-10 with ResNet-18, under a matched learning rate and decoupled weight decay, mean-preserving MGO-SC consistently improves over its AdamW baseline across clean, symmetric, and asymmetric label noise (e.g., +1.4 points at 40% symmetric noise; paired bootstrap 95% CI excluding zero; Wilcoxon p<0.01), while adding no second-order information or per-sample filtering. We report matched-seed effect sizes, nonparametric bootstrap confidence intervals, and paired significance tests, and we release code and raw logs.