cs.LGOct 21, 2025

MAGDrop: Momentum Adaptive Gradient Dropout With Provable Generalization Guarantees

Authors: Adeel Safder, Bum Jun Kim

Organizations: Quaid-i-Azam University, Islamabad

Abstract

Standard dropout applies a fixed, uniform noise rate to all units, ignoring available gradient information. We propose momentum adaptive gradient dropout (MAGDrop), a Bayesian linear layer that adapts per-unit weight noise by tracking an exponential moving average of gradient magnitudes. Units with large recent gradient magnitudes receive less noise to preserve learned features; units with below-average gradient magnitudes receive more noise to promote exploration. This adaptation transforms the classifier head into a data-dependent Bayesian predictor whose mean predictor has no additional inference cost. Our primary contribution is a certificate-aware low-rank posterior, MAGDrop-Subspace, that addresses the tension between strong accuracy and tight bounds from probably approximately correct (PAC)-Bayes theory. A full-head stochastic posterior incurs a Kullback--Leibler (KL) divergence of approximately 8,625 nats. By confining the posterior to an rr-dimensional subspace constructed from an independent prior-construction sample S0S_0 and certifying on an independent sample SS with S0⊥SS_0\perp S, MAGDrop-Subspace reduces KL by approximately a factor of 800 while keeping accuracy close to the baseline. On CIFAR-10 with a 13-layer convolutional neural network and r=50r=50, MAGDrop-Subspace achieves a PAC-Bayes Gibbs-risk bound of 0.1882 at 90% confidence using binary KL inversion, compared with 0.4439 for the full-head baseline. Mean-predictor accuracy is 82.31%, compared with 83.74% for the baseline. Complementary accuracy experiments on ImageNet-1K show that MAGDrop reaches top-1 accuracy similar to the strong baseline: 78.57% compared with 78.51% on the base Vision Transformer with 16×1616\times16 patches, and 82.14% compared with 82.23% on ConvNeXt-Base.

Figures & tables

Explore similar work

CardsList
  1. DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum

    Jun 3, 2026Naima Tasnim, Lalitha Sankar, Oliver KosutDifferentially Private Stochastic Gradient DescentGradient Clipping

  2. Robust and Fast Training via Per-Sample Clipping

    May 4, 2026Davide Nobile, Philipp GrohsGradient ClippingNonconvex Stochastic Optimization