MAGDrop: Momentum Adaptive Gradient Dropout With Provable Generalization Guarantees
Organizations: Quaid-i-Azam University, Islamabad
Abstract
Standard dropout applies a fixed, uniform noise rate to all units, ignoring available gradient information. We propose momentum adaptive gradient dropout (MAGDrop), a Bayesian linear layer that adapts per-unit weight noise by tracking an exponential moving average of gradient magnitudes. Units with large recent gradient magnitudes receive less noise to preserve learned features; units with below-average gradient magnitudes receive more noise to promote exploration. This adaptation transforms the classifier head into a data-dependent Bayesian predictor whose mean predictor has no additional inference cost. Our primary contribution is a certificate-aware low-rank posterior, MAGDrop-Subspace, that addresses the tension between strong accuracy and tight bounds from probably approximately correct (PAC)-Bayes theory. A full-head stochastic posterior incurs a Kullback--Leibler (KL) divergence of approximately 8,625 nats. By confining the posterior to an -dimensional subspace constructed from an independent prior-construction sample and certifying on an independent sample with , MAGDrop-Subspace reduces KL by approximately a factor of 800 while keeping accuracy close to the baseline. On CIFAR-10 with a 13-layer convolutional neural network and , MAGDrop-Subspace achieves a PAC-Bayes Gibbs-risk bound of 0.1882 at 90% confidence using binary KL inversion, compared with 0.4439 for the full-head baseline. Mean-predictor accuracy is 82.31%, compared with 83.74% for the baseline. Complementary accuracy experiments on ImageNet-1K show that MAGDrop reaches top-1 accuracy similar to the strong baseline: 78.57% compared with 78.51% on the base Vision Transformer with patches, and 82.14% compared with 82.23% on ConvNeXt-Base.
Figures & tables
| Symbols | Description |
|---|---|
| , , | Output classes, input dimension, and subspace dimension |
| , | Per-unit posterior and Phase 1 prior means in |
| , | Phase 1 head and orthonormal basis |
| Latent posterior mean | |
| Latent-mean learning rate | |
| , | Posterior variances per head unit and latent direction |
| Random QR | PCA | |||||
| Rank | Gibbs risk | KL (nats) | PAC-Bayes- bound | Gibbs risk | KL (nats) | PAC-Bayes- bound |
| 0.1691 | 0.1825 | 1.03 | ||||
| 0.1750 | 0.1894 | 2.19 | ||||
| 0.1705 | 0.1882 | 7.17 | ||||
| 0.1685 | 22.04 | 0.1902 | ||||
| 0.1650 | 67.83 | 0.1980 | ||||
| Configuration | Time (s) | Change (%) |
| Training per epoch | ||
| Phase 1, deterministic | 2.41 | – |
| Phase 2, subspace, | 2.35 | |
| Phase 2, baseline, 6,000 parameters | 2.43 | |
| Inference on the full test set | ||
| Deterministic, one pass | 1.55 | |
| Method | Configuration | Gibbs risk | KL (nats) | PAC-Bayes- bound |
| External references: 13-layer CNN , Pérez-Ortiz et al. [25, Table 5] | ||||
| 50% prior data | – | – | ||
| 50% prior data | – | – | ||
| 70% prior data | – | – | ||
| Full-head stochastic posteriors with large KL | ||||
| Full-head baseline | Fixed | |||
| Group | Configuration | Gibbs risk | KL (nats) | PAC-Bayes- bound | |
|---|---|---|---|---|---|
| A: Clamp | Wider, | ||||
| Default † , | |||||
| Tighter, | |||||
| B: Momentum | Fast, | ||||
| Slow, | |||||
| C: Coefficient | Gentle, | 0.1596 | 0.1758 ± 0.0027 |
| Method | Accuracy (%) | Gibbs risk | KL (nats) | PAC-Bayes- bound |
|---|---|---|---|---|
| ResNet-20 deterministic | 84.43 | – | – | – |
| MAGDrop- Subspace | 0.1996 ± 0.0107 | |||
| Subspace, fixed |
| Model | Method | Top-1 accuracy (%) |
|---|---|---|
| ViT-Base/16 | Baseline | 78.51 |
| MAGDrop | 78.57 | |
| ConvNeXt-Base | Baseline | 82.23 |
| MAGDrop | 82.14 |
Bayesian Neural Networks
Bayesian neural networks place probability distributions over neural-network parameters or functions to quantify predictive uncertainty and support robust decisions.
See all 101 papers