Standard dropout applies a fixed, uniform noise rate to all units, ignoring available gradient information. We propose momentum adaptive gradient dropout (MAGDrop), a Bayesian linear layer that adapts per-unit weight noise by tracking an exponential moving average of gradient magnitudes. Units with large recent gradient magnitudes receive less noise to preserve learned features; units with below-average gradient magnitudes receive more noise to promote exploration. This adaptation transforms the classifier head into a data-dependent Bayesian predictor whose mean predictor has no additional inference cost. Our primary contribution is a certificate-aware low-rank posterior, MAGDrop-Subspace, that addresses the tension between strong accuracy and tight bounds from probably approximately correct (PAC)-Bayes theory. A full-head stochastic posterior incurs a Kullback--Leibler (KL) divergence of approximately 8,625 nats. By confining the posterior to an r-dimensional subspace constructed from an independent prior-construction sample S0 and certifying on an independent sample S with S0⊥S, MAGDrop-Subspace reduces KL by approximately a factor of 800 while keeping accuracy close to the baseline. On CIFAR-10 with a 13-layer convolutional neural network and r=50, MAGDrop-Subspace achieves a PAC-Bayes Gibbs-risk bound of 0.1882 at 90% confidence using binary KL inversion, compared with 0.4439 for the full-head baseline. Mean-predictor accuracy is 82.31%, compared with 83.74% for the baseline. Complementary accuracy experiments on ImageNet-1K show that MAGDrop reaches top-1 accuracy similar to the strong baseline: 78.57% compared with 78.51% on the base Vision Transformer with 16×16 patches, and 82.14% compared with 82.23% on ConvNeXt-Base.
Figures & tables
Fig. 1: Overview of MAGDrop and MAGDrop-Subspace. (a) Gradient-adaptive noise; circle widths show spread around mean weights. (b) Low-dimensional posterior for Gibbs-risk certification.
Symbols
Description
K , d , r
Output classes, input dimension, and subspace dimension
μj , μj(0)
Per-unit posterior and Phase 1 prior means in Rd
W0 , U
Phase 1 head and orthonormal basis U∈RKd×r
μz∈Rr
Latent posterior mean
ηz=10−5
Latent-mean learning rate
σj2 , σz,i2
Posterior variances per head unit and latent direction
TABLE I: Summary of Notation
Random QR
PCA
Rank r
Gibbs risk
KL (nats)
PAC-Bayes- kl bound
Gibbs risk
KL (nats)
PAC-Bayes- kl bound
r=10
0.1691
1.3
0.1825
0.2384
1.03
0.2529
r=20
0.1750
2.9
0.1894
0.1985
2.19
0.2130
r=50
0.1705
10.77
0.1882
0.1778
7.17
0.1942
r=100
0.1735
26.86
0.1969
0.1685
22.04
0.1902
r=200
0.1682
85.85
0.2050
0.1650
67.83
0.1980
TABLE II: Random QR and PCA Across Subspace Ranks
Configuration
Time (s)
Change (%)
Training per epoch
Phase 1, deterministic
2.41
–
Phase 2, subspace, r=50
2.35
−2.5
Phase 2, baseline, 6,000 parameters
2.43
+0.8
Inference on the full test set
Deterministic, one pass
1.55
0.0
TABLE III: CIFAR-10 Timing for a 13-Layer CNN on Tesla T4
Fig. 2: Random-QR MAGDrop-Subspace on CIFAR-10 with a 13-layer CNN. (a) KL and (b) PAC-Bayes- kl bounds versus rank; (c) variance ablations at r=50 ; (d) risk–KL frontier with rank labels and a broken risk axis. Bands and error bars show three-seed means ± SD. The dashed line and star denote the full-head baseline.
Group
Configuration
Gibbs risk
KL (nats)
PAC-Bayes- kl bound
Δ
A: Clamp
Wider, [0.90,1.10]
0.1608
9.0
0.1775(±0.0038)
−0.0054
Default † , [0.95,1.05]
0.1661
8.9
0.1829(±0.0031)
0.0000
Tighter, [0.99,1.01]
0.1601
8.8
0.1767(±0.0057)
−0.0062
B: Momentum
Fast, β=0.70
0.1628
8.5
0.1793(±0.0035)
−0.0036
Slow, β=0.95
0.1618
9.2
0.1786(±0.0081)
−0.0043
C: Coefficient
Gentle, c=0.10
0.1596
8.0
0.1758 ± 0.0027
−0.0071
TABLE V: MAGDrop-Subspace Hyperparameter Sensitivity at r=50 with Random QR
Method
Accuracy (%)
Gibbs risk
KL (nats)
PAC-Bayes- kl bound
ResNet-20 deterministic
84.43
–
–
–
MAGDrop- Subspace
82.00
0.1833
6.56
0.1996 ± 0.0107
Subspace, fixed γ
82.02
0.1837
6.73
0.2001(±0.0035)
TABLE VI: ResNet-20 Certificates on CIFAR-10 with Random QR, r=50
Model
Method
Top-1 accuracy (%)
ViT-Base/16
Baseline
78.51
MAGDrop
78.57
ConvNeXt-Base
Baseline
82.23
MAGDrop
82.14
TABLE VII: ImageNet-1K Top-1 Accuracy
Fig. 3: ImageNet-1K results from single 300-epoch runs using EMA model weights. (a)–(b) ConvNeXt-Base accuracy and validation loss digitized from training plots; (c) best top-1 accuracy.
Fig. 4: Basis geometry on CIFAR-10 with ResNet-20. (a) Directional Gauss–Newton curvature at W0 on S for seed 42; shading spans the top nine eigenvalues, and the line marks the mean eigenvalue. (b) Top-nine eigenspace capture; lines show three-seed means and faint points show individual seeds.
Fig. 5: Controlled random-QR/PCA comparison on CIFAR-10 with ResNet-20. (a) KL, (b) mean-predictor test error, (c) empirical Gibbs risk on S , and (d) PAC-Bayes- kl bound. Lines show three-seed means; faint markers show individual seeds.
Fig. 6: Posterior sensitivity of six anchor-correct CIFAR-10 test images with ResNet-20 at Phase 2 initialization ( r=10 , seed 42). Switching rates use 8,192 fresh draws. Segments show the anchor margin ±2 exact Gaussian SDs against the most-voted wrong class under PCA in these draws; shading marks negative margins.
Bayesian neural networks place probability distributions over neural-network parameters or functions to quantify predictive uncertainty and support robust decisions.
Differentially private stochastic gradient descent (DP-SGD) has become the standard framework for privacy-preserving machine learning, yet its reliance on a fixed gradient clipping threshold to limit sensitivity remains a significant practical limitation. Adaptive clipping algorithms such as AdaClip shift and scale the gradient prior to clipping and adding noise so that the clipped gradient yields a more informative descent direction. The shift and scaling parameters are selected adaptively based on the empirical mean and variance. However, in existing adaptive clipping algorithms, these empirical estimates have not been also used for momentum to accelerate training itself. On the other hand, DP-Adam is an algorithm that exploits Adam-like momentum updates based on the gradient mean and variance to accelerate training, but does not exploit these estimates for adaptive clipping. In this work, we propose Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum (DP-MacAdam), a novel algorithm that combines these two approaches so as to use the same mean and variance estimates for both clipping and momentum. We perform an analysis showing that DP-MacAdam estimates the gradient variances in a bias-free manner. In addition, we empirically evaluate the privacy and accuracy of DP-MacAdam, demonstrating that it achieves improved model utility compared to DP-SGD, AdaClip, and DP-Adam baselines, without requiring manual tuning of the clipping threshold.
Naima Tasnim, Lalitha Sankar, Oliver Kosut
Arizona State University, Tempe, AZ, United States
We propose a robust gradient estimator based on per-sample gradient clipping and analyze its properties both theoretically and empirically. We show that the resulting method, per-sample clipped SGD (PS-Clip-SGD), achieves optimal in-expectation convergence rates for non-convex optimization problems under heavy-tailed gradient noise. Moreover, we establish high-probability convergence guarantees that match the in-expectation rates up to polylogarithmic factors in the failure probability. We complement our theoretical results with multiple numerical experiments. In particular, we demonstrate that PS-Clip-SGD outperforms both vanilla SGD with momentum and standard gradient clipping when training AlexNet on the CIFAR-100 dataset, even after accounting for the additional computational time caused by per-sample clipping. We also empirically show that, in the presence of gradient accumulation, applying clipping at the mini-batch level can improve training performance while incurring virtually no additional computational cost. This finding is particularly interesting, as it contradicts the common practice of applying clipping only after all accumulation steps have been completed.
Davide Nobile, Philipp Grohs
Department of Mathematics University of Vienna · RICAM Austrian Academy of Sciences
Adaptive optimizers are widely used in deep learning, yet their behavior can degrade under heavy stochasticity and label noise where the sign and magnitude of gradients can be unstable. We study MGO-SC (Moment-Guided Optimizer with Sign-Consistency), a lightweight modification of Adam-style methods that derives a coordinate-wise consistency gate from the magnitude of a sign exponential moving average (EMA): coordinates whose gradient sign is stable are trusted, while coordinates whose sign fluctuates are down-weighted. We show that the naive form of this gate, which can only shrink updates, loses effective step size and underperforms when the learning rate is matched to the baseline. We therefore introduce a mean-preserving variant that reallocates trust across coordinates without changing the net per-tensor step magnitude. On CIFAR-10 with ResNet-18, under a matched learning rate and decoupled weight decay, mean-preserving MGO-SC consistently improves over its AdamW baseline across clean, symmetric, and asymmetric label noise (e.g., +1.4 points at 40% symmetric noise; paired bootstrap 95% CI excluding zero; Wilcoxon p<0.01), while adding no second-order information or per-sample filtering. We report matched-seed effect sizes, nonparametric bootstrap confidence intervals, and paired significance tests, and we release code and raw logs.
Alicem Koyun
Master of Science in Computer Science, Westcliff University, Irvine, USA