Probing SAM Perturbations Through Statistical Replay
Organizations: TU Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany · Institute for Artificial Intelligence in Medicine, UK Essen, Germany
Abstract
Sharpness Aware Minimization (SAM) updates model parameters using gradients evaluated at adversarially perturbed weights, but it remains unclear which aspects of this procedure are sufficient to reproduce its generalization behavior. In our ResNet 18 experiments on CIFAR 10, simple curvature regularizers based on the Hessian trace and largest eigenvalue remain below SAM in validation accuracy, motivating a direct examination of its perturbations. We introduce statistical replay, a diagnostic that summarizes perturbations from reference SAM trajectories using elementwise means and standard deviations over local windows and constructs new perturbations during independent SGD training. Pure replay uses only these summaries, whereas mixed replay adds isotropic Gaussian noise. Pure replay does not reproduce SAM's validation behavior, while mixed replay and pure isotropic perturbations can approach its validation performance at selected scales without reproducing its training dynamics. An equal radius random direction within SAM performs similarly to SGD, and source, temporal, and layer interventions further suggest that replay behavior depends on perturbation construction. Together, these preliminary results weaken several simple explanations of SAM while leaving an important part of its mechanism unexplained.
Figures & tables
| Method | WD | Late Train | Late Val. | Late (%) |
| Standard SAM | 99.996 | 80.474 | 93.33 | |
| Random direction SAM | 99.031 | 76.475 | 0.00 | |
| Standard SGD | 99.032 | 76.693 | 0.00 | |
| Standard SGD | 99.959 | 78.077 | 0.00 | |
| Isotropic | 98.483 | 79.523 | 6.67 | |
| Isotropic | 95.665 | 80.109 | 64.00 |
| Pure replay | Mixed replay | ||||
| Ordering | Train | Val. | Train | Val. | |
| Batch | 99.981 | 78.066 | 99.391 | 79.622 | |
| 99.976 | 78.278 | 98.726 | 79.529 | ||
| 99.965 | 78.281 | 97.230 | 79.281 | ||
| Epoch | 99.972 | 77.897 | 99.388 | 79.601 | |
| 99.985 | 78.674 | 98.700 | 79.609 | ||
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Seed | Train Acc. | Val. Acc. | Best Val. | Late | |
| 1 | 99.408 | 79.671 | 80.020 | 2/50 | |
| 42 | 99.397 | 79.979 | 80.290 | 25/50 | |
| 15213 | 99.410 | 79.665 | 80.010 | 0/50 | |
| 1 | 98.695 | 79.839 | 80.320 | 8/50 | |
| 42 | 98.761 | 79.782 | 80.560 | 2/50 | |
| 15213 | 98.662 | 79.876 | 80.340 | 12/50 |
| Seed | Train Acc. | Val. Acc. | Best Val. | Late | |
| 1 | 99.965 | 78.576 | 79.290 | 0/50 | |
| 42 | 99.965 | 78.304 | 79.010 | 0/50 | |
| 15213 | 99.963 | 78.579 | 79.110 | 0/50 | |
| 1 | 99.966 | 78.565 | 79.450 | 0/50 | |
| 42 | 99.977 | 78.396 | 79.070 | 0/50 | |
| 15213 | 99.956 | 78.152 | 78.960 | 0/50 |
| Method | Seed | Train Acc. | Val. Acc. | Best Val. | Late |
| Standard SAM | 1 | 99.999 | 80.280 | 80.810 | 45/50 |
| 42 | 99.998 | 80.693 | 81.270 | 50/50 | |
| 15213 | 99.992 | 80.449 | 80.920 | 45/50 |
| Method | Seed | Train Acc. | Val. Acc. | Best Val. | Late |
| Random-direction SAM | 1 | 99.035 | 76.377 | 78.070 | 0/50 |
| 42 | 99.023 | 76.424 | 77.870 | 0/50 | |
| 15213 | 99.036 | 76.625 | 78.190 | 0/50 |
| Method | Seed | Train Acc. | Val. Acc. | Best Val. | Late |
| SGD | 1 | 99.962 | 77.878 | 78.930 | 0/50 |
| 42 | 99.968 | 78.411 | 79.210 | 0/50 | |
| 15213 | 99.946 | 77.943 | 78.640 | 0/50 | |
| SGD + WD | 1 | 99.036 | 76.983 | 78.150 | 0/50 |
| 42 | 99.021 | 76.529 | 77.560 | 0/50 | |
| 15213 | 99.040 | 76.567 | 77.760 | 0/50 |
| Seed | Train Acc. | Val. Acc. | Best Val. | Late | |
| 1 | 98.457 | 79.301 | 79.810 | 0/50 | |
| 42 | 98.523 | 79.859 | 80.340 | 10/50 | |
| 15213 | 98.470 | 79.408 | 79.980 | 0/50 | |
| 1 | 95.930 | 80.082 | 80.430 | 36/50 | |
| 42 | 95.605 | 79.834 | 80.440 | 10/50 | |
| 15213 | 95.460 | 80.412 | 80.850 | 50/50 |
| Replay | SAM Source | Late Train | Late Val. | Best Val. | Late | |
| Pure | Seed 1 | 99.987 | 78.193 | 78.910 | 0/50 | |
| Seed 15213 | 99.994 | 78.347 | 78.680 | 0/50 | ||
| Seed 1 | 99.984 | 78.477 | 79.110 | 0/50 | ||
| Seed 15213 | 99.983 | 78.436 | 79.000 | 0/50 | ||
| Seed 1 | 99.960 | 78.147 | 78.790 | 0/50 | ||
| Seed 15213 | 99.956 | 78.107 | 79.070 | 0/50 |
| Replay | Setting | Train Acc. | Val. Acc. | Best Val. | Late (%) | |
| Pure | Aligned | |||||
| Batch shuffle | ||||||
| Aligned | ||||||
| Batch shuffle | ||||||
| Aligned | ||||||
| Batch shuffle |
| Replay | Setting | Train Acc. | Val. Acc. | Best Val. | Late (%) | |
| Pure | Aligned | |||||
| Epoch shuffle | ||||||
| Aligned | ||||||
| Epoch shuffle | ||||||
| Aligned | ||||||
| Epoch shuffle |
| Replay | Late Train | Late Val. | Best Val. | Late | |
| Pure | 99.995 | 78.202 | 78.570 | 0/50 | |
| Pure | 99.984 | 78.645 | 79.150 | 0/50 | |
| Pure | 99.992 | 78.433 | 78.970 | 0/50 | |
| Mixed | 99.407 | 79.498 | 79.940 | 0/50 | |
| Mixed | 98.718 | 79.790 | 80.180 | 6/50 | |
| Mixed | 96.759 | 79.542 | 80.150 | 0/50 |
| Replay | Layers | Late Train | Late Val. | Best Val. | Late (%) | |
| Pure | All | |||||
| L1–L4 | ||||||
| L1–L4 + FC | ||||||
| FC only | ||||||
| All | ||||||
| L1–L4 |
| Method | WD | Late Train | Late Val. | Best Val. | Late (%) |
| Standard SAM | |||||
| SGD | |||||
| Pure Iso | |||||
| Method | Late Train | Late Val. | Best Val. | Late (%) |
| Fixed Trace ( ) | ||||
| Adaptive Trace (30% cap) | ||||
| Spectral ( ) |
Sharpness-Aware Minimization
Sharpness-Aware Minimization (SAM) trains models toward locally flat loss regions to improve generalization and robustness.
See all 42 papers