A Function-Centric Perspective on Flat and Sharp Minima

Israel Mason-Williams; Gabryel Mason-Williams; Helen Yannakoudakis

cs.LGOct 14, 2025

A Function-Centric Perspective on Flat and Sharp Minima

Authors: Israel Mason-Williams, Gabryel Mason-Williams, Helen Yannakoudakis

Organizations: Department of Informatics, UKRI Safe and Trusted AI, London, United Kingdom · Queen Mary University of London, London, United Kingdom · Department of Informatics, King’s College London, London, United Kingdom

Abstract

Flat minima are strongly associated with improved generalisation in deep neural networks. However, this connection has proven nuanced in recent studies, with both theoretical counterexamples and empirical exceptions emerging in the literature. In this paper, we revisit the role of sharpness in model performance and examine sharpness as a function-dependent property rather than as a direct indicator of poor generalisation. We conduct extensive empirical studies ranging from single-objective optimisation, synthetic non-linear binary classification tasks, to modern image classification tasks. In single-objective optimisation, we show that flatness and sharpness are relative to the function being learned: equally optimal solutions can exhibit markedly different local geometry. In synthetic non-linear binary classification tasks, we show that increasing decision-boundary tightness can increase sharpness even when models generalise perfectly, suggesting that sharpness is not reducible to memorisation alone. Finally, in large-scale experiments, we find that sharper minima often emerge when models are regularised (e.g., via weight decay, data augmentation, or SAM), and can coincide with better generalisation, calibration, robustness, and functional consistency. Our findings are consistent with a function-centric interpretation of minima geometry: flatness should not be treated as a universal proxy for generalisation, and sharper minima can reflect desirable inductive biases associated with improved predictive behaviour, calling for a function-centric reappraisal of minima geometry.

Figures & tables

Panel of Figure 1 — Figure 1: Scatter plots of 240 minima for ResNet-18 (top) and VGG19 (bottom) on CIFAR-10 across batch size ${128,256}$ and learning rate ${10^{-3},10^{-2}}$ : (a) Fisher–Rao norm vs. train loss, (b) Relative Flatness vs. train loss, and (c) generalisation gap vs. train loss (log scale). Full results for ResNet and VGG in Appendix D.1 and E.1 , respectively.

Figure 2: Himmelblau’s function landscape.

Global Minimum	Condition Number	Hessian Trace	Hessian determinant	Max Eigenvalue
(3.0, 2.0)	3.200	108.000	2116.000	82.284
(-2.805118, 3.131312)	1.242	145.39	5222.890	80.550
(-3.77931, -3.283186)	1.892	204.500	9460.560	133.786
(3.584428, -1.848126)	3.674	134.110	3024.540	105.419

Table 1: Local geometric properties at the four global minima of Himmelblau’s function.

Figure 3: Landscapes for six single-objective functions.

Function	Condition Number	Hessian Trace	Hessian determinant	Max Eigenvalue
Sphere	1.000	4.000	4.000	2.000
Rosenbrock	2508.010	1002.000	400.000	1001.600
Rastrigin	1.000	793.568	157438.000	396.784
Beale	162.473	49.281	14.766	48.980
Booth	9.000	20.000	36.000	18.000
Three hump camel	2.784	6.000	7.000	4.414

Table 2: Sharpness at the global minimum for six single-objective optimisation functions.

Figure 4: Training an MLP on single-objective problems over epochs: mean sharpness, training loss, and absolute generalisation gap (averaged over 10 runs).

Figure 5: Visualising the decision boundaries of an MLP trained on the make_circles data set. The proportion of randomised data is increased in 0.1 intervals between 0.0 and 0.9. Blue indicates the decision boundary for class 1 and red indicates the decision boundary for class 2. As the data is increasingly randomised, the learned decision boundaries become more complex (irregular and tightly constrained).

Figure 6: Fisher-Rao norm (Left) and Relative Flatness (Right) of the resulting minima after increasing the proportion of randomised data.

Figure 7: Visualising the decision boundaries of an MLP trained on the make_circles data set. The scale factor is increased in 0.1 intervals between 0.0 and 0.9. Blue indicates the decision boundary for class 1 and red indicates the decision boundary for class 2. As the Scale factor increases, the decision boundaries between class 1 and 2 become progressively tighter.

Figure 8: Fisher-Rao norm (Left) and Relative Flatness (Right) of the resulting minima after increasing the Scale factor of the data.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline	28.050 $\pm{0.175}$	0.720 $\pm{0.002}$	0.186 $\pm{0.001}$	58.614 $\pm{0.201}$	0.282 $\pm{0.001}$	1.366E-05 $\pm{1.206E-06}$	0.032 $\pm{0.001}$	34.607 $\pm{0.757}$
Baseline + SAM	20.588 $\pm{0.125}$	0.794 $\pm{0.001}$	0.108 $\pm{0.001}$	66.342 $\pm{0.164}$	0.168 $\pm{0.000}$	5.823E-05 $\pm{9.056E-06}$	0.107 $\pm{0.006}$	75.093 $\pm{1.693}$
Augmentation	10.399 $\pm{0.067}$	0.886 $\pm{0.001}$	0.077 $\pm{0.001}$	68.755 $\pm{0.219}$	0.121 $\pm{0.001}$	1.905E-01 $\pm{2.203E-02}$	3.940 $\pm{0.207}$	2903.220 $\pm{89.243}$
Augmentation + SAM	6.864 $\pm{0.038}$	0.908 $\pm{0.000}$	0.014 $\pm{0.001}$	71.419 $\pm{0.283}$	0.069 $\pm{0.000}$	1.303E-01 $\pm{1.547E-02}$	5.571 $\pm{0.035}$	4970.972 $\pm{30.139}$
Weight Decay	27.942 $\pm{0.196}$	0.721 $\pm{0.002}$	0.174 $\pm{0.002}$	58.562 $\pm{0.227}$	0.281 $\pm{0.001}$	3.391E-05 $\pm{4.494E-06}$	0.065 $\pm{0.004}$	59.767 $\pm{3.009}$
Weight Decay + SAM	19.788 $\pm{0.149}$	0.802 $\pm{0.001}$	0.096 $\pm{0.001}$	67.079 $\pm{0.117}$	0.162 $\pm{0.001}$	8.733E-05 $\pm{1.430E-05}$	0.127 $\pm{0.006}$	88.807 $\pm{2.336}$

Table 3: Results for ResNet18 trained on CIFAR10. Bolded values indicate the best performance per metric. For sharpness metrics, lower values correspond to flatter models.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay	✗ (N/A)	✗ (N/A)	✓ (1.0)	✗ (N/A)	✗ (N/A)	✓ (0.96)	✓ (1.0)	✓ (1.0)
Weight Decay + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)

Table 4: Significance results for the control conditions against the baseline for ResNet18 trained on CIFAR10. ✓ indicates significant difference compared to the baseline; ✗ indicates no significance, the value in the brackets represents the effect sizes using the absolute RBC.

Table 5: Loss landscape visualisation Li et al. (2018) of ResNet18 landscape on CIFAR10, exploring the loss in the domain of the perturbations

[-1,1]^{2}

with 51 steps in both directions.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline	47.010 $\pm{0.166}$	0.530 $\pm{0.002}$	0.220 $\pm{0.001}$	38.760 $\pm{0.085}$	0.452 $\pm{0.000}$	2.607E-04 $\pm{3.147E-05}$	0.294 $\pm{0.028}$	32.085 $\pm{0.313}$
Baseline + SAM	44.421 $\pm{0.168}$	0.556 $\pm{0.002}$	0.191 $\pm{0.002}$	41.888 $\pm{0.098}$	0.410 $\pm{0.000}$	4.231E-04 $\pm{4.973E-05}$	0.399 $\pm{0.014}$	123.791 $\pm{4.185}$
Augmentation	29.642 $\pm{0.133}$	0.697 $\pm{0.002}$	0.185 $\pm{0.001}$	44.613 $\pm{0.169}$	0.288 $\pm{0.001}$	1.110E-01 $\pm{9.173E-03}$	3.587 $\pm{0.150}$	2766.925 $\pm{178.669}$
Augmentation + SAM	28.999 $\pm{0.092}$	0.705 $\pm{0.001}$	0.145 $\pm{0.001}$	45.428 $\pm{0.217}$	0.269 $\pm{0.000}$	1.081E-01 $\pm{1.636E-02}$	4.179 $\pm{0.032}$	4196.832 $\pm{52.606}$
Weight Decay	47.838 $\pm{0.301}$	0.521 $\pm{0.003}$	0.099 $\pm{0.005}$	37.868 $\pm{0.265}$	0.474 $\pm{0.001}$	5.192E-04 $\pm{8.009E-05}$	0.861 $\pm{0.116}$	136.969 $\pm{7.484}$
Weight Decay + SAM	45.644 $\pm{0.117}$	0.543 $\pm{0.001}$	0.106 $\pm{0.002}$	40.604 $\pm{0.222}$	0.444 $\pm{0.001}$	1.528E-03 $\pm{1.427E-04}$	1.788 $\pm{0.069}$	360.271 $\pm{16.190}$

Table 6: Results for ResNet18 trained on CIFAR100. Bolded values indicate the best performance per metric. For sharpness metrics, lower values correspond to flatter models.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (0.93)	✓ (0.89)	✓ (1.0)
Augmentation	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay	✗ (N/A)	✗ (N/A)	✓ (1.0)	✗ (N/A)	✗ (N/A)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay + SAM	✓ (0.96)	✓ (0.96)	✓ (1.0)	✓ (1.0)	✓ (0.92)	✓ (1.0)	✓ (1.0)	✓ (1.0)

Table 7: Significance results for the control conditions against the baseline for ResNet18 trained on CIFAR100. ✓ indicates significant difference compared to the baseline; ✗ indicates no significance, the value in the brackets represents the effect sizes using the absolute RBC

Table 8: Loss landscape visualisation Li et al. (2018) of ResNet18 landscape on CIFAR100, exploring the loss in the domain of the perturbations

[-1,1]^{2}

with 51 steps in both directions.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm
Baseline	49.643 $\pm{0.103}$	0.503 $\pm{0.001}$	0.257 $\pm{0.001}$	28.347 $\pm{0.945}$	0.385 $\pm{0.000}$	3.202E-04 $\pm{9.872E-06}$	0.479 $\pm{0.002}$
Baseline + SAM	46.255 $\pm{0.128}$	0.537 $\pm{0.001}$	0.223 $\pm{0.001}$	28.984 $\pm{0.865}$	0.344 $\pm{0.000}$	3.080E-04 $\pm{8.424E-06}$	0.427 $\pm{0.004}$
Augmentation	19.993 $\pm{0.091}$	0.508 $\pm{0.001}$	0.102 $\pm{0.001}$	34.457 $\pm{0.875}$	0.544 $\pm{0.000}$	1.680E+00 $\pm{8.776E-02}$	25.887 $\pm{0.098}$
Augmentation + SAM	16.777 $\pm{0.084}$	0.520 $\pm{0.001}$	0.044 $\pm{0.001}$	35.324 $\pm{0.865}$	0.514 $\pm{0.000}$	1.446E+00 $\pm{6.332E-02}$	25.193 $\pm{0.034}$
Weight Decay	49.689 $\pm{0.092}$	0.503 $\pm{0.001}$	0.202 $\pm{0.001}$	28.997 $\pm{0.959}$	0.384 $\pm{0.000}$	2.297E-04 $\pm{9.718E-06}$	0.998 $\pm{0.002}$
Weight Decay + SAM	46.061 $\pm{0.111}$	0.539 $\pm{0.001}$	0.177 $\pm{0.001}$	34.549 $\pm{0.057}$	0.339 $\pm{0.000}$	3.784E-04 $\pm{9.996E-06}$	0.736 $\pm{0.004}$

Table 9: Results for ResNet18 (Pre-Trained) on TinyImageNet. Bolded values indicate the best performance per metric. For sharpness metrics, lower values correspond to flatter models.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm
Baseline + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✗ (N/A)	✓ (1.0)	✗ (N/A)	✗ (N/A)
Augmentation	✓ (1.0)	✓ (0.8)	✓ (1.0)	✓ (1.0)	✗ (N/A)	✓ (1.0)	✓ (1.0)
Augmentation + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✗ (N/A)	✓ (1.0)	✓ (1.0)
Weight Decay	✗ (N/A)	✗ (N/A)	✓ (1.0)	✗ (N/A)	✓ (0.51)	✗ (N/A)	✓ (1.0)
Weight Decay + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)

Table 10: Significance results for the control conditions against the baseline for ResNet18 trained on TinyImageNet. ✓ indicates significant difference compared to the baseline; ✗ indicates no significance, the value in the brackets represents the effect sizes using the absolute RBC

Panel of Figure 9 — Figure 9: Scatter plots of 60 minima for ResNet-18 on CIFAR-10 varying the SAM ( $\rho$ ) hyperparameter ( ${0.5,0.25,0.05,0.025,0.005,0.0025}$ ), using batch size ${256}$ and learning rate ${1^{-3}}$ : (a) Fisher–Rao norm vs. train loss, (b) Relative Flatness vs. train loss, and (c) generalisation gap vs. train loss (log scale).

$\rho$ Radius	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
0.000	28.050 $\pm{0.175}$	0.720 $\pm{0.002}$	0.186 $\pm{0.001}$	58.614 $\pm{0.201}$	0.282 $\pm{0.001}$	1.366E-05 $\pm{1.206E-06}$	0.032 $\pm{0.001}$	34.607 $\pm{0.757}$
0.5000	1.605 $\pm{0.646}$	0.629 $\pm{0.024}$	0.079 $\pm{0.007}$	52.847 $\pm{1.667}$	0.221 $\pm{0.009}$	6.814E-02 $\pm{7.326E-03}$	14.767 $\pm{0.375}$	4156.344 $\pm{279.557}$
0.2500	9.751 $\pm{0.640}$	0.835 $\pm{0.002}$	0.026 $\pm{0.003}$	68.479 $\pm{0.302}$	0.089 $\pm{0.002}$	3.884E-02 $\pm{4.981E-03}$	8.712 $\pm{0.623}$	4876.348 $\pm{314.164}$
0.0500	20.588 $\pm{0.125}$	0.794 $\pm{0.001}$	0.108 $\pm{0.001}$	66.342 $\pm{0.164}$	0.168 $\pm{0.000}$	5.823E-05 $\pm{9.056E-06}$	0.107 $\pm{0.006}$	75.093 $\pm{1.693}$
0.0250	22.602 $\pm{0.109}$	0.774 $\pm{0.001}$	0.124 $\pm{0.001}$	64.224 $\pm{0.154}$	0.195 $\pm{0.000}$	2.587E-05 $\pm{1.987E-06}$	0.065 $\pm{0.001}$	70.223 $\pm{0.941}$
0.0050	25.793 $\pm{0.137}$	0.742 $\pm{0.001}$	0.167 $\pm{0.001}$	60.985 $\pm{0.280}$	0.250 $\pm{0.000}$	4.861E-05 $\pm{7.166E-06}$	0.082 $\pm{0.009}$	57.886 $\pm{5.223}$

Table 11: Results for ResNet-18 trained on CIFAR10 with batch size 256 and a learning rate of

\mathbf{1e^{-3}}

, while varying the SAM

\rho

hyperperameter (0.5, 0.25 ,0.05 ,0.025, 0.005, 0.0025). Numbers in bold indicate best scores per metric. For sharpness metrics lower values represent flatter models.

$\rho$ Radius	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
0.5000	✓ (1.0)	✗ (N/A)	✓ (1.0)	✗ (N/A)	✓ (0.69)	✓ (1.0)	✓ (1.0)	✓ (1.0)
0.2500	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
0.0500	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
0.0250	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
0.0050	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (0.96)
0.0025	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✗ (N/A)	✗ (N/A)	✗ (N/A)

Table 12: Significance Table for ResNet-18 trained on CIFAR10 with batch size 256 and a learning rate of

\mathbf{1e^{-3}}

, while varying the SAM

\rho

hyperperameter (0.5, 0.25 ,0.05 ,0.025, 0.005, 0.0025). ✓ indicates significant difference compared to the baseline (

\rho

= 0.000); ✗ indicates significance, the value in the brackets represents the effect sizes using the absolute RBC.

Panel of Figure 10 — Figure 10: Increasing the portion of random training data on CIFAR10 with Resnet18 from 0.0 to 1.0: plotting train accuracy and sharpness metrics against test accuracy.

Percentage of Random Data	Train Accuracy	Train Loss	SAM Sharpness	Fisher-Rao norm	Relative Flatness
0% (Baseline)	1.000 $\pm{0.000}$	9.997E-05 $\pm{1.381E-06}$	1.344E-05 $\pm{1.968E-06}$	0.031 $\pm{0.001}$	33.700 $\pm{0.949}$
20%	1.000 $\pm{0.000}$	1.039E-04 $\pm{1.714E-06}$	2.594E-01 $\pm{2.349E-03}$	107.830 $\pm{0.183}$	38.615 $\pm{0.693}$
40%	1.000 $\pm{0.000}$	1.071E-04 $\pm{1.759E-06}$	4.180E-01 $\pm{2.405E-02}$	160.742 $\pm{0.292}$	42.111 $\pm{0.613}$
60%	1.000 $\pm{0.000}$	1.085E-04 $\pm{2.294E-06}$	8.927E-01 $\pm{1.848E-02}$	203.816 $\pm{0.212}$	44.990 $\pm{0.972}$
80%	1.000 $\pm{0.000}$	1.107E-04 $\pm{2.535E-06}$	4.984E-01 $\pm{3.913E-02}$	242.238 $\pm{0.260}$	47.011 $\pm{0.878}$
100%	1.000 $\pm{0.000}$	1.094E-04 $\pm{1.490E-06}$	3.450E-01 $\pm{3.319E-02}$	283.181 $\pm{0.129}$	47.095 $\pm{0.543}$

Table 13: Results for ResNet-18 trained on CIFAR10 with increasingly randomised data . For sharpness metrics lower values represent flatter models.

Appendix figures & tables41 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11: Scatter plots an MLP trained on the sphere, rosenbrock, rastrigin, beale, booth, three-hump camel, and himmelblaus functions for 10 different data sets till reaching a target train loss of 300: (left) mean SAM sharpness vs. train loss, (centre)

|

generalisation gap

|

vs. train loss, and (right)

|

generalisation gap

|

vs. mean SAM sharpness.

Figure 12: Scatter plots an MLP trained on the sphere, rosenbrock, rastrigin, booth, three-hump camel, and himmelblaus functions for 10 different data sets till reaching a target train loss of 150: (left) mean SAM sharpness vs. train loss, (centre)

|

generalisation gap

|

vs. train loss, and (right)

|

generalisation gap

|

vs. mean SAM sharpness.

Figure 13: Scatter plots an MLP trained on the sphere, rastrigin, booth, three-hump camel, and himmelblaus functions for 10 different data sets till reaching a target train loss of 100: (left) mean SAM sharpness vs. train loss, (centre)

|

generalisation gap

|

vs. train loss, and (right)

|

generalisation gap

|

vs. mean SAM sharpness.

Figure 14: Scatter plots an MLP trained on the sphere, booth, three-hump camel, and himmelblaus functions for 10 different data sets till reaching a target train loss of 10: (left) mean SAM sharpness vs. train loss, (centre)

|

generalisation gap

|

vs. train loss, and (right)

|

generalisation gap

|

vs. mean SAM sharpness.

Figure 15: Scatter plots an MLP trained on the sphere, booth, three-hump camel, and himmelblaus functions for 10 different data sets till reaching a target train loss of 1: (left) mean SAM sharpness vs. train loss, (centre)

|

generalisation gap

|

vs. train loss, and (right)

|

generalisation gap

|

vs. mean SAM sharpness.

Table 14: Loss landscape visualisation ( Li et al., 2018 ) of ResNet18 landscape on CIFAR10 exploring the loss in the domain of the perturbations

[-1,1]^{2}

with 51 steps in both directions on models trained with augmentation visualising landscape with standard training data and augmented training data.

Control Condition	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Augmentation	1.905E-01 $\pm{2.203E-02}$	3.940 $\pm{0.207}$	2903.220 $\pm{89.243}$
Augmentation + SAM	1.303E-01 $\pm{1.547E-02}$	5.571 $\pm{0.035}$	4970.972 $\pm{30.139}$

Appendix

Table 15: Sharpness Calculation for ResNet18 landscape on CIFAR10 trained with batch size of 256 and learning rate of 0.001 using augmented training data for sharpness calculations.

Control Condition	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Augmentation	1.591E-02 $\pm{1.609E-03}$	3.962 $\pm{0.292}$	2972.554 $\pm{137.079}$
Augmentation + SAM	2.035E-02 $\pm{1.203E-03}$	5.084 $\pm{0.032}$	5105.327 $\pm{43.058}$

Appendix

Table 16: Sharpness Calculation for ResNet18 landscape on CIFAR10 trained with batch size of 256 and learning rate of 0.001 using standard training data for sharpness calculations.

Table 17: Loss landscape visualisation ( Li et al., 2018 ) of VGG19 landscape on CIFAR10 exploring the loss in the domain of the perturbations

[-1,1]^{2}

with 51 steps in both directions on models trained with augmentation visualising landscape with standard training data and augmented training data.

Control Condition	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Augmentation	1.967E-01 $\pm{}$ 2.298E-02	3.505 $\pm{}$ 0.155	688.897 $\pm{}$ 26.348
Augmentation + SAM	9.777E-02 $\pm{}$ 1.126E-02	4.278 $\pm{}$ 0.027	1609.212 $\pm{}$ 22.719

Appendix

Table 18: Sharpness Calculation for VGG19 on CIFAR10 trained with batch size of 256 and learning rate of 0.001 using augmented training data for sharpness calculations.

Control Condition	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Augmentation	9.589E-03 $\pm{}$ 1.438E-03	2.322 $\pm{}$ 0.125	481.312 $\pm{}$ 23.358
Augmentation + SAM	1.497E-02 $\pm{}$ 7.363E-04	3.756 $\pm{}$ 0.030	1413.712 $\pm{}$ 22.212

Appendix

Table 19: Sharpness Calculation for VGG19 on CIFAR10 trained with batch size of 256 and learning rate of 0.001 using standard training data for sharpness calculations.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline	28.050 $\pm{0.175}$	0.720 $\pm{0.002}$	0.186 $\pm{0.001}$	58.614 $\pm{0.201}$	0.282 $\pm{0.001}$	1.366E-05 $\pm{1.206E-06}$	0.032 $\pm{0.001}$	34.607 $\pm{0.757}$
Baseline + SAM	20.588 $\pm{0.125}$	0.794 $\pm{0.001}$	0.108 $\pm{0.001}$	66.342 $\pm{0.164}$	0.168 $\pm{0.000}$	5.823E-05 $\pm{9.056E-06}$	0.107 $\pm{0.006}$	75.093 $\pm{1.693}$
Augmentation	10.399 $\pm{0.067}$	0.886 $\pm{0.001}$	0.077 $\pm{0.001}$	68.755 $\pm{0.219}$	0.121 $\pm{0.001}$	1.905E-01 $\pm{2.203E-02}$	3.940 $\pm{0.207}$	2903.220 $\pm{89.243}$
Augmentation + SAM	6.864 $\pm{0.038}$	0.908 $\pm{0.000}$	0.014 $\pm{0.001}$	71.419 $\pm{0.283}$	0.069 $\pm{0.000}$	1.303E-01 $\pm{1.547E-02}$	5.571 $\pm{0.035}$	4970.972 $\pm{30.139}$
Weight Decay	27.942 $\pm{0.196}$	0.721 $\pm{0.002}$	0.174 $\pm{0.002}$	58.562 $\pm{0.227}$	0.281 $\pm{0.001}$	3.391E-05 $\pm{4.494E-06}$	0.065 $\pm{0.004}$	59.767 $\pm{3.009}$
Weight Decay + SAM	19.788 $\pm{0.149}$	0.802 $\pm{0.001}$	0.096 $\pm{0.001}$	67.079 $\pm{0.117}$	0.162 $\pm{0.001}$	8.733E-05 $\pm{1.430E-05}$	0.127 $\pm{0.006}$	88.807 $\pm{2.336}$

Appendix

Table 20: Results for ResNet-18 trained on CIFAR10 with batch size 256 and a learning rate of

\mathbf{1e^{-3}}

. Numbers in bold indicate best scores for metrics. For sharpness metrics lower values represent flatter models.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline	16.203 $\pm{0.266}$	0.838 $\pm{0.003}$	0.109 $\pm{0.004}$	70.814 $\pm{0.390}$	0.138 $\pm{0.001}$	7.648E-06 $\pm{3.794E-06}$	0.015 $\pm{0.006}$	16.641 $\pm{5.960}$
Baseline + SAM	14.549 $\pm{0.059}$	0.855 $\pm{0.001}$	0.084 $\pm{0.001}$	72.618 $\pm{0.161}$	0.110 $\pm{0.000}$	2.019E-05 $\pm{2.683E-06}$	0.042 $\pm{0.002}$	49.022 $\pm{1.901}$
Augmentation	7.593 $\pm{0.092}$	0.921 $\pm{0.001}$	0.056 $\pm{0.003}$	72.923 $\pm{0.223}$	0.078 $\pm{0.001}$	1.091E-01 $\pm{1.958E-02}$	2.390 $\pm{0.268}$	1604.778 $\pm{103.972}$
Augmentation + SAM	6.920 $\pm{0.056}$	0.931 $\pm{0.001}$	0.037 $\pm{0.001}$	73.483 $\pm{0.212}$	0.058 $\pm{0.000}$	2.267E-02 $\pm{2.070E-03}$	1.165 $\pm{0.014}$	1173.090 $\pm{15.607}$
Weight Decay	16.791 $\pm{0.122}$	0.832 $\pm{0.001}$	0.071 $\pm{0.001}$	68.538 $\pm{0.159}$	0.157 $\pm{0.000}$	3.673E-05 $\pm{4.662E-06}$	0.097 $\pm{0.001}$	99.041 $\pm{0.736}$
Weight Decay + SAM	14.022 $\pm{0.089}$	0.860 $\pm{0.001}$	0.050 $\pm{0.001}$	73.100 $\pm{0.137}$	0.116 $\pm{0.000}$	3.238E-04 $\pm{6.384E-05}$	0.446 $\pm{0.013}$	178.169 $\pm{4.103}$

Appendix

Table 21: Results for ResNet-18 trained on CIFAR10 with batch size 256 and a learning rate of

\mathbf{1e^{-2}}

. Numbers in bold indicate best scores for metrics. For sharpness metrics lower values represent flatter models.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline + SAM	✓ (0.96)	✓ (0.96)	✓ (0.93)	✓ (0.96)	✓ (0.99)	✓ (0.75)	✓ (0.96)	✓ (1.0)
Augmentation	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (0.90)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (0.96)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay	✗ (N/A)	✗ (N/A)	✓ (1.0)	✗ (N/A)	✗ (N/A)	✓ (0.96)	✓ (1.0)	✓ (1.0)
Weight Decay + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (0.96)	✓ (0.99)	✓ (1.0)	✓ (1.0)	✓ (1.0)

Appendix

Table 22: Significance results for the control conditions against the baseline for ResNet18 trained on CIFAR10 with batch size 256 and a learning rate of

\mathbf{1e^{-2}}

. ✓ indicates significant difference compared to the baseline; ✗ indicates significance, the value in the brackets represents the effect sizes using the absolute RBC.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline	23.325 $\pm{0.140}$	0.767 $\pm{0.001}$	0.154 $\pm{0.001}$	63.035 $\pm{0.204}$	0.227 $\pm{0.000}$	5.181E-06 $\pm{5.856E-07}$	0.013 $\pm{0.000}$	27.916 $\pm{0.340}$
Baseline + SAM	16.714 $\pm{0.125}$	0.833 $\pm{0.001}$	0.083 $\pm{0.001}$	69.769 $\pm{0.108}$	0.126 $\pm{0.000}$	2.640E-05 $\pm{4.345E-06}$	0.072 $\pm{0.006}$	139.589 $\pm{2.679}$
Augmentation	9.110 $\pm{0.079}$	0.905 $\pm{0.001}$	0.065 $\pm{0.001}$	71.516 $\pm{0.308}$	0.099 $\pm{0.000}$	9.266E-02 $\pm{5.276E-03}$	2.465 $\pm{0.105}$	3735.018 $\pm{173.247}$
Augmentation + SAM	6.869 $\pm{0.022}$	0.921 $\pm{0.000}$	0.013 $\pm{0.000}$	72.870 $\pm{0.207}$	0.058 $\pm{0.000}$	8.913E-02 $\pm{8.054E-03}$	4.070 $\pm{0.027}$	7532.582 $\pm{69.191}$
Weight Decay	23.504 $\pm{0.136}$	0.765 $\pm{0.001}$	0.137 $\pm{0.001}$	62.879 $\pm{0.214}$	0.231 $\pm{0.000}$	2.241E-05 $\pm{3.426E-06}$	0.047 $\pm{0.000}$	80.599 $\pm{0.548}$
Weight Decay + SAM	16.433 $\pm{0.096}$	0.836 $\pm{0.001}$	0.072 $\pm{0.001}$	70.226 $\pm{0.158}$	0.124 $\pm{0.000}$	4.797E-05 $\pm{7.285E-06}$	0.110 $\pm{0.004}$	194.034 $\pm{2.840}$

Appendix

Table 23: Results for ResNet-18 trained on CIFAR10 with batch size 128 and a learning rate of

\mathbf{1e^{-3}}

. Numbers in bold indicate best scores for metrics. For sharpness metrics lower values represent flatter models.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay	✗ (N/A)	✗ (N/A)	✓ (1.0)	✗ (N/A)	✗ (N/A)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)

Appendix

Table 24: Significance results for the control conditions against the baseline for ResNet18 trained on CIFAR10 with batch size 128 and a learning rate of

\mathbf{1e^{-3}}

. ✓ indicates significant difference compared to the baseline; ✗ indicates significance, the value in the brackets represents the effect sizes using the absolute RBC.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline	15.027 $\pm{0.069}$	0.850 $\pm{0.001}$	0.109 $\pm{0.001}$	71.960 $\pm{0.158}$	0.125 $\pm{0.000}$	1.094E-06 $\pm{9.618E-08}$	0.003 $\pm{0.000}$	8.785 $\pm{0.142}$
Baseline + SAM	13.231 $\pm{0.065}$	0.868 $\pm{0.001}$	0.081 $\pm{0.001}$	73.053 $\pm{0.164}$	0.099 $\pm{0.000}$	1.021E-05 $\pm{5.519E-07}$	0.024 $\pm{0.001}$	70.694 $\pm{2.273}$
Augmentation	7.455 $\pm{0.062}$	0.923 $\pm{0.001}$	0.057 $\pm{0.001}$	72.594 $\pm{0.152}$	0.076 $\pm{0.000}$	8.274E-02 $\pm{6.784E-03}$	2.086 $\pm{0.140}$	2864.657 $\pm{151.088}$
Augmentation + SAM	6.678 $\pm{0.060}$	0.933 $\pm{0.001}$	0.036 $\pm{0.001}$	73.565 $\pm{0.245}$	0.056 $\pm{0.000}$	2.173E-02 $\pm{2.042E-03}$	1.012 $\pm{0.014}$	2354.005 $\pm{38.058}$
Weight Decay	12.695 $\pm{0.072}$	0.873 $\pm{0.001}$	0.057 $\pm{0.001}$	70.979 $\pm{0.124}$	0.103 $\pm{0.000}$	1.265E-04 $\pm{3.140E-06}$	0.159 $\pm{0.003}$	355.345 $\pm{12.866}$
Weight Decay + SAM	12.606 $\pm{0.069}$	0.874 $\pm{0.001}$	0.036 $\pm{0.001}$	72.795 $\pm{0.162}$	0.107 $\pm{0.000}$	5.880E-04 $\pm{6.127E-05}$	0.745 $\pm{0.017}$	439.467 $\pm{8.439}$

Appendix

Table 25: Results for ResNet-18 trained on CIFAR10 with batch size 128 and a learning rate of

\mathbf{1e^{-2}}

. Numbers in bold indicate best scores for metrics. For sharpness metrics lower values represent flatter models.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (0.96)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (0.82)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (0.96)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay	✓ (1.0)	✓ (1.0)	✓ (1.0)	✗ (N/A)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (0.93)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)

Appendix

Table 26: Significance results for the control conditions against the baseline for ResNet18 trained on CIFAR10 with batch size 128 and a learning rate of

\mathbf{1e^{-2}}

. ✓ indicates significant difference compared to the baseline; ✗ indicates significance, the value in the brackets represents the effect sizes using the absolute RBC.

Panel of Figure 16 — Figure 16: Plot of 240 minima using reparametrisation invariant sharpness metrics against train loss and generalisation gap against train loss using log-scale for the VGG19 with different training hyperparameters (batch size of 256, 128 and learning rate of 0.001 and $1e^{-2}$ ) trained on CIFAR10.

Control Condition	Genralisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline	21.805 $\pm{0.128}$	0.782 $\pm{0.001}$	0.160 $\pm{0.001}$	64.316 $\pm{0.193}$	0.204 $\pm{0.000}$	7.649E-06 $\pm{2.207E-06}$	0.022 $\pm{0.003}$	7.374 $\pm{0.470}$
Baseline + SAM	18.444 $\pm{0.097}$	0.815 $\pm{0.001}$	0.108 $\pm{0.001}$	66.655 $\pm{0.296}$	0.150 $\pm{0.000}$	1.495E-03 $\pm{1.703E-04}$	0.938 $\pm{0.036}$	140.164 $\pm{3.149}$
Augmentation	11.289 $\pm{0.066}$	0.879 $\pm{0.001}$	0.084 $\pm{0.001}$	68.497 $\pm{0.199}$	0.121 $\pm{0.000}$	1.967E-01 $\pm{2.298E-02}$	3.505 $\pm{0.155}$	688.897 $\pm{26.348}$
Augmentation + SAM	8.139 $\pm{0.074}$	0.903 $\pm{0.001}$	0.019 $\pm{0.001}$	71.268 $\pm{0.196}$	0.075 $\pm{0.000}$	9.777E-02 $\pm{1.126E-02}$	4.278 $\pm{0.027}$	1609.212 $\pm{22.719}$
Weight Decay	21.801 $\pm{0.121}$	0.782 $\pm{0.001}$	0.151 $\pm{0.001}$	64.405 $\pm{0.217}$	0.202 $\pm{0.000}$	1.315E-05 $\pm{1.143E-06}$	0.048 $\pm{0.001}$	16.494 $\pm{0.292}$
Weight Decay + SAM	18.394 $\pm{0.067}$	0.816 $\pm{0.001}$	0.104 $\pm{0.001}$	66.827 $\pm{0.286}$	0.151 $\pm{0.000}$	3.210E-03 $\pm{6.174E-04}$	1.121 $\pm{0.080}$	157.592 $\pm{5.360}$

Appendix

Table 27: Results for VGG19 trained on CIFAR10 with batch size 256 and a learning rate of

\mathbf{1e^{-3}}

. Numbers in bold indicate best scores for metrics. For sharpness metrics lower values represent flatter models.

Control Condition	Genralisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay	✗ (N/A)	✗ (N/A)	✓ (1.0)	✗ (N/A)	✓ (0.41)	✗ (N/A)	✓ (0.96)	✓ (1.0)
Weight Decay + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)

Appendix

Table 28: Significance results for the control conditions against the baseline for VGG19 trained on CIFAR10 with batch size 256 and a learning rate of

\mathbf{1e^{-3}}

. ✓ indicates significant difference compared to the baseline; ✗ indicates significance, the value in the brackets represents the effect sizes using the absolute RBC.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline	13.507 $\pm{0.063}$	0.865 $\pm{0.001}$	0.105 $\pm{0.001}$	71.476 $\pm{0.125}$	0.119 $\pm{0.000}$	1.604E-06 $\pm{1.637E-07}$	0.004 $\pm{0.000}$	0.807 $\pm{0.032}$
Baseline + SAM	13.183 $\pm{0.115}$	0.868 $\pm{0.001}$	0.081 $\pm{0.001}$	71.908 $\pm{0.121}$	0.103 $\pm{0.000}$	4.290E-05 $\pm{5.280E-06}$	0.077 $\pm{0.002}$	25.287 $\pm{0.720}$
Augmentation	8.565 $\pm{0.030}$	0.910 $\pm{0.000}$	0.065 $\pm{0.000}$	71.491 $\pm{0.442}$	0.092 $\pm{0.000}$	1.146E-01 $\pm{9.917E-03}$	2.555 $\pm{0.154}$	396.136 $\pm{22.479}$
Augmentation + SAM	7.969 $\pm{0.080}$	0.920 $\pm{0.001}$	0.040 $\pm{0.001}$	73.037 $\pm{0.160}$	0.071 $\pm{0.000}$	3.122E-02 $\pm{3.347E-03}$	1.087 $\pm{0.014}$	429.679 $\pm{6.763}$
Weight Decay	15.241 $\pm{0.124}$	0.836 $\pm{0.003}$	0.118 $\pm{0.002}$	68.294 $\pm{0.390}$	0.184 $\pm{0.002}$	2.302E-02 $\pm{4.987E-03}$	5.120 $\pm{0.586}$	237.081 $\pm{22.113}$
Weight Decay + SAM	13.188 $\pm{0.130}$	0.868 $\pm{0.001}$	0.063 $\pm{0.001}$	71.618 $\pm{0.284}$	0.117 $\pm{0.000}$	5.764E-04 $\pm{5.989E-05}$	0.742 $\pm{0.022}$	124.537 $\pm{3.152}$

Appendix

Table 29: Results for VGG19 trained on CIFAR10 with batch size 256 and a learning rate of

\mathbf{1e^{-2}}

. Numbers in bold indicate best scores for metrics. For sharpness metrics lower values represent flatter models.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline + SAM	✓ (0.75)	✓ (0.75)	✓ (1.0)	✓ (0.78)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation	✓ (1.0)	✓ (1.0)	✓ (1.0)	✗ (N/A)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay	✗ (N/A)	✗ (N/A)	✗ (N/A)	✗ (N/A)	✗ (N/A)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay + SAM	✗ (N/A)	✗ (N/A)	✓ (1.0)	✗ (N/A)	✓ (0.29)	✓ (1.0)	✓ (1.0)	✓ (1.0)

Appendix

Table 30: Significance results for the control conditions against the baseline for VGG19 trained on CIFAR10 with batch size 256 and a learning rate of

\mathbf{1e^{-2}}

. ✓ indicates significant difference compared to the baseline; ✗ indicates significance, the value in the brackets represents the effect sizes using the absolute RBC.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline	18.026 $\pm{0.066}$	0.820 $\pm{0.001}$	0.140 $\pm{0.001}$	68.325 $\pm{0.110}$	0.167 $\pm{0.000}$	8.460E-07 $\pm{5.355E-08}$	0.008 $\pm{0.000}$	5.824 $\pm{0.058}$
Baseline + SAM	15.059 $\pm{0.085}$	0.849 $\pm{0.001}$	0.089 $\pm{0.001}$	69.791 $\pm{0.174}$	0.115 $\pm{0.000}$	5.689E-04 $\pm{1.248E-04}$	0.649 $\pm{0.032}$	242.483 $\pm{6.334}$
Augmentation	9.988 $\pm{0.088}$	0.895 $\pm{0.001}$	0.074 $\pm{0.001}$	70.671 $\pm{0.288}$	0.107 $\pm{0.000}$	1.708E-01 $\pm{1.967E-02}$	2.851 $\pm{0.121}$	1158.004 $\pm{40.307}$
Augmentation + SAM	7.594 $\pm{0.050}$	0.916 $\pm{0.000}$	0.017 $\pm{0.000}$	72.194 $\pm{0.175}$	0.066 $\pm{0.000}$	7.664E-02 $\pm{1.149E-02}$	3.469 $\pm{0.023}$	2487.050 $\pm{14.772}$
Weight Decay	17.485 $\pm{0.066}$	0.825 $\pm{0.001}$	0.127 $\pm{0.001}$	68.655 $\pm{0.156}$	0.158 $\pm{0.000}$	4.918E-06 $\pm{6.710E-07}$	0.044 $\pm{0.001}$	29.566 $\pm{0.647}$
Weight Decay + SAM	14.803 $\pm{0.082}$	0.851 $\pm{0.001}$	0.082 $\pm{0.001}$	69.699 $\pm{0.181}$	0.114 $\pm{0.000}$	1.089E-03 $\pm{3.140E-04}$	0.802 $\pm{0.022}$	298.386 $\pm{7.049}$

Appendix

Table 31: Results for VGG19 trained on CIFAR10 with batch size 128 and a learning rate of

\mathbf{1e^{-3}}

. Numbers in bold indicate best scores for metrics. For sharpness metrics lower values represent flatter models.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay	✓ (1.0)	✓ (1.0)	✓ (1.0)	✗ (N/A)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)

Appendix

Table 32: Significance results for the control conditions against the baseline for VGG19 trained on CIFAR10 with batch size 128 and a learning rate of

\mathbf{1e^{-3}}

. ✓ indicates significant difference compared to the baseline; ✗ indicates significance, the value in the brackets represents the effect sizes using the absolute RBC.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline	12.661 $\pm{0.066}$	0.873 $\pm{0.001}$	0.100 $\pm{0.001}$	71.689 $\pm{0.113}$	0.111 $\pm{0.000}$	3.227E-07 $\pm{3.198E-08}$	0.003 $\pm{0.000}$	1.331 $\pm{0.031}$
Baseline + SAM	12.248 $\pm{0.078}$	0.878 $\pm{0.001}$	0.077 $\pm{0.001}$	71.838 $\pm{0.256}$	0.100 $\pm{0.000}$	8.424E-06 $\pm{9.410E-07}$	0.054 $\pm{0.003}$	40.853 $\pm{2.177}$
Augmentation	8.333 $\pm{0.060}$	0.913 $\pm{0.001}$	0.063 $\pm{0.001}$	72.114 $\pm{0.259}$	0.091 $\pm{0.000}$	1.605E-01 $\pm{2.042E-02}$	2.510 $\pm{0.122}$	772.205 $\pm{19.954}$
Augmentation + SAM	7.791 $\pm{0.072}$	0.922 $\pm{0.001}$	0.038 $\pm{0.001}$	72.470 $\pm{0.190}$	0.072 $\pm{0.000}$	3.497E-02 $\pm{4.347E-03}$	1.076 $\pm{0.019}$	1155.243 $\pm{35.334}$
Weight Decay	14.589 $\pm{0.143}$	0.839 $\pm{0.003}$	0.110 $\pm{0.002}$	66.692 $\pm{0.603}$	0.187 $\pm{0.001}$	7.018E-03 $\pm{8.372E-04}$	6.253 $\pm{0.596}$	608.657 $\pm{44.645}$
Weight Decay + SAM	12.295 $\pm{0.112}$	0.877 $\pm{0.001}$	0.052 $\pm{0.001}$	71.167 $\pm{0.301}$	0.116 $\pm{0.001}$	5.407E-04 $\pm{1.109E-04}$	1.193 $\pm{0.043}$	406.163 $\pm{15.541}$

Appendix

Table 33: Results for VGG19 trained on CIFAR10 with batch size 128 and a learning rate of

\mathbf{1e^{-2}}

. Numbers in bold indicate best scores for metrics. For sharpness metrics lower values represent flatter models.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✗ (N/A)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation	✓ (1.0)	✓ (1.0)	✓ (1.0)	✗ (N/A)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (0.75)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay	✗ (N/A)	✗ (N/A)	✗ (N/A)	✗ (N/A)	✗ (N/A)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay + SAM	✓ (0.89)	✓ (0.82)	✓ (1.0)	✗ (N/A)	✗ (N/A)	✓ (1.0)	✓ (1.0)	✓ (1.0)

Appendix

Table 34: Significance results for the control conditions against the baseline for VGG19 trained on CIFAR10 with batch size 128 and a learning rate of

\mathbf{1e^{-2}}

. ✓ indicates significant difference compared to the baseline; ✗ indicates significance, the value in the brackets represents the effect sizes using the absolute RBC.

Panel of Table 35 — Table 35: Loss landscape visualisation ( Li et al., 2018 ) of VGG19 landscape on CIFAR10 exploring the loss in the domain of the perturbations $[-1,1]^{2}$ with 51 steps in both directions

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline	42.454 $\pm{0.092}$	0.575 $\pm{0.001}$	0.253 $\pm{0.000}$	40.749 $\pm{0.124}$	0.396 $\pm{0.000}$	2.123E-04 $\pm{2.649E-05}$	0.158 $\pm{0.017}$	8.384 $\pm{0.151}$
Baseline + SAM	43.815 $\pm{0.224}$	0.561 $\pm{0.002}$	0.232 $\pm{0.002}$	39.690 $\pm{0.196}$	0.399 $\pm{0.001}$	7.520E-04 $\pm{5.791E-05}$	0.529 $\pm{0.017}$	67.485 $\pm{1.802}$
Augmentation	32.519 $\pm{0.156}$	0.646 $\pm{0.002}$	0.222 $\pm{0.002}$	40.832 $\pm{0.321}$	0.358 $\pm{0.001}$	2.835E-01 $\pm{1.439E-02}$	7.156 $\pm{0.270}$	1430.826 $\pm{53.977}$
Augmentation + SAM	32.008 $\pm{0.099}$	0.656 $\pm{0.001}$	0.157 $\pm{0.001}$	41.276 $\pm{0.089}$	0.326 $\pm{0.001}$	1.971E-01 $\pm{1.170E-02}$	5.653 $\pm{0.073}$	2085.080 $\pm{31.648}$
Weight Decay	41.579 $\pm{0.107}$	0.584 $\pm{0.001}$	0.138 $\pm{0.000}$	41.266 $\pm{0.112}$	0.384 $\pm{0.000}$	3.302E-04 $\pm{3.256E-05}$	0.678 $\pm{0.008}$	45.728 $\pm{0.073}$
Weight Decay + SAM	44.631 $\pm{0.228}$	0.553 $\pm{0.002}$	0.189 $\pm{0.002}$	38.961 $\pm{0.191}$	0.429 $\pm{0.001}$	2.630E-03 $\pm{2.111E-04}$	2.138 $\pm{0.084}$	153.194 $\pm{6.495}$

Appendix

Table 36: Results for VGG-19 trained on CIFAR100, the Mean and

\pm{}

1 SEM are recorded over 10 models. Numbers in bold indicate best scores for metrics. For sharpness metrics lower values represent flatter models.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline + SAM	✗ (N/A)	✗ (N/A)	✓ (1.0)	✗ (N/A)	✗ (N/A)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation	✓ (1.0)	✓ (1.0)	✓ (1.0)	✗ (N/A)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (0.71)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (0.89)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay + SAM	✗ (N/A)	✗ (N/A)	✓ (1.0)	✗ (N/A)	✗ (N/A)	✓ (1.0)	✓ (1.0)	✓ (1.0)

Appendix

Table 37: Significance results for the control conditions against the baseline for VGG19 trained on CIFAR100. ✓ indicates significant difference compared to the baseline; ✗ indicates no significance, the value in the brackets represents the effect sizes using the absolute RBC.

Panel of Table 38 — Table 38: Loss landscape visualisation ( Li et al., 2018 ) VGG19 landscape on CIFAR100 exploring the loss in the domain of the perturbations $[-1,1]^{2}$ with 51 steps in both directions

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm
Baseline	39.588 $\pm{0.063}$	0.604 $\pm{0.001}$	0.303 $\pm{0.001}$	39.082 $\pm{0.042}$	0.238 $\pm{0.000}$	2.642E-04 $\pm{1.986E-05}$	0.337 $\pm{0.118}$
Baseline + SAM	36.131 $\pm{0.048}$	0.638 $\pm{0.000}$	0.199 $\pm{0.001}$	41.815 $\pm{0.038}$	0.186 $\pm{0.000}$	3.364E-04 $\pm{2.684E-05}$	0.419 $\pm{0.097}$
Augmentation	20.952 $\pm{0.080}$	0.578 $\pm{0.001}$	0.119 $\pm{0.001}$	42.359 $\pm{0.123}$	0.473 $\pm{0.000}$	1.893E+00 $\pm{7.702E-02}$	20.033 $\pm{0.076}$
Augmentation + SAM	17.927 $\pm{0.048}$	0.594 $\pm{0.000}$	0.056 $\pm{0.002}$	44.449 $\pm{0.125}$	0.440 $\pm{0.000}$	1.665E+00 $\pm{5.137E-02}$	19.230 $\pm{0.035}$
Weight Decay	39.622 $\pm{0.069}$	0.604 $\pm{0.001}$	0.265 $\pm{0.000}$	39.148 $\pm{0.026}$	0.222 $\pm{0.000}$	2.679E-04 $\pm{1.084E-05}$	0.207 $\pm{0.026}$
Weight Decay + SAM	35.922 $\pm{0.050}$	0.641 $\pm{0.001}$	0.180 $\pm{0.001}$	42.106 $\pm{0.048}$	0.185 $\pm{0.000}$	3.072E-04 $\pm{4.621E-06}$	0.342 $\pm{0.015}$

Appendix

Table 39: Results for VGG19-BN (Pre-Trained) on TinyImageNet. Numbers in bold indicate best scores for metrics. For sharpness metrics lower values represent flatter models.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm
Baseline + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✗ (N/A)	✗ (N/A)
Augmentation	✓ (1.0)	✗ (N/A)	✓ (1.0)	✓ (1.0)	✗ (N/A)	✓ (1.0)	✓ (1.0)
Augmentation + SAM	✓ (1.0)	✗ (N/A)	✓ (1.0)	✓ (1.0)	✗ (N/A)	✓ (1.0)	✓ (1.0)
Weight Decay	✗ (N/A)	✗ (N/A)	✓ (1.0)	✗ (N/A)	✓ (1.0)	✗ (N/A)	✗ (N/A)
Weight Decay + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✗ (N/A)	✗ (N/A)

Appendix

Table 40: Significance results for the control conditions against the baseline for VGG19 trained on TinyImageNet. ✓ indicates significant difference compared to the baseline; ✗ indicates no significance the value in the brackets represents the effect sizes using the absolute RBC.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline	39.040 $\pm{0.177}$	0.610 $\pm{0.002}$	0.308 $\pm{0.002}$	54.805 $\pm{0.147}$	0.408 $\pm{0.001}$	8.769E-05 $\pm{4.974E-06}$	0.221 $\pm{0.003}$	347.198 $\pm{6.425}$
Baseline + SAM	39.935 $\pm{0.144}$	0.600 $\pm{0.001}$	0.276 $\pm{0.001}$	54.792 $\pm{0.113}$	0.421 $\pm{0.001}$	1.458E-03 $\pm{8.995E-05}$	1.576 $\pm{0.083}$	1459.292 $\pm{82.220}$
Augmentation	1.305 $\pm{0.076}$	0.724 $\pm{0.001}$	0.019 $\pm{0.001}$	64.092 $\pm{0.152}$	0.217 $\pm{0.001}$	4.741E-01 $\pm{3.822E-02}$	22.809 $\pm{0.117}$	38465.647 $\pm{139.905}$
Augmentation + SAM	-1.199 $\pm{0.097}$	0.668 $\pm{0.002}$	0.030 $\pm{0.001}$	60.535 $\pm{0.179}$	0.201 $\pm{0.001}$	4.352E-01 $\pm{2.420E-02}$	22.372 $\pm{0.042}$	18412.664 $\pm{617.822}$
Weight Decay	38.746 $\pm{0.196}$	0.613 $\pm{0.002}$	0.301 $\pm{0.002}$	55.077 $\pm{0.159}$	0.402 $\pm{0.001}$	1.359E-04 $\pm{1.030E-05}$	0.328 $\pm{0.003}$	422.966 $\pm{6.897}$
Weight Decay + SAM	39.881 $\pm{0.162}$	0.600 $\pm{0.002}$	0.268 $\pm{0.001}$	54.797 $\pm{0.125}$	0.419 $\pm{0.001}$	2.890E-03 $\pm{3.102E-04}$	2.250 $\pm{0.099}$	1908.688 $\pm{97.800}$

Appendix

Table 41: Results for ViT trained on CIFAR10, the Mean and

\pm{}

1 SEM are recorded over 10 models. Numbers in bold indicate best scores for metrics. For sharpness metrics lower values represent flatter models.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline + SAM	✗ (N/A)	✗ (N/A)	✓ (1.0)	✗ (N/A)	✗ (N/A)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay + SAM	✗ (N/A)	✗ (N/A)	✓ (1.0)	✗ (N/A)	✗ (N/A)	✓ (1.0)	✓ (1.0)	✓ (1.0)

Appendix

Table 42: Significance results for the control conditions against the baseline for ViT trained on CIFAR10. ✓ indicates significant difference compared to the baseline; ✗ indicates significance, the value in the brackets represents the effect sizes using the absolute RBC.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline	69.048 $\pm{0.164}$	0.309 $\pm{0.002}$	0.402 $\pm{0.002}$	25.936 $\pm{0.088}$	0.723 $\pm{0.000}$	3.428E-04 $\pm{4.954E-05}$	0.646 $\pm{0.061}$	112.185 $\pm{4.246}$
Baseline + SAM	67.376 $\pm{0.126}$	0.326 $\pm{0.001}$	0.386 $\pm{0.001}$	27.628 $\pm{0.097}$	0.697 $\pm{0.000}$	4.539E-04 $\pm{6.066E-05}$	0.821 $\pm{0.070}$	124.472 $\pm{30.314}$
Augmentation	37.472 $\pm{0.249}$	0.508 $\pm{0.001}$	0.227 $\pm{0.001}$	38.680 $\pm{0.091}$	0.483 $\pm{0.001}$	5.995E-01 $\pm{8.815E-02}$	17.321 $\pm{0.192}$	17401.462 $\pm{143.009}$
Augmentation + SAM	32.136 $\pm{0.262}$	0.523 $\pm{0.001}$	0.146 $\pm{0.002}$	40.275 $\pm{0.097}$	0.446 $\pm{0.000}$	4.649E-01 $\pm{2.505E-02}$	19.664 $\pm{0.127}$	17812.985 $\pm{55.523}$
Weight Decay	67.524 $\pm{0.148}$	0.325 $\pm{0.001}$	0.324 $\pm{0.001}$	27.364 $\pm{0.103}$	0.700 $\pm{0.000}$	8.440E-04 $\pm{1.251E-04}$	1.563 $\pm{0.073}$	251.148 $\pm{15.330}$
Weight Decay + SAM	67.227 $\pm{0.077}$	0.327 $\pm{0.001}$	0.284 $\pm{0.001}$	27.739 $\pm{0.069}$	0.695 $\pm{0.001}$	4.323E-03 $\pm{3.837E-04}$	5.181 $\pm{0.260}$	1554.595 $\pm{91.649}$

Appendix

Table 43: Results for ViT trained on CIFAR100, the Mean and

\pm{}

1 SEM are recorded over 10 models. Numbers in bold indicate best scores for metrics. For sharpness metrics lower values represent flatter models.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Corruption Accuracy	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✗ (N/A)	✗ (N/A)	✗ (N/A)
Augmentation	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Augmentation + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)
Weight Decay	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (0.93)	✓ (1.0)	✓ (1.0)
Weight Decay + SAM	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)	✓ (1.0)

Appendix

Table 44: Significance results for the control conditions against the baseline for ViT trained on CIFAR100. ✓ indicates significant difference compared to the baseline; ✗ indicates no significance, the value in the brackets represents the effect sizes using the absolute RBC.

Figure 17: Training an MLP on single-objective problems mean SAM sharpness vs epochs with varying

\rho

. First

\rho=0.0005

, second

\rho=0.005

, third

\rho=0.05

and fourth

\rho=0.5

. The results are averaged over 10 runs.

Table 62

Control Conditon	Generalisation Gap	Test Accuracy	Test ECE	Prediction Disagreement	Fisher Rao norm	SAM Sharpness	Relative Flatness
Baseline	12.393 $\pm{0.158}$	0.876 $\pm{0.002}$	0.080 $\pm{0.001}$	0.109 $\pm{0.001}$	0.044 $\pm{0.000}$	3.658E-05 $\pm{3.077E-06}$	13.968 $\pm{0.081}$
Baseline + SAM	12.282 $\pm{0.147}$	0.874 $\pm{0.001}$	0.054 $\pm{0.001}$	0.117 $\pm{0.001}$	0.457 $\pm{0.012}$	5.861E-03 $\pm{1.046E-03}$	188.148 $\pm{3.816}$
Augmentation	10.239 $\pm{0.110}$	0.888 $\pm{0.001}$	0.062 $\pm{0.001}$	0.114 $\pm{0.001}$	0.767 $\pm{0.021}$	4.238E-01 $\pm{5.088E-02}$	218.364 $\pm{3.721}$
Augmentation + SAM	5.187 $\pm{0.166}$	0.871 $\pm{0.002}$	0.016 $\pm{0.001}$	0.094 $\pm{0.001}$	1.810 $\pm{0.024}$	6.950E-01 $\pm{7.846E-02}$	1080.307 $\pm{6.488}$
Weight Decay	12.320 $\pm{0.155}$	0.877 $\pm{0.002}$	0.078 $\pm{0.001}$	0.108 $\pm{0.001}$	0.050 $\pm{0.000}$	4.043E-05 $\pm{3.204E-06}$	15.427 $\pm{0.095}$
Weight Decay + SAM	12.242 $\pm{0.190}$	0.874 $\pm{0.002}$	0.052 $\pm{0.002}$	0.116 $\pm{0.001}$	0.499 $\pm{0.013}$	6.311E-03 $\pm{1.028E-03}$	203.940 $\pm{3.967}$

Appendix

Table 47: Results for GPT2 architecture trained on the NAMES dataset.

Control Condition	Generalisation Gap	Test Accuracy	Test ECE	Prediction Disagreement	SAM Sharpness	Fisher-Rao norm	Relative Flatness
Baseline + SAM	✗(N/A)	✗(N/A)	✓(1.0)	✗(N/A)	✓(1.0)	✓(1.0)	✓(1.0)
Augmentation	✓(1.0)	✓(0.96)	✓(1.0)	✗(N/A)	✓(1.0)	✓(1.0)	✓(1.0)
Augmentation + SAM	✓(1.0)	✗(N/A)	✓(1.0)	✓(1.0)	✓(1.0)	✓(1.0)	✓(1.0)
Weight Decay	✗(N/A)	✗(N/A)	✓(0.82)	✓(0.42)	✓(1.0)	✓(1.0)	✓(1.0)
Weight Decay + SAM	✗(N/A)	✗(N/A)	✓(1.0)	✗(N/A)	✓(1.0)	✓(1.0)	✓(1.0)

Appendix

Table 48: Significance results for the control conditions against the baseline for GPT2 trained on NAMES. ✓ indicates significant difference compared to the baseline; ✗ indicates no significance, the value in the brackets represents the effect sizes using the absolute RBC.

Explore similar work

Mar 24, 2026

Are Flat Minima an Illusion?

Flat minima are an account of why deep networks generalise. However flatness is a matter of form (parameters), while generalisation is of function. The same function can be a result of many different parameterisations. I demonstrate this by rescaling ReLU networks, changing raw Hessian trace by up to

99

times while every prediction remains fixed. Raw curvature cannot identify a function-level explanation. Previous theoretical work traced generalisation to the weakness of constraints implied by function, meaning the freedom a model retains within the bounds of what it has learned to be correct. A policy is weaker when more future commitments remain compatible with what it has learned, allowing more freedom to adapt. To measure this for neural networks, I freeze the last hidden representation and ask whether each of 512 sampled label bundles can be met by a replacement affine classifier. The resulting joint completion score is invariant under invertible linear mixing and translation of feature coordinates. Across two predeclared cohorts of 100 networks, it predicts held-out accuracy with rank correlations

0.29

and

0.47

. Raw Hessian trace and relative flatness have no multiplicity-corrected association. To put it provocatively, freedom is correlated with adaptability, while flatness is a matter of description.

Michael Timothy Bennett

School of Computing, The Australian National University, Canberra, ACT 2601, Australia

Neural Network Generalization Flat Minima

Ask PDF

Jun 18, 2026

Fisher-Geometric Sharpness and the Implicit Bias of SGD toward Flat Minima

A widely held intuition in deep learning is that stochastic gradient descent (SGD) implicitly favors flat minima and that flat minima generalize better, but standard Euclidean measures of flatness such as the trace or maximum eigenvalue of the loss Hessian are not invariant under reparametrizations that preserve the network function, which undermines the theoretical foundations of this narrative. In this study we resolve this issue by grounding flatness in the Riemannian geometry of the statistical manifold induced by the Fisher Information Matrix (FIM). We define Riemannian sharpness mathematically and prove that it is invariant under smooth, function-preserving reparametrizations, which directly addresses the critique of Dinh et al. in the paper ``Sharp minima can generalize for deep nets''.We note that this invariance is a property of the true FIM; the diagonal empirical estimator used in practice (and in all experiments below) inherits invariance only approximately, and exact invariance under arbitrary reparametrizations would require structured estimators such as K-FAC. We formalize the gradient noise of mini-batch SGD as having a covariance structure proportional to the FIM, derive the stationary distribution of the resulting stochastic differential equation, and then show that the probability mass is exponentially concentrated at Riemannian-flat minima. A PAC-Bayes generalization bound controlled explicitly by SR formally links this geometric bias to test performance. Our experiments on MNIST and CIFAR-10 confirm that SR reliably tracks generalization in ways that Euclidean sharpness does not, and that its scaling with

η/B

matches the theoretical predictions. Together these results provide a rigorous, reparametrization-invariant account of why flat minima generalize.

Md Sakir Ahmed, Kumaresh Sarmah, Hemen Dutta

Department of Electronics and Communication Technology Gauhati University Guwahati, Assam, India · Department of Mathematics Gauhati University Guwahati, Assam, India

Information Geometry Implicit Bias Flat Minima Loss Landscape Geometry Stochastic Gradient Descent

Ask PDF

Oct 8, 2026

Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking

Flat loss landscapes have long been linked to better generalization in neural networks. However, its role as a causal mechanism for generalization is less established. Grokking provides an unique testbed to understand this distinction: models are prone to fit observed data using non-generalizing structure and remain in that regime for prolonged periods, transitioning to generalization only under particular training conditions. In this work, we study whether flat loss landscapes can act as a driving mechanism in this transition. While recent work has argued for flatness as a necessary geometric condition for this transition, we find that biasing training toward flatter solutions using sharpness-aware minimization (SAM) is insufficient to reliably induce this transition, despite producing flatter solutions. However, when SAM is paired with mechanisms that drive generalization such as weight decay, an interesting property emerges: SAM can accelerate the transition to generalizing solutions by up to 4x at the epoch-level. We theoretically untangle this relationship between SAM and weight decay using a minimal interpolating two-layer ReLU model with both memorizing and generalizing solutions. We show that even in this simple setup, flatness alone cannot distinguish a memorizing solution from a generalizing one, while weight decay favors generalizing solutions. However, under a local stability analysis, there exists a window where a memorizing interpolant is locally stable under gradient descent but unstable under SAM in the low-norm regime, which can explain SAM's ability to accelerate this transition. Overall, our results provide a more interpretable account of the role of flatness in driving generalization, especially in settings where models are vulnerable to minimizing loss through learning non-generalizing structure.

Mohnish Harwani

Purdue University

Neural Network Generalization Flat Minima Grokking Sharpness-Aware Minimization

Ask PDF