Are Flat Minima an Illusion?

Michael Timothy Bennett

cs.LGMar 24, 2026

Are Flat Minima an Illusion?

Authors: Michael Timothy Bennett

Organizations: School of Computing, The Australian National University, Canberra, ACT 2601, Australia

Abstract

Flat minima are an account of why deep networks generalise. However flatness is a matter of form (parameters), while generalisation is of function. The same function can be a result of many different parameterisations. I demonstrate this by rescaling ReLU networks, changing raw Hessian trace by up to $99$ times while every prediction remains fixed. Raw curvature cannot identify a function-level explanation. Previous theoretical work traced generalisation to the weakness of constraints implied by function, meaning the freedom a model retains within the bounds of what it has learned to be correct. A policy is weaker when more future commitments remain compatible with what it has learned, allowing more freedom to adapt. To measure this for neural networks, I freeze the last hidden representation and ask whether each of 512 sampled label bundles can be met by a replacement affine classifier. The resulting joint completion score is invariant under invertible linear mixing and translation of feature coordinates. Across two predeclared cohorts of 100 networks, it predicts held-out accuracy with rank correlations $0.29$ and $0.47$ . Raw Hessian trace and relative flatness have no multiplicity-corrected association. To put it provocatively, freedom is correlated with adaptability, while flatness is a matter of description.

Figures & tables

Measure	Oriented $\rho$	95% CI	Holm $q$
Joint bank score $J_{H}$	$+0.290$	[ $+0.094$ , $+0.468$ ]	$0.004$
Raw Hessian trace	$+0.091$	[ $-0.123$ , $+0.298$ ]	$1.000$
Final-layer relative flatness	$+0.168$	[ $-0.032$ , $+0.361$ ]	$0.389$
Spectral-product margin	$+0.504$	[ $+0.329$ , $+0.652$ ]	$1.60\times 10^{-4}$
Weight norm	$+0.398$	[ $+0.217$ , $+0.565$ ]	$2.40\times 10^{-4}$

Table 1: MNIST cohort. Larger oriented scores predict greater test accuracy. Split-bank Spearman-Brown reliability is

0.940

Measure	Oriented $\rho$	95% CI	Holm $q$
Joint bank score $J_{H}$	$+0.468$	[ $+0.301$ , $+0.607$ ]	$4.00\times 10^{-5}$
Raw Hessian trace	$+0.056$	[ $-0.142$ , $+0.252$ ]	$1.000$
Final-layer relative flatness	$-0.039$	[ $-0.235$ , $+0.164$ ]	$1.000$
Spectral-product margin	$+0.546$	[ $+0.381$ , $+0.680$ ]	$1.60\times 10^{-4}$
Weight norm	$+0.361$	[ $+0.174$ , $+0.524$ ]	$1.40\times 10^{-3}$

Table 2: Fashion-MNIST cohort. Larger oriented scores predict greater test accuracy. Split-bank Spearman-Brown reliability is

0.956

Cohort	Selected seed	Bank score $J_{H}$	Expected path-score lower bound
MNIST	$68$	$0.434$	$0.241$
Fashion-MNIST	$19$	$0.467$	$0.269$

Table 3: PAC-Bayes certificates for the bank-selected network. Each cohort uses

n=128

, a uniform prior over 100 frozen networks, and

\delta=0.025

. The two bounds therefore hold together with probability at least

0.95

. Each bound concerns expected feasibility over future paths drawn by the same sampling rule.

Panel of Figure 1 — Figure 1: Joint bank score and held-out accuracy. Natural-label cohorts show positive ranking associations. The random-label association centres near zero despite greater mean bank score.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Baseline	$\rho_{J}-\rho_{b}$	$98.75\%$ CI
MNIST	Raw Hessian trace	$+0.198$	[ $-0.149$ , $+0.553$ ]
MNIST	Final-layer relative flatness	$+0.122$	[ $-0.201$ , $+0.448$ ]
MNIST	Spectral-product margin	$-0.215$	[ $-0.467$ , $+0.037$ ]
MNIST	Weight norm	$-0.108$	[ $-0.422$ , $+0.194$ ]
Fashion-MNIST	Raw Hessian trace	$+0.412$	[ $+0.106$ , $+0.701$ ]
Fashion-MNIST	Final-layer relative flatness	$+0.507$	[ $+0.178$ , $+0.796$ ]

Appendix

Table 4: Paired differences between the joint-measure correlation and each oriented baseline correlation. Intervals are Bonferroni

98.75\%

intervals within each dataset.

Dataset	$\rho_{\rm low}$	$\rho_{\rm high}$	Half-bank $\rho$	Spearman–Brown
MNIST	$+0.290$	$+0.290$	$+0.887$	$0.940$
Fashion-MNIST	$+0.468$	$+0.468$	$+0.915$	$0.956$

Appendix

Table 5: Estimator diagnostics. The LP columns report correlations at the lower and upper feasibility bounds.

Dataset	Order 2	Order 4	Order 8	Order 16
MNIST	$0.671$	$0.402$	$0.073$	$0.000$
Fashion-MNIST	$0.741$	$0.487$	$0.133$	$0.000$
Random labels	$0.843$	$0.628$	$0.206$	$0.003$

Appendix

Table 6: Mean feasible fraction by commitment order. Rates near zero or one show that an order distinguishes few networks.

Map	Value	Test accuracy	Hessian trace
$T_{\beta}$	1	0.9392	66.1
$T_{\beta}$	2	0.9392	81.3
$T_{\beta}$	5	0.9392	388.8
$T_{\beta}$	10	0.9392	1607.1
$T_{\beta}$	20	0.9392	6525.9
$T_{\gamma}$	1	0.9392	62.9

Appendix

Table 7: One network trained on 6000 points. Function-preserving rescaling changes raw Hessian trace while preserving every prediction.

Explore similar work

A Function-Centric Perspective on Flat and Sharp Minima
Oct 14, 2025Israel Mason-Williams, Gabryel Mason-Williams, Helen YannakoudakisFlat Minima Loss Landscape Geometry
Department of Informatics, UKRI Safe and Trusted AI, London, United Kingdom · Queen Mary University of London, London, United Kingdom · Department of Informatics, King’s College London, London, United Kingdom
Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking
Oct 8, 2026Mohnish HarwaniNeural Network Generalization Flat Minima
Purdue University
A Flatness-Generalization Relation in the Teacher-Student Tree-Committee Machine
Sep 25, 2026Brandon Livio Annesi, Davide Straziota, Enrico Maria MalatestaFlat Minima Neural Network Generalization
Department of Quantitative Life Sciences, ICTP Trieste, Italy

Oct 14, 2025J/K move · Enter open · S save

A Function-Centric Perspective on Flat and Sharp Minima

Israel Mason-Williams, Gabryel Mason-Williams, Helen Yannakoudakis

Department of Informatics, UKRI Safe and Trusted AI, London, United Kingdom · Queen Mary University of London, London, United Kingdom · Department of Informatics, King’s College London, London, United Kingdom

Flat Minima Loss Landscape Geometry Neural Network Generalization

Open paper AskPDF arXiv

Flat minima are strongly associated with improved generalisation in deep neural networks. However, this connection has proven nuanced in recent studies, with both theoretical counterexamples and empirical exceptions emerging in the literature. In this paper, we revisit the role of sharpness in model performance and examine sharpness as a function-dependent property rather than as a direct indicator of poor generalisation. We conduct extensive empirical studies ranging from single-objective optimisation, synthetic non-linear binary classification tasks, to modern image classification tasks. In single-objective optimisation, we show that flatness and sharpness are relative to the function being learned: equally optimal solutions can exhibit markedly different local geometry. In synthetic non-linear binary classification tasks, we show that increasing decision-boundary tightness can increase sharpness even when models generalise perfectly, suggesting that sharpness is not reducible to memorisation alone. Finally, in large-scale experiments, we find that sharper minima often emerge when models are regularised (e.g., via weight decay, data augmentation, or SAM), and can coincide with better generalisation, calibration, robustness, and functional consistency. Our findings are consistent with a function-centric interpretation of minima geometry: flatness should not be treated as a universal proxy for generalisation, and sharper minima can reflect desirable inductive biases associated with improved predictive behaviour, calling for a function-centric reappraisal of minima geometry.

Abstract

Explore similar work

A Function-Centric Perspective on Flat and Sharp Minima

Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking

A Flatness-Generalization Relation in the Teacher-Student Tree-Committee Machine