The flatness of the loss landscape at a minimizer is a widely used heuristic for reasoning about neural-network generalization, yet evidence for this relation is mostly empirical and controversial. We study this relation in a teacher-student tree committee machine, where both the ERM estimator and the Hessian spectrum are analytically tractable in the proportional high-dimensional limit. First, we use a zero-temperature Gibbs formulation to obtain predictions for the observables of the typical minimizers of the empirical loss. Secondly, we use Edwards-Jones formalism to derive the limiting Hessian resolvent around these typical minimizers. All predictions agree with finite-size gradient-descent simulations. Finally, we study three measures of flatness, namely the left and right edges and the spectral mean, and check if a decrease in generalization error as the dataset size is increased corresponds to an increase in flatness. We find that the answer strongly depends on the learning task and on the ratio of the number of parameters to the number of data points. In regression, the spectral mean and right edge correlate with the generalization error, while the left edge does so only in the overparametrized regime. In classification this correlation reliably holds only in the highly overparametrized phase, while for underparametrized networks it can even reverse.
Figures & tables
Figure 1: Schematic representation of a tree committee machine.
Figure 2: Generalization error and order parameters for regression. The RS theory predicts the test error ϵg , squared norm Q , and teacher–student overlap r as functions of sample complexity α and regularization strength λ . Curves are theoretical predictions; markers report finite-size ( N=2000,K=20 ) GD simulations, averaged over 20 seeds. Error bars are not shown because they are very small.
Figure 3: Classification order parameters. Solid curves denote RS solutions, while dashed curves denote 1RSB solutions. Markers report finite-size GD simulations for N=2000,K=20 , averaged over 20 seeds. Error bars are not shown because they are very small.
Figure 4: Empirical spectrum vs theoretical spectrum for N=2000,K=20 and λ=0.01 . The left panel shows regression; the middle and right panels show classification.
Figure 5: Flatness observables versus generalization error in regression. Markers are given by GD simulations with N=2000,K=20 . In the phases where the empirical spectrum develops outliers, we show the smallest eigenvalue of the continuous bulk.
Figure 6: Flatness observables versus generalization error in classification. Solid curves are RS branches and dashed ones are 1RSB branches. Markers are given by GD simulations with N=2000,K=20 . In the phases where the empirical spectrum develops outliers, we show the smallest eigenvalue of the continuous bulk.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Classification case with square loss function. Dependence of the squared norm Q on the inverse temperature β for λ=0.001 and α=5 . Left panel: comparison between the replica symmetric solution and the 1RSB solution. Right panel: comparison between the 1RSB solution and the finite-step k -RSB approximation with k=20 . In the k -RSB solution we have optimized also on the breaking points ms , s=0,…,k−1 . In both cases the curves coincide at high temperature and separate as β increases.
Figure 8: Classification case with square loss function. Overlap function qs vs breaking points ms for α=5 , λ=0.001 and several values of β (left panel). On the right panel we have plotted the same data of the left panel vs βms≡δms . As predicted in ( 44 ) this is the right scaling of the breaking-points for large β . In black we also depict the exact solution in the limit β→∞ . In the inset we show a numerical verification of ( 44 ), plotting β(Q−qk) vs β and showing that it approaches a finite limit δq (black dashed line).
Figure 9: Classification case with square loss function. Left panel: squared norm Q versus α in the large β regime, for λ=0.001 . We compare the 1RSB (orange curve) and k -RSB with k=20 (green). In the k -RSB solution we have optimized also on the breaking points δms , s=0,…,k−1 ; we therefore expect the green curve to be a good proxy of the full-RSB line (i.e. k→∞ ). More than one step of RSB are therefore only slightly influencing the squared norm. We point out here that other order parameters are for all purposes left unchanged. Right panel: overlap vs breaking-points for several values of α . The maximum value achieved by the overlap is Q .
Flat loss landscapes have long been linked to better generalization in neural networks. However, its role as a causal mechanism for generalization is less established. Grokking provides an unique testbed to understand this distinction: models are prone to fit observed data using non-generalizing structure and remain in that regime for prolonged periods, transitioning to generalization only under particular training conditions. In this work, we study whether flat loss landscapes can act as a driving mechanism in this transition. While recent work has argued for flatness as a necessary geometric condition for this transition, we find that biasing training toward flatter solutions using sharpness-aware minimization (SAM) is insufficient to reliably induce this transition, despite producing flatter solutions. However, when SAM is paired with mechanisms that drive generalization such as weight decay, an interesting property emerges: SAM can accelerate the transition to generalizing solutions by up to 4x at the epoch-level. We theoretically untangle this relationship between SAM and weight decay using a minimal interpolating two-layer ReLU model with both memorizing and generalizing solutions. We show that even in this simple setup, flatness alone cannot distinguish a memorizing solution from a generalizing one, while weight decay favors generalizing solutions. However, under a local stability analysis, there exists a window where a memorizing interpolant is locally stable under gradient descent but unstable under SAM in the low-norm regime, which can explain SAM's ability to accelerate this transition. Overall, our results provide a more interpretable account of the role of flatness in driving generalization, especially in settings where models are vulnerable to minimizing loss through learning non-generalizing structure.
The flatness hypothesis suggests that flatness of the loss landscape, as measured by the eigenvalues of the loss Hessian, correlates with better neural network generalization. While various algorithms reduce these eigenvalues, most focus on procedural design, leaving it unclear how data distributions and NN parameters structurally determine directions toward flat minima. Characterizing these directions analytically is generally intractable. To overcome this mathematical difficulty, recent studies derived the Wolkowicz-Styan (WS) upper bound on the maximum eigenvalue of the cross-entropy loss Hessian in three-layer NNs. Although this upper bound is differentiable, its gradient was not derived. Therefore, we analytically derive the gradient of the WS upper bound to characterize directions leading to flat minima. Based on this, we propose Hessian Spectral Range (HSR) Regularization, which updates parameters along the steepest descent direction of the WS bound. Experiments demonstrate that HSR Regularization narrows the Hessian eigenvalue spectrum, avoids sharp minima and saddle points, and promotes convergence to flat minima. Although the applicability of this method is currently limited to cross-entropy loss and three-layer architectures, to the best of the authors' knowledge, this is the first study to report a closed-form gradient that promotes convergence to flat minima without numerical approximations. Therefore, the theoretical analysis of this gradient is expected to contribute to the further development of NNs.
Yuto Omae, Kazuki Sakai, Yohei Kakimoto +3
Nihon University (Japan) · National Institute of Technology, Nagaoka College (Japan) · Tokyo City University (Japan)
A widely held intuition in deep learning is that stochastic gradient descent (SGD) implicitly favors flat minima and that flat minima generalize better, but standard Euclidean measures of flatness such as the trace or maximum eigenvalue of the loss Hessian are not invariant under reparametrizations that preserve the network function, which undermines the theoretical foundations of this narrative. In this study we resolve this issue by grounding flatness in the Riemannian geometry of the statistical manifold induced by the Fisher Information Matrix (FIM). We define Riemannian sharpness mathematically and prove that it is invariant under smooth, function-preserving reparametrizations, which directly addresses the critique of Dinh et al. in the paper ``Sharp minima can generalize for deep nets''.We note that this invariance is a property of the true FIM; the diagonal empirical estimator used in practice (and in all experiments below) inherits invariance only approximately, and exact invariance under arbitrary reparametrizations would require structured estimators such as K-FAC. We formalize the gradient noise of mini-batch SGD as having a covariance structure proportional to the FIM, derive the stationary distribution of the resulting stochastic differential equation, and then show that the probability mass is exponentially concentrated at Riemannian-flat minima. A PAC-Bayes generalization bound controlled explicitly by SR formally links this geometric bias to test performance. Our experiments on MNIST and CIFAR-10 confirm that SR reliably tracks generalization in ways that Euclidean sharpness does not, and that its scaling with η/B matches the theoretical predictions. Together these results provide a rigorous, reparametrization-invariant account of why flat minima generalize.
Md Sakir Ahmed, Kumaresh Sarmah, Hemen Dutta
Department of Electronics and Communication Technology Gauhati University Guwahati, Assam, India · Department of Mathematics Gauhati University Guwahati, Assam, India