The flatness of the loss landscape at a minimizer is a widely used heuristic for reasoning about neural-network generalization, yet evidence for this relation is mostly empirical and controversial. We study this relation in a teacher-student tree committee machine, where both the ERM estimator and the Hessian spectrum are analytically tractable in the proportional high-dimensional limit. First, we use a zero-temperature Gibbs formulation to obtain predictions for the observables of the typical minimizers of the empirical loss. Secondly, we use Edwards-Jones formalism to derive the limiting Hessian resolvent around these typical minimizers. All predictions agree with finite-size gradient-descent simulations. Finally, we study three measures of flatness, namely the left and right edges and the spectral mean, and check if a decrease in generalization error as the dataset size is increased corresponds to an increase in flatness. We find that the answer strongly depends on the learning task and on the ratio of the number of parameters to the number of data points. In regression, the spectral mean and right edge correlate with the generalization error, while the left edge does so only in the overparametrized regime. In classification this correlation reliably holds only in the highly overparametrized phase, while for underparametrized networks it can even reverse.
Figures & tables
Figure 1: Schematic representation of a tree committee machine.
Figure 2: Generalization error and order parameters for regression. The RS theory predicts the test error ϵg , squared norm Q , and teacher–student overlap r as functions of sample complexity α and regularization strength λ . Curves are theoretical predictions; markers report finite-size ( N=2000,K=20 ) GD simulations, averaged over 20 seeds. Error bars are not shown because they are very small.
Figure 3: Classification order parameters. Solid curves denote RS solutions, while dashed curves denote 1RSB solutions. Markers report finite-size GD simulations for N=2000,K=20 , averaged over 20 seeds. Error bars are not shown because they are very small.
Figure 4: Empirical spectrum vs theoretical spectrum for N=2000,K=20 and λ=0.01 . The left panel shows regression; the middle and right panels show classification.
Figure 5: Flatness observables versus generalization error in regression. Markers are given by GD simulations with N=2000,K=20 . In the phases where the empirical spectrum develops outliers, we show the smallest eigenvalue of the continuous bulk.
Figure 6: Flatness observables versus generalization error in classification. Solid curves are RS branches and dashed ones are 1RSB branches. Markers are given by GD simulations with N=2000,K=20 . In the phases where the empirical spectrum develops outliers, we show the smallest eigenvalue of the continuous bulk.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Classification case with square loss function. Dependence of the squared norm Q on the inverse temperature β for λ=0.001 and α=5 . Left panel: comparison between the replica symmetric solution and the 1RSB solution. Right panel: comparison between the 1RSB solution and the finite-step k -RSB approximation with k=20 . In the k -RSB solution we have optimized also on the breaking points ms , s=0,…,k−1 . In both cases the curves coincide at high temperature and separate as β increases.
Figure 8: Classification case with square loss function. Overlap function qs vs breaking points ms for α=5 , λ=0.001 and several values of β (left panel). On the right panel we have plotted the same data of the left panel vs βms≡δms . As predicted in ( 44 ) this is the right scaling of the breaking-points for large β . In black we also depict the exact solution in the limit β→∞ . In the inset we show a numerical verification of ( 44 ), plotting β(Q−qk) vs β and showing that it approaches a finite limit δq (black dashed line).
Figure 9: Classification case with square loss function. Left panel: squared norm Q versus α in the large β regime, for λ=0.001 . We compare the 1RSB (orange curve) and k -RSB with k=20 (green). In the k -RSB solution we have optimized also on the breaking points δms , s=0,…,k−1 ; we therefore expect the green curve to be a good proxy of the full-RSB line (i.e. k→∞ ). More than one step of RSB are therefore only slightly influencing the squared norm. We point out here that other order parameters are for all purposes left unchanged. Right panel: overlap vs breaking-points for several values of α . The maximum value achieved by the overlap is Q .
Department of Electronics and Communication Technology Gauhati University Guwahati, Assam, India · Department of Mathematics Gauhati University Guwahati, Assam, India