Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks
Authors: Jie Huang, Bruno Loureiro, Stefano Sarao Mannelli
Organizations: Physics, Chalmers University of Technology and University of Gothenburg · Ecole Normale Superieure, PSL & CNRS · Data Science and AI, Computer Science and Engineering, Chalmers University of Technology and University of Gothenburg · School of Computer Science and Applied Mathematics, University of the Witwatersrand
We study the population loss landscape of two-layer ReLU networks of the form ∑k=1KReLU(wk⊤x) in a realisable teacher-student setting with Gaussian covariates. We show that local minima admit an exact low-dimensional representation in terms of summary statistics, yielding a sharp and interpretable characterisation of the landscape. We further establish a direct link with one-pass SGD: local minima correspond to attractive fixed points of the dynamics in summary statistics space. This perspective reveals a hierarchical organisation of minima into discrete families and shows how overparameterisation changes their stability and reachability under gradient-based dynamics. In this overparameterised regime, global minima become increasingly accessible, attracting the dynamics and reducing convergence to spurious solutions. Overall, our results reveal intrinsic limitations of common simplifying assumptions, which may miss essential features of the loss landscape even in minimal neural network models.
Figures & tables
Figure 1: Geometry and statistics of local minima. (a) Schematic hierarchy of local-minimum families in the well-specified regime. Each plane represents a loss level and the crosses mark representative minima. (b) Validation of the theoretical predictions. The histogram shows the distribution of population risk reached by gradient flow ( 104 runs) in the high-dimensional standard normal teacher initialisations. Vertical dashed lines correspond to the analytical loss levels of the families derived in Result 2 . Insets show heatmaps of the student-student ( Q ) and student-teacher ( R ) overlaps for a representative solution within the dominant family.
Figure 2: Loss families and theoretical value. Histogram of final loss values obtained from ODE dynamics at M=17 for K=M (top), K=M+1 (middle), and K=M+2 (bottom) starting from 104 initialisations in orthonormal teacher configuration. Dashed vertical lines indicate the corresponding theoretical loss values obtained from the Result 2 For K=M and K=M+1 , the theoretical values capture the locations of the local minima families and the minor discrepancies may be attributed to numerical effects. In contrast, for K=M+2 , local minima are strongly suppressed in M=17 and the loss distribution concentrates near zero.
Figure 3: Perturbative analysis of fixed points. Distance from the initial configuration obtained by adding a Gaussian perturbation of standard deviation σ in weight space. The plot consider the average distance after the network has been allowed to relax with gradient descent for 1,000 steps and learning rate 0.01 . The mean (solid line) and standard deviation (shaded area) are evaluated by collecting 20 independent perturbations. The different line represent different level of over-parameterisation. The exact-parametirisation case appears to be less that 0.001 from the original configuration, a distance that is compatible with the limited epochs allowed by gradient descent, and it is therefore considered stable from small enough σ . On contrary, the overparameteritdsed case show instability even at small σ .
MinimaOrder
K=17M=17
K=18M=17
K=19M=17
k1=0
13.09%
59.29%
99.63%
k1=1
27.52%
0.00%
0.00%
k1=2
29.05%
2.10%
0.05%
k1=3
18.94%
10.83%
0.31%
k1=4
7.55%
8.99%
0%
Table 1: Percentage of 104 random initialisations that converge to minima with k1 anti-aligned units in different (K,M) ; k1=0 corresponds to the global minimum.
Figure 4: Training dynamics and loss quantisation across parameterisation regimes. Evolution of population risk under gradient flow for M=20 teacher units (1,000 random initialisations per condition, learning rate η=0.1 , orthonormal teacher configuration). All regimes exhibit entrapment in discrete high-loss plateaux. However, while the well-specified case (K=20, orange) is dominated by these suboptimal attractors, mild over-parameterisation ( K=21 , purple; K=22 , blue) progressively destabilises intermediate plateaus, allowing a larger fraction of trajectories to escape towards the global minimum. The observed loss bands correspond to the theoretical loss levels of the spurious families predicted by Result 2 .
Optimiser
K=17M=17
K=18M=17
K=19M=17
GD
13.25%
64.18%
77.50%
2L-GD
13.24%
67.91%
99.48%
nGD
14.12%
58.35%
N/C
onGD
N/C
N/C
N/C
Table 2: Convergence frequency across optimisation schemes. The table reports the percentage of runs reaching the global minimum out of 104 initialisations. Columns represent different parameterisation regimes (exact vs. over-parameterised). Consistent with Safran and Shamir (2018) , over-parameterisation aids convergence in unconstrained dynamics. A more thorough comparison is reported in Appendix G . However, constrained dynamics (nGD, onGD) show severe slowdowns, resulting in non-convergence (N/C) within the allocated computational budget ( 1.2×107 steps), a comprehensive analysis is detailed in Appendix G.1 .
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Comparison between mean-field ODE predictions and network simulations. Panel (a) shows the training loss, while panels (b) and (c) display representative entries of the order parameters R and Q as functions of training steps. Solid lines denote simulation results and markers indicate ODE predictions.
Figure 6: Gaussian mixture analysis of diagonal order parameters. The left panel shows the distribution of diagonal elements of the matrix Q , which is well described by a two-component Gaussian mixture model. The right panel shows the distribution of diagonal elements of the matrix T , which follows a single Gaussian distribution centred at 1 . The data are obtained by grouping order parameters within the same minima family, re-arranging the corresponding (Q,R) matrices to maximise the maximise the absolute values of diagonal of Q and extracting the diagonal entries.
Figure 7: Different order parameters at K=18 and M=17 . The plots show the values of R (first row) and Q (second row) for minima representative of the different families. From left to right, we see results for k1=0,2,3,4,5,6 .
Figure 8: Order parameters obtained from solvers at K=18 and M=17 . The plots show the values of R (first row) and Q (second row) for minima computed by the solver. Columns are ordered from left to right by the number of anti-aligned units, k1=0,2,3,4,5,6 .
Figure 9: Impact of finite dimensionality on the loss distribution. Histograms of the final population loss obtained by simulating the ODEs starting from actual neural network weight initialisations at dimensions d∈{196,392,784} , compared against the infinite-dimensional limit (orthonormal teacher, T=I ). Dashed vertical lines indicate the corresponding theoretically predicted loss values. While smaller dimensions introduce a broadening effect due to variance, the discrete families of minima remain robustly centred around the predicted macroscopic levels.
Figure 10: Student-teacher overlap matrix ( R ) across different dimensions. Heatmaps of the R matrix for 10 randomly sampled configurations converging to the first-order local minimum ( k1=1 ). The top row represents the idealised d→∞ limit, while subsequent rows correspond to finite dimensions d=784,392,and 196 . The block-symmetric structure predicted by our theoretical ansatz clearly emerges across all regimes, with finite-size fluctuations increasing at lower dimensions.
Figure 11: Order Parameters of Leaky ReLU in K=M=12
Figure 12: Order Parameters of erf in K=M=12
Figure 13: Loss distribution in different constraints These histograms show the empirical density of the final population loss after 20,000,000 running steps. (a) Normalized Gradient Descent (nGD) with K=19,M=17 . Despite overparameterisation, the loss distribution is strictly multimodal, exhibiting discrete peaks at high loss values. (b) Orthonormalised Gradient Descent (onGD) with K=17,M=17 . The distribution is unimodal and broad, concentrated entirely in a high-error regime (centered at 0.22 ).
Figure 14: Order parameters of local minima in nGD ( K=19,M=17 ). The columns correspond to different final losses. Left Two Columns (Best Minima): The R matrices exhibit a near-perfect diagonal structure, indicating successful retrieval of the 17 teacher features. However, the irreducible error persists because the 2 excess students (constrained to Qii=1 ) cannot decay to zero. Right Four Columns (Suboptimal Minima): These states represent the dominant local minima cluster. They exhibit ”blocking” defects (e.g., 2 -to- 1 assignments seen as red blocks in R ), where extra units are misallocated, further elevating the loss.
Figure 15: Final order parameters for onGD ( K=M=17 ). The columns correspond to different final losses. Bottom Row: The student-student overlap matrices Q retain a strict identity structure ( Q=I ), satisfying the orthogonality constraint. Top Row: The student-teacher overlap matrices R exhibit a disordered, noise-like pattern. Unlike successful learning scenarios shown in Fig. 7 , the entries remain diffuse and small. This lack of structural alignment confirms that the student units fail to retrieve the teacher vectors in this constrained setting.
Figure 16: Dynamics of onGD in a setting with small hidden layers ( K=M=2 ). (a) The training loss (log scale) decreases monotonically and converges to a near-zero value ( ∼10−3 ), indicating successful optimisation. (b) The final student-teacher overlap matrix R exhibits a clear diagonal structure (red blocks indicate high positive correlation). This confirms that, unlike the case with larger hidden layers ( K=M=17 ), the orthogonality constraint perfectly retrieves the teacher weights in low dimensions.
Table 3: Statistics of minima under mean-field dynamics. Proportions of local, global, and non-minima found by integrating the mean-field ODEs across 10,000 random seeds for each (K,M) pair.
K
M
Local (%)
Global (%)
None (%)
ε
δ
2
2
0.00
99.77
0.23
1e-2
1e-2
3
3
0.15
99.85
0.00
1e-2
1e-2
4
4
0.02
99.98
0.00
1e-2
1e-2
5
5
0.01
99.99
0.00
1e-2
1e-2
6
6
0.00
100.00
0.00
1e-2
1e-2
7
7
0.62
99.38
0.00
1e-2
1e-2
Appendix
Table 4: Statistics of minima in nGD under mean-field dynamics for case K=M. Proportions of local, global, and non-minima obtained by integrating the mean-field ODEs under Normalized GD over 10,000 random seeds for each (K,M) with K=M . A result is classified as a global minimum when its error is smaller than ε . For the result whose errors are larger than ε , if is gradients ( dQij/dt or dRin/dt ) are larger than δ , it will be classified as None. Otherwise it will be classified as a local minimum. From the analysis in Section F , the global minima of nGD are very large.
K
M
Local (%)
Global (%)
None (%)
ε
δ
3
2
2.73
97.27
0.00
0.123
1e-1
4
3
48.95
51.05
0.00
0.11
1e-1
5
4
1.34
98.66
0.00
0.105
1e-1
6
5
1.40
98.60
0.00
0.105
1e-1
7
6
2.87
97.12
0.01
1e-1
1e-1
8
7
1.80
98.19
0.01
1e-1
1e-1
Appendix
Table 5: Statistics of minima in nGD under mean-field dynamics for case K=M+1. Proportions of local, global, and non-minima obtained by integrating the mean-field ODEs under Normalized GD over 10,000 random seeds for each (K,M) with K=M+1 . A result is classified as a global minimum when its error is smaller than ε . For the result whose errors are larger than ε , if is gradients ( dQij/dt or dRin/dt ) are larger than δ , it will be classified as None. Otherwise it will be classified as a local minimum. From the analysis in Section F , the global minima of nGD are very large.
Figure 17: Distribution of loss for small networks in onGD under mean-field dynamic. Histograms of the final population loss obtained after long-time integration of the mean-field ODEs under Orthonormalised GD over 10,000 random initialisations. Each panel corresponds to a small student-teacher size configuration (K,M) . The top row shows the equal case K=M and the bottom row shows the overparameterised case K=M+1 . All panels share the same horizontal axis for direct comparison.
Figure 18: Distribution of loss for wider networks in onGD under mean-field dynamic. Histograms of the final population loss obtained after long-time integration of the mean-field ODEs under Orthonormalised GD over 10,000 random initialisations. Each panel corresponds to a student-teacher size configuration (K,M) with larger widths than those shown in Fig. 17 . The top row shows the equal case K=M and the bottom row shows the overparameterised case K=M+1 . All panels share the same horizontal axis for direct comparison.
Understanding loss landscapes is central to explaining neural-network training, yet their structure remains only partially understood even in simple models. We study the Gaussian population loss of shallow, bias-free ReLU networks in the teacher--student setting. This provides a simple model for studying essential aspects such as feature learning and overparameterization. For teacher networks with positive output weights and planar features, we show that including a learned linear skip removes all spurious local minima with non-negative student output weights once the student network is at least as wide as the teacher network. In contrast, without the skip, we construct a fixed teacher network with positive output weights and only three hidden neurons in input dimension two whose spurious local minima persist at every student width at least three. Thus, a learned linear skip can remove spurious minima that persist under arbitrary overparameterization. Furthermore, we show that a positive output weight student network always learns the subspace spanned by the teacher features: student features at local minima with non-negative student output weights lie in the span of the teacher features. For ReLU networks in two dimensions, even heavily overparameterized student networks have effective width controlled by the teacher width: every critical point with positive student output weights has at most twice as many distinct student feature directions as teacher neurons. Finally, we transfer the benignity result to empirical minima over parameter balls of any prescribed radius, with the required sampling accuracy depending on that radius.
Jakob Paul Zimmermann, Andrei Balakin, Moritz Grillo +1
Freie Universität Berlin · Technische Universität Berlin
The main result presented is a formula for the population loss in the student-teacher kernel model that is applicable to shallow ReLU networks with bias. This extends previous work of Choo and Saul (2009) and Brutzkus and Globerson (2017). The formula makes essential use of Owen's T-function. The necessary theory of the T-function is given and a high precision coding using MPFR for the T-function, based on an algorithm of Komelj (2023), is available on request. It is shown that various families of spurious minima described in past papers of Arjevani and the author extend to biased networks and that the loss is always strictly decreased when bias is added. The change in landscape geometry caused by adding bias appears to be relatively mild. Only the simplest examples are described in this paper where it is assumed that the number of inputs is equal to the number of neurons (this restriction is for reasons of length). A review of relevant previous results on unbiased networks is included. Aside from Gaussian statistics, the main mathematical tools and ideas come from analytic geometry (analytic and subanalytic sets, the Curve Selection Lemma).
Understanding deep neural networks remains a central challenge in machine learning. In particular, the theoretical properties of even two-layer ReLU networks, especially in the presence of weight decay, remain poorly understood. To this end, we derive a sufficient condition on the hyperparameter settings under which the global minima collapse to the zero solution. Interestingly, our experiments reveal that using AdamW as an optimizer prevents the collapse of the learned parameters, whereas using SGD does not, which may help explain the success of AdamW in deep learning training. In addition, when restricting the input dimension to one, we derive an analytical solution for the globally optimal parameter sets of two-layer ReLU networks and show that ℓ2-regularization has a width-invariant effect on connectivity, but its dimensionality-reducing effect becomes stronger as the network width increases. These results provide insight into how width-dependent hyperparameters influence the geometry of regularized loss landscapes.
Haruka Eshima, Makoto Yamada
Okinawa Institute of Science and Technology, Okinawa, Japan