Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks

Jie Huang; Bruno Loureiro; Stefano Sarao Mannelli

stat.MLApr 10, 2026

Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks

Authors: Jie Huang, Bruno Loureiro, Stefano Sarao Mannelli

Organizations: Physics, Chalmers University of Technology and University of Gothenburg · Ecole Normale Superieure, PSL & CNRS · Data Science and AI, Computer Science and Engineering, Chalmers University of Technology and University of Gothenburg · School of Computer Science and Applied Mathematics, University of the Witwatersrand

Abstract

We study the population loss landscape of two-layer ReLU networks of the form $\sum_{k=1}^K \mathrm{ReLU}(w_k^\top x)$ in a realisable teacher-student setting with Gaussian covariates. We show that local minima admit an exact low-dimensional representation in terms of summary statistics, yielding a sharp and interpretable characterisation of the landscape. We further establish a direct link with one-pass SGD: local minima correspond to attractive fixed points of the dynamics in summary statistics space. This perspective reveals a hierarchical organisation of minima into discrete families and shows how overparameterisation changes their stability and reachability under gradient-based dynamics. In this overparameterised regime, global minima become increasingly accessible, attracting the dynamics and reducing convergence to spurious solutions. Overall, our results reveal intrinsic limitations of common simplifying assumptions, which may miss essential features of the loss landscape even in minimal neural network models.

Figures & tables

Panel of Figure 1 — Figure 1: Geometry and statistics of local minima. (a) Schematic hierarchy of local-minimum families in the well-specified regime. Each plane represents a loss level and the crosses mark representative minima. (b) Validation of the theoretical predictions. The histogram shows the distribution of population risk reached by gradient flow ( $10^{4}$ runs) in the high-dimensional standard normal teacher initialisations. Vertical dashed lines correspond to the analytical loss levels of the families derived in Result 2 . Insets show heatmaps of the student-student ( $Q$ ) and student-teacher ( $R$ ) overlaps for a representative solution within the dominant family.

Panel of Figure 2 — Figure 2: Loss families and theoretical value. Histogram of final loss values obtained from ODE dynamics at $M=17$ for $K=M$ (top), $K=M+1$ (middle), and $K=M+2$ (bottom) starting from $10^{4}$ initialisations in orthonormal teacher configuration. Dashed vertical lines indicate the corresponding theoretical loss values obtained from the Result 2 For $K=M$ and $K=M+1$ , the theoretical values capture the locations of the local minima families and the minor discrepancies may be attributed to numerical effects. In contrast, for $K=M+2$ , local minima are strongly suppressed in $M=17$ and the loss distribution concentrates near zero.

Figure 3: Perturbative analysis of fixed points. Distance from the initial configuration obtained by adding a Gaussian perturbation of standard deviation

\sigma

in weight space. The plot consider the average distance after the network has been allowed to relax with gradient descent for

1,000

steps and learning rate

0.01

. The mean (solid line) and standard deviation (shaded area) are evaluated by collecting

20

independent perturbations. The different line represent different level of over-parameterisation. The exact-parametirisation case appears to be less that

0.001

from the original configuration, a distance that is compatible with the limited epochs allowed by gradient descent, and it is therefore considered stable from small enough

\sigma

. On contrary, the overparameteritdsed case show instability even at small

\sigma

$\begin{matrix}Minima\\ Order\end{matrix}$	$\begin{matrix}K=17\\ M=17\end{matrix}$	$\begin{matrix}K=18\\ M=17\end{matrix}$	$\begin{matrix}K=19\\ M=17\end{matrix}$
$k_{1}=0$	13.09%	59.29%	99.63%
$k_{1}=1$	27.52%	0.00%	0.00%
$k_{1}=2$	29.05%	2.10%	0.05%
$k_{1}=3$	18.94%	10.83%	0.31%
$k_{1}=4$	7.55%	8.99%	0%

Table 1: Percentage of

10^{4}

random initialisations that converge to minima with

k_{1}

anti-aligned units in different

(K,M)

;

k_{1}=0

corresponds to the global minimum.

Figure 4: Training dynamics and loss quantisation across parameterisation regimes. Evolution of population risk under gradient flow for $M=20$ teacher units (1,000 random initialisations per condition, learning rate $\eta=0.1$ , orthonormal teacher configuration). All regimes exhibit entrapment in discrete high-loss plateaux. However, while the well-specified case (K=20, orange) is dominated by these suboptimal attractors, mild over-parameterisation ( $K=21$ , purple; $K=22$ , blue) progressively destabilises intermediate plateaus, allowing a larger fraction of trajectories to escape towards the global minimum. The observed loss bands correspond to the theoretical loss levels of the spurious families predicted by Result 2 .

Optimiser	$\begin{matrix}K=17\\ M=17\end{matrix}$	$\begin{matrix}K=18\\ M=17\end{matrix}$	$\begin{matrix}K=19\\ M=17\end{matrix}$
GD	13.25%	64.18%	77.50%
2L-GD	13.24%	67.91%	99.48%
nGD	14.12%	58.35%	N/C
onGD	N/C	N/C	N/C

Table 2: Convergence frequency across optimisation schemes. The table reports the percentage of runs reaching the global minimum out of

10^{4}

initialisations. Columns represent different parameterisation regimes (exact vs. over-parameterised). Consistent with Safran and Shamir (2018) , over-parameterisation aids convergence in unconstrained dynamics. A more thorough comparison is reported in Appendix G . However, constrained dynamics (nGD, onGD) show severe slowdowns, resulting in non-convergence (N/C) within the allocated computational budget (

1.2\times 10^{7}

steps), a comprehensive analysis is detailed in Appendix G.1 .

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Comparison between mean-field ODE predictions and network simulations. Panel (a) shows the training loss, while panels (b) and (c) display representative entries of the order parameters $R$ and $Q$ as functions of training steps. Solid lines denote simulation results and markers indicate ODE predictions.

Figure 6: Gaussian mixture analysis of diagonal order parameters. The left panel shows the distribution of diagonal elements of the matrix

Q

, which is well described by a two-component Gaussian mixture model. The right panel shows the distribution of diagonal elements of the matrix

T

, which follows a single Gaussian distribution centred at

1

. The data are obtained by grouping order parameters within the same minima family, re-arranging the corresponding

(Q,R)

matrices to maximise the maximise the absolute values of diagonal of

Q

and extracting the diagonal entries.

Figure 7: Different order parameters at $K=18$ and $M=17$ . The plots show the values of $R$ (first row) and $Q$ (second row) for minima representative of the different families. From left to right, we see results for $k_{1}=0,2,3,4,5,6$ .

Figure 8: Order parameters obtained from solvers at $K=18$ and $M=17$ . The plots show the values of $R$ (first row) and $Q$ (second row) for minima computed by the solver. Columns are ordered from left to right by the number of anti-aligned units, $k_{1}=0,2,3,4,5,6$ .

Figure 9: Impact of finite dimensionality on the loss distribution. Histograms of the final population loss obtained by simulating the ODEs starting from actual neural network weight initialisations at dimensions

d\in\{196,392,784\}

, compared against the infinite-dimensional limit (orthonormal teacher,

T=\mathbb{I}

). Dashed vertical lines indicate the corresponding theoretically predicted loss values. While smaller dimensions introduce a broadening effect due to variance, the discrete families of minima remain robustly centred around the predicted macroscopic levels.

Figure 10: Student-teacher overlap matrix ( $R$ ) across different dimensions. Heatmaps of the $R$ matrix for 10 randomly sampled configurations converging to the first-order local minimum ( $k_{1}=1$ ). The top row represents the idealised $d\to\infty$ limit, while subsequent rows correspond to finite dimensions $d=784,392,\text{and }196$ . The block-symmetric structure predicted by our theoretical ansatz clearly emerges across all regimes, with finite-size fluctuations increasing at lower dimensions.

Figure 11: Order Parameters of Leaky ReLU in $K=M=12$

Figure 12: Order Parameters of erf in $K=M=12$

Panel of Figure 13 — Figure 13: Loss distribution in different constraints These histograms show the empirical density of the final population loss after 20,000,000 running steps. (a) Normalized Gradient Descent (nGD) with $K=19,M=17$ . Despite overparameterisation, the loss distribution is strictly multimodal, exhibiting discrete peaks at high loss values. (b) Orthonormalised Gradient Descent (onGD) with $K=17,M=17$ . The distribution is unimodal and broad, concentrated entirely in a high-error regime (centered at $0.22$ ).

Figure 14: Order parameters of local minima in nGD ( $K=19,M=17$ ). The columns correspond to different final losses. Left Two Columns (Best Minima): The $R$ matrices exhibit a near-perfect diagonal structure, indicating successful retrieval of the 17 teacher features. However, the irreducible error persists because the 2 excess students (constrained to $Q_{ii}=1$ ) cannot decay to zero. Right Four Columns (Suboptimal Minima): These states represent the dominant local minima cluster. They exhibit ”blocking” defects (e.g., $2$ -to- $1$ assignments seen as red blocks in $R$ ), where extra units are misallocated, further elevating the loss.

Figure 15: Final order parameters for onGD ( $K=M=17$ ). The columns correspond to different final losses. Bottom Row: The student-student overlap matrices $Q$ retain a strict identity structure ( $Q=I$ ), satisfying the orthogonality constraint. Top Row: The student-teacher overlap matrices $R$ exhibit a disordered, noise-like pattern. Unlike successful learning scenarios shown in Fig. 7 , the entries remain diffuse and small. This lack of structural alignment confirms that the student units fail to retrieve the teacher vectors in this constrained setting.

Figure 16: Dynamics of onGD in a setting with small hidden layers ( $K=M=2$ ). (a) The training loss (log scale) decreases monotonically and converges to a near-zero value ( $\sim 10^{-3}$ ), indicating successful optimisation. (b) The final student-teacher overlap matrix $R$ exhibits a clear diagonal structure (red blocks indicate high positive correlation). This confirms that, unlike the case with larger hidden layers ( $K=M=17$ ), the orthogonality constraint perfectly retrieves the teacher weights in low dimensions.

Table 3: Statistics of minima under mean-field dynamics. Proportions of local, global, and non-minima found by integrating the mean-field ODEs across 10,000 random seeds for each

(K,M)

pair.

K	M	Local (%)	Global (%)	None (%)	$\boldsymbol{\varepsilon}$	$\boldsymbol{\delta}$
2	2	0.00	99.77	0.23	1e-2	1e-2
3	3	0.15	99.85	0.00	1e-2	1e-2
4	4	0.02	99.98	0.00	1e-2	1e-2
5	5	0.01	99.99	0.00	1e-2	1e-2
6	6	0.00	100.00	0.00	1e-2	1e-2
7	7	0.62	99.38	0.00	1e-2	1e-2

Appendix

Table 4: Statistics of minima in nGD under mean-field dynamics for case K=M. Proportions of local, global, and non-minima obtained by integrating the mean-field ODEs under Normalized GD over 10,000 random seeds for each

(K,M)

with

K=M

. A result is classified as a global minimum when its error is smaller than

\varepsilon

. For the result whose errors are larger than

\varepsilon

, if is gradients (

dQ_{ij}/dt

dR_{in}/dt

) are larger than

\delta

, it will be classified as None. Otherwise it will be classified as a local minimum. From the analysis in Section F , the global minima of nGD are very large.

K	M	Local (%)	Global (%)	None (%)	$\boldsymbol{\varepsilon}$	$\boldsymbol{\delta}$
3	2	2.73	97.27	0.00	0.123	1e-1
4	3	48.95	51.05	0.00	0.11	1e-1
5	4	1.34	98.66	0.00	0.105	1e-1
6	5	1.40	98.60	0.00	0.105	1e-1
7	6	2.87	97.12	0.01	1e-1	1e-1
8	7	1.80	98.19	0.01	1e-1	1e-1

Appendix

Table 5: Statistics of minima in nGD under mean-field dynamics for case K=M+1. Proportions of local, global, and non-minima obtained by integrating the mean-field ODEs under Normalized GD over 10,000 random seeds for each

(K,M)

with

K=M+1

. A result is classified as a global minimum when its error is smaller than

\varepsilon

. For the result whose errors are larger than

\varepsilon

, if is gradients (

dQ_{ij}/dt

dR_{in}/dt

) are larger than

\delta

, it will be classified as None. Otherwise it will be classified as a local minimum. From the analysis in Section F , the global minima of nGD are very large.

Figure 17: Distribution of loss for small networks in onGD under mean-field dynamic. Histograms of the final population loss obtained after long-time integration of the mean-field ODEs under Orthonormalised GD over 10,000 random initialisations. Each panel corresponds to a small student-teacher size configuration

(K,M)

. The top row shows the equal case

K=M

and the bottom row shows the overparameterised case

K=M+1

. All panels share the same horizontal axis for direct comparison.

Figure 18: Distribution of loss for wider networks in onGD under mean-field dynamic. Histograms of the final population loss obtained after long-time integration of the mean-field ODEs under Orthonormalised GD over 10,000 random initialisations. Each panel corresponds to a student-teacher size configuration

(K,M)

with larger widths than those shown in Fig. 17 . The top row shows the equal case

K=M

and the bottom row shows the overparameterised case

K=M+1

. All panels share the same horizontal axis for direct comparison.

Overparameterization

See all 22 papers

Removing spurious minima for planar features by skip connections
Oct 1, 2026Jakob Paul Zimmermann, Andrei Balakin, Moritz Grillo +1Shallow Neural Networks ReLU Neural Networks
Freie Universität Berlin · Technische Universität Berlin
Population loss in shallow ReLU networks: Bias & families of critical points
Sep 25, 2026Michael FieldShallow Neural Networks ReLU Neural Networks
Effects of width-dependent model hyperparameters and $\ell_2$ -regularization on the loss landscape of two-layer ReLU networks
Jul 18, 2026Haruka Eshima, Makoto YamadaReLU Neural Networks Weight Decay
Okinawa Institute of Science and Technology, Okinawa, Japan

Oct 1, 2026J/K move · Enter open · S save

Removing spurious minima for planar features by skip connections

Jakob Paul Zimmermann, Andrei Balakin, Moritz Grillo, Georg Loho

Freie Universität Berlin · Technische Universität Berlin

Shallow Neural Networks ReLU Neural Networks Neural Network Optimization Nonconvex Optimization Teacher-Student Learning

Open paper AskPDF arXiv

Understanding loss landscapes is central to explaining neural-network training, yet their structure remains only partially understood even in simple models. We study the Gaussian population loss of shallow, bias-free ReLU networks in the teacher--student setting. This provides a simple model for studying essential aspects such as feature learning and overparameterization. For teacher networks with positive output weights and planar features, we show that including a learned linear skip removes all spurious local minima with non-negative student output weights once the student network is at least as wide as the teacher network. In contrast, without the skip, we construct a fixed teacher network with positive output weights and only three hidden neurons in input dimension two whose spurious local minima persist at every student width at least three. Thus, a learned linear skip can remove spurious minima that persist under arbitrary overparameterization. Furthermore, we show that a positive output weight student network always learns the subspace spanned by the teacher features: student features at local minima with non-negative student output weights lie in the span of the teacher features. For ReLU networks in two dimensions, even heavily overparameterized student networks have effective width controlled by the teacher width: every critical point with positive student output weights has at most twice as many distinct student feature directions as teacher neurons. Finally, we transfer the benignity result to empirical minima over parameter balls of any prescribed radius, with the required sampling accuracy depending on that radius.

Abstract

Overparameterization

Explore similar work

Removing spurious minima for planar features by skip connections

Population loss in shallow ReLU networks: Bias & families of critical points

Effects of width-dependent model hyperparameters and ℓ2\ell_2ℓ2​-regularization on the loss landscape of two-layer ReLU networks

Effects of width-dependent model hyperparameters and $\ell_2$ -regularization on the loss landscape of two-layer ReLU networks