Neural network optimization strategies and the topography of the loss landscape

Jianneng Yu; Alexandre V. Morozov

cs.LGFeb 24, 2026

Neural network optimization strategies and the topography of the loss landscape

Authors: Jianneng Yu, Alexandre V. Morozov

Organizations: Department of Physics & Astronomy, Rutgers, The State University of New Jersey, 136 Frelinghuysen Rd., Piscataway, NJ 08854, U.S.A.

Abstract

Neural networks are trained by optimizing multi-dimensional sets of fitting parameters on non-convex loss landscapes. Low-loss regions of the landscapes correspond to the parameter sets that perform well on the training data. A key issue in machine learning is the performance of trained neural networks on previously unseen test data. Here, we investigate neural network training by stochastic gradient descent (SGD) - a non-convex global optimization algorithm which relies only on the gradient of the objective function. We contrast SGD solutions with those obtained via a non-stochastic quasi-Newton method, which utilizes curvature information to determine step direction and Golden Section Search to choose step size. We use several computational tools to investigate neural network parameters obtained by these two optimization methods, including kernel Principal Component Analysis and a novel, general-purpose algorithm for finding low-height paths between pairs of points on loss or energy landscapes, FourierPathFinder. We find that the choice of the optimizer profoundly affects the nature of the resulting solutions. SGD solutions tend to be separated by lower barriers than quasi-Newton solutions, even if both sets of solutions are regularized by early stopping to ensure adequate performance on test data. When allowed to fit extensively on the training data, quasi-Newton solutions occupy deeper minima on the loss landscapes that are not reached by SGD. These solutions are less generalizable to the test data however. Overall, SGD explores smooth basins of attraction, while quasi-Newton optimization is capable of finding deeper, more isolated minima that are more spread out in the parameter space. Our findings help understand both the topography of the loss landscapes and the fundamental role of landscape exploration strategies in creating robust, transferrable neural network models.

Figures & tables

Figure 1: Generation and visualization of optimized NN parameter sets. (a) A subset of input training/test data. A single $28\times 28$ MNIST image [ 6 ] is used as NN input. (b) Representative NN architecture, with an input layer, two hidden layers, and an output layer. (c) A conceptual sketch of the corresponding NN loss landscape, with two local minima (red dots) located in a shallow valley. The basins of attraction of the two minima are separated by a relatively low barrier.

Figure 2: LSTM loss curves . Representative LSTM training (a) and test (b) loss curves ( $\langle l(x^{\text{train}},\omega)\rangle$ and $\langle l(x^{\text{test}},\omega)\rangle$ , respectively) as a function of the number of epochs. In both panels, dashed vertical lines mark the epochs where the loss curves of the same color reach their minima. The weight configurations $\omega^{\text{train}}_{\text{SGD}},\omega^{\text{test}}_{\text{SGD}}$ and $\omega^{\text{train}}_{\text{BFGS}},\omega^{\text{test}}_{\text{BFGS}}$ denote the sets of NN parameters found at these minima (optimized with SGD and L-BFGS-GSS, respectively).

NN	L-BFGS	SGD	L-BFGS	SGD	$N_{\text{prm}}$
	$\langle\overline{l(x^{\text{train}},\omega^{\text{train}})}\rangle$		$\langle\overline{l(x^{\text{test}},\omega^{\text{test}})}\rangle$
FCP	1.26e-08	6.21e-04	9.69e-02	8.25e-02	42200
LeNet	3.35e-08	1.76e-05	4.17e-02	3.89e-02	61706
Autoencoder	1.69e-02	4.57e-02	1.65e-02	4.52e-02	52224
LSTM	1.63e-06	1.35e-02	1.55e-01	1.28e-01	15456

Table 1: Average training and test loss. For each NN architecture, we list the training loss

\langle l(x^{\text{train}},\omega^{\text{train}})\rangle

and the test loss

\langle l(x^{\text{test}},\omega^{\text{test}})\rangle

averaged over

48

independent runs. Also listed are

N_{\text{prm}}

, the total number of NN fitting parameters (weights and biases) in each of the four architectures.

Figure 3: Low-loss paths on a 2D landscape. (a) Two-dimensional loss landscape composed of two positive and two negative Gaussian peaks: $f(x,y)=-\sum_{i=1}^{2}\exp[-3|\mathbf{r}-\mathbf{c}_{i}|^{2}]+\sum_{j=1}^{2}\exp[-15|\mathbf{r}-\mathbf{d}_{j}|^{2}]+C$ , where $\mathbf{r}=(x,y)$ , $\mathbf{c}_{1}=(-0.5,-0.5)$ , $\mathbf{c}_{2}=(0.5,0.0)$ , $\mathbf{d}_{1}=(-0.2,-0.4)$ , $\mathbf{d}_{2}=(0.0,0.3)$ , and $C=1.019$ . Four representative paths connecting two landscape minima: $\mathbf{w}_{1}=(-0.62,-0.54)$ and $\mathbf{w}_{2}=(0.49,-0.02)$ are shown: a linear interpolation path (dashed blue line) and three FourierPathFinder optimized paths ( $\lambda=10$ , solid blue curve; $\lambda=100$ , solid orange curve; $\lambda=1000$ , solid green curve). Dots indicate function values at discrete time steps $t_{m}\in[0,1]$ along the path: $f(x(t_{m}),y(t_{m}))$ , $m=1\dots M$ ( $M=100$ ). (b) Loss values $f(x(t),y(t))$ as a function of the curve parameter $t$ along the four paths in panel (a): the linear interpolation path (dashed blue curve) and three FourierPathFinder paths (solid curves with the colors matching the paths in panel (a)). Path heights $H_{i}$ (Eq. ( 5 )) are labeled with black dots, with $H_{0}=1.102$ (straight line), $H_{1}=0.646$ (optimized path, $\lambda=10$ ), $H_{2}=0.640$ (optimized path, $\lambda=100$ ), and $H_{3}=0.652$ (optimized path, $\lambda=1000$ ).

Figure 4: Distribution of barrier heights along optimized paths on the training landscape. Shown are distributions of the FourierPathFinder path heights (Eq. ( 5 )) for FCP (a), LeNet (b), Autoencoder (c), and LSTM (d). Histograms in each panel show heights of $300$ low-loss paths connecting randomly chosen pairs of optimized parameter vectors in $\{\omega^{\text{train},i}_{\text{BFGS}}\}_{i=1}^{48}$ (navy blue), $\{\omega^{\text{test},i}_{\text{BFGS}}\}_{i=1}^{48}$ (light blue), $\{\omega^{\text{train},i}_{\text{SGD}}\}_{i=1}^{48}$ (light red), and $\{\omega^{\text{test},i}_{\text{SGD}}\}_{i=1}^{48}$ (gold). The paths are computed on the training landscape, $\langle l(x^{\text{train}},\omega)\rangle$ .

Figure 5: Distributions of SGD test and BFGS training weight vector lengths. Shown are the histograms of $L_{2}$ distances between individual weight vectors $\omega_{i}$ and the common origin $\bar{\omega}$ , $|\omega^{i}-\bar{\omega}|$ . Distributions of the BFGS training ( $\{\omega^{\text{train},i}_{\text{BFGS}}\}_{i=1}^{48}$ ) and SGD test ( $\{\omega^{\text{test},i}_{\text{SGD}}\}_{i=1}^{48}$ ) weight vector lengths are plotted in blue and light red, respectively, for FCP (a), LeNet (b), Autoencoder (c), and LSTM (d). Dotted vertical lines indicate the positions of $|\bar{\omega}_{\text{BFGS}}-\bar{\omega}|=|\bar{\omega}_{\text{SGD}}-\bar{\omega}|$ .

Figure 6: kPCA projections of SGD test and BFGS training weight vectors. Shown are the first two principal components, $\text{PC}_{1}$ and $\text{PC}_{2}$ , obtained by kPCA with the RBF kernel (Methods). The projections are applied to $\{\omega^{\text{train},i}_{\text{BFGS}}\}_{i=1}^{48}$ (blue points) and $\{\omega^{\text{test},i}_{\text{SGD}}\}_{i=1}^{48}$ (red points) sets of optimized weight vectors, for FCP (a), LeNet (b), Autoencoder (c), and LSTM (d).

Figure S1 : Additional examples of loss curves. Same as Fig. 2 but for FCP (training: a, test: b), LeNet (training: c, test: d), and Autoencoder (training: e, test: f) NN architectures.

Figure S2 : Distribution of barrier heights along optimized paths on the test landscape. Shown are distributions of the FourierPathFinder path heights (Eq. ( 5 )) for FCP (a), LeNet (b), Autoencoder (c), and LSTM (d). Histograms in each panel show heights of $300$ low-loss paths connecting randomly chosen pairs of optimized parameter vectors in $\{\omega^{\text{test},i}_{\text{BFGS}}\}_{i=1}^{48}$ (light blue) and $\{\omega^{\text{test},i}_{\text{SGD}}\}_{i=1}^{48}$ (gold). The paths are computed on the test landscape, $\langle l(x^{\text{test}},\omega)\rangle$ .

Figure S3 : Distributions of SGD and BFGS test weight vector lengths. Shown are the histograms of $L_{2}$ distances between individual weight vectors $\omega_{i}$ and the common origin $\bar{\omega}$ , $|\omega^{i}-\bar{\omega}|$ . Distributions of the BFGS ( $\{\omega^{\text{test},i}_{\text{BFGS}}\}_{i=1}^{48}$ ) and SGD ( $\{\omega^{\text{test},i}_{\text{SGD}}\}_{i=1}^{48}$ ) test weight vector lengths are plotted in blue and light red, respectively, for FCP (a), LeNet (b), Autoencoder (c), and LSTM (d). Dotted vertical lines indicate the positions of $|\bar{\omega}_{\text{BFGS}}-\bar{\omega}|=|\bar{\omega}_{\text{SGD}}-\bar{\omega}|$ .

Figure S4 : kPCA projections of SGD and BFGS test weight vectors. Shown are the first two principal components,

\text{PC}_{1}

and

\text{PC}_{2}

, obtained by kPCA with the RBF kernel (Methods). The projections are applied to

\{\omega^{\text{test},i}_{\text{BFGS}}\}_{i=1}^{48}

(blue points) and

\{\omega^{\text{test},i}_{\text{SGD}}\}_{i=1}^{48}

(red points) sets of optimized weight vectors, for FCP (a), LeNet (b), Autoencoder (c), and LSTM (d).

	Training weights				Test weights
NN	SGD		L-BFGS-GSS		SGD		L-BFGS-GSS
	$\mu$	$\sigma$	$\mu$	$\sigma$	$\mu$	$\sigma$	$\mu$	$\sigma$
FCP	-1.44e-03	1.39e-01	-2.47e-03	2.34e-01	-3.17e-03	1.03e-01	-4.11e-03	1.22e-01
LeNet	-3.57e-03	8.77e-02	1.04e-03	1.08e-01	-2.83e-03	7.06e-02	2.33e-04	8.00e-02
Autoencoder	-7.06e-03	8.54e-02	-3.65e-02	1.60e-01	-7.07e-03	8.54e-02	-3.65e-02	1.61e-01
LSTM	1.35e-02	3.24e-01	1.06e-02	7.33e-01	1.16e-02	2.59e-01	2.92e-03	4.76e-01

Table S1 : Statistics of training and test weight distributions. Means (

\mu

) and standard deviations (

\sigma

) of

W^{\text{train}}_{\text{SGD}}

W^{\text{train}}_{\text{BFGS}}

W^{\text{test}}_{\text{SGD}}

W^{\text{test}}_{\text{BFGS}}

– combined vectors of optimized weights and biases for each NN architecture.

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 327 papers

Aug 28, 2026

From the Loss Landscape to Diverse Feature Learning in Neural Networks

Over the course of the last decade, neural networks have grown from an academic curiosity to moving the markets of nations. Despite this explosion in both research and deployment, relatively little is understood about how they achieve the solutions they do. This is both scientifically relevant, and pressing for society. When neural networks make decisions across self-driving, construction, law, hiring and health, there have been and will continue to be unintended consequences. However, attempting to generalize the failures of the largest and most important production systems makes for a very difficult task. Yet signs of these failures exist at all scales of neural networks, so we should be able to study a much more tractable setting. All neural networks must undergo an optimization process, called training, to be useful. To a great degree, understanding neural networks is understanding their optimization: through what process and exposure to which data did they arrive at their results. Yet our knowledge on this topic as a field is quite imprecise. In particular, a curious phenomenon called mode connectivity, the ability to connect neural networks in the loss surface, defies explanation entirely. This dissertation elucidates, explains and exploits this special structure in the loss landscape...

David Aram Yunis

TOYOTA TECHNOLOGICAL INSTITUTE AT CHICAGO Chicago, Illinois

Loss Landscape Mode Connectivity Neural Network Optimization Neural Network Training Dynamics Representation Learning

Ask PDF

Aug 25, 2026

Across the Loss Landscape with Progressive Growth

Deep neural networks generalize well despite their highly nonconvex, overparameterized loss landscapes, a phenomenon often associated with the geometry of the minima found by stochastic optimization. We study how incremental grow-and-optimize strategies bias training toward flatter regions by viewing growth as progressive constraint relaxation. Starting from a low-dimensional submodel, we iteratively expand the trainable parameters by unlocking nested random subspaces while freezing the orthogonal complement at the network initialization, re-optimizing after each expansion until the full architecture is reached. Under standard local regularity conditions around non-degenerate minima, we prove that local sublevel sets are well approximated by ellipsoids and that basin accessibility under frozen constraints can be characterized by an explicit effective curvature in the frozen directions. This leads to an explanation of the bias: progressive growth increases the relative weight of wide basins and suppresses sharp ones through a volume effect induced by the frozen constraints. We empirically validate these predictions in controlled toy landscapes and in a realistic ResNet/CIFAR-100 setting and confirm that although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection. The code is available at https://github.com/p0lcAi/Across-the-Loss-Landscape.

Paul Caillon, Christophe Cerisara, Alexandre Allauzen

MILES Team, LAMSADE, Université Paris Dauphine-PSL · LORIA CNRS, Campus Scientifique, rue du Jardin Botanique, 54500 Vandoeuvre-les-Nancy · ESPCI-PSL

Deep Learning Optimization Loss Landscape Neural Network Growth Neural Network Training Dynamics Neural Network Generalization Flat Minima

Ask PDF

May 18, 2026

Tight Complexity of Muon under Heavy-Tailed Noise

A growing lesson from neural network optimization is that optimizer design should respect how the model is parametrized. The layerwise input-output structure of neural networks motivates Muon and Scion, whose updates also support hyperparameter transfer. At the same time, stochastic gradient noise in deep learning is often far from sub-Gaussian and may exhibit heavy tails. These observations have shaped recent algorithmic principles for training neural networks, yet their joint theoretical consequences are underexplored. In particular, it is unclear what dimension dependence is unavoidable for gradient-based methods when the problem class is defined using an input-output norm and the stochastic gradient noise is heavy-tailed, and if higher-order smoothness can accelerate optimization. We study these questions through nonconvex smooth stochastic optimization over

\mathbb{R}^{m\times n}

equipped with general norms and under

p^\mathrm{th}

-moment heavy-tailed noise, where the goal is to achieve an

ε

-stationary point in the dual norm. Our first contribution is a dimension-dependent lower bound: when

\max\{m,n\}

is large enough, any method using only the stochastic gradients requires

Ω(\min\{m, n\}ε^{-\frac{3p-2}{p-1}})

oracle calls for the problem class defined by the spectral norm, which is a common input-output matrix norm. We prove that a batched Muon method with the spectral norm can achieve the matching upper bound of

O(\min\{m, n\}ε^{-\frac{3p-2}{p-1}})

. To exploit higher-order smoothness, we propose a transported Muon method and improve the bound to

O(\min\{m, n\}ε^{-\frac{5p-3}{2p-2}})

when the norm is spectral and the Hessian is Lipschitz. Finally, we incorporate practical heuristics into our transported method and evaluate it across multiple architectures and model sizes, showing its compatibility with neural network training.

Jiayu Zhang, Tianyi Lin

Department of Industrial Engineering and Operations Research, Columbia University

Stochastic Optimization Neural Network Optimization Nonconvex Stochastic Optimization Heavy-Tailed Noise Stochastic Optimization Convergence Heavy-Tailed Stochastic Optimization

Ask PDF

Abstract

Neural Network Optimization

Explore similar work

From the Loss Landscape to Diverse Feature Learning in Neural Networks

Across the Loss Landscape with Progressive Growth

Tight Complexity of Muon under Heavy-Tailed Noise