Across the Loss Landscape with Progressive Growth

Paul Caillon; Christophe Cerisara; Alexandre Allauzen

cs.LGAug 25, 2026

Across the Loss Landscape with Progressive Growth

Authors: Paul Caillon, Christophe Cerisara, Alexandre Allauzen

Organizations: MILES Team, LAMSADE, Université Paris Dauphine-PSL · LORIA CNRS, Campus Scientifique, rue du Jardin Botanique, 54500 Vandoeuvre-les-Nancy · ESPCI-PSL

Abstract

Deep neural networks generalize well despite their highly nonconvex, overparameterized loss landscapes, a phenomenon often associated with the geometry of the minima found by stochastic optimization. We study how incremental grow-and-optimize strategies bias training toward flatter regions by viewing growth as progressive constraint relaxation. Starting from a low-dimensional submodel, we iteratively expand the trainable parameters by unlocking nested random subspaces while freezing the orthogonal complement at the network initialization, re-optimizing after each expansion until the full architecture is reached. Under standard local regularity conditions around non-degenerate minima, we prove that local sublevel sets are well approximated by ellipsoids and that basin accessibility under frozen constraints can be characterized by an explicit effective curvature in the frozen directions. This leads to an explanation of the bias: progressive growth increases the relative weight of wide basins and suppresses sharp ones through a volume effect induced by the frozen constraints. We empirically validate these predictions in controlled toy landscapes and in a realistic ResNet/CIFAR-100 setting and confirm that although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection. The code is available at https://github.com/p0lcAi/Across-the-Loss-Landscape.

Figures & tables

Panel of Figure 1 — Figure 1: Toy validation of the growth bias under controlled basin geometry. Top row: multiplicity regime. Bottom row: energy–entropy trade-off regime. Left column: probability of selecting one of the top- $5$ flattest basins as a function of the number of growth stages. Right column: expected curvature of the selected basin, measured by $\log\det(H)$ . In the multiplicity regime, progressive growth increasingly selects flatter basins. In the trade-off regime, this trend reverses when sharp basins are closer to initialization, illustrating the energy–entropy competition predicted by the theory.

Panel of Figure 2 — Figure 2: Predictive performance and patience sensitivity under progressive growth. Left: Final test accuracy as a function of the number of growth stages. Increasing the number of stages does not improve performance over the full-model baseline and generally reduces final test accuracy. Right: Patience ablation for the aggressive $S=10$ training-loss schedule. Patience has only a mild effect on final accuracy, but changes the mean transition barrier more noticeably, indicating that it mainly affects stage-to-stage geometric stability.

Method	Trigger	$S$	Test acc. (%) $\uparrow$	Barrier ( $\times 10^{-2}$ ) $\downarrow$	Retention $\uparrow$	$\lambda_{\max}^{\text{new}}/\lambda_{\max}^{\text{act}}\downarrow$
Full ResNet-18		1	$76.25\pm 0.23$	–	–	–
Growth	val_acc	3	$74.99\pm 0.48$	$3.37\pm 3.44$	$0.833\pm 0.289$	$0.313\pm 0.056$
Growth	val_acc	5	$74.94\pm 0.70$	$3.82\pm 5.93$	$0.917\pm 0.144$	$0.201\pm 0.014$
Growth	val_acc	10	$73.94\pm 0.86$	$2.67\pm 4.26$	$0.963\pm 0.064$	$0.0896\pm 0.0058$
Growth	train_loss	3	$75.54\pm 0.07$	$0.00\pm 0.00$	$1.000\pm 0.000$	$0.305\pm 0.015$
Growth	train_loss	5	$74.40\pm 0.58$	$12.47\pm 4.16$	$0.750\pm 0.000$	$0.174\pm 0.014$

Table 1: ResNet-18 on CIFAR-100. Progressive growth does not improve final test accuracy over the full-model baseline, but it induces clear geometric effects. The full-model baseline uses the exact same ResNet-18 architecture and training budget, but without any freezing or growth: all parameters are active from initialization. We report mean

\pm

standard deviation over 3 seeds. Barrier values are reported in units of

10^{-2}

. For moderate schedules, consecutive stages are often strongly retained and barriers remain small. Across all schedules, the newly released directions are substantially flatter than the already active ones, as shown by the ratio

\lambda_{\max}^{\text{new}}/\lambda_{\max}^{\text{act}}<1

Figure 3: Analysis of transition quality in the ResNet experiments. Each point corresponds to a single stage transition. Left: interpolation barrier versus absolute endpoint gap. Right: leakage versus absolute endpoint gap. In both cases, more disruptive transitions are associated with larger endpoint changes after expansion, showing that the proposed transition-level metrics are predictive of how strongly a growth step perturbs the current solution.

Figure 4: Transition geometry under progressive growth in ResNet-18 on CIFAR-100. (a) Mean interpolation barrier between consecutive stage solutions. (b) Mean retention across stage transitions. (c) Ratio between the top Hessian eigenvalue on the newly released subspace and that on the already active subspace. Validation-accuracy-based growth yields consistently mild transitions with low barriers, while training-loss-based growth becomes less stable as the number of stages increases. Across all schedules, the newly released directions are substantially flatter than the already active ones, as shown by

\lambda_{\max}^{\mathrm{new}}/\lambda_{\max}^{\mathrm{act}}\ll 1

Method	Test acc. (%) $\uparrow$	Barrier $(\times 10^{-2})$ $\downarrow$	Retention $\uparrow$	Leakage $\downarrow$	$\lambda_{\max}^{\mathrm{new}}/\lambda_{\max}^{\mathrm{act}}$ $\downarrow$
Full ResNet-18	$76.25\pm 0.23$	–	–	–	–
Fixed subspace, $S=5$	$3.13\pm 0.77$	–	–	–	–
One-shot unfreeze, $S=5$	$75.84\pm 0.33$	$0.17\pm 0.29$	$1.0000\pm 0.0000$	$0.0000\pm 0.0000$	$0.7913\pm 0.0076$

Table 2: Internal controls on ResNet-18/CIFAR-100. The full baseline uses the same architecture, optimizer, data split, augmentation, epoch budget, and learning-rate schedule, but all parameters are active from initialization. Fixed-subspace training keeps the initial trainable subspace fixed throughout optimization. One-shot unfreezing starts from the same initial constrained submodel as progressive growth and follows the same validation-accuracy trigger, but releases all remaining parameters in a single transition instead of unlocking them stage by stage. These controls separate the effect of delayed capacity release from gradual constraint relaxation.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Method	Test acc. (%) $\uparrow$	Barrier $(\times 10^{-2})$ $\downarrow$	Retention $\uparrow$	Leakage $\downarrow$	$\lambda_{\max}^{\mathrm{new}}/\lambda_{\max}^{\mathrm{act}}$ $\downarrow$
Full MLP	$91.58\pm 0.16$	–	–	–	–
Fixed subspace, $S=5$	$91.50\pm 0.37$	–	–	–	–
One-shot unfreeze, $S=5$	$91.72\pm 0.14$	$0.00\pm 0.00$	$1.0000\pm 0.0000$	$0.0000\pm 0.0000$	$0.5733\pm 0.1552$
Progressive growth, $S=5$	$91.60\pm 0.12$	$0.02\pm 0.03$	$1.0000\pm 0.0000$	$0.0000\pm 0.0000$	$0.1691\pm 0.0938$
Progressive growth, $S=10$	$91.53\pm 0.14$	$0.00\pm 0.00$	$1.0000\pm 0.0000$	$0.0000\pm 0.0000$	$0.1142\pm 0.0329$

Appendix

Table 3: Controlled MLP experiment on a two-moons classification task. All methods achieve comparable test accuracy and nearly zero interpolation barriers, reflecting the simplicity of the task. Nevertheless, progressive growth releases directions with substantially lower restricted curvature than one-shot unfreezing, suggesting that the curvature effect is not specific to convolutional residual architectures.

Explore similar work

Oct 8, 2026cs.LG

Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking

Flat loss landscapes have long been linked to better generalization in neural networks. However, its role as a causal mechanism for generalization is less established. Grokking provides an unique testbed to understand this distinction: models are prone to fit observed data using non-generalizing structure and remain in that regime for prolonged periods, transitioning to generalization only under particular training conditions. In this work, we study whether flat loss landscapes can act as a driving mechanism in this transition. While recent work has argued for flatness as a necessary geometric condition for this transition, we find that biasing training toward flatter solutions using sharpness-aware minimization (SAM) is insufficient to reliably induce this transition, despite producing flatter solutions. However, when SAM is paired with mechanisms that drive generalization such as weight decay, an interesting property emerges: SAM can accelerate the transition to generalizing solutions by up to 4x at the epoch-level. We theoretically untangle this relationship between SAM and weight decay using a minimal interpolating two-layer ReLU model with both memorizing and generalizing solutions. We show that even in this simple setup, flatness alone cannot distinguish a memorizing solution from a generalizing one, while weight decay favors generalizing solutions. However, under a local stability analysis, there exists a window where a memorizing interpolant is locally stable under gradient descent but unstable under SAM in the low-norm regime, which can explain SAM's ability to accelerate this transition. Overall, our results provide a more interpretable account of the role of flatness in driving generalization, especially in settings where models are vulnerable to minimizing loss through learning non-generalizing structure.

Mohnish Harwani

Purdue University

Neural Network Generalization Flat Minima Grokking Sharpness-Aware Minimization

Ask PDF

May 28, 2025cs.LG

Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization

The optimization of neural networks under weight decay remains poorly understood from a theoretical standpoint. While weight decay is standard practice in modern training procedures, most theoretical analyses focus on unregularized settings. In this work, we investigate the loss landscape of the

\ell_2

-regularized training loss for two-layer ReLU networks. We show that the landscape becomes benign -- i.e., free of spurious local minima -- under large overparametrization, specifically when the network width

m

satisfies

m \gtrsim \min(n^d, 2^n)

, where

n

is the number of data points and

d

the input dimension. More precisely in this regime, almost all constant activation regions contain a global minimum and no spurious local minima. We further show that this level of overparametrization is not only sufficient but also necessary via the example of orthogonal data. Finally, we demonstrate that such loss landscape results primarily hold relevance in the large initialization regime. In contrast, for small initializations -- corresponding to the feature learning regime -- optimization can still converge to spurious local minima, despite the global benignity of the landscape.

Etienne Boursier, Matthew Bowditch, Matthias Englert +1

INRIA LMO, Université Paris-Saclay Orsay, France · Mathematics Institute University of Warwick Coventry, UK · Department of Computer Science University of Warwick Coventry, UK

Weight Decay Neural Network Initialization Neural Network Optimization Global Optimization Loss Landscape Geometry

Ask PDF

Apr 28, 2026cs.LG

The Role of Symmetry in Optimizing Overparameterized Networks

Overparameterization is central to the success of deep learning, yet the mechanisms by which it improves optimization remain incompletely understood. We analyze weight-space symmetries in neural networks and show that overparameterization introduces additional symmetries that benefit optimization in two distinct ways. First, we prove that these symmetries act as a form of diagonal preconditioning on the Hessian, enabling the existence of better-conditioned minima within each equivalence class of functionally identical solutions. Second, we show that overparameterization increases the probability mass of global minima near typical initializations, making these favourable solutions more reachable. These results offer a potential link between loss landscape geometry and simplicity bias. Empirically, we observe wider networks have lower top eigenvalues, smaller condition numbers and faster convergence, matching our analysis. Our analysis provides a unified framework for understanding overparameterization and width growth as a geometric transformation of the loss landscape.

Kusha Sareen, Mohammad Pedramfar, Sékou-Oumar Kaba +2

McGill University · Mila - Quebec Artificial Intelligence Institute

Neural Network Optimization Weight-Space Analysis Loss Landscape Geometry

Ask PDF