Loss Landscape Geometry

Follow this topic Ask about this topic

Loss landscape geometry studies the structure of objective functions, including minima, saddle points, curvature, symmetries, and the connectedness of low-loss regions. Researchers analyze these features with tools such as Hessian spectra, sharpness measures, landscape visualization, and theoretical models, then test how they affect gradient methods, stochastic optimization, and search algorithms. The work spans neural-network training and settings such as matrix factorization, forecasting, quantum optimization, and inverse problems. A central tension is that flatness or benign minima do not by themselves guarantee good generalization or easy optimization: outcomes also depend on parameterization, data, initialization, and optimizer dynamics.

Latest papers

34 papers5 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking
Feb 18, 2026Yongzhong XuGrokking Neural Network Training Dynamics
A Function-Centric Perspective on Flat and Sharp Minima
Oct 14, 2025Israel Mason-Williams, Gabryel Mason-Williams, Helen YannakoudakisFlat Minima Loss Landscape Geometry
Department of Informatics, UKRI Safe and Trusted AI, London, United Kingdom · Queen Mary University of London, London, United Kingdom · Department of Informatics, King’s College London, London, United Kingdom
Temporal horizons in forecasting: a performance-learnability trade-off
Jun 4, 2025Pau Vilimelis Aceituno, Jack William Miller, Noah Marti +2Long-Term Time Series Forecasting Time Series Forecasting
Institute of Neuroinformatics, ETH Zürich and University of Zürich, Winterhurerstrasse 190, Zürich 8057, Switzerland · School of Computing, National Australian University, 108 North Rd, Acton ACT 2601, Australia · Unit of Land Change Science, Swiss Federal Research Institute for Forest, Snow and Landscape Zürcherstrasse 111, Birmensdorf 8903, Switzerland
Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization
May 28, 2025Etienne Boursier, Matthew Bowditch, Matthias Englert +1Loss Landscape Overparameterized Neural Network
INRIA LMO, Université Paris-Saclay Orsay, France · Mathematics Institute University of Warwick Coventry, UK · Department of Computer Science University of Warwick Coventry, UK

Feb 18, 2026J/K move · Enter open · S save

Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking

Yongzhong Xu

Grokking Neural Network Training Dynamics Gradient Descent Dynamics Loss Landscape Geometry

Open paper AskPDF arXiv

Grokking -- the delayed transition from memorization to generalization in small algorithmic tasks -- remains poorly understood. We present a geometric analysis of optimization dynamics in transformers trained on modular arithmetic. PCA of attention weight trajectories reveals that training evolves predominantly within a low-dimensional execution subspace, with a single principal component capturing 68-83% of trajectory variance. To probe loss-landscape geometry, we measure commutator defects -- the non-commutativity of successive gradient steps -- and project them onto this learned subspace. We find that curvature grows sharply in directions orthogonal to the execution subspace while the trajectory remains largely confined to it. Importantly, curvature growth consistently precedes generalization across learning rates and hyperparameter regimes, with the lead time obeying a power law in the grokking timescale. Causal intervention experiments show that motion along the learned subspace is necessary for grokking, while artificially increasing curvature is insufficient. Together, these results support a geometric account in which grokking reflects escape from a metastable regime characterized by low-dimensional confinement and transverse curvature accumulation. All findings replicate across this learning-rate range, a qualitatively different slow regime (lr=5e-5, wd=0.1, 3 layers), and three random seeds, though alignment dynamics differ quantitatively between regimes. Causal intervention experiments establish that orthogonal gradient flow is necessary but not sufficient for grokking: suppressing it prevents generalization with a monotonic dose-response across four operations, while artificially boosting curvature defects has no effect.

Latest papers

Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking

A Function-Centric Perspective on Flat and Sharp Minima

Temporal horizons in forecasting: a performance-learnability trade-off

Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization