Anisotropic Loss Landscapes

Follow this topic Ask about this topic

Momentum

11 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 54

All topics

Learning the identity: a case study of how SGD selects among functional decompositions
Sep 30, 2026Andy Arditi, Weian Xie, David Bau +1Stochastic Gradient Descent Anisotropic Loss Landscapes
Northeastern University · Massachusetts Institute of Technology
Awakening of the Buddha: Subspace Learning During Population-Loss Plateaus
Sep 30, 2026Akash KumarAnisotropic Loss Landscapes Singular Learning Theory
Department of Computer Science & Engineering University of California-San Diego
Normative Loss Landscape Navigation: A Trajectory-Based Approach to Mitigating Forgetting in Incremental Learning
Sep 28, 2026Isabelle Aguilar, Zayn Andre Zainal, Luis Fernando Herbozo Contreras +2Class-Incremental Learning Continual Learning
Quantization-Robust Unlearning through the Lens of Retain-Forget Loss Landscapes Interaction
Sep 23, 2026Jialu Wang, Jianing Deng, Shuqing Luo +6Large Language Model Unlearning Large Language Model Quantization
University of North Carolina at Chapel Hill · University of Pittsburgh · University of Arizona +1
Special Lagrangian cones in Deep Learning
Sep 17, 2026Tejas Kotwal, Govind MenonManifold Perspective Anisotropic Loss Landscapes
Institute for Advanced Study, Princeton
Beyond Quadratic Loss: The Stability Phase Diagram of Adam
Sep 16, 2026Gaoxiang Tang, Huanran Chen, Ziming LiuAdam Anisotropic Loss Landscapes
IIIS, Tsinghua University · College AI, Tsinghua University · Shanghai Qizhi Institute +1
SAM-on-the-Curve: Sharpness-Aware Mode Connectivity for Robust Weight-Space Interpolation
Sep 15, 2026Alejandro Calatrava, Xu Zhang, Ren WangSharpness-Aware Minimization Anisotropic Loss Landscapes
Benign Loss Landscapes Can Coexist with Worst-Case Hardness
Sep 14, 2026Zach Furman, Stephan Wäldchen, Yangda Bei +1Anisotropic Loss Landscapes Tensor Networks
University of Melbourne · Iliad
Teacher Geometry Shapes Learnability in Teacher-Student Networks
Sep 10, 2026Kai J. Sandbrink, Flavio Martinelli, Alexander van Meegen +2Two-Layer Neural Networks Teacher
Department of Experimental Psychology, University of Oxford · Department of Physics, RWTH Aachen
A Function-Space Approach to the Statistical Mechanics of Learning Dynamics
Sep 9, 2026Yizhou Zhang, Weichen Wu, Lun Du +1Learning Dynamics Anisotropic Loss Landscapes
Variational AI · The Voleon Group · Independent Researcher +1
Kolmogorov--Arnold stability for discontinuous functions
Sep 7, 2026Sviatoslav V. DzhenzherKolmogorov-Arnold Networks Anisotropic Loss Landscapes
Moscow Institute of Physics and Technology 141701, Institutskii lane, 9, Dolgoprudny, Russia.
Kähler landscapes for complex neural network descents and guarantees including a search and destroy of the Calabi-Yau manifold
Aug 20, 2026Andrew GracykAnisotropic Loss Landscapes Manifolds
Department of Mathematics Purdue University West Lafayette, IN 47907, USA
Predicting When Random Low-Dimensional Reparameterizations Train Neural Networks
Aug 12, 2026Andrew Cheng, Ali Eslamian, Jie Cheng +2Neural Network Representations Reparameterization
Department of Computer Science, Tsinghua University, Beijing, China · Department of Computer Science, The University of Manchester, Manchester, UK · Department of Computer Science, University of Kentucky, Lexington, KY, USA +3
Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks
Aug 6, 2026Björn Ladewig, Ibrahim Talha Ersoy, Karoline WiesnerFeature Learning Anisotropic Loss Landscapes
Institute of Physics and Astronomy, University of Potsdam
Statistical Mechanics of Learning on Product Wasserstein Manifolds
Aug 2, 2026Srinivasa Rao P Vangmayi P ReddyQuantum Learning Wasserstein Distance
Curlvee Technolabs, India; Former Scientist, C-DAC.
The Grokked Illusion: True Equilibrium Mitigates Catastrophic Forgetting
Jul 31, 2026Xiaotian Zhang, Lai Shun Chan, Yue Shang +2Catastrophic Forgetting Anisotropic Loss Landscapes
Department of Physics, City University of Hong Kong, Hong Kong, China · Department of Physics and Astronomy, University of Pennsylvania, Philadelphia, PA, USA · Innovation Campus Delaware, Air Liquide, Newark, DE, USA
A Defense of the Quadratic Model
Jul 23, 2026Alexandru Meterez, Pranav Ajit Nair, Depen Morwani +3Anisotropic Loss Landscapes Large Language Model Training
1Kempner Institute at Harvard University · †: Work done while at Harvard. · 2MIT
Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay
Jul 23, 2026Xiaolong Li, Zhangchen Zhou, Zhi-Qin John XuWeight Decay Anisotropic Loss Landscapes
Institute of Natural Sciences, School of Mathematical Sciences, Shanghai Jiao Tong University.
1-Lipschitz Neural Networks on Hadamard Manifolds
Jul 21, 2026Davide Murari, Marta Ghirardelli, Ben Adcock +3Lipschitz Continuity Anisotropic Loss Landscapes
Department of Applied Mathematics and Theoretical Physics, University of Cambridge · Department of Mathematical Sciences, Norwegian University of Science and Technology (NTNU) · Department of Mathematics, Simon Fraser University
Effects of width-dependent model hyperparameters and $\ell_2$ -regularization on the loss landscape of two-layer ReLU networks
Jul 18, 2026Haruka Eshima, Makoto YamadaRectified Linear Unit Networks Anisotropic Loss Landscapes
Okinawa Institute of Science and Technology, Okinawa, Japan
How the Hessian-Spectrum of Neural Networks Depends on Data
Jul 15, 2026Jasraj Singh, Enea Monzio Compagnoni, Antonio OrvietoHessian Two-Layer Neural Networks
ELLIS Institute, Tübingen, Germany · Universität Basel, Basel, Switzerland · MPI-IS, Tübingen, Germany
Overcoming Fourier Locking in Quantum Data Re-uploading Classifiers via Spectral Homotopy
Jul 13, 2026Spencer TopelAnisotropic Loss Landscapes Fourier
Moth, Brooklyn, New York
Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks
Jul 8, 2026Marcel Kühn, Bernd RosenowHessian Anisotropic Loss Landscapes
Institut für Theoretische Physik, Universität Leipzig, 04103 Leipzig, Germany · ScaDS.AI Dresden/Leipzig, Universität Leipzig, 04105 Leipzig, Germany
Avoiding unsafe sets when training with Langevin Dynamics
Jul 8, 2026Adam M. ObermanAnisotropic Loss Landscapes Langevin Dynamics
Department of Mathematics and Statistics, McGill University, Montreal, QC, Canada; Mila, Quebec AI Institute; and LawZero
Neural Network Quantization by Learning Low-Loss Subspaces
Jun 23, 2026Vladimir Protsenko, Mikhalina Kharkevich, Alexander Vashchilko +1Subspace Anisotropic Loss Landscapes
Huawei, Montreal, Canada
Noise-Debiased Thermodynamic Variance for Local Learning Coefficient Probes
Jun 21, 2026Yingjia CaiAnisotropic Loss Landscapes Singular Learning Theory
Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin
Jun 8, 2026Hanyang Li, Jianhao Ma, Ying CuiQuantization-Aware Training Post-Training Quantization
Department of IEOR, University of California, Berkeley · Department of Statistics and Data Science, University of Pennsylvania
Lost in the Non-convex Loss Landscape: How to Fine-tune the Large Time Series Model?
Jun 7, 2026Xu Zhang, Peang Wang, Wei WangModel Fine-Tuning Anisotropic Loss Landscapes
Shanghai Key Laboratory of Data Science College of Computer Science and Artificial Intelligence Fudan University, Shanghai 200082, China
Flatness and Generalization: Learning Multi-Index Models with Homogeneous Neural Networks
Jun 3, 2026Harsh Vardhan, Hossein Taheri, Arya MazumdarTwo-Layer Neural Networks Anisotropic Loss Landscapes
A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networks
Jun 3, 2026Tian Ding, Dawei Li, Ruoyu SunTwo-Layer Neural Networks Anisotropic Loss Landscapes
Shenzhen International Center of Industrial and Applied Mathematics, Shenzhen Research Institute of Big Data, China; Shenzhen Loop Area Institute, China; AutoKernel, China. · Shenzhen Loop Area Institute, China · AutoKernel, China +3
Quadratic integrate-and-fire neurons exhibit less fragmented loss landscapes and outperform leaky integrate-and-fire neurons in spike-based gradient descent
Jun 2, 2026Carlo Wenig, Raoul-Martin Memmesheimer, Christian KlosLfsr-Based Stochastic Leaky Integrate-And-Fire Neuron Neurons
University of Bonn · University of Tübingen · University of Illinois Urbana-Champaign
Exploiting weight-space symmetries for approximating curvature
May 30, 2026Artem Artemev, Rui Xia, Benjamin M. Boyd +4Local Curvature Hessian
MediaTek, Cambridge, UK · University of Cambridge, Cambridge, UK · Vector Institute, Toronto, Canada
On the Construction and Implications of Low-Loss Valleys in LoRA-based Bayesian Inference
May 28, 2026Daniel Dold, Emanuel Sommer, Julius Kobialka +2Parameter-Efficient Fine-Tuning Methods Anisotropic Loss Landscapes
HTWG Konstanz · LMU Munich · Munich Center for Machine Learning (MCML)
Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization
May 28, 2026Ruoran Xu, Borong She, Xiaobo Jin +1Stochastic Convex Optimization Adam
Xi’an Jiaotong-Liverpool University
Learning to Perturb Hidden Representations for Generalizable Deep Learning
May 28, 2026Hua LiAnisotropic Loss Landscapes Hidden States
Henan University
Convex Basins in Single-Index Model Loss Landscapes: Applications to Robust Recovery under Strong Adversarial Corruption
May 28, 2026Santanu Das, Sagnik Chatterjee, Jatin BatraAnisotropic Loss Landscapes Basin
School of Technology and Computer Science, Tata Institute of Fundamental Research, Mumbai, India.
Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent
May 22, 2026Anherutowa CalvoHessian Local Curvature
19D Labs
A lift for input-convex neural net training
May 22, 2026Ali SiahkoohiAnisotropic Loss Landscapes Model Weights
Department of Computer Science University of Central Florida
Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics
May 21, 2026Igor Ignashin, Anna Radovskaya, Andrew Semenov +7Stochastic Gradient Descent Stochastic Differential Equations
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · P.N. Lebedev Physical Institute of the Russian Academy of Sciences · Innopolis University
LLM Pretraining Shapes a Generalizable Manifold: Insights into Cross-Modal Transfer to Time Series
May 19, 2026Alexis Roger, Prateek Humane, Zhenghan Tai +4Large Language Model Pretraining Generative Pretrained Transformers
Université de Montréal Mila - Quebec AI Institute
Landscape-Awareness for Geometric View Diffusion Model
May 19, 2026Yan-Ting Chen, Hao-Wei Chen, Tsu-Ching Hsiao +1Sparse-View Anisotropic Loss Landscapes
Elsa Lab, National Taiwan University
Angel or Demon: Investigating the Plasticity Interventions' Impact on Backdoor Threats in Deep Reinforcement Learning
May 14, 2026Oubo Ma, Ruixiao Lin, Yang Dai +4Offline Reinforcement Learning Plasticity
1Zhejiang University · 2National University of Defense Technol · 3Xi’an Jiaotong University
Beyond Perplexity: A Geometric and Spectral Study of Low-Rank Pre-Training
May 13, 2026Namrata Shivagunde, Vijeta Deshpande, Sherin Muckatira +1Low-Rank Structure Language Model Perplexity
University of Massachusetts Lowell
DP-KFC: Data-Free Preconditioning for Privacy-Preserving Deep Learning
May 13, 2026Marc Molina Van den Bosch, Riccardo Taiello, Albert Sund Aillet +3Spectral Preconditioning Fisher Information Matrix
CERN, Geneva, Switzerland · BCN MedTech, Department of Engineering, Universitat Pompeu Fabra, Barcelona, Spain · ICREA, Barcelona, Spain
Finite Volume-Informed Neural Network Framework for 2D Shallow Water Equations: Rugged Loss Landscapes and the Importance of Data Guidance
May 9, 2026Xiaofeng LiuParametric Physics-Informed Neural Network Reynolds-Averaged Navier-Stokes
Department of Civil and Environmental Engineering, Pennsylvania State University, University Park, PA, USA · Institute for Computational and Data Sciences, Pennsylvania State University, University Park, PA, USA
Susceptibilities and Patterning: A Primer on Linear Response in Bayesian Learning
May 8, 2026Chris Elliott, Daniel MurfetBayesian Neural Networks Bayesian
Timaeus
Weight-Decay Turns Transformer Loss Landscapes Villani: Functional-Analytic Foundations for Optimization and Generalization
May 7, 2026Abhijit Das, Sayantan DuttaAnisotropic Loss Landscapes Weight Decay
GE HealthCare, Bangalore 560066, Karnataka, India
Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization
May 7, 2026Andy Zeyi Liu, Elliot Paquette, John SousModel Activations Large Language Model Training
Department of Applied Physics, Yale University, New Haven, CT 06511, United States · Department of Mathematics and Statistics, McGill University · Department of Applied Physics, Yale University, New Haven, Connecticut 06511, USA +1
Curvature-Aligned Probing for Local Loss-Landscape Stabilization
Apr 16, 2026Nikita Kiselev, Andrey GrabovoyAnisotropic Loss Landscapes Curvature-Aware Spectral Framework
Department of Intelligent Systems Moscow Institute of Physics and Technology
Sampling at intermediate temperatures is optimal for training large language models in protein structure prediction
Mar 31, 2026L. Ghiringhelli, A. Zambon, G. TianaProtein Structure Prediction Transformer Architectures
Department of Physics, via Celoria 16, 20133 Milano, Italy · Department of Physics and INFN, via Celoria 16, 20133 Milano, Italy
Do Flat Minima Improve Sparse Novel View Synthesis?
Nov 22, 2025Youngsik Yun, Dongjun Gu, Youngjung UhNovel View Synthesis Sparse-View
Yonsei University, Seoul, Korea
Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization
May 28, 2025Etienne Boursier, Matthew Bowditch, Matthias Englert +1Anisotropic Loss Landscapes Overparameterization
INRIA LMO, Université Paris-Saclay Orsay, France · Mathematics Institute University of Warwick Coventry, UK · Department of Computer Science University of Warwick Coventry, UK
AYLA: Architecting a loss landscape in shallow neural networks to accelerate feature recovery
Apr 2, 2025Behnam Gheshlaghi, Shahin AtakishiyevAnisotropic Loss Landscapes Feature Learning
Bit-Flip Attacks on Vision-Language-Action Models: Action-Decoding Architecture Shapes the Vulnerability
Date pendingYudong Gao, Linghan Chen, Wenhan Wu +5Real-World Cloud Fault Injection Dataset Anisotropic Loss Landscapes
The Hong Kong University of Science and Technology · Adelaide University · Wuhan University +3

Sep 30, 2026, cs.LGJ/K move · Enter open · S save

Learning the identity: a case study of how SGD selects among functional decompositions

Andy Arditi, Weian Xie, David Bau, Liu Ziyin

Northeastern University · Massachusetts Institute of Technology

Stochastic Gradient Descent Anisotropic Loss Landscapes Weight Decay Residual Networks Decomposition Case Study

Open paper AskPDF arXiv

One might think that learning the identity function with a deep linear residual network is trivial - the path along residual connections already implements the identity, and so the network need only drive its weights to zero. However, this zero-weight solution is just one point on an entire manifold of population-loss minimizers, each corresponding to a different decomposition of the identity across the network's layers. Although the population loss does not distinguish among these solutions, stochastic gradient descent (SGD) reproducibly favors particular ones. For instance, under anisotropic label noise, the learned layers exhibit a noise-dependent spectrum; even with weight decay, SGD does not generally recover the zero-weight solution. Changing only the parametrization, while leaving the set of realizable functions unchanged, yields different behavior: factoring each weight matrix as a product of two matrices causes the weights to collapse to zero, even without explicit weight decay. While perhaps mysterious and unintuitive at first, these phenomena can be understood through the lens of entropic loss, which augments the population loss with a term proportional to the expected squared norm of the minibatch gradient (Ziyin et al., 2025). On the identity manifold, the population loss is constant, while the entropic term distinguishes among these decompositions. We characterize its minimizers analytically and use them to derive predictions for the structure of solutions favored by SGD. Networks trained with SGD closely match these predictions. Overall, the identity learning task studied here serves as a clean and simple case study of how the lens of entropic loss can clarify why SGD favors particular decompositions of the same input-output function.

Momentum

Related topics

Latest papers 54