Loss Landscape

Follow this topic Ask about this topic

Recent momentum

+0%

5 papers in the last 28 days · 0.1% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

2 new papers

A weekly snapshot of new work published in Loss Landscape.

Period ending 2026-09-14

1 new paper

A weekly snapshot of new work published in Loss Landscape.

Period ending 2026-09-07

1 new paper

A weekly snapshot of new work published in Loss Landscape.

53 papers

Latest in Loss Landscape

Browse all topics →

Beyond Quadratic Loss: The Stability Phase Diagram of Adam
Sep 16, 2026Gaoxiang Tang, Huanran Chen, Ziming LiuAdam Loss Landscape
Benign Loss Landscapes Can Coexist with Worst-Case Hardness
Sep 14, 2026Zach Furman, Stephan Wäldchen, Yangda Bei +1Loss Landscape Tensor Networks
Teacher Geometry Shapes Learnability in Teacher-Student Networks
Sep 10, 2026Kai J. Sandbrink, Flavio Martinelli, Alexander van Meegen +2Teacher-Student Log-Probability Gap Two-Layer Neural Networks
Post-Training Science for Supervised Fine-Tuning
Sep 1, 2026Charles O'Neill, Mudith Jayasekara, Harry PartridgeSupervised Fine-Tuning LLM Post-Training Methods
Mode Connectivity Beyond Classifiers: Evidence from Generative and Contrastive Models
Aug 31, 2026Chengzheyi Yao, Yongzhao Zhang, Yongding TianLoss Landscape Contrastive Learning
Kähler landscapes for complex neural network descents and guarantees including a search and destroy of the Calabi-Yau manifold
Aug 20, 2026Andrew GracykLoss Landscape Manifolds
Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks
Aug 6, 2026Björn Ladewig, Ibrahim Talha Ersoy, Karoline WiesnerFeature Learning Loss Landscape
Tunneling the Loss Landscape: Bypassing Memorization with Monte Carlo Parameter Swapping
Aug 3, 2026Lai Shun Chan, Xiaotian Zhang, Yue Shang +2Grokking Loss Landscape
The Grokked Illusion: True Equilibrium Mitigates Catastrophic Forgetting
Jul 31, 2026Xiaotian Zhang, Lai Shun Chan, Yue Shang +2Catastrophic Forgetting Loss Landscape
On the robustness of noisy solutions in non-convex neural networks
Jul 29, 2026Enrico M. Malatesta, Alessandra Passalacqua, Riccardo ZecchinaLoss Landscape Parallel Liquid-Inspired Temporal Relaxation
A Defense of the Quadratic Model
Jul 23, 2026Alexandru Meterez, Pranav Ajit Nair, Depen Morwani +3Large Models Condition Number
Loss Landscape Topology Reveals Why Simple Baselines are Competitive at 3D Point Cloud Segmentation Under Class Imbalance
Jul 23, 2026Antonis Savva, Christos Kyrkou, Theocharis Theocharides3D Semantic Segmentation Class Imbalance
Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay
Jul 23, 2026Xiaolong Li, Zhangchen Zhou, Zhi-Qin John XuLoss Landscape Model Weights
Effects of width-dependent model hyperparameters and $\ell_2$ -regularization on the loss landscape of two-layer ReLU networks
Jul 18, 2026Haruka Eshima, Makoto YamadaRectified Linear Unit Networks Loss Landscape
Overcoming Fourier Locking in Quantum Data Re-uploading Classifiers via Spectral Homotopy
Jul 13, 2026Spencer TopelQuantum Circuit Simulation Fourier
Avoiding unsafe sets when training with Langevin Dynamics
Jul 8, 2026Adam M. ObermanLoss Landscape Langevin Dynamics
Level-Crossing Density as a Mesh-Free High-Frequency Auxiliary Loss for Implicit Neural Representations
Jul 7, 2026Gunner Levi HoweNeural Fields Discretization
Noise-Debiased Thermodynamic Variance for Local Learning Coefficient Probes
Jun 21, 2026Yingjia CaiLoss Landscape Singular Learning Theory
Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts
Jun 17, 2026Tho Tran Huu, Huu-Tuan Nguyen, Thien-Hai Nguyen +4Mixture-of-Experts Models Smoothing
Loss Landscape Poisoning: Targeted Extraction of Unseen Training Data from LLMs
Jun 15, 2026Md Abdullah Al Mamun, Ngoc Phu Doan, Pedram Zaree +2Language Model Evasion Attacks Differential Privacy
Loss Landscape Diagnosis for Gradient-Based Gray-Scott System Inversion: Disentangling the Roles of PINN Components
Jun 9, 2026Yan YangConventional Physics-Informed Neural Networks Loss Landscape
Lost in the Non-convex Loss Landscape: How to Fine-tune the Large Time Series Model?
Jun 7, 2026Xu Zhang, Peang Wang, Wei WangFinetuning Time Series
Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability
Jun 3, 2026Vincent Bürgin, Daniel Herbst, Ya-Wei Eileen Lin +1Two-Layer Neural Networks Symmetry
Flatness and Generalization: Learning Multi-Index Models with Homogeneous Neural Networks
Jun 3, 2026Harsh Vardhan, Hossein Taheri, Arya MazumdarTwo-Layer Neural Networks Neural Network
A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networks
Jun 3, 2026Tian Ding, Dawei Li, Ruoyu SunTwo-Layer Neural Networks Loss Landscape
Quadratic integrate-and-fire neurons exhibit less fragmented loss landscapes and outperform leaky integrate-and-fire neurons in spike-based gradient descent
Jun 2, 2026Carlo Wenig, Raoul-Martin Memmesheimer, Christian KlosSpiking Neural Networks Loss Landscape
On the Construction and Implications of Low-Loss Valleys in LoRA-based Bayesian Inference
May 28, 2026Daniel Dold, Emanuel Sommer, Julius Kobialka +2Bayesian Loss Landscape
Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization
May 28, 2026Ruoran Xu, Borong She, Xiaobo Jin +1Adam Neural Network Optimization
Convex Basins in Single-Index Model Loss Landscapes: Applications to Robust Recovery under Strong Adversarial Corruption
May 28, 2026Santanu Das, Sagnik Chatterjee, Jatin BatraLoss Landscape Gaussian Mixture Models
Unveiling Multi-regime Patterns in SciML: Distinct Failure Modes and Regime-specific Optimization
May 27, 2026Yuxin Wang, Yuanzhe Hu, Xiaokun Zhong +7Low-Data Regimes Machine Learning
How the Optimizer Shapes Learned Solutions in Equivariant Neural Networks
May 26, 2026Teodor-Mihai Stupariu, Andrei ManolacheEquivariant Neural Networks Neural Network Optimization
Model Merging on Loss Landscape: A Geometry Perspective
May 26, 2026Juanwu Lu, Anand Bhaskar, Brian Axelrod +2Continual Model Merging Model Merging
Towards the Connection between Activation Sparsity and Flat Minima
May 25, 2026Ze Peng, Jian Zhang, Lei Qi +2Sparse Expert Activation Modern Deep Networks
Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent
May 22, 2026Anherutowa CalvoCondition Number Curvature
A lift for input-convex neural net training
May 22, 2026Ali SiahkoohiLoss Landscape Neural Network Optimization
Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics
May 21, 2026Igor Ignashin, Anna Radovskaya, Andrew Semenov +7Stochastic Gradient Descent Stochastic Differential Equations
LLM Pretraining Shapes a Generalizable Manifold: Insights into Cross-Modal Transfer to Time Series
May 19, 2026Alexis Roger, Prateek Humane, Zhenghan Tai +4Generative Pretrained Transformers Time Series
Landscape-Awareness for Geometric View Diffusion Model
May 19, 2026Yan-Ting Chen, Hao-Wei Chen, Tsu-Ching Hsiao +1Multi-View Diffusion Model Reward-Guided Diffusion
Angel or Demon: Investigating the Plasticity Interventions' Impact on Backdoor Threats in Deep Reinforcement Learning
May 14, 2026Oubo Ma, Ruixiao Lin, Yang Dai +4Backdoor Attacks Deep Reinforcement Learning
Beyond Perplexity: A Geometric and Spectral Study of Low-Rank Pre-Training
May 13, 2026Namrata Shivagunde, Vijeta Deshpande, Sherin Muckatira +1Low-Rank Factorization Language Model Perplexity
DP-KFC: Data-Free Preconditioning for Privacy-Preserving Deep Learning
May 13, 2026Marc Molina Van den Bosch, Riccardo Taiello, Albert Sund Aillet +3Differential Privacy Spectral Preconditioning
Finite Volume-Informed Neural Network Framework for 2D Shallow Water Equations: Rugged Loss Landscapes and the Importance of Data Guidance
May 9, 2026Xiaofeng LiuConventional Physics-Informed Neural Networks Shallow Water Equations
Susceptibilities and Patterning: A Primer on Linear Response in Bayesian Learning
May 8, 2026Chris Elliott, Daniel MurfetBayesian Loss Landscape
Flatness and Gradient Alignment Are Both Necessary: Spectral-Aware Gradient-Aligned Exploration for Multi-Distribution Learning
May 8, 2026Aristotelis Ballas, Christos DiouDistributional Learning Loss Landscape
Weight-Decay Turns Transformer Loss Landscapes Villani: Functional-Analytic Foundations for Optimization and Generalization
May 7, 2026Abhijit Das, Sayantan DuttaLoss Landscape Regularization Techniques
Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes
May 7, 2026Liu Hanqing, Jianjun Cao, Yuanze Li +1Loss Landscape Full-Precision Performance
A Theory of Saddle Escape in Deep Nonlinear Networks
May 2, 2026Divit Rawal, Michael R. DeWeeseDeep Network Loss Landscape
The Role of Symmetry in Optimizing Overparameterized Networks
Apr 28, 2026Kusha Sareen, Mohammad Pedramfar, Sékou-Oumar Kaba +2Over-Parameterized Neural Networks Loss Landscape
Curvature-Aligned Probing for Local Loss-Landscape Stabilization
Apr 16, 2026Nikita Kiselev, Andrey GrabovoyCurvature-Aware Quadratic Banks Loss Landscape
Do Flat Minima Improve Sparse Novel View Synthesis?
Nov 22, 2025Youngsik Yun, Dongjun Gu, Youngjung UhNovel View Synthesis Sharpness-Aware Minimization
Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU
Oct 29, 2025Jingzhou LiuTwo-Layer Neural Networks Rectified Linear Unit
Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization
May 28, 2025Etienne Boursier, Matthew Bowditch, Matthias Englert +1Loss Landscape Over-Parameterized Neural Networks
Bit-Flip Attacks on Vision-Language-Action Models: Action-Decoding Architecture Shapes the Vulnerability
Date pendingYudong Gao, Linghan Chen, Wenhan Wu +5Vision-Language-Action Models Fault Injection Experiments

Sep 16, 2026 · cs.LGJ/K move · Enter open · S save

Beyond Quadratic Loss: The Stability Phase Diagram of Adam

Gaoxiang Tang, Huanran Chen, Ziming Liu

Loss spikes are recurrent instabilities in neural-network training and can arise from multiple mechanisms. For Adam in particular, macroscopic loss spikes have been linked to optimizer dynamics, yet how its two momentum timescales govern them remains unclear. We investigate this dependence by mapping training dynamics across the

(β_1,β_2)

plane. Across a range of model--task settings, an approximately linear boundary,

1-β_2=C(1-β_1)

, separates spiky from non-spiky dynamics, whereas a one-dimensional quadratic loss produces approximately cubic slope. A one-dimensional superquadratic loss

L(x)\propto|x|^n

recovers the near-linear scaling and links the boundary coefficient to the effective loss exponent

n

. We further show that confident cross-entropy losses develop a core--wall landscape comprising a narrow quadratic core followed by a steep wall, which produces effective superquadratic behavior at the scale of an optimizer update. Together, these results connect Adam loss spikes to both the mismatch between momentum timescales and finite-scale superquadratic loss geometry beyond the Hessian.

Adam Loss Landscape Loss Function Phase Transitions Temporal Scales

Open paper Ask about it PDF