Loss Landscape

Latest papers

22 papers2 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

Rethinking Activation Functions: Reducing Bias Can Improve Learning
Oct 10, 2026Zirui Peng, Heyi Zheng, Ziming LiuLoss Landscape Neural Network Activation Functions
Department of Physics, Peking University · Department of Mathematics, Peking University · College of AI, Tsinghua University +1
Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking
Oct 8, 2026Mohnish HarwaniNeural Network Generalization Flat Minima
Purdue University
A Flatness-Generalization Relation in the Teacher-Student Tree-Committee Machine
Sep 25, 2026Brandon Livio Annesi, Davide Straziota, Enrico Maria MalatestaFlat Minima Neural Network Generalization
Department of Quantitative Life Sciences, ICTP Trieste, Italy
Towards Understanding Momentum Acceleration in River-Valley Loss Landscape
Sep 25, 2026Miao Lu, Zeyu Bian, Kaiyue Wen +4Momentum Methods Gradient Descent Dynamics
Stanford University · University of California, San Diego · University of Chicago +2
Population loss in shallow ReLU networks: Bias & families of critical points
Sep 25, 2026Michael FieldShallow Neural Networks ReLU Neural Networks
Benign Loss Landscapes Can Coexist with Worst-Case Hardness
Sep 11, 2026Zach Furman, Stephan Wäldchen, Yangda Bei +1Tensor Networks Statistical Learning Theory
University of Melbourne · Iliad
From the Loss Landscape to Diverse Feature Learning in Neural Networks
Aug 28, 2026David Aram YunisLoss Landscape Mode Connectivity
TOYOTA TECHNOLOGICAL INSTITUTE AT CHICAGO Chicago, Illinois
Across the Loss Landscape with Progressive Growth
Aug 25, 2026Paul Caillon, Christophe Cerisara, Alexandre AllauzenDeep Learning Optimization Loss Landscape
MILES Team, LAMSADE, Université Paris Dauphine-PSL · LORIA CNRS, Campus Scientifique, rue du Jardin Botanique, 54500 Vandoeuvre-les-Nancy · ESPCI-PSL
Dataset Complexity Shapes Finite-Distance Loss Geometry in Neural Networks
Aug 23, 2026Jaeyong Bae, Hawoong JeongLoss Landscape Loss Landscape Geometry
Department of Physics, Korea Advanced Institute of Science and Technology, Daejeon, Korea · Center of Complex Systems, Korea Advanced Institute of Science and Technology, Daejeon, Korea
Loss Landscape Features That Make Adam Stall: Definitions, Estimators, and the Preconditioned Hessian View
Aug 23, 2026Rodion PodorozhnyLoss Landscape Adaptive Gradient Methods
Texas State University
Understanding Machine Unlearning Through the Lens of Mode Connectivity
Jul 27, 2026Jiali Cheng, Hadi AmiriLoss Landscape Mode Connectivity
University of Massachusetts Lowell
Loss Landscape Topology Reveals Why Simple Baselines are Competitive at 3D Point Cloud Segmentation Under Class Imbalance
Jul 23, 2026Antonis Savva, Christos Kyrkou, Theocharis TheocharidesClass-Imbalanced Learning Loss Landscape
KIOS Research and Innovation Center of Excellence, University of Cyprus, 1 Panepistimiou Avenue, 2109 Aglantzia, Nicosia · Department of Electrical and Computer Engineering, University of Cyprus, 1 Panepistimiou Avenue, 2109 Aglantzia, Nicosia
Loss Landscape Diagnosis for Gradient-Based Gray-Scott System Inversion: Disentangling the Roles of PINN Components
Jun 9, 2026Yan YangLoss Landscape Dynamical Systems
Independent Researcher.
Unveiling Multi-regime Patterns in SciML: Distinct Failure Modes and Regime-specific Optimization
May 27, 2026Yuxin Wang, Yuanzhe Hu, Xiaokun Zhong +7Loss Landscape Neural Network Optimization
Dartmouth College · University of California, San Diego · University of California, Berkeley +3
Beyond Perplexity: A Geometric and Spectral Study of Low-Rank Pre-Training
May 13, 2026Namrata Shivagunde, Vijeta Deshpande, Sherin Muckatira +1Language Model Pretraining Loss Landscape
University of Massachusetts Lowell
Curvature-Aligned Probing for Local Loss-Landscape Stabilization
Apr 16, 2026Nikita Kiselev, Andrey GrabovoyNeural Network Optimization Neural Network Training Dynamics
Department of Intelligent Systems Moscow Institute of Physics and Technology
Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks
Apr 10, 2026Jie Huang, Bruno Loureiro, Stefano Sarao MannelliOverparameterization Loss Landscape
Physics, Chalmers University of Technology and University of Gothenburg · Ecole Normale Superieure, PSL & CNRS · Data Science and AI, Computer Science and Engineering, Chalmers University of Technology and University of Gothenburg +1
How and why does deep ensemble coupled with transfer learning increase performance in bipolar disorder and schizophrenia classification?
Apr 2, 2026Sara Petiton, Antoine Grigis, Benoit Dufumier +1Loss Landscape Deep Ensembles
NeuroSpin/GAIA · NeuroSpin, CEA Saclay, Université Paris-Saclay, France · EPFL, NeuroSpin/GAIA +1
On the Loss Landscape Geometry of Regularized Deep Matrix Factorization: Uniqueness and Sharpness
Mar 28, 2026Anil Kamber, Rahul ParhiLoss Landscape Loss Landscape Geometry
UC San Diego
A Loss Landscape Visualization Framework for Interpreting Reinforcement Learning: An ADHDP Case Study
Mar 15, 2026Jingyi Liu, Jian Guo, Eberhard GillLoss Landscape Interpretability in RL
Faculty of Aerospace Engineering, Delft University of Technology, Kluyverweg 1, Delft, 2629 HS, The Netherlands
Topological Exploration of High-Dimensional Empirical Risk Landscapes: general approach, and applications to phase retrieval
Feb 19, 2026Antoine Maillard, Tony Bonnaire, Giulio BiroliLoss Landscape Loss Landscape Geometry
INRIA Paris & DI ENS, PSL University, Paris, France. · Université Paris-Saclay, CNRS, Institut d’Astrophysique Spatiale, 91405 Orsay, France · Laboratoire de Physique de l’École normale supérieure, ENS, Université PSL, CNRS, Sorbonne Université, Université Paris Cité, F-75005 Paris, France
Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization
May 28, 2025Etienne Boursier, Matthew Bowditch, Matthias Englert +1Loss Landscape Overparameterized Neural Network
INRIA LMO, Université Paris-Saclay Orsay, France · Mathematics Institute University of Warwick Coventry, UK · Department of Computer Science University of Warwick Coventry, UK

Oct 10, 2026J/K move · Enter open · S save

Rethinking Activation Functions: Reducing Bias Can Improve Learning

Zirui Peng, Heyi Zheng, Ziming Liu

Department of Physics, Peking University · Department of Mathematics, Peking University · College of AI, Tsinghua University+1

Loss Landscape Neural Network Activation Functions Neural Network Training Dynamics

Open paper AskPDF arXiv

Many widely used activation functions, including ReLU, GELU, Swish, and Sigmoid, produce predominantly positive outputs, resulting in a nonzero post-activation mean that we term activation bias. Despite its ubiquity, the effect of activation bias on learning dynamics remains poorly understood. Using controlled models, we show that activation bias accelerates initial loss reduction but is accompanied by overshooting, non-smooth trajectories, and a steeper loss landscape. This early advantage also coincides with the model's use of task-relevant information encoded in the activation bias component. In contrast, reducing activation bias slows initial learning but ultimately yields a lower final loss. Experiments on larger vision and language models exhibit a similar pattern: smaller activation bias is associated with better late-stage performance. Thus, this seemingly minor property can substantially redirect learning and warrants explicit consideration in future activation function design.