Weight Decay

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 6Week of Sep 21

Latest papers 46

All topics

Learning the identity: a case study of how SGD selects among functional decompositions
Sep 30, 2026Andy Arditi, Weian Xie, David Bau +1Stochastic Gradient Descent Anisotropic Loss Landscapes
Northeastern University · Massachusetts Institute of Technology
The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation
Sep 30, 2026S. Aaron McClendon, Jorge Gallego-Feliciano, Antonios SaravanosWeight Decay Adam
Aimpoint Digital Labs, Atlanta, GA, USA · New York University, New York, NY, USA
Imprint Reader: From Weight-Update Readout to Behavioral Intervention
Sep 28, 2026Guanxu Chen, Qihao Lin, Jing ShaoLatent Feature Interventions Readout
Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University
Prediction with Expert Advice: Anytime Regret with Many Experts Matches the Fixed-Time Constant
Sep 23, 2026Yang Cai, Vineet Gupta, Yanchen Jiang +4\Widetilde{\Mathcal{O}}(\Sqrt{T})$ Regret Regret
Google Research · Yale University · Google DeepMind
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
Sep 22, 2026Lenz Pracher, Pascal de Jong, Oskar Lieshaus +2Neural Tangent Kernel Feature Learning
Arnold-Sommerfeld-Center for Theoretical Physics, Ludwig-Maximilians-Universität München, Munich, Germany · Department of Applied Physics, Standford University, Standford, CA, USA · Department of Mathematics, University of Cambridge, Cambridge, UK
Double descent is the principle of least action
Sep 16, 2026Congzhou M ShaDescent Weight Decay
Penn Medicine Doylestown Hospital, 595 W State St, Doylestown, PA 18901, United States
Symmetry without a manifold: intrinsic dimension on orbits
Sep 15, 2026Chon-Fai Kam, Miloud Bessafi, Frédéric CadetIntrinsic Dimensionality Scaling Laws
Dipartimento di Fisica e Chimica, Universit`a degli Studi di Palermo, via Archirafi 36, I-90123 Palermo, Italy · EnergyLab, Universit´e de La R´eunion, F-97715 Saint-Denis, France · Universit´e Paris Cit´e and Universit´e de La R´eunion, BIGR, INSERM UMR S1134, F-75014 Paris, France +1
Weighted Empirical Risk Minimization for Machine Learning under Long-Range Dependence: Exact Pathwise Rates and Learning-Error Geometry
Sep 9, 2026Elina MoldavskayaEmpirical Risk Minimization Weight Decay
Technion–Israel Institute of Technology, Israel
When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay
Sep 8, 2026Hasan Amin, Wei-Kai Chang, Rajiv KhannaWeight Decay Batch
Purdue University
A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
Sep 7, 2026Yuqing Wang, Ioannis G. Kevrekidis, Mikhail BelkinTwo-Layer Neural Networks Generalization Bounds
Johns Hopkins University · University of California San Diego
Two Dimensions Govern Agnostic Multiclass Transductive Learning
Aug 26, 2026Pahan DewasurendraTransductive Learning Multiclass Classification
Johns Hopkins University
Distributed Optimization with Streaming Data: A Temporal Weighting Perspective
Aug 10, 2026Muhammad Faraz Ul Abrar, Nicolò Michelusi, Erik G. LarssonDecentralized Optimization Weight Decay
School of Electrical, Computer and Energy Engineering, Arizona State University · Department of Electrical Engineering (ISY), Linköping University, 58183 Linköping, Sweden
Grokking on the Weight-Decay Clock: A Rate Hierarchy from Softly Broken Symmetries
Jul 27, 2026Taeyoung KimGrokking Weight Decay
Center for AI and Natural Sciences Korea Institute for Advanced Study Seoul 02455, Republic of Korea
Scale Weight Decay and Train Better
Jul 26, 2026Anuj ApteWeight Decay Batch
Global Technology Applied Research, JPMorganChase, New York, NY 10001
Searching the Space of Feed-Forward Neural-Network Weight-Update Rules with Fixed Depth Symbolic Regression
Jul 23, 2026Charles Brum, Edward FinkelsteinSymbolic Regression Neural Network
San Diego, California, USA · UCI, USA
Natural Invariant Measures for Chaotic Game Dynamics: Finding Order in Chaos
Jul 23, 2026Jakub Bielawski, Thiparat Chotibut, Fryderyk Falniowski +2Chaos Probability Measures
Department of Mathematics, Krakow University of Economics, Rakowicka 27, 31-510 Kraków, Poland · Chula Intelligent and Complex Systems Lab, Department of Physics, Faculty of Science, Chulalongkorn University,Bangkok 10330, Thailand · Department of Mathematical Sciences, Indiana University Indianapolis,402 N. Blackford Street, Indianapolis, Indiana 46202, USA +1
Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay
Jul 23, 2026Xiaolong Li, Zhangchen Zhou, Zhi-Qin John XuWeight Decay Anisotropic Loss Landscapes
Institute of Natural Sciences, School of Mathematical Sciences, Shanghai Jiao Tong University.
Neural Collapse Is Forbidden: Information Floors in Language Models
Jul 10, 2026Bruno AbrahaoNeural Collapse Language Modeling
NYU Shanghai · Leonard N. Stern School of Business, New York University
Disco-LoRA: Disentangled Composition of Content, Style, and Motion for Multi-concept Video Customization
Jun 25, 2026Xuancheng Xu, Gengyun Jia, Bing-Kun BaoDuet Motion
Nanjing University of Posts and Telecommunications · Hefei University of Technology · Peng Cheng Laboratory
What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy
Jun 16, 2026Truong Xuan KhanhGrokking Weight Decay
H&K Research Studio, Clevix LLC · Hanoi, Vietnam
Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization
Jun 15, 2026Kaiyue Wen, Xingyu Dang, Kaifeng Lyu +2Adaptive Optimizers Weight Decay
†Stanford University. · ‡Princeton University. · §Tsinghua University.
The Weight Norm Sets the Grokking Timescale: A Causal Delay Law
Jun 11, 2026Truong Xuan Khanh, Doan Hoang Viet, Luu Duc Trung +1Grokking Weight Decay
H&K Research Studio / Clevix LLC, Hanoi, Vietnam · Bac A Bank, Hanoi · Banking Academy of Vietnam
Last-Iterate Convergence of Optimistic Multiplicative Weight Update
Jun 10, 2026Francesco OrabonaConvex Optimization Primal-Dual Methods
King Abdullah University of Science and Technology (KAUST) Thuwal, 23955-6900, Kingdom of Saudi Arabia
Fixed-Parameter Tractability of Private Synthetic Data Generation
Jun 9, 2026Badih Ghazi, Cristóbal Guzmán, Pritish Kamath +3Synthetic Data Distributional Fidelity
Google Deepmind · Institute for Mathematical and Computational Engineering, Faculty of Mathematics and School of Engineering, Pontificia Universidad Cat´olica de Chile · Google Research
Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
Jun 4, 2026Hu Tan, Kuo Gai, Shihua ZhangDeep Network Rectified Linear Unit
1State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · 3Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China +1
Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View
Jun 3, 2026Mingyu LiLow-Rank Structure Weight Decay
Beijing Normal University
Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies
May 28, 2026Kajetan Schweighofer, Conor F. Hayes, Roberto Dailey +2Large Language Model Fine-Tuning Catastrophic Forgetting
Cognizant AI Lab · UT Austin & Cognizant AI Lab
Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate
May 24, 2026Huangyu Xu, Jingqin Yang, Qianqian Xu +1Sparsity Regularization
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China · IIIS, Tsinghua University, Beijing, China +3
Feature Learning in Wide Neural Networks under $μ$ P: Identifiability and Sparse-Dictionary Decomposition of the Mean-Field Limit
May 23, 2026Akmal XodarevTwo-Layer Neural Networks Feature Learning
Independent Researcher · Tashkent, Uzbekistan
Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics
May 19, 2026Lucky VermaWeight Decay Transformer Attention
Independent Researcher
Does Weight Decay Enhance Training Stability?
May 15, 2026Marius Saether, Amir Kolic, Tomaso Poggio +1Weight Decay Phase Transitions
NTNU, MIT · MIT
When and Why is Optimistic Multiplicative Weights Slow? The Geometry of Energy Dissipation
May 13, 2026John Lazarsfeld, Anas Barakat, Georgios Piliouras +2Imperfect-Information Games Convergence
SUTD · Google DeepMind · NUS CQT +2
OUI as a Structural Observable: Towards an Activation-Centric View of Neural Network Training
May 12, 2026Alberto Fernández-Hernández, Jose I. Mestre, Cristian Pérez-Corral +3Deep Network Neural Network
Universitat Politècnica de València, Valencia, Spain · Universitat Jaume I, Castelló de la Plana, Spain · Openchip & Software Technologies S.L., Spain
Neural Weight Norm = Kolmogorov Complexity
May 11, 2026Tiberiu MusatWeight Decay Model Weights
ETH Zürich
Nearly-Optimal Algorithm for Adversarial Kernelized Bandits
May 11, 2026Shogo IwazakiBandits Bounded Adversary
LY Corporation Tokyo, Japan
OUIDecay: Adaptive Layer-wise Weight Decay for CNNs Using Online Activation Patterns
May 11, 2026Alberto Fernández-Hernández, Jose I. Mestre, Cristian Pérez-Corral +3Weight Decay Convolutional Neural Networks
Universitat Politècnica de València, Valencia, Spain · Universitat Jaume I, Castelló de la Plana, Spain · Openchip & Software Technologies S.L., Spain
Decentralized Time-Varying Optimization for Streaming Data via Temporal Weighting
May 7, 2026Muhammad Faraz Ul Abrar, Nicolò Michelusi, Erik G. LarssonDecentralized Optimization Weight Decay
School of Electrical, Computer and Energy Engineering, Arizona State University · Department of Electrical Engineering (ISY), Linköping University
Weight-Decay Turns Transformer Loss Landscapes Villani: Functional-Analytic Foundations for Optimization and Generalization
May 7, 2026Abhijit Das, Sayantan DuttaAnisotropic Loss Landscapes Weight Decay
GE HealthCare, Bangalore 560066, Karnataka, India
Critical Windows of Complexity Control: When Transformers Decide to Reason or Memorize
May 6, 2026Sarwan AliTransformer Architectures Weight Decay
Columbia University, Irving Medical Center, NY, USA
TsallisPGD: Adaptive Gradient Weighting for Adversarial Attacks on Semantic Segmentation
May 5, 2026Alexander Matyasko, Xin Lou, Indriyati Atmosukarto +1Efficient Semantic Segmentation Semantic Segmentation
Singapore Institute of Technology, Singapore
Prescriptive Scaling Laws for Data Constrained Training
May 2, 2026Justin Lovelace, Christian Belardi, Srivatsa Kundurthy +2Scaling Laws Weight Decay
Department of Computer Science Cornell University Ithaca, NY
Learning to Forget: Continual Learning with Adaptive Weight Decay
Apr 29, 2026Aditya A. Ramesh, Alex Lewandowski, Jürgen SchmidhuberWeight Decay Continual Learning
The Swiss AI Lab, IDSIA USI-SUPSI, Lugano, Switzerland · University of Alberta, Alberta Machine Intelligence Institute, Edmonton, Canada · Center of Excellence for Generative AI, KAUST, Thuwal, Saudi Arabia
Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuning
Apr 21, 2026Xin Ning, Qiankun Li, Xiaolong Huang +5Neural Network Optimization Parameter-Efficient Fine-Tuning Methods
Institute of Semiconductors, Chinese Academy of Sciences, Beijing, 10083, China · College of Computing and Data Science (CCDS), Nanyang Technological University, 639798, Singapore · Mila-Quebec AI Institute, Montreal, QC H2S 3H1, Canada +3
Uniform Approximation of Functions with Asymmetric Growth and Decay by Deep Weighted Polynomials
Jun 26, 2025Kingsley Yeon, Steven B. DamelinApproximation Mathematical Finance
Department of Statistics and CCAM, University of Chicago, Chicago, 60637, IL, USA · Department of Mathematics, ZBMATH-OPEN, Hermann-von-Helmholtz-Platz 1, Eggenstein-Leopoldshafen, 76344, Germany
Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization
May 28, 2025Etienne Boursier, Matthew Bowditch, Matthias Englert +1Anisotropic Loss Landscapes Overparameterization
INRIA LMO, Université Paris-Saclay Orsay, France · Mathematics Institute University of Warwick Coventry, UK · Department of Computer Science University of Warwick Coventry, UK
Stability and Convergence of Optimistic Exponential Weights with Asymmetric Step Sizes in Bimatrix Games
Date pendingHédi Hadiji, Sarah SachsImperfect-Information Games Equilibrium
Laboratoire des Signaux et Systèmes, CentraleSupélec, Paris, France · School of Mathematics, University of Bristol, Bristol, United Kingdom

Sep 30, 2026, cs.LGJ/K move · Enter open · S save

Learning the identity: a case study of how SGD selects among functional decompositions

Andy Arditi, Weian Xie, David Bau, Liu Ziyin

Northeastern University · Massachusetts Institute of Technology

Stochastic Gradient Descent Anisotropic Loss Landscapes Weight Decay Residual Networks Decomposition Case Study

Open paper AskPDF arXiv

One might think that learning the identity function with a deep linear residual network is trivial - the path along residual connections already implements the identity, and so the network need only drive its weights to zero. However, this zero-weight solution is just one point on an entire manifold of population-loss minimizers, each corresponding to a different decomposition of the identity across the network's layers. Although the population loss does not distinguish among these solutions, stochastic gradient descent (SGD) reproducibly favors particular ones. For instance, under anisotropic label noise, the learned layers exhibit a noise-dependent spectrum; even with weight decay, SGD does not generally recover the zero-weight solution. Changing only the parametrization, while leaving the set of realizable functions unchanged, yields different behavior: factoring each weight matrix as a product of two matrices causes the weights to collapse to zero, even without explicit weight decay. While perhaps mysterious and unintuitive at first, these phenomena can be understood through the lens of entropic loss, which augments the population loss with a term proportional to the expected squared norm of the minibatch gradient (Ziyin et al., 2025). On the identity manifold, the population loss is constant, while the entropic term distinguishes among these decompositions. We characterize its minimizers analytically and use them to derive predictions for the structure of solutions favored by SGD. Networks trained with SGD closely match these predictions. Overall, the identity learning task studied here serves as a clean and simple case study of how the lens of entropic loss can clarify why SGD favors particular decompositions of the same input-output function.

Momentum

Related topics

Latest papers 46