Weight Decay

Follow this topic Ask about this topic

Weight decay penalizes or shrinks model parameters during training, often to reduce overfitting and shape the solutions a model learns. Common approaches include coupled L2 penalties, decoupled AdamW-style updates, scheduled or adaptive layer-wise decay, and spectral penalties that encourage low-rank weights. Theory and experiments examine its effects on loss landscapes, stability, compression, and delayed generalization in settings such as language-model training and modular-arithmetic grokking. A central tension is that decay can improve generalization or induce useful structure, yet its effect depends strongly on the optimizer, architecture, schedule, and scale-invariant parameterization, and excessive decay can destabilize training.

Latest papers

28 papers1 paper in the last four weeks, down 75% on the four weeks before. 0.0% of all new papers.

Tune the Baseline First: Most of Muon's Scaled Weight Decay Gain Is a Tuning Gap, but Its Small Real Advantage Grows With Model Size
Nov 2, 2026Esther XinWeight Decay Muon Optimizer
Independent Researcher
Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights
Oct 8, 2026Dmitrii Andriianov, Andrey Veprikov, Aleksandr BeznosikovWeight Decay Nuclear Norm Minimization
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · SB AI Lab · Innopolis University
Weight Decay and Neuron Condensation: A Three-Stage Analysis of Two-Layer ReLU Networks
Oct 3, 2026Cheng Xu, Pengxiao Lin, Zhangchen Zhou +1Neural Network Training Dynamics Weight Decay
School of Mathematical Sciences, Shanghai Jiao Tong University · Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University
The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation
Sep 30, 2026S. Aaron McClendon, Jorge Gallego-Feliciano, Antonios SaravanosWeight Decay Attention Sinks
Aimpoint Digital Labs, Atlanta, GA, USA · New York University, New York, NY, USA
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
Sep 22, 2026Lenz Pracher, Pascal de Jong, Oskar Lieshaus +2Neural Network Generalization Weight Decay
Arnold-Sommerfeld-Center for Theoretical Physics, Ludwig-Maximilians-Universität München, Munich, Germany · Department of Applied Physics, Standford University, Standford, CA, USA · Department of Mathematics, University of Cambridge, Cambridge, UK
When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay
Sep 8, 2026Hasan Amin, Wei-Kai Chang, Rajiv KhannaLearning Rate Scheduling Weight Decay
Purdue University
Grokking on the Weight-Decay Clock: A Rate Hierarchy from Softly Broken Symmetries
Jul 27, 2026Taeyoung KimNeural Network Generalization Weight Decay
Center for AI and Natural Sciences Korea Institute for Advanced Study Seoul 02455, Republic of Korea
Scale Weight Decay and Train Better
Jul 26, 2026Anuj ApteWeight Decay Neural Network Optimization
Global Technology Applied Research, JPMorganChase, New York, NY 10001
Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay
Jul 23, 2026Xiaolong Li, Zhangchen Zhou, Zhi-Qin John XuWeight Decay Neural Network Optimization
Institute of Natural Sciences, School of Mathematical Sciences, Shanghai Jiao Tong University.
Effects of width-dependent model hyperparameters and $\ell_2$ -regularization on the loss landscape of two-layer ReLU networks
Jul 18, 2026Haruka Eshima, Makoto YamadaReLU Neural Networks Weight Decay
Okinawa Institute of Science and Technology, Okinawa, Japan
Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization
Jun 15, 2026Kaiyue Wen, Xingyu Dang, Kaifeng Lyu +2Language Model Pretraining Weight Decay
Stanford University. · Princeton University. · Tsinghua University.
Learned Structure Shows Up Earlier in the Task's Own Basis: A Controlled Look at a Synchronisation Precursor in Grokking
Jun 11, 2026Achyuthan SivasankarWeight Decay Frequency-Domain Feature Learning
New York University
Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View
Jun 3, 2026Mingyu LiSpectral Regularization Weight Decay
Beijing Normal University
Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models
May 26, 2026Mingze Wang, Shuchen Zhu, Yuxin Fang +3Language Model Pretraining Weight Decay
ByteDance Seed · Peking University
Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics
May 19, 2026Lucky VermaWeight Decay Neural Network Memorization
Independent Researcher
Does Weight Decay Enhance Training Stability?
May 15, 2026Marius Saether, Amir Kolic, Tomaso Poggio +1Edge of Stability Deep Learning Optimization
NTNU, MIT · MIT
Neural Weight Norm = Kolmogorov Complexity
May 11, 2026Tiberiu MusatWeight Decay Kolmogorov Complexity
ETH Zürich
OUIDecay: Adaptive Layer-wise Weight Decay for CNNs Using Online Activation Patterns
May 11, 2026Alberto Fernández-Hernández, Jose I. Mestre, Cristian Pérez-Corral +3Deep Learning Optimization Weight Decay
Universitat Politècnica de València, Valencia, Spain · Universitat Jaume I, Castelló de la Plana, Spain · Openchip & Software Technologies S.L., Spain
Weight-Decay Turns Transformer Loss Landscapes Villani: Functional-Analytic Foundations for Optimization and Generalization
May 7, 2026Abhijit Das, Sayantan DuttaWeight Decay Neural Network Optimization
GE HealthCare, Bangalore 560066, Karnataka, India
Critical Windows of Complexity Control: When Transformers Decide to Reason or Memorize
May 6, 2026Sarwan AliDeep Learning Optimization Weight Decay
Columbia University, Irving Medical Center, NY, USA
Prescriptive Scaling Laws for Data Constrained Training
May 2, 2026Justin Lovelace, Christian Belardi, Srivatsa Kundurthy +2Weight Decay Language Model Scaling Laws
Department of Computer Science, Cornell University, Ithaca, NY
Learning to Forget: Continual Learning with Adaptive Weight Decay
Apr 29, 2026Aditya A. Ramesh, Alex Lewandowski, Jürgen SchmidhuberDeep Learning Optimization Weight Decay
The Swiss AI Lab, IDSIA USI-SUPSI, Lugano, Switzerland · University of Alberta, Alberta Machine Intelligence Institute, Edmonton, Canada · Center of Excellence for Generative AI, KAUST, Thuwal, Saudi Arabia
The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression
Apr 8, 2026Yongzhong XuNeural Network Training Dynamics Grokking
On the Loss Landscape Geometry of Regularized Deep Matrix Factorization: Uniqueness and Sharpness
Mar 28, 2026Anil Kamber, Rahul ParhiLoss Landscape Loss Landscape Geometry
UC San Diego
Stable and Steerable Sparse Autoencoders with Weight Regularization
Mar 4, 2026Piotr Jedryszek, Oliver M. CrookSparse Autoencoders LLM Interpretability
Department of Biology, University of Oxford, Oxford, UK · Kavli Institute for Nanoscience Discovery, University of Oxford, Oxford, UK · Department of Chemistry, University of Oxford, Oxford, UK
The Geometry of Multi-Task Grokking: Transverse Instability, Superposition, and Weight Decay Phase Structure
Feb 19, 2026Yongzhong XuGrokking Neural Network Training Dynamics
Optimizer choice matters for the emergence of Neural Collapse
Feb 18, 2026Jim Zhao, Tin Sum Cheng, Wojciech Masarczyk +1Neural Collapse Neural Network Training Dynamics
University of Basel · Warsaw University of Technology IDEAS Research Institute
Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization
May 28, 2025Etienne Boursier, Matthew Bowditch, Matthias Englert +1Loss Landscape Overparameterized Neural Network
INRIA LMO, Université Paris-Saclay Orsay, France · Mathematics Institute University of Warwick Coventry, UK · Department of Computer Science University of Warwick Coventry, UK

Nov 2, 2026J/K move · Enter open · S save

Tune the Baseline First: Most of Muon's Scaled Weight Decay Gain Is a Tuning Gap, but Its Small Real Advantage Grows With Model Size

Esther Xin

Independent Researcher

Weight Decay Muon Optimizer Hyperparameter Optimization

Open paper AskPDF arXiv

Scaled weight decay for Muon (Muon-SW) has been reported to train language models more efficiently than constant decoupled weight decay, but such comparisons usually reuse one learning-rate (LR) and weight-decay (WD) setting for both methods. We re-run the comparison under the MAI-Base-2 protocol: LR and WD are tuned jointly through their product for each method, on a source-scaled mixture-of-experts (MoE) scaling ladder (1.9M to 15M active parameters) that holds tokens per active parameter and data repetition fixed across sizes. We find: (1) Most of the gain is a tuning gap. At the shared setting Muon-SW wins by 0.035 bits per byte (BPB), a 1.52x compute saving; after both methods are tuned, the gap is 0.0026 BPB (95% CI [0.0018, 0.0034]), a 1.03x saving, so about 90% of the advantage disappears. (2) The small advantage that remains is real: at the smallest size, all four paired seeds favour Muon-SW at the tuned settings. (3) That advantage grows with model size: across an 8x range of active parameters, the tuned gap rises from 0.0036 to 0.0051 BPB while the untuned gap shrinks from 0.035 to 0.024, so the surviving share rises from about 10% to 21%. (4) The best weight decay moves with scale: the optimal LRWD product grows for both methods, faster for Muon-SW (ratio 1.40 to 1.91), and a law fitted on the two smallest sizes overshoots the third by 1.3-1.5x, costing up to 0.002 BPB. The whole study runs on one NVIDIA T4 GPU; we release the code and all run results.