Weight Decay

Follow this topic Ask about this topic

Latest papers 22

4 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

All topics

Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights
Oct 8, 2026Dmitrii Andriianov, Andrey Veprikov, Aleksandr BeznosikovWeight Decay Nuclear Norm Minimization
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · SB AI Lab · Innopolis University
Weight Decay and Neuron Condensation: A Three-Stage Analysis of Two-Layer ReLU Networks
Oct 3, 2026Cheng Xu, Pengxiao Lin, Zhangchen Zhou +1Neural Network Training Dynamics Weight Decay
School of Mathematical Sciences, Shanghai Jiao Tong University · Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University
The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation
Sep 30, 2026S. Aaron McClendon, Jorge Gallego-Feliciano, Antonios SaravanosWeight Decay Attention Sinks
Aimpoint Digital Labs, Atlanta, GA, USA · New York University, New York, NY, USA
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
Sep 22, 2026Lenz Pracher, Pascal de Jong, Oskar Lieshaus +2Neural Network Generalization Weight Decay
Arnold-Sommerfeld-Center for Theoretical Physics, Ludwig-Maximilians-Universität München, Munich, Germany · Department of Applied Physics, Standford University, Standford, CA, USA · Department of Mathematics, University of Cambridge, Cambridge, UK
When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay
Sep 8, 2026Hasan Amin, Wei-Kai Chang, Rajiv KhannaLearning Rate Scheduling Weight Decay
Purdue University
Grokking on the Weight-Decay Clock: A Rate Hierarchy from Softly Broken Symmetries
Jul 27, 2026Taeyoung KimNeural Network Generalization Weight Decay
Center for AI and Natural Sciences Korea Institute for Advanced Study Seoul 02455, Republic of Korea
Scale Weight Decay and Train Better
Jul 26, 2026Anuj ApteWeight Decay Neural Network Optimization
Global Technology Applied Research, JPMorganChase, New York, NY 10001
Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay
Jul 23, 2026Xiaolong Li, Zhangchen Zhou, Zhi-Qin John XuWeight Decay Neural Network Optimization
Institute of Natural Sciences, School of Mathematical Sciences, Shanghai Jiao Tong University.
Effects of width-dependent model hyperparameters and $\ell_2$ -regularization on the loss landscape of two-layer ReLU networks
Jul 18, 2026Haruka Eshima, Makoto YamadaReLU Neural Networks Weight Decay
Okinawa Institute of Science and Technology, Okinawa, Japan
Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization
Jun 15, 2026Kaiyue Wen, Xingyu Dang, Kaifeng Lyu +2Language Model Pretraining Weight Decay
†Stanford University. · ‡Princeton University. · §Tsinghua University.
Circuit Synchronization Precedes Generalization: A Causal Precursor to Grokking
Jun 11, 2026Achyuthan SivasankarWeight Decay Frequency-Domain Feature Learning
New York University
Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View
Jun 3, 2026Mingyu LiSpectral Regularization Weight Decay
Beijing Normal University
Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models
May 26, 2026Mingze Wang, Shuchen Zhu, Yuxin Fang +3Language Model Pretraining Weight Decay
ByteDance Seed · Peking University
Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics
May 19, 2026Lucky VermaWeight Decay Neural Network Memorization
Independent Researcher
Does Weight Decay Enhance Training Stability?
May 15, 2026Marius Saether, Amir Kolic, Tomaso Poggio +1Edge of Stability Deep Learning Optimization
NTNU, MIT · MIT
Neural Weight Norm = Kolmogorov Complexity
May 11, 2026Tiberiu MusatWeight Decay
ETH Zürich
OUIDecay: Adaptive Layer-wise Weight Decay for CNNs Using Online Activation Patterns
May 11, 2026Alberto Fernández-Hernández, Jose I. Mestre, Cristian Pérez-Corral +3Deep Learning Optimization Weight Decay
Universitat Politècnica de València, Valencia, Spain · Universitat Jaume I, Castelló de la Plana, Spain · Openchip & Software Technologies S.L., Spain
Weight-Decay Turns Transformer Loss Landscapes Villani: Functional-Analytic Foundations for Optimization and Generalization
May 7, 2026Abhijit Das, Sayantan DuttaWeight Decay Neural Network Optimization
GE HealthCare, Bangalore 560066, Karnataka, India
Critical Windows of Complexity Control: When Transformers Decide to Reason or Memorize
May 6, 2026Sarwan AliDeep Learning Optimization Weight Decay
Columbia University, Irving Medical Center, NY, USA
Prescriptive Scaling Laws for Data Constrained Training
May 2, 2026Justin Lovelace, Christian Belardi, Srivatsa Kundurthy +2Weight Decay Language Model Scaling Laws
Department of Computer Science Cornell University Ithaca, NY
Learning to Forget: Continual Learning with Adaptive Weight Decay
Apr 29, 2026Aditya A. Ramesh, Alex Lewandowski, Jürgen SchmidhuberDeep Learning Optimization Weight Decay
The Swiss AI Lab, IDSIA USI-SUPSI, Lugano, Switzerland · University of Alberta, Alberta Machine Intelligence Institute, Edmonton, Canada · Center of Excellence for Generative AI, KAUST, Thuwal, Saudi Arabia
Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization
May 28, 2025Etienne Boursier, Matthew Bowditch, Matthias Englert +1Weight Decay Neural Network Initialization
INRIA LMO, Université Paris-Saclay Orsay, France · Mathematics Institute University of Warwick Coventry, UK · Department of Computer Science University of Warwick Coventry, UK

Oct 8, 2026, cs.LGJ/K move · Enter open · S save

Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights

Dmitrii Andriianov, Andrey Veprikov, Aleksandr Beznosikov

Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · SB AI Lab · Innopolis University

Weight Decay Nuclear Norm Minimization Low-Rank Compression LLM Compression

Open paper AskPDF arXiv

Standard weight decay treats each weight matrix as a vector and ignores its spectral structure. We introduce spectral weight decay, a post-step decoupled nuclear-norm update that applies additive rather than multiplicative spectral shrinkage. We connect the update to approximate proximal descent and show that its sensitivity to update order can exceed that of conventional

\ell_2

weight decay near rank deficiency. Across LLaMA models with

124

M to

500

M parameters, spectral weight decay lowers effective rank and improves SVD-LLM compression at matched validation loss. At

500

M and a

4\%

distortion budget, it reaches

1.89\times

compression and

1.18\times

GPU inference speedup, compared with

1.14\times

and

1.01\times

after standard weight decay. Under fixed-horizon training with

60\%

label noise, it also improves final mean clean-test accuracy over matched

\ell_2

regularization by up to

17.8

points on MNIST and

4.6

points across four BERT-base tasks. Code is available at https://github.com/brain-lab-research/SpectralWD.