Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights

Dmitrii Andriianov; Andrey Veprikov; Aleksandr Beznosikov

cs.LGOct 8, 2026

Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights

Authors: Dmitrii Andriianov, Andrey Veprikov, Aleksandr Beznosikov

Organizations: Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · SB AI Lab · Innopolis University

Abstract

Standard weight decay treats each weight matrix as a vector and ignores its spectral structure. We introduce spectral weight decay, a post-step decoupled nuclear-norm update that applies additive rather than multiplicative spectral shrinkage. We connect the update to approximate proximal descent and show that its sensitivity to update order can exceed that of conventional $\ell_2$ weight decay near rank deficiency. Across LLaMA models with $124$ M to $500$ M parameters, spectral weight decay lowers effective rank and improves SVD-LLM compression at matched validation loss. At $500$ M and a $4\%$ distortion budget, it reaches $1.89\times$ compression and $1.18\times$ GPU inference speedup, compared with $1.14\times$ and $1.01\times$ after standard weight decay. Under fixed-horizon training with $60\%$ label noise, it also improves final mean clean-test accuracy over matched $\ell_2$ regularization by up to $17.8$ points on MNIST and $4.6$ points across four BERT-base tasks. Code is available at https://github.com/brain-lab-research/SpectralWD.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	$124$ M	$257$ M	$500$ M
Transformer blocks	$12$	$16$	$22$
Hidden dimension	$768$	$1{,}024$	$1{,}280$
Attention heads	$12$	$16$	$20$
Feed-forward dimension	$2{,}048$	$2{,}816$	$3{,}584$
Embedding vocabulary size	$50{,}304$	$50{,}304$	$50{,}304$
Training steps	$19{,}000$	$39{,}000$	$76{,}294$

Appendix

Table 1: LLaMA architectures and reference pretraining configurations. Token budgets are rounded.

Method	Compression	val loss	$\Delta$ val loss	ARC-E	HellaSwag	PIQA
baseline (uncompressed)	–	$3.1129$	–	$0.4947$	$0.3004$	$0.6088$
truncated SVD	$1.43\times$	$3.1488$	$+0.0359$	$0.4947$	$0.2999$	$0.6099$
SliceGPT	$1.22\times$	$4.6347$	$+1.5218$	$0.4140$	$0.2815$	$0.5680$
ASVD	$\mathbf{1.45\times}$	$3.1416$	$+0.0286$	$0.4877$	$0.2998$	$\mathbf{0.6121}$
SVD-LLM	$1.42\times$	$\mathbf{3.1325}$	$\mathbf{+0.0196}$	$\mathbf{0.4982}$	$0.2991$	$0.6088$
Dobi-SVD	$1.21\times$	$3.1336$	$+0.0207$	$0.4912$	$\mathbf{0.3005}$	$0.6110$

Appendix

Table 2: Five compression methods applied to the

124

M spectral-weight-decay model (

\lambda=1

). Lower losses are better. Higher compression rates and downstream accuracies are better. Bold marks the best compressed value in each column.

Method	$10\%$ noise	$25\%$ noise	$40\%$ noise	$60\%$ noise
No WD	$87.99\pm 0.93$	$79.52\pm 1.67$	$65.29\pm 3.27$	$45.71\pm 1.79$
$\ell_{2}$ WD	$86.89\pm 1.70$	$77.90\pm 2.00$	$67.12\pm 2.71$	$45.78\pm 1.29$
Spectral WD (post-step)	$87.99\pm 0.93$	$81.22\pm 2.36$	$75.01\pm 3.01$	$63.56\pm 1.04$
Spectral WD (pre-step)	$87.31\pm 1.13$	$81.63\pm 1.77$	$75.52\pm 2.35$	$61.63\pm 2.74$

Appendix

Table 3: MNIST MLP: final clean-test accuracy, mean

\pm

one sample standard deviation.

Method	$10\%$ noise	$25\%$ noise	$40\%$ noise	$60\%$ noise
No WD	$91.29\pm 0.64$	$82.42\pm 1.18$	$71.50\pm 1.13$	$51.65\pm 2.12$
$\ell_{2}$ WD	$91.07\pm 1.55$	$82.98\pm 0.54$	$72.69\pm 0.84$	$51.20\pm 2.78$
Spectral WD (post-step)	$94.06\pm 1.03$	$89.96\pm 0.97$	$81.38\pm 3.15$	$65.60\pm 3.70$
Spectral WD (pre-step)	$94.83\pm 0.62$	$90.00\pm 1.17$	$80.02\pm 2.03$	$67.33\pm 1.52$

Appendix

Table 4: MNIST GRU: final clean-test accuracy, mean

\pm

one sample standard deviation.

Dataset	Noise	No WD	$\ell_{2}$ -SP	Spectral WD (post-step)	Spectral WD (pre-step)
AG News	$10\%$	$88.49\pm 0.10$	$89.63\pm 0.03$	$90.57\pm 0.10$	$90.55\pm 0.11$
	$25\%$	$79.23\pm 0.47$	$88.60\pm 0.21$	$89.77\pm 0.11$	$89.76\pm 0.13$
	$40\%$	$66.02\pm 0.50$	$86.65\pm 0.55$	$88.56\pm 0.33$	$88.61\pm 0.31$
	$60\%$	$44.05\pm 0.85$	$78.40\pm 1.91$	$81.07\pm 1.14$	$81.08\pm 1.10$
DBpedia-14	$10\%$	$97.97\pm 0.11$	$98.72\pm 0.01$	$98.86\pm 0.03$	$98.85\pm 0.04$
	$25\%$	$95.19\pm 0.18$	$98.52\pm 0.13$	$98.72\pm 0.10$	$98.72\pm 0.10$

Appendix

Table 5: BERT-base: final clean-test accuracy, mean

\pm

one sample standard deviation.

Explore similar work

Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View
Jun 3, 2026Mingyu LiSpectral Regularization Weight Decay
Beijing Normal University
Scale Weight Decay and Train Better
Jul 26, 2026Anuj ApteWeight Decay Neural Network Optimization
Global Technology Applied Research, JPMorganChase, New York, NY 10001
Predicting LLM Compression Degradation from Spectral Statistics
Apr 20, 2026Mingxue XuLLM Compression Low-Rank Matrix Decomposition
Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom

Jun 3, 2026, cs.LGJ/K move · Enter open · S save

Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View

Mingyu Li

Beijing Normal University

Spectral Regularization Weight Decay Transformer Nuclear Norm Minimization Grokking

Open paper AskPDF arXiv

Modern Transformer architectures frequently employ normalization mechanisms such as RMSNorm and Query-Key Normalization, making parts of the model approximately scale-invariant with respect to weight magnitudes. In this regime, standard Frobenius-norm weight decay acts purely along the radial direction of the weight space and cannot directly simplify the function represented by the normalized layer. We study grokking in small algorithmic tasks through this lens and propose \emph{Low-Rank Decay} (LRD), a nuclear-norm-like spectral regularizer whose subgradient -- the polar factor

UV^\top

-- retains a tangential component even in the scale-invariant setting. This distinction has a concrete dynamical consequence: after the model memorizes the training set and task gradients vanish, L2 decay can no longer reshape the weight spectrum, whereas LRD continues to compress singular values in an

\ell_1

-like fashion. On modular arithmetic tasks, we find that LRD induces rapid effective-rank collapse in Query/Key matrices and expands the data-fraction boundary at which delayed generalization (grokking) occurs. We further provide a spectral-geometric interpretation through the ``needle-to-fan'' expansion of the nuclear-norm subdifferential near low-rank strata.

Abstract

Explore similar work

Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View

Scale Weight Decay and Train Better

Predicting LLM Compression Degradation from Spectral Statistics