Stabilizing the Dynamic Low-Rank Training

Zhonghan Xu; Ling Wang; Junhao Chen; Jianwei Zhao; Jinwei Yang

cs.LGSep 26, 2026

Stabilizing the Dynamic Low-Rank Training

Authors: Zhonghan Xu, Ling Wang, Junhao Chen, Jianwei Zhao, Jinwei Yang

Organizations: School of Communications and Information Engineering University of Electronic Science and Technology of China

Abstract

Training neural networks directly in a low-rank parameterization is an appealing route to reducing memory, compute, and storage simultaneously during both training and inference. Dynamic low-rank training (DLRT), which confines weights to a rank- $r$ manifold via the Galerkin projection of the gradient flow, is particularly attractive because it identifies efficient subnetworks on the fly without specialized initialization or post-factorization. However, DLRT fails to find trainable networks under high compression. In this paper, we derive the gradient flow of the best rank- $r$ approximation and point out that the offset of DLRT comes from a curvature-coupling term which is large and thus non-negligible under aggressive compression. Guided by this analysis, we propose a stable dynamic low-rank training method, named SDLRT, which maintains a lightweight compensation buffer that reinjects the top neglected singular directions. Additionally, we introduce a negative feedback on the truncation tolerance to stabilize each layer's rank. Experimentally, SDLRT reliably finds trainable subnetworks where DLRT collapses and as a PEFT adapter on DeBERTa-v3, it achieves the best average score on SuperGLUE at only $2.8\%$ parameter overhead over LoRA.

Figures & tables

Figure 1: DLRT collapses under aggressive compression on VGG-19/CIFAR-100 ( $\approx 69.6\%$ compression). (a) Accuracy curves: DLRT lags the baseline by $25.9\%$ in test accuracy; training accuracy also plateaus far below baseline, indicating insufficient capacity rather than over-fitting. (b) Per-layer rank evolution; $\blacktriangledown$ marks the first epoch each layer hits its minimum rank—every layer collapses within the first 100 epochs. (c) Singular-value spectra of the last two weight matrices across epochs $\{0,10,\dots,190\}$ . The red shaded tail beyond $i>r$ is the energy that DLRT permanently discards: $98.8\%/98.6\%$ of the total $\|W\|_{F}^{2}$ for the two layers.

Figure 2: The procedures of SVD-based low-rank training (left) and the stable SVD-based low-rank training with compensation (right).

Panel of Figure 3 — Figure 3: Comparison of accuracy under different training and testing compression on CIFAR-10. SDLRT and $\text{SDLRT}_{\text{2dim}}$ achieve the top two highest scores. Error bars represent the minimum and maximum across 5 samples.

Panel of Figure 4 — Figure 4: On VGG-19/CIFAR-100, SDLRT can stably identify winning tickets under extreme compression rates, whereas DLRT cannot. Error bars represent the minimum and maximum across 5 samples.

Method	CB	COPA	WSC	RTE	WiC	BoolQ	Avg.
SDLRT (ours)	$83.63_{\pm 1.34}$	${91.00}_{\pm 1.67}$	$\mathbf{91.51}_{\pm 1.13}$	$86.52_{\pm 0.59}$	$\mathbf{73.93}_{\pm 0.88}$	$84.29_{\pm 0.20}$	$\mathbf{85.15}$
DLRT	$81.85_{\pm 1.75}$	$\mathbf{91.17}_{\pm 0.98}$	$90.06_{\pm 1.32}$	$86.28_{\pm 1.07}$	$73.41_{\pm 0.61}$	$\textbf{84.33}_{\pm 0.54}$	$84.52$
LoRA	$\mathbf{85.72}_{\pm 2.26}$	$90.50_{\pm 0.55}$	$84.93_{\pm 10.53}$	$\mathbf{87.30}_{\pm 0.93}$	$73.07_{\pm 0.81}$	$84.14_{\pm 0.21}$	$84.28$
AdaLoRA	$82.74_{\pm 2.16}$	$90.67_{\pm 2.07}$	$63.46_{\pm 0.00}$	$85.92_{\pm 0.82}$	$73.49_{\pm 0.54}$	$83.92_{\pm 0.14}$	$80.03$
LoRA+	$85.12_{\pm 2.69}$	$89.83_{\pm 1.83}$	$89.58_{\pm 2.23}$	$87.24_{\pm 1.16}$	$73.27_{\pm 0.57}$	$83.80_{\pm 0.56}$	$84.81$

Table 1: Results (%) on fine-tuning DeBERTa-base with SuperGLUE datasets.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Distribution of the validation accuracy on DeBERTa-v3-base across six SuperGLUE tasks over six random seeds. The diamond marker indicates the mean and the circular marker indicates outliers. The box shows the interquartile range ( $25\%$ – $75\%$ quantiles) and the median.

Method	Memory	Training Time (Average per epoch)	Test Accuracy $(\%)$
DLRT	$2680\text{MB}$	$56.88$ s	$53.87\%$
SDLRT	$2710\text{MB}$	$57.53$ s	$61.86\%$

Appendix

Table 2: Comparison of memory and time usage.

Low-Rank Approximation

Low-rank approximation represents matrices, tensors, and model computations with fewer components to reduce storage and computation while preserving useful structure.

See all 93 papers

SLORR: Simple and Efficient In-Training Low-Rank Regularization
Jul 9, 2026David González-Martínez, Shiwei LiuLLM Compression Low-Rank Matrix Decomposition
Max Planck Institute for Intelligent Systems · University of Tübingen · ELLIS Institute Tübingen +1
DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation
Oct 13, 2026Yangming Zhang, Ling Yi, Datong Li +4Low-Rank Adaptation Parameter-Efficient Fine-Tuning
Chongqing University of Posts and Telecommunications, China · Anhui University of Science and Technology · Northeastern University, China
Stabilizing Native Low-Rank LLM Pretraining
Feb 12, 2026Paul Janson, Edouard Oyallon, Eugene BelilovskyLanguage Model Pretraining Low-Rank Matrix Decomposition
Concordia University, Montreal, Canada · Mila Quebec AI Institute, Montreal, Canada · Sorbonne University, CNRS, Paris, France

Jul 9, 2026J/K move · Enter open · S save

SLORR: Simple and Efficient In-Training Low-Rank Regularization

David González-Martínez, Shiwei Liu

Max Planck Institute for Intelligent Systems · University of Tübingen · ELLIS Institute Tübingen+1

LLM Compression Low-Rank Matrix Decomposition Nuclear Norm Minimization Low-Rank Compression Low-Rank Approximation Neural Network Compression

Open paper AskPDF arXiv

Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. Existing training-time low-rank regularizers can improve compressibility, but they often require SVDs of large weight matrices, modify the model architecture (introducing additional trainable parameters), or rely on stateful cached quantities. To address these limitations, we introduce SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm. SLORR directly regularizes the original weight matrices using GPU-friendly approximations for the forward and backward passes of the regularizers, for which we provide approximation guarantees. We first evaluate SLORR on ImageNet-1K across short-horizon continued training of ResNet-50, ViT-B/16, and ViT-L/16, and pretraining of ResNet-18, where SLORR induces compressibility while introducing less than 8% training overhead. We further evaluate SLORR-Hoyer in LLM pretraining at 135M and 560M scales: SLORR-trained compressed models preserve performance substantially better than unregularized models while adding less than 1% average training overhead.

Abstract

Low-Rank Approximation

Explore similar work

SLORR: Simple and Efficient In-Training Low-Rank Regularization

DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation

Stabilizing Native Low-Rank LLM Pretraining