Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FAC

Abdulla Jasem Almansoori; Maria Ivanova; Andrey Veprikov; Aleksandr Beznosikov; Samuel Horváth; Martin Takáč

cs.LGFeb 18, 2026

Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FAC

Authors: Abdulla Jasem Almansoori, Maria Ivanova, Andrey Veprikov, Aleksandr Beznosikov, Samuel Horváth, Martin Takáč

Organizations: MBZUAI, Abu Dhabi, UAE · Yandex School of Data Analysis, Moscow, Russia · SB AI Lab, Moscow, Russia · Moscow Independent Research Institute of Artificial Intelligence (MIRAI), Moscow, Russia ·

Abstract

Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights, dramatically reducing trainable parameters and memory. In this work, we address the gap between training with full steps with low-rank projections (SVDLoRA) and LoRA fine-tuning. We propose LoRSum, a memory-efficient subroutine that closes this gap for gradient descent by casting LoRA optimization as a proximal sub-problem and solving it efficiently with alternating least squares updates, which we prove to be an implicit block power method. We recover several recently proposed preconditioning methods for LoRA as special cases, and show that LoRSum can also be used for updating a low-rank momentum. In order to address full steps with preconditioned gradient descent, we propose a scaled variant of LoRSum that uses structured metrics such as K-FAC and Shampoo, and we show that storing the diagonal of these metrics still allows them to perform well while remaining memory-efficient. Experiments on a synthetic task, CIFAR-100, and language-model fine-tuning on GLUE, SQuAD v2, and WikiText-103, show that our method can match or improve LoRA baselines given modest compute overhead, while avoiding full-matrix SVD projections and retaining LoRA-style parameter efficiency.

Figures & tables

Method	MNLI	QNLI	QQP	SST2	COLA	STSB
Full	87.67 $\pm$ 0.19	92.82 $\pm$ 0.06	91.10 $\pm$ 0.06	94.44 $\pm$ 0.08	62.00 $\pm$ 0.25	90.51 $\pm$ 0.02
SVDLoRA	86.40 $\pm$ 0.48	92.13 $\pm$ 0.08	88.68 $\pm$ 0.03	94.09 $\pm$ 0.08	63.35 $\pm$ 0.65	91.00 $\pm$ 0.03
LoRA	86.39 $\pm$ 0.04	91.70 $\pm$ 0.01	88.57 $\pm$ 0.08	93.75 $\pm$ 0.57	61.86 $\pm$ 0.73	90.56 $\pm$ 0.13
RPLoRA	85.67 $\pm$ 0.09	84.64 $\pm$ 8.91	87.06 $\pm$ 0.62	93.58 $\pm$ 0.00	58.74 $\pm$ 3.53	80.58 $\pm$ 2.17
Proj. PSI-LoRA	84.91 $\pm$ 0.03	90.46 $\pm$ 0.21	86.96 $\pm$ 0.11	93.35 $\pm$ 0.16	59.29 $\pm$ 0.17	89.93 $\pm$ 0.00
Scaled PSI-LoRA	86.57 $\pm$ 0.06	92.18 $\pm$ 0.16	89.10 $\pm$ 0.07	94.15 $\pm$ 0.16	61.41 $\pm$ 1.29	90.83 $\pm$ 0.26

Table 1: RoBERTa-base GLUE: best main metric (mean

\pm

std). Learning rates

\eta

is tuned over 3 values per method per task and method on 20% of the training budget. Then, the overall best

\eta

is used to train on the full budget. See Figure 6 for full learning curves and

\eta

Figure 1: Linear task with full-batch gradients and no momentum (left) vs. mini-batch gradients with momentum (right).

\textsc{PSI-LoRA}\times K

means

K

alternating iterations.

Method	Top Accuracy
SVDLoRA (SGD)	32.053 $\pm$ 0.179
Full (SGD)	37.456 $\pm$ 0.168
LoRA (AdamW)	31.393 $\pm$ 0.056
RPLoRA (AdamW)	32.895 $\pm$ 0.530
PSI-LoRA x 1	30.143 $\pm$ 0.377
PSI-LoRA x 2	30.302 $\pm$ 0.305

Table 2: CIFAR-100 (PatchMLP, batch size 64): top test accuracy over time over 3 seeds. Bold is best, underlined is second best.

Method	$\eta$	EM	F1	Eval loss
Ours	0.5	60.80 $\pm$ 0.9410	62.41 $\pm$ 1.02	0.7546 $\pm$ 0.0019
Full	1e-4	56.57 $\pm$ 3.64	57.55 $\pm$ 4.28	0.7145 $\pm$ 0.0288
LoRA	5e-4	56.52 $\pm$ 1.60	57.56 $\pm$ 1.86	0.7951 $\pm$ 0.0144

Table 3: T5-base SQuAD v2, tuned over 3 learning rates.

Figure 2: Perplexity of GPT-2 fine-tuning on WikiText-103 with LoRA adapters for 3 learning rates per method. Scaled PSI-LoRA is more robust to learning rate choice and achieves better perplexity for all learning rates.

Figure 3: Ablation of Scaled PSI-LoRA on GLUE-MNLI.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Extra state	Step cost	Dense
LoRA + SGD	$r(d_{\text{in}}+d_{\text{out}})$	$O((d_{\text{in}}+d_{\text{out}})r)$	No
LoRA + AdamW	$2r(d_{\text{in}}+d_{\text{out}})$	$O((d_{\text{in}}+d_{\text{out}})r)$	No
LoFT	$2r(d_{\text{in}}+d_{\text{out}})$	$O((d_{\text{in}}+d_{\text{out}})r^{2})$	No
Precond-LoRA	$r(d_{\text{in}}+d_{\text{out}})$	$O((d_{\text{in}}+d_{\text{out}})r^{2})$	No
SVDLoRA + AdamW	$2d_{\text{in}}d_{\text{out}}$	$O((d_{\text{in}}+d_{\text{out}})^{2}r)$	Yes
PSI-LoRA	$r_{m}(d_{\text{in}}+d_{\text{out}})$	$O((d_{\text{in}}+d_{\text{out}})r^{2}K)$	No

Appendix

Table 4: Per-layer optimizer state and dominant per-step costs (adapter rank

r

, effective batch size

B

). We assume that

r_{m}\leq r

for PSI-LoRA momentum rank.

\mathbf{U}_{t+1}\mathbf{V}_{t+1}^{\top}=\mathbf{W}_{t+1}\leftarrow\textsc{LoRSum}(\bar{\mathbf{W}}_{t+1};\,K,\rho).

Appendix

Algorithm 4 PSI-LoRA step with low-rank momentum

\mathbf{v}_{s,t}\leftarrow\beta_{2}\mathbf{v}_{s,t-1}+(1-\beta_{2})\mathrm{diag}\left(\tfrac{1}{B}\mathbf{S}_{t}^{\top}\mathbf{S}_{t}\right),\qquad\mathbf{v}_{x,t}\leftarrow\beta_{2}\mathbf{v}_{x,t-1}+(1-\beta_{2})\mathrm{diag}\left(\tfrac{1}{B}\mathbf{X}_{t}^{\top}\mathbf{X}_{t}\right).

Appendix

Algorithm 6 Scaled PSI-LoRA step with low-rank momentum and diagonal K-FAC metric with fractional power

Figure 6: Learning curves of validation main metrics for RoBERTa-base on GLUE tasks. Legend shows learning rates for each method.

Figure 7: Stochastic linear task: increasing the rank budget of the low-rank momentum buffer improves the approximation quality and moves PSI-LoRA closer to the SVDLoRA oracle.

Proximal Optimization

See all 14 papers

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition
Sep 29, 2026Yihao Ouyang, Shiwei Li, Haozhao Wang +5Low-Rank Adaptation Parameter-Efficient Fine-Tuning
Huazhong University of Science and Technology, Wuhan, China · Hebei University of Technology, Tianjin, China
Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence
May 29, 2026Valérie Castin, Kimia Nadjahi, Pierre Ablin +1LLM Fine-Tuning Low-Rank Adaptation
1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.
SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation
Jun 15, 2026Junghun Oh, Sungyong Baik, Kyoung Mu LeeLow-Rank Adaptation Parameter-Efficient Fine-Tuning
Dept. of ECE, ASRI Seoul National University · Dept. of Artificial Intelligence Dept. of Data Science Hanyang University

Sep 29, 2026J/K move · Enter open · S save

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition

Yihao Ouyang, Shiwei Li, Haozhao Wang, Xiandi Luo+4

Huazhong University of Science and Technology, Wuhan, China · Hebei University of Technology, Tianjin, China

Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT), yet a performance gap can remain relative to full fine-tuning (FFT). Many LoRA variants improve the initialization or optimization of low-rank factors. At each training step, however, their first-order weight-space directions are constrained by the current parameterization. We characterize the corresponding LoRA-accessible gradient space and show that it coincides with the tangent space induced by the current LoRA parameterization. This characterization yields an orthogonal decomposition of the full weight gradient at the current model parameters. We term the component orthogonal to this space the normal gradient. Based on this decomposition, we propose GDLoRA (Gradient-Decomposed Low-Rank Adaptation). GDLoRA reconstructs the full weight gradient from forward activations and backward signals, extracts its normal component, and directly updates the base weights with this component, while retaining standard AdamW optimization for the LoRA factors. GDLoRA incorporates complementary normal gradients without increasing standard LoRA's optimizer-state memory budget under matched adapter and optimizer configurations. Experiments on natural language understanding, mathematical reasoning, commonsense reasoning, and image classification show that GDLoRA consistently improves over LoRA and narrows the performance gap to FFT. The code is available at https://anonymous.4open.science/r/GDLoRA.

Abstract

Proximal Optimization

Explore similar work

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence

SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation