Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning by learning low-rank weight updates. Increasing rank expands adaptation capacity, but does not guarantee that all learned directions improve downstream performance. In a diagnostic experiment with rank-16 LoRA, we show that retaining only the top singular components achieves the best performance, whereas adding further components degrades it. Motivated by this, we propose Residual-Gradient Adaptation (ReGRA), a two-stage method for making more effective use of a fixed rank budget. After the first stage, ReGRA estimates task gradients on a calibration set and removes their projections onto the two-sided subspace induced by the first-stage update. It then aggregates the residual-gradient directions to initialize a second-stage adapter without changing the first-stage update. An orthogonality regularizer encourages the two updates to capture complementary directions. Both updates can be merged into the pretrained weights, incurring no additional inference overhead. Across diverse benchmark sets, ReGRA achieves stronger performance than alternatives with the same total rank and even matches LoRA adapters with twice the rank. Ablation studies and further analysis support the contribution of its core design choices. Code is available at https://github.com/Fanyao-Li/ReGRA.
Figures & tables
Figure 1: Effect of retained rank on accuracy. Model performance does not improve monotonically with increasing retained rank.
Figure 2: Overall framework of Residual-Gradient Adaptation (ReGRA). ReGRA first trains Adapter 1 to learn an initial task-specific update. It then projects the task gradient away from the subspace already captured by Adapter 1 and extracts the dominant residual directions via SVD. These residual directions are used to initialize and guide Adapter 2 toward complementary task-relevant information. The two adapters are finally combined with the frozen backbone to form the final model.
Figure 3: Illustration of the standard LoRA.
Method
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST-2
STS-B
Avg.
RoBERTa-base
LoRA ( r=16 )
60.24
85.75
87.25
91.45
87.94
75.09
93.69
90.31
83.97
LoRA ( r=8 )
58.04
85.11
87.50
90.99
87.16
72.92
93.92
90.07
83.21
DoRA ( r=8 )
56.53
86.20
88.23
91.21
85.43
74.36
93.92
90.05
83.24
PiSSA ( r=8 )
56.01
84.16
89.12
90.49
87.43
72.56
93.23
89.56
82.82
LoRA-One ( r=8 )
58.39
85.28
89.12
91.15
85.51
75.81
94.38
90.54
83.77
KeepLoRA ( r=8 )
58.55
84.76
90.35
90.93
84.98
67.87
93.69
90.42
82.69
Table 1: Performance (%) on eight GLUE natural language understanding tasks using RoBERTa-base and RoBERTa-large. Avg. is the unweighted mean across tasks. We calculate the performance gain of ReGRA against the rank-8 LoRA on the same backbone. For ReGRA, r=8 denotes the rank of each stage. The best average for each backbone is marked in bold.
Method
AddSub
MultiArith
SingleEq
SVAMP
GSM8K
AQuA
Avg.
Llama-3-8B
LoRA ( r=16 )
79.74
92.83
87.59
69.80
60.50
39.37
71.64
LoRA ( r=8 )
76.20
92.66
87.99
69.30
61.41
37.40
70.83
DoRA ( r=8 )
77.72
94.16
86.41
72.10
60.34
39.76
71.75
PiSSA ( r=8 )
74.93
89.00
85.23
67.80
58.07
43.30
69.72
LoRA-One ( r=8 )
79.24
91.33
87.99
70.30
60.65
40.15
71.61
KeepLoRA ( r=8 )
80.50
87.33
88.18
70.50
60.80
37.79
70.85
Table 2: Accuracy (%) on six mathematical reasoning benchmarks using Llama-3-8B and Qwen3-8B-Base. Avg. is the unweighted mean across tasks. Subscript values beside ReGRA scores show percentage-point differences from rank-8 LoRA on the same backbone. For ReGRA, r=8 denotes the rank of each stage. The best average for each backbone is marked in bold.
Figure 6
Figure 5: Functional importance of LoRA singular components. For rank-16 LoRA, we apply SVD to each learned update ΔW and remove one component at a time. Bars show performance changes relative to the full update (dashed zero line) for Llama-3-8B on MBPP (a) and MBPP+ (b), and Qwen3-8B-Base on Math (c). Removing the leading component reduces performance by 3.97 percentage points on both code tasks and 8.03 percentage points on Math. Red and blue indicate gains and losses, respectively; broken axes display the larger leading-component effect.
Figure 6: Effects of isolated Stage-2 SVD components on Llama-3-8B. Each Ci retains only the i th rank-one component of every Stage-2 update matrix on a fixed Stage-1 model; components are evaluated independently, not cumulatively. Full retains all eight components. Bars show percentage-point changes from Stage 1-only in (a) mean accuracy across six math benchmarks and (b) MBPP+ pass1 on 378 problems. Positive and negative bars indicate gains and losses, respectively. Results are from one deterministic evaluation per condition.
Figure 7: Progressive rank expansion on LLaMA-3-8B. At each stage, previously learned adapters are frozen and a new rank-2 adapter is trained.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Epochs
Stage-1 LR
Stage-2 LR
Batch size
Max. length
RoBERTa-base, GLUE
Task-specific
2×10−4
2×10−4
32
128
RoBERTa-large, GLUE
Task-specific
2×10−4
5×10−5 *
32
128
Llama-3-8B, math
2
10−4
10−4
2
256
Qwen3-8B-Base, math
2
10−4
10−5
2
256
Llama-3-8B, code
1
2×10−5
5×10−6
4
1024
Llama-3-8B, commonsense
1
10−4
2×10−5
4
256
Appendix
Table 4: Training settings for ReGRA. The Stage-2 learning rate for RoBERTa-large RTE is 10−4 rather than 5×10−5 .
Setting
M
b
q
GLUE, both backbones
8
32
16
Llama-3-8B, math
1
1
8
Qwen3-8B-Base, math
4
2
8
Llama-3-8B, code
16
1
16
Llama-3-8B, commonsense
16
4
32
Appendix
Table 5: Residual-gradient calibration settings for ReGRA.
Method
OBQA
ARC-C
Wino.
PIQA
SIQA
ARC-E
BoolQ
Hella.
Avg.
Llama-3-8B
LoRA ( r=16 )
88.40
81.10
87.29
88.74
81.63
93.14
74.19
93.35
85.98
LoRA ( r=8 )
86.80
81.31
87.61
88.57
81.12
93.10
69.93
92.73
85.15
DoRA ( r=8 )
86.40
81.14
87.21
89.28
81.57
93.43
73.82
92.57
85.68
PiSSA ( r=8 )
81.00
77.90
86.02
88.46
80.80
92.38
73.18
91.55
83.91
LoRA-One ( r=8 )
88.20
81.05
87.37
88.84
81.67
93.22
74.31
92.81
85.93
ReGRA ( r=8 )
88.20 +1.40
82.00 +0.69
86.90 -0.71
89.34 +0.77
81.93 +0.81
92.30 -0.80
74.22 +4.29
93.07 +0.34
86.00 +0.85
Appendix
Table 6: Accuracy (%) on eight commonsense reasoning benchmarks using Llama-3-8B. Avg. is the unweighted mean across tasks. Subscript values beside ReGRA scores show percentage-point differences from rank-8 LoRA. The best average is marked in bold.
Figure 8: Stage 1 errors uniquely corrected by isolated Stage-2 components. Each cell counts cases answered incorrectly by Stage 1, corrected by the indicated component, and not corrected by any of the other seven isolated components. Math counts task–example pairs across six benchmarks; Code counts MBPP+ problems.
Figure 9: Coverage of the full Stage-2 model’s newly solved cases. For each case answered incorrectly by Stage 1 and correctly by the full Stage-2 model, the bars show how many isolated components also answer it correctly. Math includes 391 task–example pairs; Code includes 15 MBPP+ problems.
Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT), yet a performance gap can remain relative to full fine-tuning (FFT). Many LoRA variants improve the initialization or optimization of low-rank factors. At each training step, however, their first-order weight-space directions are constrained by the current parameterization. We characterize the corresponding LoRA-accessible gradient space and show that it coincides with the tangent space induced by the current LoRA parameterization. This characterization yields an orthogonal decomposition of the full weight gradient at the current model parameters. We term the component orthogonal to this space the normal gradient. Based on this decomposition, we propose GDLoRA (Gradient-Decomposed Low-Rank Adaptation). GDLoRA reconstructs the full weight gradient from forward activations and backward signals, extracts its normal component, and directly updates the base weights with this component, while retaining standard AdamW optimization for the LoRA factors. GDLoRA incorporates complementary normal gradients without increasing standard LoRA's optimizer-state memory budget under matched adapter and optimizer configurations. Experiments on natural language understanding, mathematical reasoning, commonsense reasoning, and image classification show that GDLoRA consistently improves over LoRA and narrows the performance gap to FFT. The code is available at https://anonymous.4open.science/r/GDLoRA.
Yihao Ouyang, Shiwei Li, Haozhao Wang +5
Huazhong University of Science and Technology, Wuhan, China · Hebei University of Technology, Tianjin, China
Low-rank adaptation (LoRA) often underperforms full fine-tuning. We investigate this performance gap through a tangent-space lens: factorized updates are confined to the tangent space of the current low-rank factors, leaving gradient components outside this space unexploited. An analysis of ten representative LoRA variants reveals that these residual components account for most of the gradient energy and exhibit strong spectral concentration. Motivated by this finding, we propose Dynamic Tangent LoRA (DT-LoRA), which partitions a fixed rank budget into a stable branch and a dynamic branch to capture the principal residual gradient directions. We show that these directions define the optimal update of a local quadratic model under orthogonal-complement and rank constraints. Matrix-free randomized SVD recovers them without materializing the residual gradient, while controlled write-back preserves accumulated updates and stabilizes training within the allocated rank. DT-LoRA adds no trainable parameters or inference cost over LoRA. Experiments across computer vision and natural language processing tasks demonstrate consistent improvements over LoRA and its variants. Moreover, despite being a general-purpose adaptation method, DT-LoRA outperforms task-specific approaches to open-vocabulary domain generalization for semantic segmentation, achieving state-of-the-art results. Code is available at https://github.com/Snipine/DT-LoRA.
Yangming Zhang, Ling Yi, Datong Li +4
Chongqing University of Posts and Telecommunications, China · Anhui University of Science and Technology · Northeastern University, China
Low-Rank Adaptation (LoRA) is a prominent fine-tuning method for large models, achieving competitive performance with reduced memory overhead. However, a persistent performance gap remains between LoRA and full fine-tuning. Recent studies have sought to narrow this gap by employing one-step gradient approximations of pretrained weights to align LoRA updates with the principal directions or intrinsic dimensionalities of full fine-tuning updates. Nevertheless, these approaches fail to capture the full dynamics of the gradients. In this paper, we propose LoRA-GA2, an effective fine-tuning algorithm that fully leverages multi-step gradient information. Specifically, we introduce a lightweight probe for multi-step gradients of pretrained weights that incurs no additional GPU memory cost and only marginal time overhead. We further employ a spectrum-aware, importance-based rank allocation and optimal initialization derived from multi-step gradients. Extensive experimental results demonstrate that LoRA-GA2 consistently outperforms existing LoRA variants while preserving the efficiency advantages of vanilla LoRA. For instance, LoRA-GA2 surpasses the leading baseline by an average of 0.66 points on the GLUE benchmark, and outperforms the strongest baseline by 1.03 points on GSM8K and 0.87 points on HumanEval, respectively.
Haonan He, Xinyue Fan
University of Science and Technology of China · Independent Researcher