Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning by learning low-rank weight updates. Increasing rank expands adaptation capacity, but does not guarantee that all learned directions improve downstream performance. In a diagnostic experiment with rank-16 LoRA, we show that retaining only the top singular components achieves the best performance, whereas adding further components degrades it. Motivated by this, we propose Residual-Gradient Adaptation (ReGRA), a two-stage method for making more effective use of a fixed rank budget. After the first stage, ReGRA estimates task gradients on a calibration set and removes their projections onto the two-sided subspace induced by the first-stage update. It then aggregates the residual-gradient directions to initialize a second-stage adapter without changing the first-stage update. An orthogonality regularizer encourages the two updates to capture complementary directions. Both updates can be merged into the pretrained weights, incurring no additional inference overhead. Across diverse benchmark sets, ReGRA achieves stronger performance than alternatives with the same total rank and even matches LoRA adapters with twice the rank. Ablation studies and further analysis support the contribution of its core design choices. Code is available at https://github.com/Fanyao-Li/ReGRA.
Figures & tables
Figure 1: Effect of retained rank on accuracy. Model performance does not improve monotonically with increasing retained rank.
Figure 2: Overall framework of Residual-Gradient Adaptation (ReGRA). ReGRA first trains Adapter 1 to learn an initial task-specific update. It then projects the task gradient away from the subspace already captured by Adapter 1 and extracts the dominant residual directions via SVD. These residual directions are used to initialize and guide Adapter 2 toward complementary task-relevant information. The two adapters are finally combined with the frozen backbone to form the final model.
Figure 3: Illustration of the standard LoRA.
Method
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST-2
STS-B
Avg.
RoBERTa-base
LoRA ( r=16 )
60.24
85.75
87.25
91.45
87.94
75.09
93.69
90.31
83.97
LoRA ( r=8 )
58.04
85.11
87.50
90.99
87.16
72.92
93.92
90.07
83.21
DoRA ( r=8 )
56.53
86.20
88.23
91.21
85.43
74.36
93.92
90.05
83.24
PiSSA ( r=8 )
56.01
84.16
89.12
90.49
87.43
72.56
93.23
89.56
82.82
LoRA-One ( r=8 )
58.39
85.28
89.12
91.15
85.51
75.81
94.38
90.54
83.77
KeepLoRA ( r=8 )
58.55
84.76
90.35
90.93
84.98
67.87
93.69
90.42
82.69
Table 1: Performance (%) on eight GLUE natural language understanding tasks using RoBERTa-base and RoBERTa-large. Avg. is the unweighted mean across tasks. We calculate the performance gain of ReGRA against the rank-8 LoRA on the same backbone. For ReGRA, r=8 denotes the rank of each stage. The best average for each backbone is marked in bold.
Method
AddSub
MultiArith
SingleEq
SVAMP
GSM8K
AQuA
Avg.
Llama-3-8B
LoRA ( r=16 )
79.74
92.83
87.59
69.80
60.50
39.37
71.64
LoRA ( r=8 )
76.20
92.66
87.99
69.30
61.41
37.40
70.83
DoRA ( r=8 )
77.72
94.16
86.41
72.10
60.34
39.76
71.75
PiSSA ( r=8 )
74.93
89.00
85.23
67.80
58.07
43.30
69.72
LoRA-One ( r=8 )
79.24
91.33
87.99
70.30
60.65
40.15
71.61
KeepLoRA ( r=8 )
80.50
87.33
88.18
70.50
60.80
37.79
70.85
Table 2: Accuracy (%) on six mathematical reasoning benchmarks using Llama-3-8B and Qwen3-8B-Base. Avg. is the unweighted mean across tasks. Subscript values beside ReGRA scores show percentage-point differences from rank-8 LoRA on the same backbone. For ReGRA, r=8 denotes the rank of each stage. The best average for each backbone is marked in bold.
Figure 6
Figure 5: Functional importance of LoRA singular components. For rank-16 LoRA, we apply SVD to each learned update ΔW and remove one component at a time. Bars show performance changes relative to the full update (dashed zero line) for Llama-3-8B on MBPP (a) and MBPP+ (b), and Qwen3-8B-Base on Math (c). Removing the leading component reduces performance by 3.97 percentage points on both code tasks and 8.03 percentage points on Math. Red and blue indicate gains and losses, respectively; broken axes display the larger leading-component effect.
Figure 6: Effects of isolated Stage-2 SVD components on Llama-3-8B. Each Ci retains only the i th rank-one component of every Stage-2 update matrix on a fixed Stage-1 model; components are evaluated independently, not cumulatively. Full retains all eight components. Bars show percentage-point changes from Stage 1-only in (a) mean accuracy across six math benchmarks and (b) MBPP+ pass1 on 378 problems. Positive and negative bars indicate gains and losses, respectively. Results are from one deterministic evaluation per condition.
Figure 7: Progressive rank expansion on LLaMA-3-8B. At each stage, previously learned adapters are frozen and a new rank-2 adapter is trained.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Epochs
Stage-1 LR
Stage-2 LR
Batch size
Max. length
RoBERTa-base, GLUE
Task-specific
2×10−4
2×10−4
32
128
RoBERTa-large, GLUE
Task-specific
2×10−4
5×10−5 *
32
128
Llama-3-8B, math
2
10−4
10−4
2
256
Qwen3-8B-Base, math
2
10−4
10−5
2
256
Llama-3-8B, code
1
2×10−5
5×10−6
4
1024
Llama-3-8B, commonsense
1
10−4
2×10−5
4
256
Appendix
Table 4: Training settings for ReGRA. The Stage-2 learning rate for RoBERTa-large RTE is 10−4 rather than 5×10−5 .
Setting
M
b
q
GLUE, both backbones
8
32
16
Llama-3-8B, math
1
1
8
Qwen3-8B-Base, math
4
2
8
Llama-3-8B, code
16
1
16
Llama-3-8B, commonsense
16
4
32
Appendix
Table 5: Residual-gradient calibration settings for ReGRA.
Method
OBQA
ARC-C
Wino.
PIQA
SIQA
ARC-E
BoolQ
Hella.
Avg.
Llama-3-8B
LoRA ( r=16 )
88.40
81.10
87.29
88.74
81.63
93.14
74.19
93.35
85.98
LoRA ( r=8 )
86.80
81.31
87.61
88.57
81.12
93.10
69.93
92.73
85.15
DoRA ( r=8 )
86.40
81.14
87.21
89.28
81.57
93.43
73.82
92.57
85.68
PiSSA ( r=8 )
81.00
77.90
86.02
88.46
80.80
92.38
73.18
91.55
83.91
LoRA-One ( r=8 )
88.20
81.05
87.37
88.84
81.67
93.22
74.31
92.81
85.93
ReGRA ( r=8 )
88.20 +1.40
82.00 +0.69
86.90 -0.71
89.34 +0.77
81.93 +0.81
92.30 -0.80
74.22 +4.29
93.07 +0.34
86.00 +0.85
Appendix
Table 6: Accuracy (%) on eight commonsense reasoning benchmarks using Llama-3-8B. Avg. is the unweighted mean across tasks. Subscript values beside ReGRA scores show percentage-point differences from rank-8 LoRA. The best average is marked in bold.
Figure 8: Stage 1 errors uniquely corrected by isolated Stage-2 components. Each cell counts cases answered incorrectly by Stage 1, corrected by the indicated component, and not corrected by any of the other seven isolated components. Math counts task–example pairs across six benchmarks; Code counts MBPP+ problems.
Figure 9: Coverage of the full Stage-2 model’s newly solved cases. For each case answered incorrectly by Stage 1 and correctly by the full Stage-2 model, the bars show how many isolated components also answer it correctly. Math includes 391 task–example pairs; Code includes 15 MBPP+ problems.