Parameter-Efficient Fine-Tuning (PEFT) methods, especially LoRA, are widely used for adapting pre-trained models to downstream tasks due to their computational and storage efficiency. However, in the context of LoRA and its variants, the potential of activation subspaces corresponding to tail eigenvectors remains substantially under-exploited, which may lead to suboptimal fine-tuning performance. In this work, we propose Astra (Activation-Space Tail-Eigenvector Low-Rank Adaptation), a novel PEFT method that leverages the tail eigenvectors of the model output activations-estimated from a small task-specific calibration set-to construct task-adaptive low-rank adapters. By constraining updates to the subspace spanned by these tail eigenvectors, Astra achieves faster convergence and improved downstream performance with a significantly reduced parameter budget. Extensive experiments across natural language understanding (NLU) and natural language generation (NLG) tasks demonstrate that Astra consistently outperforms existing PEFT baselines across 16 benchmarks and even surpasses full fine-tuning (FFT) in certain scenarios.
Figures & tables
#Params
MNLI 393k
SST-2 67k
QNLI 105k
CoLA 8.5k
MRPC 3.7K
Average
Full FT
223M
86.95 ±0.04
97.02 ±0.03
98.78 ±0.02
84.52 ±0.01
84.19 ±0.05
90.29
LoRA
3.2M
86.97 ±0.01
96.62 ±0.02
98.75 ±0.03
49.95 ±1.33
47.67 ±0.06
75.99
DoRA
3.4M
87.05 ±0.02
97.19 ±0.01
98.79 ±0.02
84.23 ±0.03
49.88 ±0.05
83.43
rsLoRA
3.2M
87.06 ±0.01
97.13 ±0.02
98.79 ±0.02
83.89 ±0.02
49.63 ±0.04
83.30
PiSSA
3.2M
87.01 ±0.01
97.08 ±0.01
98.82 ±0.01
84.80 ±0.01
82.84 ±0.01
90.11
CorDA
3.2M
87.11 ±0.03
97.19 ±0.02
98.81 ±0.05
84.71 ±0.22
69.12 ±0.23
87.39
Table 1: Performance of T5-base fine-tuned with different adaptation methods on 5 datasets of the GLUE benchmark. We report accuracy for all tasks, and the results are averaged over three runs with different random seeds. Bold values indicate the best performance.
Model
Method
#Params
GSM8K
Math
HumanEval
HumanEval+
MBPP
MBPP+
Average
LLaMA2-7B
Full FT
6738M
58.76
12.04
32.9
31.1
43.9
36.8
35.92
LoRA
320M
41.40
5.42
22.0
20.1
34.9
27.2
25.17
MiLoRA
320M
39.12
5.06
20.1
18.9
36.8
29.4
24.90
PiSSA
320M
51.63
7.36
23.2
20.1
36.7
29.5
28.08
CorDA
320M
52.99
8.08
25.0
23.2
36.2
29.6
29.18
Ours
320M
55.19
8.98
25.0
23.2
38.4
31.2
30.33
Table 2: Comparison of full fine-tuning (Full FT) and several LoRA variants on 2 mathematical reasoning and 4 code generation benchmarks. The best PEFT results are highlighted in bold .
Model
Method
#Params
BoolQ
PIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Average
LLaMA2-7B
Full FT
6738M
82.81
75.08
55.57
73.64
72.69
41.72
32.00
61.93
LoRA
320M
79.97
78.35
57.30
68.82
78.07
46.16
32.40
63.01
PiSSA
320M
83.03
78.18
57.52
70.72
78.41
47.35
33.40
64.09
MiLoRA
320M
79.66
78.13
57.53
69.22
77.31
45.39
32.40
62.81
CorDA
320M
82.87
78.45
56.24
71.82
75.67
43.09
33.00
63.02
Ours
320M
83.76
78.51
57.28
71.74
79.63
48.72
34.20
64.83
Table 3: Zero-shot performance of LLaMA2-7B and LLaMA3-8B fine-tuned with different adaptation methods on seven commonsense reasoning benchmarks. The best PEFT results are shown in bold .
Figure 1: Training loss and gradient norm curves for FFT, LoRA (rank=128), and Astra with varying ranks on the MetaMathQA dataset. Our method (rank=8) performs even better than LoRA (rank=128), and higher ranks lead to faster loss reduction, approaching the performance of FFT.
Figure 2: (a) and (b) report the performance of different LoRA variants on GSM8K and MATH under various ranks, respectively. (c) shows the final training loss on the MetaMathQA dataset under various ranks. (d) illustrates the performance using different calibration data.
Eigenvectors
GSM8K
MATH
Random
40.49
5.64
Top
40.71
5.48
Upper Quartile (Q3)
40.49
5.64
Medium
38.74
5.60
Lower Quartile (Q1)
42.76
5.70
Tail
55.19
8.98
Table 4: Performance of LLaMA2-7B fine-tuned with adapters initialized using eigenvectors from different quantiles of the eigenvalue spectrum.
Figure 3: Comparison of effective rank before and after fine-tuning.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Driven-Type
Signal
Gradient Free
Calibration Data
PiSSA
weight
weight
✔
No
MiLoRA
weight
weight
✔
No
CorDA
data
Input Context
✔
Downstream
LoRA-GA
data
Gradient
✘
Downstream
Astra
data
Output Activation
✔
Downstream/General
Appendix
Table 5: Comparison of our selective LoRA initialization variants in the experimental section.
Corpus
Task
#Train
#Val
#Test
#Labels
Domain
CoLA
Acceptability
8.55k
1.04k
1.06k
2
misc.
SST-2
Sentiment
67.3k
872
1.82k
2
Movie Reviews
MRPC
Paraphrase
3.67k
408
1.73k
2
News
MNLI
NLI
393k
19.65k
19.65k
3
misc.
QNLI
QA/NLI
105k
5.46k
5.46k
2
Wikipedia
Appendix
Table 6: Statistical overview of the GLUE benchmark datasets used in our experiments.
Table 10
Figure 4: Training loss and gradient-norm curves of LLaMA2-7B fine-tuned with different adapters initialized using different eigenvectors. The results demonstrate that initializing the adapter with tail eigenvectors leads to the fastest and lowest loss convergence
Figure 5: Training loss and gradient-norm curves of LLaMA2-7B fine-tuned with different adaptation methods on the first 100,000 samples from MetaMathQA, CodeFeedback and Commonsense170K datasets for one epoch.
Figure 6: Training loss and gradient-norm curves of LLaMA3-8B fine-tuned with different adaptation methods on the first 100,000 samples from MetaMathQA, CodeFeedback and Commonsense170K datasets for one epoch.
Figure 7: Response on a humanities problem from MT-Bench
Figure 8: Response on a multi-turn math problem from MT-Bench
Figure 9: Response on a roleplay problem from MT-Bench
Figure 10: Response on a humanities problem from MT-Bench