The high cost of full-parameter fine-tuning (FFT) of Large Language Models (LLMs) has led to a series of parameter-efficient fine-tuning (PEFT) methods. However, it remains unclear which methods provide the best cost-performance trade-off at different model scales. We introduce Astraios, a suite of 28 instruction-tuned OctoCoder models using 7 tuning methods and 4 model sizes up to 16 billion parameters. Through investigations across 5 tasks and 8 different datasets encompassing both code comprehension and code generation tasks, we find that FFT generally leads to the best downstream performance across all scales, and PEFT methods differ significantly in their efficacy based on the model scale. LoRA usually offers the most favorable trade-off between cost and performance. Further investigation into the effects of these methods on both model robustness and code security reveals that larger models tend to demonstrate reduced robustness and less security. At last, we explore the relationships among updated parameters, cross-entropy loss, and task performance. We find that the tuning effectiveness observed in small models generalizes well to larger models, and the validation loss in instruction tuning can be a reliable indicator of overall downstream performance.
Figures & tables
Figure 1: Mean task performance of Astraios models across 5 representative tasks and 8 datasets. We indicate the average percentage of total parameters updated for each PEFT method.
Figure 2: Percentage (%) of total parameters updated for each PEFT method in Astraios models.
Type
Name
1B
3B
7B
16B
Low-Rank
LoRA ( Hu et al., 2021 )
3,588,096
7,372,800
12,472,320
17,776,640
Prompt
P-Tuning ( Liu et al., 2023 )
12,650,496
23,882,496
50,466,816
113,448,960
Adapter
(IA) 3 ( Liu et al., 2022 )
251,904
516,096
870,912
1,239,040
Adapter H ( Houlsby et al., 2019 )
50,331,648
103,809,024
176,160,768
251,658,240
Adapter P ( Pfeiffer et al., 2020 )
25,165,824
51,904,512
88,080,384
125,829,120
Parallel ( He et al., 2021 )
26,738,688
54,263,808
90,832,896
128,450,560
Table 1: Summary of tuning methods and the trainable parameters of different model scales.
Figure 3: Final loss across model sizes.
Figure 4: Test loss of Astraios models across training time measured by Global Step .
Method
Defect Detection
Clone Detection
1B
3B
7B
16B
1B
3B
7B
16B
LoRA
44.15
44.90
49.05
31.95
9.30
12.05
14.10
8.80
P-Tuning
53.70
27.75
40.55
11.00
19.27
23.52
13.35
3.24
Adapter H
45.75
45.80
46.25
41.75
8.59
8.17
12.05
8.18
Adapter P
45.55
46.05
46.85
27.35
8.88
8.63
12.05
9.00
Parallel
34.50
33.50
52.55
42.30
9.55
8.94
10.16
17.21
Table 2: Results of Astraios models on Defect Detection and Clone Detection. The best performance is highlighted in bold . The second best performance is underlined .
Figure 7
Method
Code Synthesis
Code Repair
Code Explanation
1B
3B
7B
16B
1B
3B
7B
16B
1B
3B
7B
16B
Python
LoRA
17.26
25.37
32.01
38.08
3.29
11.16
21.74
27.50
20.49
22.53
25.34
30.52
P-Tuning
15.79
24.33
29.39
35.58
1.86
13.69
20.34
18.72
9.48
11.92
14.60
15.43
Adapter H
15.70
23.87
28.26
33.29
3.14
15.55
22.50
22.28
17.77
22.35
24.24
26.07
Adapter P
17.04
24.76
30.67
34.97
3.69
12.87
19.54
26.46
16.07
24.05
22.87
30.67
Parallel
15.98
26.65
28.81
35.88
4.91
8.11
16.13
26.43
19.70
23.14
23.93
31.10
Table 3: Pass@1 results of Astraios models on HumanEvalPack Python and Java splits. The best performance is highlighted in bold . The second best performance is underlined .
Method
Format
Function
Syntax
Docstring
1B
3B
7B
16B
1B
3B
7B
16B
1B
3B
7B
16B
1B
3B
7B
16B
Robust Pass
LoRA
28.05
35.98
43.29
51.22
12.80
15.24
23.78
29.27
8.54
13.41
15.85
18.29
10.98
15.24
17.68
20.73
P-Tuning
18.29
29.88
39.63
48.78
7.32
15.85
21.34
23.78
6.71
11.59
14.02
17.68
6.71
14.63
18.29
21.34
Adapter H
10.98
34.15
40.24
46.95
4.88
14.02
17.07
23.78
7.32
11.59
12.20
15.85
6.10
12.80
14.63
17.68
Adapter P
9.76
35.37
43.90
50.00
1.22
15.85
21.34
26.22
4.88
12.20
14.63
18.29
3.05
15.24
19.51
20.12
Parallel
26.22
32.32
42.68
50.00
10.37
11.59
21.95
26.83
7.93
12.80
14.63
17.07
8.54
15.24
17.68
21.95
Table 4: RP@1 and RC@1 results of Astraios models on ReCode. The best performance is highlighted in bold . The second best performance is underlined .
Figure 7: Mean RC@1 of Astraios on ReCode. Lower RC@1 indicates better robustness. We indicate the percentage of total parameters updated for each PEFT method.
Method
Valid% ( ↑ )
Insecure% ( ↓ )
1B
3B
7B
16B
1B
3B
7B
16B
LoRA
85.9
89.1
75.9
87.1
23.1
26.2
20.9
35.0
P-Tuning
70.1
68.6
86.8
82.0
32.8
25.9
28.1
34.5
Adapter H
84.5
90.9
87.5
86.8
29.0
26.0
31.9
34.1
Adapter P
83.9
92.1
82.8
86.3
31.7
25.2
26.6
37.8
Parallel
88.9
94.1
70.0
86.0
30.2
19.3
22.3
32.6
Table 5: Valid and Insecure rates of Astraios models on AATK benchmark. We note that the insecure rate is calculated based on the valid programs. The best performance is highlighted in bold . The second best performance is underlined .
Figure 8: Relationships between cross-entropy loss and the number of updated parameters.
Figure 9: Relationships between cross-entropy loss and overall task performance.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Name
Public Link
Base Models
StarCoderBase 1B
https://huggingface.co/bigcode/starcoderbase-1b
StarCoderBase 3B
https://huggingface.co/bigcode/starcoderbase-3b
StarCoderBase 7B
https://huggingface.co/bigcode/starcoderbase-7b
StarCoderBase
https://huggingface.co/bigcode/starcoderbase
Instruction Tuning Data
Appendix
Table 6: Used and produced artifacts.
Figure 10: Test loss of selected models across training time measured by Global Step .
Figure 11: Final loss across model sizes.
Figure 12: Pass@1 results of Astraios models on HumanEvalPack.
Figure 18
Model Size
Train Loss
Test Loss
τ
rp
rs
τ
rp
rs
1B
.4286
.3113
.6071
.3333
.3358
.4643
3B
.5238
.3433
.7143
.2381
.3835
.4286
7B
.5238
.3555
.7143
.2381
.4091
.4286
16B
.5238
.3524
.7143
.2381
.3986
.4286
Overall
.4339 (.00)
.3328 (.08)
.5616 (.00)
.3598 (.01)
.3308 (.09)
.4953 (.01)
Appendix
Table 7: Correlations between trainable parameters and final loss. Correlations are computed with the negated loss ( − loss), so a positive coefficient means that more trainable parameters are associated with lower loss. p -values are provided in gray .
Model Size
Train Loss
Test Loss
τ
rp
rs
τ
rp
rs
1B
-.2381
-.4319
-.2857
-.0476
-.4328
-.0357
3B
.5238
.7819
.7143
.8095
.7859
.9286
7B
.5238
.7165
.6786
.8095
.8230
.9286
16B
.3333
.8096
.5000
.8095
.9211
.8929
Overall
.7302 (.00)
.9027 (.00)
.9201 (.00)
.8466 (.00)
.9277 (.00)
.9579 (.00)
Appendix
Table 8: Correlations between final loss and overall task performance. Correlations are computed with the negated loss ( − loss), so a positive coefficient means that lower loss is associated with better task performance. p -values are provided in gray .
Department of Computer Science, School of Computing, Institute of Science Tokyo · National Institute of Advanced Industrial Science and Technology · Research and Development Center for Large Language Models, NII
Department of Computer Science and Engineering, POSTECH, South Korea · Graduate School of Artificial Intelligence, POSTECH, South Korea · Microsoft Research Asia, Beijing, China