The high cost of full-parameter fine-tuning (FFT) of Large Language Models (LLMs) has led to a series of parameter-efficient fine-tuning (PEFT) methods. However, it remains unclear which methods provide the best cost-performance trade-off at different model scales. We introduce Astraios, a suite of 28 instruction-tuned OctoCoder models using 7 tuning methods and 4 model sizes up to 16 billion parameters. Through investigations across 5 tasks and 8 different datasets encompassing both code comprehension and code generation tasks, we find that FFT generally leads to the best downstream performance across all scales, and PEFT methods differ significantly in their efficacy based on the model scale. LoRA usually offers the most favorable trade-off between cost and performance. Further investigation into the effects of these methods on both model robustness and code security reveals that larger models tend to demonstrate reduced robustness and less security. At last, we explore the relationships among updated parameters, cross-entropy loss, and task performance. We find that the tuning effectiveness observed in small models generalizes well to larger models, and the validation loss in instruction tuning can be a reliable indicator of overall downstream performance.
Figures & tables
Figure 1: Mean task performance of Astraios models across 5 representative tasks and 8 datasets. We indicate the average percentage of total parameters updated for each PEFT method.
Figure 2: Percentage (%) of total parameters updated for each PEFT method in Astraios models.
Type
Name
1B
3B
7B
16B
Low-Rank
LoRA ( Hu et al., 2021 )
3,588,096
7,372,800
12,472,320
17,776,640
Prompt
P-Tuning ( Liu et al., 2023 )
12,650,496
23,882,496
50,466,816
113,448,960
Adapter
(IA) 3 ( Liu et al., 2022 )
251,904
516,096
870,912
1,239,040
Adapter H ( Houlsby et al., 2019 )
50,331,648
103,809,024
176,160,768
251,658,240
Adapter P ( Pfeiffer et al., 2020 )
25,165,824
51,904,512
88,080,384
125,829,120
Parallel ( He et al., 2021 )
26,738,688
54,263,808
90,832,896
128,450,560
Table 1: Summary of tuning methods and the trainable parameters of different model scales.
Figure 3: Final loss across model sizes.
Figure 4: Test loss of Astraios models across training time measured by Global Step .
Method
Defect Detection
Clone Detection
1B
3B
7B
16B
1B
3B
7B
16B
LoRA
44.15
44.90
49.05
31.95
9.30
12.05
14.10
8.80
P-Tuning
53.70
27.75
40.55
11.00
19.27
23.52
13.35
3.24
Adapter H
45.75
45.80
46.25
41.75
8.59
8.17
12.05
8.18
Adapter P
45.55
46.05
46.85
27.35
8.88
8.63
12.05
9.00
Parallel
34.50
33.50
52.55
42.30
9.55
8.94
10.16
17.21
Table 2: Results of Astraios models on Defect Detection and Clone Detection. The best performance is highlighted in bold . The second best performance is underlined .
Figure 7
Method
Code Synthesis
Code Repair
Code Explanation
1B
3B
7B
16B
1B
3B
7B
16B
1B
3B
7B
16B
Python
LoRA
17.26
25.37
32.01
38.08
3.29
11.16
21.74
27.50
20.49
22.53
25.34
30.52
P-Tuning
15.79
24.33
29.39
35.58
1.86
13.69
20.34
18.72
9.48
11.92
14.60
15.43
Adapter H
15.70
23.87
28.26
33.29
3.14
15.55
22.50
22.28
17.77
22.35
24.24
26.07
Adapter P
17.04
24.76
30.67
34.97
3.69
12.87
19.54
26.46
16.07
24.05
22.87
30.67
Parallel
15.98
26.65
28.81
35.88
4.91
8.11
16.13
26.43
19.70
23.14
23.93
31.10
Table 3: Pass@1 results of Astraios models on HumanEvalPack Python and Java splits. The best performance is highlighted in bold . The second best performance is underlined .
Method
Format
Function
Syntax
Docstring
1B
3B
7B
16B
1B
3B
7B
16B
1B
3B
7B
16B
1B
3B
7B
16B
Robust Pass
LoRA
28.05
35.98
43.29
51.22
12.80
15.24
23.78
29.27
8.54
13.41
15.85
18.29
10.98
15.24
17.68
20.73
P-Tuning
18.29
29.88
39.63
48.78
7.32
15.85
21.34
23.78
6.71
11.59
14.02
17.68
6.71
14.63
18.29
21.34
Adapter H
10.98
34.15
40.24
46.95
4.88
14.02
17.07
23.78
7.32
11.59
12.20
15.85
6.10
12.80
14.63
17.68
Adapter P
9.76
35.37
43.90
50.00
1.22
15.85
21.34
26.22
4.88
12.20
14.63
18.29
3.05
15.24
19.51
20.12
Parallel
26.22
32.32
42.68
50.00
10.37
11.59
21.95
26.83
7.93
12.80
14.63
17.07
8.54
15.24
17.68
21.95
Table 4: RP@1 and RC@1 results of Astraios models on ReCode. The best performance is highlighted in bold . The second best performance is underlined .
Figure 7: Mean RC@1 of Astraios on ReCode. Lower RC@1 indicates better robustness. We indicate the percentage of total parameters updated for each PEFT method.
Method
Valid% ( ↑ )
Insecure% ( ↓ )
1B
3B
7B
16B
1B
3B
7B
16B
LoRA
85.9
89.1
75.9
87.1
23.1
26.2
20.9
35.0
P-Tuning
70.1
68.6
86.8
82.0
32.8
25.9
28.1
34.5
Adapter H
84.5
90.9
87.5
86.8
29.0
26.0
31.9
34.1
Adapter P
83.9
92.1
82.8
86.3
31.7
25.2
26.6
37.8
Parallel
88.9
94.1
70.0
86.0
30.2
19.3
22.3
32.6
Table 5: Valid and Insecure rates of Astraios models on AATK benchmark. We note that the insecure rate is calculated based on the valid programs. The best performance is highlighted in bold . The second best performance is underlined .
Figure 8: Relationships between cross-entropy loss and the number of updated parameters.
Figure 9: Relationships between cross-entropy loss and overall task performance.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Name
Public Link
Base Models
StarCoderBase 1B
https://huggingface.co/bigcode/starcoderbase-1b
StarCoderBase 3B
https://huggingface.co/bigcode/starcoderbase-3b
StarCoderBase 7B
https://huggingface.co/bigcode/starcoderbase-7b
StarCoderBase
https://huggingface.co/bigcode/starcoderbase
Instruction Tuning Data
Appendix
Table 6: Used and produced artifacts.
Figure 10: Test loss of selected models across training time measured by Global Step .
Figure 11: Final loss across model sizes.
Figure 12: Pass@1 results of Astraios models on HumanEvalPack.
Figure 18
Model Size
Train Loss
Test Loss
τ
rp
rs
τ
rp
rs
1B
.4286
.3113
.6071
.3333
.3358
.4643
3B
.5238
.3433
.7143
.2381
.3835
.4286
7B
.5238
.3555
.7143
.2381
.4091
.4286
16B
.5238
.3524
.7143
.2381
.3986
.4286
Overall
.4339 (.00)
.3328 (.08)
.5616 (.00)
.3598 (.01)
.3308 (.09)
.4953 (.01)
Appendix
Table 7: Correlations between trainable parameters and final loss. Correlations are computed with the negated loss ( − loss), so a positive coefficient means that more trainable parameters are associated with lower loss. p -values are provided in gray .
Model Size
Train Loss
Test Loss
τ
rp
rs
τ
rp
rs
1B
-.2381
-.4319
-.2857
-.0476
-.4328
-.0357
3B
.5238
.7819
.7143
.8095
.7859
.9286
7B
.5238
.7165
.6786
.8095
.8230
.9286
16B
.3333
.8096
.5000
.8095
.9211
.8929
Overall
.7302 (.00)
.9027 (.00)
.9201 (.00)
.8466 (.00)
.9277 (.00)
.9579 (.00)
Appendix
Table 8: Correlations between final loss and overall task performance. Correlations are computed with the negated loss ( − loss), so a positive coefficient means that lower loss is associated with better task performance. p -values are provided in gray .
AI coding assistants have significantly improved developer productivity by automatically suggesting code that aligns with user intent, and many of these tools are now integrated directly into Integrated Development Environments (IDEs). Developers interact with code in two distinct cognitive modes: Flow and Command. While developers require tools that directly complete or infill code in unfinished programs during Flow mode, they also need tools that can comprehend intentions expressed as natural-language instructions and convert them into executable code in Command mode. Although instruction-tuned Large Language Models (LLMs) dominate many application scenarios due to their abilities to infer and fulfill developers' intents, it remains unclear whether the same paradigm is equally suitable for different code-related tasks. Therefore, it is necessary to understand how instruction tuning affects the feasibility of CodeLLMs as coding assistants. To fill this gap, we conduct the first empirical study that uncovers a key trade-off caused by instruction tuning across programming modes, which we term the Instruction-Tuning Tax. Our results show that instruction tuning is not a free lunch: although instruction-tuned models are more capable of following instructions and leveraging structured guidance, these gains often come at the cost of weaker infilling performance. We further extend our study through both qualitative and quantitative analyses, including manual failure categorization, behavioral metrics that capture generation fidelity, and intermediate-checkpoint evaluation throughout the tuning process. Summarizing our results into seven findings and four implications, our study offers a new perspective on the development of AI-powered coding tools and highlights the need to carefully balance instruction-following ability with effective code generation assistance.
Using responses generated by high-performing large language models (LLMs) for instruction tuning has become a widely adopted approach. However, the existing literature overlooks a property of LLM-generated responses: they conflate world knowledge acquired during pre-training with instruction-following capabilities acquired during post-training. We hypothesize that disentangling the instruction-following capabilities from pre-trained knowledge improves the effectiveness of instruction tuning. To this end, we propose CoDIT, a method that applies contrastive decoding between a post-trained model and its pre-trained counterpart during response generation. The method suppresses pre-trained knowledge shared between the two models while amplifying the instruction-following behavior acquired via post-training, resulting in responses that more purely reflect instruction-following capabilities. Experiment results demonstrate that models trained on datasets constructed via CoDIT consistently outperform those trained on directly generated responses. Training on our datasets also yields better performance than on existing publicly available instruction-tuning datasets across multiple benchmarks. Furthermore, we theoretically and empirically show that CoDIT can be interpreted as distilling the chat vector from parameter space to text space, enabling the transfer of instruction-tuning capabilities across models of different architectures.
Tatsuya Ichinose, Youmi Ma, Masanari Oi +2
Department of Computer Science, School of Computing, Institute of Science Tokyo · National Institute of Advanced Industrial Science and Technology · Research and Development Center for Large Language Models, NII
Beyond training-time optimization, scaling test-time computation has emerged as a key paradigm to extend the reasoning capabilities of Large Language Models (LLMs). However, most existing methods adopt a rigid Planning-before-Trial (PbT) policy, which inefficiently allocates test-time compute by incurring planning overhead even on directly solvable problems. We propose Planning-after-Trial (PaT), an adaptive policy for code generation that invokes a planner only upon verification failure. This adaptive policy naturally enables a heterogeneous model configuration: a cost-efficient model handles generation attempts, while a powerful model is reserved for targeted planning interventions. Empirically, across multiple benchmarks and model families, our approach significantly advances the cost-performance Pareto frontier. Notably, our heterogeneous configuration achieves performance comparable to a large homogeneous model while reducing inference cost by approximately 69%.
Youngsik Yoon, Sungjae Lee, Seockbean Song +3
Department of Computer Science and Engineering, POSTECH, South Korea · Graduate School of Artificial Intelligence, POSTECH, South Korea · Microsoft Research Asia, Beijing, China