Large Language Models (LLMs) remain heavily centered on English, with limited performance in low-resource languages. Existing adaptation approaches, such as continual pre-training, demand significant computational resources. In the case of instructed models, high-quality instruction data is also required, both of which are often inaccessible for low-resource language communities. Under these constraints, model merging offers a lightweight alternative, but its potential in low-resource contexts has not been systematically explored. In this work, we explore whether it is possible to transfer language knowledge to an instruction-tuned LLM by merging it with a language-specific base model, thereby eliminating the need of language-specific instructions and repeated fine-tuning processes whenever stronger instructed variants become available. Through experiments covering four Iberian languages (Basque, Catalan, Galician, and Spanish) and two model families, we show that merging enables effective instruction following behavior in new languages and even supports multilingual capability through the combination of multiple language-specific models. Our results indicate that model merging is a viable and efficient alternative to traditional adaptation methods for low-resource languages, achieving competitive performance while greatly reducing computational cost.
Figures & tables
Language
Documents
Llama 3.1
Qwen3
Basque (eu)
4.2M
3.5B
3.5B
Galician (gl)
8.9M
3.5B
3.5B
Catalan (ca)
3.8M
3.7B
3.8B
Spanish (es)
3.8M
3.4B
3.5B
English (en)
0.5M
0.3B
0.3B
Table 1: Corpus sizes for each language in documents, Llama 3.1 tokens and Qwen3 tokens.
Model
EU
GL
CA
ES
EN
Llama 3.1 8B
47.37
58.91
59.14
66.36
72.80
Llama 3.1 8B eu
60.63
56.19
56.77
64.37
74.85
Llama 3.1 8B gl
43.94
59.59
56.89
63.44
71.08
Llama 3.1 8B ca
44.13
54.64
60.91
64.64
70.91
Llama 3.1 8B es
45.51
57.81
58.56
65.88
72.13
Qwen3 8B
51.02
61.80
62.81
68.57
75.27
Table 2: Base model language adaptation results. Bold indicates best among the same model architecture and underline indicates best overall.
Benchmark average
Machine Translation
Model
EU
GL
CA
ES
EN
*-EU
*-GL
*-CA
*-ES
*-EN
Llama 3.1 8B joint-EU
61.75
58.13
57.81
64.59
73.71
15.03
25.71
29.00
23.86
35.42
Salamandra 2B Instruct
27.95
37.11
43.18
34.68
37.13
6.69
25.27
28.41
21.69
31.50
Salamandra 7B Instruct
44.94
53.60
56.55
52.79
57.27
11.31
29.94
34.12
25.46
37.67
ALIA 40B Instruct
60.64
64.98
64.68
62.93
66.04
15.78
29.86
33.10
26.56
37.85
Llama 3.1 8B Instruct
49.29
60.11
61.56
68.22
73.87
7.18
26.55
30.12
24.01
35.49
Table 3: Main experimental results on multiple-choice benchmarks (Accuracy) and machine translation (BLEU). Bold indicates best among the same backbone model and underline indicates best overall.
Model
EU
GL
CA
ES
Llama 3.1 8B joint-EU
46.82
2.0
48.36
2.6
46.71
0.7
57.06
0.2
Salamandra 2B Instruct
23.87
0.9
24.78
0.7
25.20
0.7
25.15
0.9
Salamandra 7B Instruct
32.79
1.5
35.12
0.7
34.93
2.8
33.50
2.0
ALIA 40B Instruct
35.41
0.3
38.40
0.7
41.17
0.4
43.48
1.6
Llama 3.1 8B Instruct
40.22
1.2
58.66
1.7
52.03
0.7
63.83
0.9
Llama 3.1 8B merge-*
39.98
1.1
47.86
1.3
46.96
1.2
50.08
0.7
Table 4: Strict instruction-level accuracy on IFEval (mean ± SD over 3 runs). For each test language, {model} merge-* denotes the corresponding monolingual variant adapted to that language.
Method
EU
GL
CA
ES
Avg
Benchmark average
Linear
58.36
63.94
63.99
68.87
63.79
Task Arithmetic
61.48
61.67
63.62
67.61
63.59
DARE
61.23
61.55
63.70
66.29
63.19
Breadcrumbs
61.53
61.22
63.89
67.54
63.55
Machine translation
Table 5: Merge method comparison. Results are reported using Llama 3.1 8B merge-* , where each language is evaluated with their specialized expert.
Figure 1: Proportion sweep ablation for Llama 3.1 8B merges across Iberian languages (EU, GL, CA, ES), comparing Linear (top row) and Task Arithmetic (bottom row). Each subplot shows the effect of varying the merge proportion wi on Benchmark Average, Machine Translation, and IFEval Strict.
Method
EU
GL
CA
ES
Avg
Benchmark average
Linear
55.81
58.84
62.66
65.91
60.80
Task Arithmetic
55.71
57.26
58.14
63.55
58.66
DARE
52.39
55.97
55.42
61.94
56.43
Breadcrumbs
56.38
59.35
61.33
65.41
60.62
Machine translation
Table 6: Merge method comparison. Results are reported using Qwen3 8B merge-* , where each language is evaluated with their specialized expert.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark average
Machine Translation
Model
EU
GL
CA
ES
EN
*-EU
*-GL
*-CA
*-ES
*-EN
Llama 3.1 8B joint-EU
61.75
58.13
57.81
64.59
73.71
15.03
25.71
29.00
23.86
35.42
Salamandra 2B Instruct
27.95
37.11
43.18
34.68
37.13
6.69
25.27
28.41
21.69
31.50
Salamandra 7B Instruct
44.94
53.60
56.55
52.79
57.27
11.31
29.94
34.12
25.46
37.67
ALIA 40B Instruct
60.64
64.98
64.68
62.93
66.04
15.78
29.86
33.10
26.56
37.85
Llama 3.1 8B Instruct
49.29
60.11
61.56
68.22
73.87
7.18
26.55
30.12
24.01
35.49
Appendix
Table 7: Additional results for Task Arithmetic merges on multiple-choice benchmarks (Accuracy) and machine translation (BLEU). We report baseline models together with Task Arithmetic merge variants for Llama 3.1 8B, Qwen3 8B, and Qwen3 14B. Bold indicates the best result among merged variants of the same backbone model, and underline indicates the best overall result.
Benchmark average
Machine Translation
Model
EU
GL
CA
ES
EN
*-EU
*-GL
*-CA
*-ES
*-EN
Llama 3.1 8B joint-EU
61.75
58.13
57.81
64.59
73.71
15.03
25.71
29.00
23.86
35.42
Salamandra 2B Instruct
27.95
37.11
43.18
34.68
37.13
6.69
25.27
28.41
21.69
31.50
Salamandra 7B Instruct
44.94
53.60
56.55
52.79
57.27
11.31
29.94
34.12
25.46
37.67
ALIA 40B Instruct
60.64
64.98
64.68
62.93
66.04
15.78
29.86
33.10
26.56
37.85
Llama 3.1 8B Instruct
49.29
60.11
61.56
68.22
73.87
7.18
26.55
30.12
24.01
35.49
Appendix
Table 8: Additional results showing the best-performing merge configuration per language and backbone on multiple-choice benchmarks (Accuracy) and machine translation (BLEU). For each merged model, we report the strongest variant among the explored merge methods. Bold indicates the best result among merged variants of the same backbone model, and underline indicates the best overall result.
We investigate how large language models perform on low-resource languages by benchmarking eight LLMs across five experimental conditions in English, Kazakh, and Mongolian. Using 50 hand-crafted questions spanning factual, reasoning, technical, and culturally grounded categories, we evaluate 2,000 responses on accuracy, fluency, and completeness. We find a consistent performance gap of 13.8-16.7 percentage points between English and low-resource language conditions, with models maintaining surface-level fluency while producing significantly less accurate content. Cross-lingual transfer-prompting models to reason in English before translating back-yields selective gains for bilingual architectures (+2.2pp to +4.3pp) but provides no benefit to English-dominant models. Our results demonstrate that current LLMs systematically underserve low-resource language communities, and that effective mitigation strategies are architecture-dependent rather than universal.
We investigate whether commonly available LoRA variants have an advantage over basic LoRA in multilingual instruction tuning. Experiments involving LoRA and four other variants on two datasets across diverse target languages show that there is no significant advantage in using more complex LoRA variants instead of basic LoRA, with respect to balancing cross-lingual transfer and knowledge retention. An analysis of hidden embeddings reveal that layer-wise language representation remains largely similar across LLMs fine-tuned with different LoRA techniques, suggesting that architectural novelty of LoRA techniques may not translate into better cross-lingual adaptation.
Thamali Wijewardhana, Napoleon H. Reyes, Surangika Ranathunga
School of Mathematical and Computational Sciences Massey University, Auckland, New Zealand
Large Language Models (LLMs) offer new potential for translation tasks but often experience performance degradation when handling low-resource languages. To address this limitation, we propose an approach for fine-tuning LLMs on a low-resource language, Kupang Malay. Our approach involves designing a set of instructions by leveraging explicit lexical and semantic features from a bilingual dictionary, and introducing Continual Instruction Tuning (CIT), a training paradigm that enables iterative instruction-based training. Experimental results demonstrate that our model, named Lius, yields notable improvements over standard instruction-tuned models by outperforming 4-6 points, and surpassing both Neural Machine Translation (NMT) and Multilingual LLM models by 10-13 points on several evaluation metrics. These findings highlight the potential of our approach to mitigate the reliance on large-scale parallel data in low-resource language translation.