Merge and Conquer: Instructing Multilingual Models by Adding Target Language Weights

Eneko Valero; Maria Ribalta i Albado; Oscar Sainz; Naiara Perez; German Rigau

cs.CLMar 30, 2026

Merge and Conquer: Instructing Multilingual Models by Adding Target Language Weights

Authors: Eneko Valero, Maria Ribalta i Albado, Oscar Sainz, Naiara Perez, German Rigau

Organizations: HiTZ Center - Ixa, University of the Basque Country UPV/EHU

Abstract

Large Language Models (LLMs) remain heavily centered on English, with limited performance in low-resource languages. Existing adaptation approaches, such as continual pre-training, demand significant computational resources. In the case of instructed models, high-quality instruction data is also required, both of which are often inaccessible for low-resource language communities. Under these constraints, model merging offers a lightweight alternative, but its potential in low-resource contexts has not been systematically explored. In this work, we explore whether it is possible to transfer language knowledge to an instruction-tuned LLM by merging it with a language-specific base model, thereby eliminating the need of language-specific instructions and repeated fine-tuning processes whenever stronger instructed variants become available. Through experiments covering four Iberian languages (Basque, Catalan, Galician, and Spanish) and two model families, we show that merging enables effective instruction following behavior in new languages and even supports multilingual capability through the combination of multiple language-specific models. Our results indicate that model merging is a viable and efficient alternative to traditional adaptation methods for low-resource languages, achieving competitive performance while greatly reducing computational cost.

Figures & tables

Language	Documents	Llama 3.1	Qwen3
Basque (eu)	4.2M	3.5B	3.5B
Galician (gl)	8.9M	3.5B	3.5B
Catalan (ca)	3.8M	3.7B	3.8B
Spanish (es)	3.8M	3.4B	3.5B
English (en)	0.5M	0.3B	0.3B

Table 1: Corpus sizes for each language in documents, Llama 3.1 tokens and Qwen3 tokens.

Model	EU	GL	CA	ES	EN
Llama 3.1 8B	$47.37$	$58.91$	$59.14$	$\mathbf{66.36}$	$72.80$
Llama 3.1 8B eu	$\mathbf{60.63}$	$56.19$	$56.77$	$64.37$	$\mathbf{74.85}$
Llama 3.1 8B gl	$43.94$	$\mathbf{59.59}$	$56.89$	$63.44$	$71.08$
Llama 3.1 8B ca	$44.13$	$54.64$	$\mathbf{60.91}$	$64.64$	$70.91$
Llama 3.1 8B es	$45.51$	$57.81$	$58.56$	$65.88$	$72.13$
Qwen3 8B	$51.02$	$61.80$	$62.81$	$\mathbf{68.57}$	$75.27$

Table 2: Base model language adaptation results. Bold indicates best among the same model architecture and underline indicates best overall.

	Benchmark average					Machine Translation
Model	EU	GL	CA	ES	EN	*-EU	*-GL	*-CA	*-ES	*-EN
Llama 3.1 8B joint-EU	$61.75$	$58.13$	$57.81$	$64.59$	$73.71$	$15.03$	$25.71$	$29.00$	$23.86$	$35.42$
Salamandra 2B Instruct	$27.95$	$37.11$	$43.18$	$34.68$	$37.13$	$6.69$	$25.27$	$28.41$	$21.69$	$31.50$
Salamandra 7B Instruct	$44.94$	$53.60$	$56.55$	$52.79$	$57.27$	$11.31$	$\underline{29.94}$	$\underline{34.12}$	$25.46$	$37.67$
ALIA 40B Instruct	$60.64$	$64.98$	$64.68$	$62.93$	$66.04$	$\underline{15.78}$	$29.86$	$33.10$	$\underline{26.56}$	$\underline{37.85}$
Llama 3.1 8B Instruct	$49.29$	$60.11$	$61.56$	$68.22$	$73.87$	$7.18$	$26.55$	$30.12$	$24.01$	$35.49$

Table 3: Main experimental results on multiple-choice benchmarks (Accuracy) and machine translation (BLEU). Bold indicates best among the same backbone model and underline indicates best overall.

Model	EU		GL		CA		ES
Llama 3.1 8B joint-EU	$46.82$	$2.0$	$48.36$	$2.6$	$46.71$	$0.7$	$57.06$	$0.2$
Salamandra 2B Instruct	$23.87$	$0.9$	$24.78$	$0.7$	$25.20$	$0.7$	$25.15$	$0.9$
Salamandra 7B Instruct	$32.79$	$1.5$	$35.12$	$0.7$	$34.93$	$2.8$	$33.50$	$2.0$
ALIA 40B Instruct	$35.41$	$0.3$	$38.40$	$0.7$	$41.17$	$0.4$	$43.48$	$1.6$
Llama 3.1 8B Instruct	$\mathbf{40.22}$	$1.2$	$\mathbf{58.66}$	$1.7$	$\mathbf{52.03}$	$0.7$	$\mathbf{63.83}$	$0.9$
Llama 3.1 8B merge-*	$39.98$	$1.1$	$47.86$	$1.3$	$46.96$	$1.2$	$50.08$	$0.7$

Table 4: Strict instruction-level accuracy on IFEval (mean

\pm

SD over 3 runs). For each test language, {model} merge-* denotes the corresponding monolingual variant adapted to that language.

Method	EU	GL	CA	ES	Avg
Benchmark average
Linear	$58.36$	$\mathbf{63.94}$	$\mathbf{63.99}$	$\mathbf{68.87}$	$\mathbf{63.79}$
Task Arithmetic	$61.48$	$61.67$	$63.62$	$67.61$	$63.59$
DARE	$61.23$	$61.55$	$63.70$	$66.29$	$63.19$
Breadcrumbs	$\mathbf{61.53}$	$61.22$	$63.89$	$67.54$	$63.55$
Machine translation

Table 5: Merge method comparison. Results are reported using Llama 3.1 8B merge-* , where each language is evaluated with their specialized expert.

Figure 1: Proportion sweep ablation for Llama 3.1 8B merges across Iberian languages (EU, GL, CA, ES), comparing Linear (top row) and Task Arithmetic (bottom row). Each subplot shows the effect of varying the merge proportion $w_{i}$ on Benchmark Average, Machine Translation, and IFEval Strict.

Method	EU	GL	CA	ES	Avg
Benchmark average
Linear	$55.81$	$58.84$	$\mathbf{62.66}$	$\mathbf{65.91}$	$\mathbf{60.80}$
Task Arithmetic	$55.71$	$57.26$	$58.14$	$63.55$	$58.66$
DARE	$52.39$	$55.97$	$55.42$	$61.94$	$56.43$
Breadcrumbs	$\mathbf{56.38}$	$\mathbf{59.35}$	$61.33$	$65.41$	$60.62$
Machine translation

Table 6: Merge method comparison. Results are reported using Qwen3 8B merge-* , where each language is evaluated with their specialized expert.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

	Benchmark average					Machine Translation
Model	EU	GL	CA	ES	EN	*-EU	*-GL	*-CA	*-ES	*-EN
Llama 3.1 8B joint-EU	$61.75$	$58.13$	$57.81$	$64.59$	$73.71$	$15.03$	$25.71$	$29.00$	$23.86$	$35.42$
Salamandra 2B Instruct	$27.95$	$37.11$	$43.18$	$34.68$	$37.13$	$6.69$	$25.27$	$28.41$	$21.69$	$31.50$
Salamandra 7B Instruct	$44.94$	$53.60$	$56.55$	$52.79$	$57.27$	$11.31$	$\underline{29.94}$	$\underline{34.12}$	$25.46$	$37.67$
ALIA 40B Instruct	$60.64$	$\underline{64.98}$	$\underline{64.68}$	$62.93$	$66.04$	$\underline{15.78}$	$29.86$	$33.10$	$\underline{26.56}$	$\underline{37.85}$
Llama 3.1 8B Instruct	$49.29$	$60.11$	$61.56$	$\underline{68.22}$	$73.87$	$7.18$	$26.55$	$30.12$	$24.01$	$35.49$

Appendix

Table 7: Additional results for Task Arithmetic merges on multiple-choice benchmarks (Accuracy) and machine translation (BLEU). We report baseline models together with Task Arithmetic merge variants for Llama 3.1 8B, Qwen3 8B, and Qwen3 14B. Bold indicates the best result among merged variants of the same backbone model, and underline indicates the best overall result.

	Benchmark average					Machine Translation
Model	EU	GL	CA	ES	EN	*-EU	*-GL	*-CA	*-ES	*-EN
Llama 3.1 8B joint-EU	$61.75$	$58.13$	$57.81$	$64.59$	$73.71$	$15.03$	$25.71$	$29.00$	$23.86$	$35.42$
Salamandra 2B Instruct	$27.95$	$37.11$	$43.18$	$34.68$	$37.13$	$6.69$	$25.27$	$28.41$	$21.69$	$31.50$
Salamandra 7B Instruct	$44.94$	$53.60$	$56.55$	$52.79$	$57.27$	$11.31$	$\underline{29.94}$	$\underline{34.12}$	$25.46$	$37.67$
ALIA 40B Instruct	$60.64$	$\underline{64.98}$	$64.68$	$62.93$	$66.04$	$\underline{15.78}$	$29.86$	$33.10$	$\underline{26.56}$	$\underline{37.85}$
Llama 3.1 8B Instruct	$49.29$	$60.11$	$61.56$	$68.22$	$73.87$	$7.18$	$26.55$	$30.12$	$24.01$	$35.49$

Appendix

Table 8: Additional results showing the best-performing merge configuration per language and backbone on multiple-choice benchmarks (Accuracy) and machine translation (BLEU). For each merged model, we report the strongest variant among the explored merge methods. Bold indicates the best result among merged variants of the same backbone model, and underline indicates the best overall result.

Explore similar work

Left Behind: Cross-Lingual Transfer as a Bridge for Low-Resource Languages in Large Language Models
Mar 22, 2026Abdul-Salem BeibitkhanLow-Resource Language Processing Multilingual Language Model Evaluation
North American University
Which LoRA? An Empirical Study on the Effectiveness of LoRA Techniques During Multilingual Instruction Tuning
Jun 9, 2026Thamali Wijewardhana, Napoleon H. Reyes, Surangika RanathungaLow-Rank Adaptation Cross-Lingual Transfer
School of Mathematical and Computational Sciences Massey University, Auckland, New Zealand
Lius: Translation Model Based Instructional Lingustic Using Continual Instruction Tuning In Kupang Malay
Jun 10, 2026Joanito Agili Lopo, Yunita Sari, Guntur Budi HerwantoLLM Fine-Tuning Low-Resource MT
Department of Computer Science and Electronics Universitas Gadjah Mada

Mar 22, 2026J/K move · Enter open · S save

Left Behind: Cross-Lingual Transfer as a Bridge for Low-Resource Languages in Large Language Models

Abdul-Salem Beibitkhan

North American University

Low-Resource Language Processing Multilingual Language Model Evaluation Cross-Lingual Transfer LLM Prompting

Open paper AskPDF arXiv

We investigate how large language models perform on low-resource languages by benchmarking eight LLMs across five experimental conditions in English, Kazakh, and Mongolian. Using 50 hand-crafted questions spanning factual, reasoning, technical, and culturally grounded categories, we evaluate 2,000 responses on accuracy, fluency, and completeness. We find a consistent performance gap of 13.8-16.7 percentage points between English and low-resource language conditions, with models maintaining surface-level fluency while producing significantly less accurate content. Cross-lingual transfer-prompting models to reason in English before translating back-yields selective gains for bilingual architectures (+2.2pp to +4.3pp) but provides no benefit to English-dominant models. Our results demonstrate that current LLMs systematically underserve low-resource language communities, and that effective mitigation strategies are architecture-dependent rather than universal.

Abstract

Explore similar work

Left Behind: Cross-Lingual Transfer as a Bridge for Low-Resource Languages in Large Language Models

Which LoRA? An Empirical Study on the Effectiveness of LoRA Techniques During Multilingual Instruction Tuning

Lius: Translation Model Based Instructional Lingustic Using Continual Instruction Tuning In Kupang Malay