Benchmarking Arabic--Russian Machine Translation: A Comparison of Fine-tuned NMT and Few-shot LLMs under Rich Morphology and Low Lexical Overlap

Mullosharaf K. Arabov

cs.CLAug 26, 2026

Benchmarking Arabic--Russian Machine Translation: A Comparison of Fine-tuned NMT and Few-shot LLMs under Rich Morphology and Low Lexical Overlap

Authors: Mullosharaf K. Arabov

Organizations: Kazan Federal University, Institute of Computational Mathematics and Information Technologies, Kazan, Russia

Abstract

Arabic-Russian machine translation (MT) remains under-explored due to the rich morphology of Arabic and low lexical overlap between the two languages. We benchmark seven fine-tuned neural machine translation (NMT) models against four few-shot large language models (LLMs) on a 20k/5k/5k split of a new 15.47M-pair corpus. Fine-tuned NLLB-1.3B achieves the highest BLEU (16.3) and COMET (0.738). Aya-Expanse 8B leads the few-shot LLMs (BLEU 1.7 on 500 sentences, chrF 25.7), but all LLM scores remain far below the fine-tuned NMT baselines. Error analysis identifies low lexical overlap as the dominant failure mode; among the worst translations, mT5-small produces 32% too-short outputs. Bootstrap tests confirm significant differences among most models. Our results demonstrate that fine-tuned NMT significantly outperforms few-shot LLMs for Arabic-Russian translation under low-resource conditions.

Figures & tables

Source	Sentence pairs
OPUS	14,924,037
TED	375,463
Religion (hadiths & Quran)	82,302
Dictionary (Baranov & Borisov)	43,662
Bible	31,102
Tatoeba	9,044

Table 1: Composition of the filtered Arabic–Russian Translation Corpus used in our experiments.

Model	Method	Training time (min)	Peak memory (GB)
Marian	full	10.0	3.7
mT5-small	full	14.7	14.1
mT5-base	LoRA	38.7	7.4
mT5-large	QLoRA	353.1	8.2
NLLB-600M	LoRA	43.0	7.8
NLLB-1.3B	QLoRA	373.5	5.6

Table 2: Training time and peak GPU memory for fine-tuned NMT models on 20k sentence pairs.

Model	BLEU	chrF	TER
Marian	15.48 $\pm$ 0.47	34.66 $\pm$ 0.59	98.2 $\pm$ 0.9
mT5-small	1.51 $\pm$ 0.07	0.24 $\pm$ 0.01	103.7 $\pm$ 0.4
mT5-base	6.54 $\pm$ 0.20	14.85 $\pm$ 0.34	106.7 $\pm$ 0.5
mT5-large	11.28 $\pm$ 0.38	26.04 $\pm$ 0.55	102.7 $\pm$ 0.4
NLLB-600M	15.38 $\pm$ 0.46	33.45 $\pm$ 0.61	93.0 $\pm$ 0.6
NLLB-1.3B	16.30 $\pm$ 0.50	34.98 $\pm$ 0.61	90.8 $\pm$ 0.6

Table 3: Fine-tuned NMT: BLEU, chrF, and TER (mean with 95% bootstrap CI). Best in each metric is bold.

Model	BERTScore	COMET
Marian	0.779 $\pm$ 0.002	0.735 $\pm$ 0.004
mT5-small	0.571 $\pm$ 0.001	0.280 $\pm$ 0.001
mT5-base	0.702 $\pm$ 0.002	0.564 $\pm$ 0.004
mT5-large	0.755 $\pm$ 0.002	0.682 $\pm$ 0.004
NLLB-600M	0.777 $\pm$ 0.002	0.726 $\pm$ 0.004
NLLB-1.3B	0.783 $\pm$ 0.002	0.738 $\pm$ 0.004

Table 4: Fine-tuned NMT: BERTScore and COMET (mean with 95% bootstrap CI). Best in each metric is bold.

Model	BLEU	chrF
Aya-Expanse 8B	1.29 $\pm$ 0.15	19.44 $\pm$ 0.53
Llama 3.1 8B	1.16 $\pm$ 0.12	21.09 $\pm$ 0.47
Qwen 2.5 7B	1.05 $\pm$ 0.13	15.33 $\pm$ 0.60
Mistral 7B	0.75 $\pm$ 0.08	13.28 $\pm$ 0.63

Table 5: Zero-shot LLM (500 test sentences): BLEU and chrF with 95% bootstrap CI. Best in each metric is bold.

Model	BERTScore	COMET
Aya-Expanse 8B	0.6457 $\pm$ 0.0025	0.6118 $\pm$ 0.0052
Llama 3.1 8B	0.6595 $\pm$ 0.0023	0.6196 $\pm$ 0.0053
Qwen 2.5 7B	0.6265 $\pm$ 0.0031	0.5275 $\pm$ 0.0085
Mistral 7B	0.5953 $\pm$ 0.0036	0.4675 $\pm$ 0.0104

Table 6: Zero-shot LLM (500 test sentences): BERTScore and COMET with 95% bootstrap CI. Best in each metric is bold.

Model	BLEU	chrF
Aya-Expanse 8B	1.69 $\pm$ 0.16	25.75 $\pm$ 0.65
Llama 3.1 8B	1.55 $\pm$ 0.12	22.02 $\pm$ 0.54
Qwen 2.5 7B	1.13 $\pm$ 0.13	17.61 $\pm$ 0.61
Mistral 7B	0.58 $\pm$ 0.05	13.72 $\pm$ 0.70

Table 7: Few-shot LLM (5-shot, 500 test sentences): BLEU and chrF with 95% bootstrap CI. Best in each metric is bold.

Model	BERTScore	COMET
Aya-Expanse 8B	0.6750 $\pm$ 0.0030	0.6535 $\pm$ 0.0061
Llama 3.1 8B	0.6622 $\pm$ 0.0031	0.6347 $\pm$ 0.0060
Qwen 2.5 7B	0.6402 $\pm$ 0.0034	0.5669 $\pm$ 0.0090
Mistral 7B	0.6080 $\pm$ 0.0036	0.4534 $\pm$ 0.0110

Table 8: Few-shot LLM (5-shot, 500 sentences): BERTScore and COMET with 95% bootstrap CI. Best in each metric is bold.

Fine-tuned NMT
Model	BLEU	chrF
Marian	14.90 $\pm$ 1.43	35.11 $\pm$ 1.78
mT5-small	1.39 $\pm$ 0.21	0.24 $\pm$ 0.06
mT5-base	6.97 $\pm$ 0.62	15.60 $\pm$ 1.11
mT5-large	11.39 $\pm$ 1.14	26.13 $\pm$ 1.60
NLLB-600M	15.76 $\pm$ 1.43	34.20 $\pm$ 1.85

Table 9: Lexical metrics (BLEU and chrF) on the shared 500-sentence test subset, with 95% bootstrap CI.

Fine-tuned NMT
Model	BERTScore	COMET
Marian	0.784 $\pm$ 0.007	0.745 $\pm$ 0.014
mT5-small	0.564 $\pm$ 0.002	0.280 $\pm$ 0.004
mT5-base	0.711 $\pm$ 0.006	0.570 $\pm$ 0.011
mT5-large	0.757 $\pm$ 0.008	0.680 $\pm$ 0.011
NLLB-600M	0.782 $\pm$ 0.007	0.731 $\pm$ 0.013

Table 10: Semantic metrics (BERTScore and COMET) on the same 500-sentence test subset, with 95% bootstrap CI.

Metric	Value
BLEU	5.57 (95% CI 5.35–5.83)
chrF	26.21 (95% CI 25.81–26.59)
BERTScore	0.6920 (95% CI 0.6896–0.6942)
COMET	0.6125 (95% CI 0.6102–0.6149)

Table 11: Aya-Expanse 8B on the full 5,000-sentence test set (5-shot).

	Marian	mT5-small	mT5-base	mT5-large	NLLB-600M	NLLB-1.3B	M2M100
Marian	–	$<$ 0.001	$<$ 0.001	$<$ 0.001	0.672	$<$ 0.001	0.004
mT5-small		–	$<$ 0.001	$<$ 0.001	$<$ 0.001	$<$ 0.001	$<$ 0.001
mT5-base			–	$<$ 0.001	$<$ 0.001	$<$ 0.001	$<$ 0.001
mT5-large				–	$<$ 0.001	$<$ 0.001	$<$ 0.001
NLLB-600M					–	$<$ 0.001	$<$ 0.001
NLLB-1.3B						–	$<$ 0.001

Table 12: Paired bootstrap test

p

-values for BLEU among fine-tuned NMT models (1,000 resamples).

Model	Short	Long	LowOv
Marian	—	16	84
mT5-small	32	—	68
mT5-base	—	4	96
mT5-large	2	8	90
NLLB-600M	2	4	94
NLLB-1.3B	2	6	92

Table 13: Error type distribution among fine-tuned NMT models (worst 50 translations per model, %). LowOv denotes low lexical overlap (character-level Jaccard similarity < 0.2).

Model	Too short (%)	Too long (%)
Marian	0.7	5.6
mT5-small	48.0	1.0
mT5-base	1.0	2.8
mT5-large	0.4	4.4
NLLB-600M	0.9	3.8
NLLB-1.3B	1.1	3.6

Table 14: Percentage of too-short and too-long translations across the full test set (5,000 sentences).

	BLEU	chrF	TER	BERTScore	COMET
BLEU	1.000	0.737	-0.626	0.763	0.660
chrF		1.000	-0.479	0.855	0.816
TER			1.000	-0.577	-0.444
BERTScore				1.000	0.843
COMET					1.000

Table 15: Spearman rank correlations between sentence-level automatic metrics (all fine-tuned NMT models aggregated).

Table 16: Translations of two Arabic sentences by fine - tuned NMT models and the best few - shot LLM.

Explore similar work

Bridging Scientific Heritage: An Arabic--Russian Parallel Corpus and LLM Benchmark for Sustainable Knowledge Transfer
Jun 29, 2026M. K. ArabovMultilingual Language Models Arabic NLP
Kazan Federal University, Institute of Computational Mathematics and Information Technologies, Department of Data Analysis, Kazan, Russia
Why do Large Language Models Fail in Low-resource Translation? Unraveling the Token Dynamics of Large Language Models for Machine Translation
May 8, 2026Shenbin Qian, Yves ScherrerMultilingual Language Model Evaluation Low-Resource MT
Language Technology Group, Department of Informatics University of Oslo, Norway
Biomedical Machine Translation for Low-Resource Arabic-Script Languages via Cross-Lingual Transfer and LoRA Adapter Merging
Jul 24, 2026Abdullah Alabdullah, Arash Eslamighayour, Sarp Harbalioglu +1Low-Resource MT Language Model Merging
School of Informatics, University of Edinburgh, The United Kingdom · Leids Universitair Medisch Centrum & LIACS, Universiteit Leiden, NL

Jun 29, 2026J/K move · Enter open · S save

Bridging Scientific Heritage: An Arabic--Russian Parallel Corpus and LLM Benchmark for Sustainable Knowledge Transfer

M. K. Arabov

Kazan Federal University, Institute of Computational Mathematics and Information Technologies, Department of Data Analysis, Kazan, Russia

Multilingual Language Models Arabic NLP Scientific Communication Low-Resource MT Machine Translation MT Evaluation

Open paper AskPDF arXiv

Russian and Arabic are among the major languages of scientific communication. Language barriers impede the exchange of research results between these communities, which affects international collaboration and the progress of sustainability-related research. We present a benchmark for Arabic--Russian scientific translation. The benchmark includes a hybrid parallel corpus of about 27,000 sentence pairs, compiled from scientific abstracts and general-domain texts (religion, news, conversations). We fine-tune three multilingual language models -- mT5-base (580M parameters), NLLB-200-distilled-1.3B (1.3B), and Qwen2.5-7B-Instruct (7B) -- using LoRA with ranks 8, 16, 32, and 64. The Qwen2.5-7B model with QLoRA (rank 8) yields BLEU 23.15, chrF 43.89, BERTScore 0.906, and COMET 0.758. These are +4.36 BLEU and +0.051 COMET above the zero-shot baseline. Few-shot prompting with three examples does not improve performance, indicating that domain-specific fine-tuning is required. We release the models, the corpus, and the evaluation code. By lowering the language barrier for scientific texts, the work enables knowledge exchange between Arabic-speaking and Russian-speaking researchers. It contributes to sustainable partnerships (UN SDG 17) and innovation infrastructure (SDG 9), aligning with the conference's focus on technology-driven sustainable development.

Abstract

Explore similar work

Bridging Scientific Heritage: An Arabic--Russian Parallel Corpus and LLM Benchmark for Sustainable Knowledge Transfer

Why do Large Language Models Fail in Low-resource Translation? Unraveling the Token Dynamics of Large Language Models for Machine Translation

Biomedical Machine Translation for Low-Resource Arabic-Script Languages via Cross-Lingual Transfer and LoRA Adapter Merging