The landscape of extremely low-resource machine translation (MT) is characterized by perplexing variability in reported performance, often making results across different language pairs difficult to contextualize. For researchers focused on specific language groups -- such as ancient languages -- it is nearly impossible to determine if breakthroughs reported in other contexts (e.g., native African or American languages) result from superior methodologies or are merely artifacts of benchmark collection. To address this problem, we introduce the FRED Difficulty Metrics, which include the Fertility Ratio (F), Retrieval Proxy (R), Pre-training Exposure (E), and Corpus Diversity (D) and serve as dataset-intrinsic metrics to contextualize reported scores. These metrics reveal that a significant portion of result variability is explained by train-test overlap and pre-training exposure rather than model capability. Additionally, we identify that some languages -- particularly extinct and non-Latin indigenous languages -- suffer from poor tokenization coverage (high token fertility), highlighting a fundamental limitation of transferring models from high-resource languages that lack a shared vocabulary. By providing these indices alongside performance scores, we enable more transparent evaluation of cross-lingual transfer and provide a more reliable foundation for the XLR MT community.
Figures & tables
Figure 1: Performance distribution of extremely low-resource (XLR) languages in machine translation, shown as scattered outliers. The green line represents the mean performance of mBART when fine-tuned on varying sizes of training data from five high-resource languages (see Table 5 and § 2.1 for details), with the shaded region indicating ±1 standard deviation.
Metric
Notation
Interpretation
Fertility Ratio
F=Ntoken/Nchar
Tokenization efficiency; higher = harder
Retrieval Proxy
R
Upper bound via memorization; lower = harder
Pre-training Exposure
E
Overlap with pre-training corpus; lower = harder
Corpus Diversity
D
Train-test lexical similarity; lower = harder
Table 1: Summary of the four core metrics ( F , R , E , D ) quantify intrinsic task difficulty.
Lang
Ntrain (# sent)
Ntoken
F -score
E -score
D -score
R -score
Reported scores
4-gram
BLEU
BLEU
BLEU
High-resource languages (Table 7 )
avg.
10k
30.8
0.41
96.9
1.37
3.24
16.27
ja → en
10k
31.4
0.56
114
1.65
2.86
9.03
hi → en
10k
33.1
0.28
86
0.95
2.78
14.02
fi → en
10k
27.3
0.25
85
2.03
3.21
13.46
Table 2: Overview of FRED difficulty metrics to measure the data quality of different languages (translating into high-resource direction). The column of Reported scores of high-resource language group are shown here for reference and trained by us and the low-resource pairs from different languages groups are excerpted from corresponding papers (citations are listed in the table). E -score is calculated on the target side. ∗ For Americas Group, we also reported chrF++ score in parenthesis below BLEU score. For a complete data, refer to Appendix (Table 6 ).
Feature
R2 Value
Pred Strength
R -score
0.5821
Strongest
Ntoken
0.3415
Moderate
F -score ( Ntoken/Nchar )
0.2248
Low-Moderate
D -score
0.1204
Low
E -score
0.0142
Negligible
Ntrain
0.0011
None
Table 3: Individual R2 for Hybrid Regression Model (BLEU/ChrF/chrF++).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Venue
Language
Region/Period
Reference
WMT
Multiple
Global
Kocmi et al. (2023)
AfricaNLP
Multiple
African
George et al. (2024)
AmericasNLP
Multiple
Latin America
Ebrahimi et al. (2024)
WAT/WMT
Indic
South Asia
Nakazawa et al. (2023) ; Pal et al. (2023) ; Zhou et al. (2025)
ML4AL
Ancient Egyptian
Ancient
De Cao et al. (2024)
CL Conf
Akkadian
Ancient
Chen et al. (2024)
Appendix
Table 4: Survey of venues and publications on XLR languages from the past three years.
Lang
Writing
Phonography
Dataset
BLEU / ChrF at Different Data Sizes
1k
10k
100k
1M
fi
Latin
Alphabetic
wmt18-fi-en
6.32 / 28.87
11.97 / 38.82
18.31 / 45.39
18.47 / 46.98
zh
Han
Logographic
wmt18-zh-en
6.51 / 30.05
10.77 / 38.10
15.42 / 44.67
17.59 / 47.34
ar
Arabic
Abjad
iwslt2017-ar-en
7.20 / 24.49
18.94 / 38.90
29.61 / 50.10
36.31 / 56.03
ja
Han/Kana
Moraic
iwslt2017-ja-en
3.08 / 18.97
7.17 / 31.37
11.24 / 37.49
–
hi
Devanagari
Abugida
IITB-hi-en
1.42 / 15.88
4.40 / 26.47
7.71 / 39.17
8.11 / 45.66
Appendix
Table 5: High-resource reference baselines (xx → en). BLEU / ChrF scores with capped training data (number of training sentences). Languages are selected from varied language families and writing systems to represent diverse morphological and orthographic challenges.
Lang
Parallel (# sent)
Monolingual
Ntoken
Nchar
NcharNtoken
D -score
R -score
PBSMT
E -score
Model
BLEU / ChrF
BLEU / ChrF
BLEU / ChrF
4-gram
BLEU / ChrF
High-resource languages (Table 7 )
ja → en
10k
>10M
31.4
56.3
0.56
1.65 / 5.01
2.86 / 18.38
4.49 / 21.52
114.47
9.03 / 31.37
hi → en
10k
>10M
33.1
117.2
0.28
0.95 / 10.08
2.78 / 17.34
2.36 / 11.20
85.98
14.02 / 26.47
fi → en
10k
>10M
27.3
109.8
0.25
2.03 / 16.44
3.21 / 17.16
4.63 / 24.01
85.46
13.46 / 38.82
zh → en
10k
>10M
38.0
59.0
0.66
0.78 / 1.60
2.38 / 17.46
4.18 / 24.87
90.17
13.32 / 38.10
Appendix
Table 6: Overview of automatic metrics to measure the data quality of different languages on both low-to-high and high-to-low directions. The high-resource language (ja, hi, fi, zh, ar) are shown here for reference. Ntoken , Nchar and NcharNtoken are calculated on the test dataset of the source side, TTR is calculated on the train dataset of the target side, and E -score is calculated on the test dataset on the target side. For Americas languages, ChrF++ scores are reported instead of ChrF in Model column.
Lang
N-train
D-chrF
R-chrF
PBSMT-chrF
Japanese (ja)
10000
5.01
18.38
21.52
Hindi (hi)
10000
10.08
17.34
11.20
Finnish (fi)
10000
16.44
17.16
24.01
Chinese (zh)
10000
1.60
17.46
24.87
Arabic (ar)
10000
11.36
18.56
22.71
Lang
D-chrF++
R-chrF++
PBSMT-chrF++
Appendix
Table 7: Translation Performance Metrics for high-resource language pairs – chrF as similarity function.
Lang
N-train
D-chrF
R-chrF
PBSMT-chrF
Akkadian (akk)
50000
2.61
46.18
43.25
Egyptian (egy)
10000
9.22
41.95
13.86
Lang
D-chrF++
R-chrF++
PBSMT-chrF++
Akkadian (akk)
2.21
45.10
41.48
Egyptian (egy)
9.02
41.34
12.12
Appendix
Table 8: Translation Performance Metrics – chrF and chrF++ for Ancient language pairs.
Lang
N-train
D-BLEU
R-BLEU
PBSMT-BLEU
E-4-gram
BLEU
Sakizaya (ais)
4590
6.09
12.05
9.17
0
3.11
Amis (ami)
4600
5.51
12.59
8.97
0
3.56
Bunun (bnn)
7180
6.21
15.7
13.97
0.002
5.44
Kavalan (ckv)
6573
5.04
17.61
19.81
0
7.18
Rukai (dru)
8319
5.82
10.48
11.39
0
8.44
Paiwan (pwn)
4126
5.24
9.93
9.77
0.001
3.80
Appendix
Table 9: Translation Performance Metrics for Formosan-Chinese (Mandarin) language pairs. Exposure scores are calculated by counting on the high-resource side of the language pairs. The BLEU and chrF scores are taken from Zheng et al. (2024) .
lang
N-train
D-BLEU
R-BLEU
PBSMT-BLEU
E-4-gram
BLEU
ashaninka (cni)
3883
3.98
4.94
6.36
0.62
2.35
awajun (agr)
21964
1.23
5.50
5.75
0.87
11.12
aymara (aym)
6531
1.74
3.51
5.17
0.65
8.82
bribri (bzd)
7508
2.01
4.33
6.01
0.65
4.31
chatino (ctp)
357
1.50
9.63
4.69
7.62
-
guarani (gn)
26032
1.89
4.51
6.29
0.65
8.62
Appendix
Table 10: Translation Performance Metrics for Americas Indigenous Languages - Spanish Language Pairs from 2025’s shared tasks. We use dev set as the test set for the evaluation of this corpus since the test set of the dataset this year is not publicly released. Exposure scores are calculated by counting on the high-resource side of the language pairs. The BLEU and chrF++ scores are taken from the dev set performance of the team Syntax Squad in 2025’s competition Yahan and Islam (2025) .
Lang
N-train
D-BLEU
R-BLEU
PBSMT-BLEU
E-4-gram
BLEU
Translate into English
Amharic (amh)
899
0.29
2.80
4.42
109.65
-
Hausa (hau)
3098
1.41
6.28
6.09
146.42
12.9
Igbo (ibo)
6998
1.14
5.47
7.76
86.69
21.0
Kinyarwanda (kin)
460
1.24
2.95
4.24
94.52
-
Luganda (lug)
4075
1.96
4.96
5.36
84.58
19.8
Appendix
Table 11: Translation Performance Metrics for African language pairs. Exposure scores are calculated by counting on the high-resource side of the language pairs. BLEU and chrF scores are taken from the best possible BLEU scores from mBART/mT5/T5 of table 4 of the paper Adelani et al. (2022b) .
Lang
N-train
D-BLEU
R-BLEU
PBSMT-BLEU
E-4-gram
BLEU
Assamese (as)
50000
1.44
8.19
6.03
484.92
66.36
Mizo (lus)
50000
2.08
15.77
11.10
1141.87
33.30
Manipuri (mni)
21687
1.24
19.91
14.85
330.78
69.75
Khasi (kha)
24000
2.00
4.43
7.67
727.82
20.72
Lang
D-chrF
R-chrF
PBSMT-chrF
Assamese (as)
11.59
24.89
26.30
75.88
Appendix
Table 12: Translation Performance Metrics for Indic language pairs. Exposure scores are calculated by counting on the high-resource side of the language pairs. BLEU and chrF scores are taken from the best performance in WMT 2023 shared task Pal et al. (2023) .
Lang
BLEU / chrFzh only
BLEU / chrFBoth side
Sakizaya (ais)
9.17 / 8.17
6.78 / 7.43
Amis (ami)
8.97 / 6.38
6.73 / 5.96
Bunun (bnn)
13.97 / 10.60
12.36 / 11.69
Kavalan (ckv)
19.81 / 16.03
16.26 / 14.79
Rukai (dru)
11.39 / 7.79
10.11 / 9.08
Paiwan (pwn)
9.77 / 9.82
9.02 / 9.82
Appendix
Table 13: PBSMT Scores for Formosan language pairs: translate to high-resource (zh) direction. The two columns shows the performance difference of using MBart as tokenizer on Chinese side only and using MBart on both sides.
Figure 2: Scores of PBSMT versus R-BLEU by language groups.
Evaluating machine translation (MT) quality in extremely low-resource language (ELRL) scenarios poses unique challenges, as widely used metrics such as BLEU, effective in high-resource settings, often misrepresent quality in data-scarce contexts. This work presents a comparative analysis of BLEU, an n-gram-based metric, and ChrF++, a character-based metric, for MT evaluation in ELRL settings. We examine how each metric responds to translation artifacts, including hallucinations, repetition, source-text copying, and diacritic (matra) variations across three ELRLs: Magahi, Bhojpuri, and Chhattisgarhi, with a focus on outputs from large language models (LLMs) and neural MT (NMT) systems. While recent work often relies solely on ChrF++, our findings show that BLEU, despite its lower absolute scores, provides complementary lexical-precision insights that improve interpretability.
Cross-lingual transfer is central to low-resource machine translation, but its behavior within closely related language families remains insufficiently characterized. We study transfer among five Turkic languages; Turkish, Azerbaijani, Uzbek, Kazakh, and Kyrgyz; using pairwise transfer matrices. In this setting, each model is fine-tuned with one transfer source and evaluated on a different transfer target while the translation target remains the same. Across mT5 experiments, we find that transfer is strongest between closely related Turkic pairs, especially Turkish-Azerbaijani and Kazakh-Kyrgyz. We also show that transfer direction matters, and that the same transfer source-transfer target pair can behave differently when the translation target changes. Latinization improves BLEU and chrF in several script-mismatched settings, but its effect is not uniform across metrics. Additional analyses show that transfer sources are mostly stable across different datasets and model settings.
Omer Burak Cinar, Mehmet Mert Dalkilic, Cagri Toraman
Middle East Technical University · Computer Engineering Department
Reliable Text Difficulty Assessment is a prerequisite for valid text simplification workflows and personalized learning applications. However, the development of robust assessment models is severely hindered by a critical bottleneck: the scarcity of expert-annotated corpora containing fine-grained difficulty levels (e.g., CEFR), particularly for lower-resource languages. This paper addresses this data scarcity problem in the context of a low-resource European language. We propose a cross-lingual data augmentation strategy that leverages machine translation to transfer labeled resources from high-resource languages to the target low-resource language. We train BERT-based regression models to predict difficulty scores and investigate whether synthetic, translated data can effectively supplement native training sets. Our experiments demonstrate that augmenting scarce native data with machine-translated corpora significantly improves the accuracy of difficulty estimation, offering a viable solution for languages lacking extensive expert annotations.