Evaluating machine translation (MT) quality in extremely low-resource language (ELRL) scenarios poses unique challenges, as widely used metrics such as BLEU, effective in high-resource settings, often misrepresent quality in data-scarce contexts. This work presents a comparative analysis of BLEU, an n-gram-based metric, and ChrF++, a character-based metric, for MT evaluation in ELRL settings. We examine how each metric responds to translation artifacts, including hallucinations, repetition, source-text copying, and diacritic (matra) variations across three ELRLs: Magahi, Bhojpuri, and Chhattisgarhi, with a focus on outputs from large language models (LLMs) and neural MT (NMT) systems. While recent work often relies solely on ChrF++, our findings show that BLEU, despite its lower absolute scores, provides complementary lexical-precision insights that improve interpretability.
Figures & tables
Direction
ChrF ++
BLEU
Model Name
hi → bho
27.69
6.76
Aya-101
hi → bho
27.62
11.35
mT5-Large
en → bho
20.15
7.57
Airavata
en → bho
21.53
11.55
mT5-base
en → bho
23.82
19.69
mT5-Large
hi → bho
22.37
7.01
mT5-Large 2
Table 1: BLEU–ChrF++ results for English/Hindi → Bhojpuri. 2 Trained on 20% less data.
Direction
ChrF ++
BLEU
Model Name
en → mag
27.26
9.81
Airavata
en → mag
26.63
12.54
MT5-large
hi → mag
41.43
18.09
MT5-large
hi → mag
39.9
11
MT5-large 2
en → mag
43.26
35.77
Aya-101
hi → mag
41.43
18.09
MT5-large
Table 2: BLEU–ChrF++ results for target language (English/Hindi ↔ Target). 2 Trained on 20% less data.
Direction
ChrF++
BLEU
Model Name
en → hne
18.46
7.93
Airavata
hi → hne
17.55
15.14
mT5-Large
hi → hne
19.47
7.66
Aya-101
en → hne
23.83
19.59
mT5-Large 2
en → hne
24.34
29.55
mT5-Large
en → hne
18.28
3.15
Airavata 2
Table 3: BLEU–ChrF++ results for target language (English/Hindi ↔ Target). 2 Trained on 20% less data.
Case
Original (English)
Output
BLEU
ChrF++
Decrease in both ChrF++ and BLEU
Antim sanskar mein shamil hoe ke lel char million se adhik logan Rome gel rahalei.
Output 1. Char million log maut ke dauraan Roma galay.
5.1
19.96
(More than four million people had gone to Rome to attend the funeral.)
Output 2. Amrityu ke dekhkar paanch million log Roma gelei.
2.74
15.61
Stable ChrF++ with significant change in BLEU
U ego prasuti rog visheshagya ke roop me prashikshan lalakay aur 1959 me Auckland ke rashtriya mahila aspatal me kaam kare le shuru kailkay.
Output 1. U stanpanavid ke roop me shikshit holkhin aur 1959 me Aaskel ke rashtriya mahila aspatal me kary kare lagi shuru holkhin.
23.12
45.92
(He trained as an obstetrician and began working at Auckland’s National Women’s Hospital in 1959.)
Output 2. Ek rogi ke roop me shikshit holai aur 1959 me Aulakelet ke rashtriya mahila aspatal me kaam karne shuru kalai.
12.63
44.94
Increase in ChrF++ and decrease in BLEU
Pichhle mahinE ego Rashtrapati Aayog desh ke naye chunaav ke or le jaay ke upaay ke package ke tahat poorv CEP ke isteefe ke sifarish kailkay hal.
Output 1. Kai mahinE pehle ek sansadiya commission desh ke naya chunaav tak pahunche ke lel poorv CEP ke padasthapan ke prastav delkay
7.4
27.3
(Last month, a presidential commission recommended the resignation of the former CEP as part of a package of measures to lead the country toward new elections.)
Output 2. Hal hi mein Rashtrapati Samiti poorv EP ke narazgi ke desh mein naye chunaavon ke disha mein ek pack-up mein sathe upyog kare ke lel sifarish kailkay.
2.33
30.53
Table 4: Comparison of translation outputs across representative cases, with English translations provided in parentheses. Examples illustrate six typical BLEU–ChrF++ divergence patterns. Scores are computed on the original script; transliterated outputs are shown only for readability.