Translation or Recitation? Calibrating Evaluation Scores for Machine Translation of Extremely Low-Resource Languages

Danlu Chen; Ka Sing He; Jiahe Tian; Chenghao Xiao; Zhaofeng Wu; Taylor Berg-Kirkpatrick; Freda Shi

cs.CLMar 26, 2026

Translation or Recitation? Calibrating Evaluation Scores for Machine Translation of Extremely Low-Resource Languages

Authors: Danlu Chen, Ka Sing He, Jiahe Tian, Chenghao Xiao, Zhaofeng Wu, Taylor Berg-Kirkpatrick, Freda Shi

Organizations: UC San Diego · New York University · Durham University · MIT · University of Waterloo · Vector Institute

Abstract

The landscape of extremely low-resource machine translation (MT) is characterized by perplexing variability in reported performance, often making results across different language pairs difficult to contextualize. For researchers focused on specific language groups -- such as ancient languages -- it is nearly impossible to determine if breakthroughs reported in other contexts (e.g., native African or American languages) result from superior methodologies or are merely artifacts of benchmark collection. To address this problem, we introduce the FRED Difficulty Metrics, which include the Fertility Ratio (F), Retrieval Proxy (R), Pre-training Exposure (E), and Corpus Diversity (D) and serve as dataset-intrinsic metrics to contextualize reported scores. These metrics reveal that a significant portion of result variability is explained by train-test overlap and pre-training exposure rather than model capability. Additionally, we identify that some languages -- particularly extinct and non-Latin indigenous languages -- suffer from poor tokenization coverage (high token fertility), highlighting a fundamental limitation of transferring models from high-resource languages that lack a shared vocabulary. By providing these indices alongside performance scores, we enable more transparent evaluation of cross-lingual transfer and provide a more reliable foundation for the XLR MT community.

Figures & tables

Figure 1: Performance distribution of extremely low-resource (XLR) languages in machine translation, shown as scattered outliers. The green line represents the mean performance of mBART when fine-tuned on varying sizes of training data from five high-resource languages (see Table 5 and § 2.1 for details), with the shaded region indicating

\pm 1

standard deviation.

Metric	Notation	Interpretation
Fertility Ratio	$F=N_{\text{token}}/N_{\text{char}}$	Tokenization efficiency; higher = harder
Retrieval Proxy	$R$	Upper bound via memorization; lower = harder
Pre-training Exposure	$E$	Overlap with pre-training corpus; lower = harder
Corpus Diversity	$D$	Train-test lexical similarity; lower = harder

Table 1: Summary of the four core metrics (

F

R

E

D

) quantify intrinsic task difficulty.

Lang	$N_{\text{train}}$ (# sent)	$N_{\text{token}}$	$F$ -score	$E$ -score	$D$ -score	$R$ -score	Reported scores
Lang	$N_{\text{train}}$ (# sent)	$N_{\text{token}}$	$F$ -score	4-gram	BLEU	BLEU	BLEU
High-resource languages (Table 7 )
avg.	10k	30.8	0.41	96.9	1.37	3.24	16.27
ja $\rightarrow$ en	10k	31.4	0.56	114	1.65	2.86	9.03
hi $\rightarrow$ en	10k	33.1	0.28	86	0.95	2.78	14.02
fi $\rightarrow$ en	10k	27.3	0.25	85	2.03	3.21	13.46

Table 2: Overview of FRED difficulty metrics to measure the data quality of different languages (translating into high-resource direction). The column of Reported scores of high-resource language group are shown here for reference and trained by us and the low-resource pairs from different languages groups are excerpted from corresponding papers (citations are listed in the table).

E

-score is calculated on the target side. ∗ For Americas Group, we also reported chrF++ score in parenthesis below BLEU score. For a complete data, refer to Appendix (Table 6 ).

Feature	$R^{2}$ Value	Pred Strength
$R$ -score	0.5821	Strongest
$N_{token}$	0.3415	Moderate
$F$ -score ( $N_{token}/N_{char}$ )	0.2248	Low-Moderate
$D$ -score	0.1204	Low
$E$ -score	0.0142	Negligible
$N_{\text{train}}$	0.0011	None

Table 3: Individual

R^{2}

for Hybrid Regression Model (BLEU/ChrF/chrF++).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Venue	Language	Region/Period	Reference
WMT	Multiple	Global	Kocmi et al. (2023)
AfricaNLP	Multiple	African	George et al. (2024)
AmericasNLP	Multiple	Latin America	Ebrahimi et al. (2024)
WAT/WMT	Indic	South Asia	Nakazawa et al. (2023) ; Pal et al. (2023) ; Zhou et al. (2025)
ML4AL	Ancient Egyptian	Ancient	De Cao et al. (2024)
CL Conf	Akkadian	Ancient	Chen et al. (2024)

Appendix

Table 4: Survey of venues and publications on XLR languages from the past three years.

Lang	Writing	Phonography	Dataset	BLEU / ChrF at Different Data Sizes
Lang	Writing	Phonography	Dataset	1k	10k	100k	1M
fi	Latin	Alphabetic	wmt18-fi-en	6.32 / 28.87	11.97 / 38.82	18.31 / 45.39	18.47 / 46.98
zh	Han	Logographic	wmt18-zh-en	6.51 / 30.05	10.77 / 38.10	15.42 / 44.67	17.59 / 47.34
ar	Arabic	Abjad	iwslt2017-ar-en	7.20 / 24.49	18.94 / 38.90	29.61 / 50.10	36.31 / 56.03
ja	Han/Kana	Moraic	iwslt2017-ja-en	3.08 / 18.97	7.17 / 31.37	11.24 / 37.49	–
hi	Devanagari	Abugida	IITB-hi-en	1.42 / 15.88	4.40 / 26.47	7.71 / 39.17	8.11 / 45.66

Appendix

Table 5: High-resource reference baselines (xx

\rightarrow

en). BLEU / ChrF scores with capped training data (number of training sentences). Languages are selected from varied language families and writing systems to represent diverse morphological and orthographic challenges.

Lang	Parallel (# sent)	Monolingual	$N_{\text{token}}$	$N_{\text{char}}$	$\frac{N_{\text{token}}}{N_{\text{char}}}$	$D$ -score	$R$ -score	PBSMT	$E$ -score	Model
Lang	Parallel (# sent)	Monolingual	$N_{\text{token}}$	$N_{\text{char}}$	$\frac{N_{\text{token}}}{N_{\text{char}}}$	BLEU / ChrF	BLEU / ChrF	BLEU / ChrF	4-gram	BLEU / ChrF
High-resource languages (Table 7 )
ja $\rightarrow$ en	10k	>10M	31.4	56.3	0.56	1.65 / 5.01	2.86 / 18.38	4.49 / 21.52	114.47	9.03 / 31.37
hi $\rightarrow$ en	10k	>10M	33.1	117.2	0.28	0.95 / 10.08	2.78 / 17.34	2.36 / 11.20	85.98	14.02 / 26.47
fi $\rightarrow$ en	10k	>10M	27.3	109.8	0.25	2.03 / 16.44	3.21 / 17.16	4.63 / 24.01	85.46	13.46 / 38.82
zh $\rightarrow$ en	10k	>10M	38.0	59.0	0.66	0.78 / 1.60	2.38 / 17.46	4.18 / 24.87	90.17	13.32 / 38.10

Appendix

Table 6: Overview of automatic metrics to measure the data quality of different languages on both low-to-high and high-to-low directions. The high-resource language (ja, hi, fi, zh, ar) are shown here for reference.

N_{\text{token}}

N_{\text{char}}

and

\frac{N_{\text{token}}}{N_{\text{char}}}

are calculated on the test dataset of the source side, TTR is calculated on the train dataset of the target side, and

E

-score is calculated on the test dataset on the target side. For Americas languages, ChrF++ scores are reported instead of ChrF in Model column.

Lang	N-train	D-chrF	R-chrF	PBSMT-chrF
Japanese (ja)	10000	5.01	18.38	21.52
Hindi (hi)	10000	10.08	17.34	11.20
Finnish (fi)	10000	16.44	17.16	24.01
Chinese (zh)	10000	1.60	17.46	24.87
Arabic (ar)	10000	11.36	18.56	22.71
Lang		D-chrF++	R-chrF++	PBSMT-chrF++

Appendix

Table 7: Translation Performance Metrics for high-resource language pairs – chrF as similarity function.

Lang	N-train	D-chrF	R-chrF	PBSMT-chrF
Akkadian (akk)	50000	2.61	46.18	43.25
Egyptian (egy)	10000	9.22	41.95	13.86
Lang		D-chrF++	R-chrF++	PBSMT-chrF++
Akkadian (akk)		2.21	45.10	41.48
Egyptian (egy)		9.02	41.34	12.12

Appendix

Table 8: Translation Performance Metrics – chrF and chrF++ for Ancient language pairs.

Lang	N-train	D-BLEU	R-BLEU	PBSMT-BLEU	E-4-gram	BLEU
Sakizaya (ais)	4590	6.09	12.05	9.17	0	3.11
Amis (ami)	4600	5.51	12.59	8.97	0	3.56
Bunun (bnn)	7180	6.21	15.7	13.97	0.002	5.44
Kavalan (ckv)	6573	5.04	17.61	19.81	0	7.18
Rukai (dru)	8319	5.82	10.48	11.39	0	8.44
Paiwan (pwn)	4126	5.24	9.93	9.77	0.001	3.80

Appendix

Table 9: Translation Performance Metrics for Formosan-Chinese (Mandarin) language pairs. Exposure scores are calculated by counting on the high-resource side of the language pairs. The BLEU and chrF scores are taken from Zheng et al. (2024) .

lang	N-train	D-BLEU	R-BLEU	PBSMT-BLEU	E-4-gram	BLEU
ashaninka (cni)	3883	3.98	4.94	6.36	0.62	2.35
awajun (agr)	21964	1.23	5.50	5.75	0.87	11.12
aymara (aym)	6531	1.74	3.51	5.17	0.65	8.82
bribri (bzd)	7508	2.01	4.33	6.01	0.65	4.31
chatino (ctp)	357	1.50	9.63	4.69	7.62	-
guarani (gn)	26032	1.89	4.51	6.29	0.65	8.62

Appendix

Table 10: Translation Performance Metrics for Americas Indigenous Languages - Spanish Language Pairs from 2025’s shared tasks. We use dev set as the test set for the evaluation of this corpus since the test set of the dataset this year is not publicly released. Exposure scores are calculated by counting on the high-resource side of the language pairs. The BLEU and chrF++ scores are taken from the dev set performance of the team Syntax Squad in 2025’s competition Yahan and Islam (2025) .

Lang	N-train	D-BLEU	R-BLEU	PBSMT-BLEU	E-4-gram	BLEU
Translate into English
Amharic (amh)	899	0.29	2.80	4.42	109.65	-
Hausa (hau)	3098	1.41	6.28	6.09	146.42	12.9
Igbo (ibo)	6998	1.14	5.47	7.76	86.69	21.0
Kinyarwanda (kin)	460	1.24	2.95	4.24	94.52	-
Luganda (lug)	4075	1.96	4.96	5.36	84.58	19.8

Appendix

Table 11: Translation Performance Metrics for African language pairs. Exposure scores are calculated by counting on the high-resource side of the language pairs. BLEU and chrF scores are taken from the best possible BLEU scores from mBART/mT5/T5 of table 4 of the paper Adelani et al. (2022b) .

Lang	N-train	D-BLEU	R-BLEU	PBSMT-BLEU	E-4-gram	BLEU
Assamese (as)	50000	1.44	8.19	6.03	484.92	66.36
Mizo (lus)	50000	2.08	15.77	11.10	1141.87	33.30
Manipuri (mni)	21687	1.24	19.91	14.85	330.78	69.75
Khasi (kha)	24000	2.00	4.43	7.67	727.82	20.72
Lang		D-chrF	R-chrF	PBSMT-chrF
Assamese (as)		11.59	24.89	26.30		75.88

Appendix

Table 12: Translation Performance Metrics for Indic language pairs. Exposure scores are calculated by counting on the high-resource side of the language pairs. BLEU and chrF scores are taken from the best performance in WMT 2023 shared task Pal et al. (2023) .

Lang	$\frac{\text{zh only}}{\text{BLEU / chrF}}$	$\frac{\text{Both side}}{\text{BLEU / chrF}}$
Sakizaya (ais)	9.17 / 8.17	6.78 / 7.43
Amis (ami)	8.97 / 6.38	6.73 / 5.96
Bunun (bnn)	13.97 / 10.60	12.36 / 11.69
Kavalan (ckv)	19.81 / 16.03	16.26 / 14.79
Rukai (dru)	11.39 / 7.79	10.11 / 9.08
Paiwan (pwn)	9.77 / 9.82	9.02 / 9.82

Appendix

Table 13: PBSMT Scores for Formosan language pairs: translate to high-resource (zh) direction. The two columns shows the performance difference of using MBart as tokenizer on Chinese side only and using MBart on both sides.

Figure 2: Scores of PBSMT versus R-BLEU by language groups.

Explore similar work

Evaluating Extremely Low-Resource Machine Translation: A Comparative Study of ChrF++ and BLEU Metrics
Feb 19, 2026Sanjeev Kumar, Preethi Jyothi, Pushpak BhattacharyyaMT Evaluation Machine Translation
Dept. of CSE, IIT Bombay, Mumbai, India
Cross-Lingual Transfer for Machine Translation in Turkic Languages
Jul 31, 2026Omer Burak Cinar, Mehmet Mert Dalkilic, Cagri ToramanMultilingual Language Models Low-Resource MT
Middle East Technical University · Computer Engineering Department
Translation as Augmentation: Effect of Translated Data on Assessment of Difficulty
Jul 21, 2026Yiheng Wu, Jue Hou, Roman YangarberSynthetic Data Augmentation Automated Evaluation
University of Helsinki, Finland

Feb 19, 2026J/K move · Enter open · S save

Evaluating Extremely Low-Resource Machine Translation: A Comparative Study of ChrF++ and BLEU Metrics

Sanjeev Kumar, Preethi Jyothi, Pushpak Bhattacharyya

Dept. of CSE, IIT Bombay, Mumbai, India

MT Evaluation Machine Translation Low-Resource MT Low-Resource Language Processing

Open paper AskPDF arXiv

Evaluating machine translation (MT) quality in extremely low-resource language (ELRL) scenarios poses unique challenges, as widely used metrics such as BLEU, effective in high-resource settings, often misrepresent quality in data-scarce contexts. This work presents a comparative analysis of BLEU, an n-gram-based metric, and ChrF++, a character-based metric, for MT evaluation in ELRL settings. We examine how each metric responds to translation artifacts, including hallucinations, repetition, source-text copying, and diacritic (matra) variations across three ELRLs: Magahi, Bhojpuri, and Chhattisgarhi, with a focus on outputs from large language models (LLMs) and neural MT (NMT) systems. While recent work often relies solely on ChrF++, our findings show that BLEU, despite its lower absolute scores, provides complementary lexical-precision insights that improve interpretability.

Abstract

Explore similar work

Evaluating Extremely Low-Resource Machine Translation: A Comparative Study of ChrF++ and BLEU Metrics

Cross-Lingual Transfer for Machine Translation in Turkic Languages

Translation as Augmentation: Effect of Translated Data on Assessment of Difficulty