Omnilingual MT: Machine Translation for 1,600 Languages

Omnilingual MT Team; Belen Alastruey; Niyati Bafna; Andrea Caciolai; Kevin Heffernan; Artyom Kozhevnikov; Christophe Ropers; Eduardo Sánchez

cs.CLMar 17, 2026

Omnilingual MT: Machine Translation for 1,600 Languages

Authors: Omnilingual MT Team, Belen Alastruey, Niyati Bafna, Andrea Caciolai, Kevin Heffernan, Artyom Kozhevnikov, Christophe Ropers, Eduardo Sánchez, +24 more

Organizations: FAIR at Meta

Abstract

High-quality machine translation (MT) can scale to hundreds of languages, setting a high bar for multilingual systems. However, compared to the world's 7,000 languages, current systems still offer only limited coverage: about 200 languages on the target side, and maybe a few hundreds more on the source side, supported due to cross-lingual transfer. And even these numbers have been hard to evaluate due to the lack of reliable benchmarks and metrics. We present Omnilingual Machine Translation (OMT), the first MT system supporting more than 1,600 languages. This scale is enabled by a comprehensive data strategy that integrates large public multilingual corpora with newly created datasets, including manually curated MeDLEY bitext. We explore two ways of specializing a Large Language model (LLM) for machine translation: as a decoder-only model (OMT-LLaMA) or as a module in an encoder-decoder architecture (OMT-NLLB). Notably, all our 1B to 8B parameter models match or exceed the MT performance of a 70B LLM baseline, revealing a clear specialization advantage and enabling strong translation quality in low-compute settings. Moreover, our evaluation of English-to-1,600 translations further shows that while baseline models can interpret undersupported languages, they frequently fail to generate them with meaningful fidelity; OMT-LLaMA models substantially expand the set of languages for which coherent generation is feasible. Additionally, OMT models improve in cross-lingual transfer, being close to solving the "understanding" part of the puzzle in MT for the 1,600 evaluated. Our leaderboard and main human-created evaluation datasets (BOUQuET and Met-BOUQuET) are dynamically evolving towards Omnilinguality and freely available.

Figures & tables

Name	Type	Description	Claim
OMT-LLaMA	models family	LLaMA3 -based models for MT (different sizes) + extendable recipes	1,600 MT coverage with non-trivial performance and above baselines (Section 6 ).
OMT-NLLB	model	Encoder-decoder MT model built on top of OmniSONAR	1,600 to 250 MT coverage with non-trivial performance and above baselines (Section 7 ).
BLASER 3	model	OmniSONAR -based model for MT quality estimation	Highest multilingual coverage in reference-free MT quality estimation (1,600 + languages in the source-side). Outperforming previous metrics by several points in correlation with human judgments on Met-BOUQuET (Section 8.3 ).
OmniTOX	model	OmniSONAR -based model for toxicity detection	Largest language coverage in toxicity detection (1,600 languages) while outperforming its predecessor 200-language MuTox model by +0.06 mean per-language ROC AUC, achieving 0.86 across 30 diverse languages (Section 8.5 ).
MeDLEy	dataset	MT training dataset	Large-language scale training data collection created from scratch in multiple languages and manually extended finally covering 108 extremely low-resource languages (Section 4.3 )
BOUQuET	dataset	MT benchmark	Largest language scale evaluation data collection created from scratch in multiple languages and manually extended finally covering 275 languages including a wide variety of linguistic families and resources (Section 4.4.1 )

Table 1.1 : A summary of the corresponding claims of our line of work.

Figure 3.1 : Correlation between translation quality ( OMT-LLaMA model, Bible benchmark of 1,560 languages, mean xCOMET score) and amount of parallel documents from primary sources (not mined or synthetic). We fit an isotonic regression to show the global trend.

Figure 3.2 : Graph with distribution of languages per resource bucket. Note that we count all languages for which we have some data (including monolingual data and word-level parallel data like Panlex), but the buckets are determined based on the parallel data that is at least (and predominantly) sentence-level.

Type	Source	# Sentences	# Languages
Monolingual	CC-2000-Web	$\approx$ 20M	¿2000
Monolingual	CC-2000-Pdf	$\approx$ 5M	$\approx$ 1700
Translation	Bible	$\approx$ 600M	$\approx$ 1600
	Panlex	$\approx$ 2B	$\approx$ 1000
	Tatoeba	$\approx$ 25M	$\approx$ 500
	CC-NLLB-200	$\approx$ 500M	$\approx$ 200

Table 4.1 : A summary of the main sources and volumes (in number of sentences) used for CPT as detailed in Section 6.2 .

System	En-YY				XX-En
System	High	Mid	Low	Very Low	High	Mid	Low	Very Low
Baseline Data Mix	44.68	23.97	16.98	19.97	56.96	42.29	35.71	38.21
+ BT Data (LQ)	46.1	26.11	20.32	23.6	58.02	43.81	38.12	40.93
+ BT Data (MQ)	46.33	26.6	20.82	24.34	58.17	44.01	38.26	41.29
+ BT Data (HQ)	46.85	26.19	20.73	24.08	58.52	43.64	37.72	40.98

Table 4.2 : ChrF++ when evaluating MT systems trained with different backtranslation data mixes.

System	En-YY				XX-En
System	High	Mid	Low	Very Low	High	Mid	Low	Very Low
Baseline Data Mix	44.68	23.97	16.98	19.97	56.96	42.29	35.71	38.21
+ BT Data (5%)	46.54	25.16	18.41	21.1	59.68	43.1	37.53	39.67
+ BT Data (10%)	46.67	25.46	18.84	21.3	59.84	43.37	37.83	40.05
+ BT Data (20%)	46.93	25.78	19.25	21.13	60.11	43.78	38.31	41.11
+ BT Data (50%)	47.41	26.7	20.13	20.72	60.49	44.34	38.95	40.82

Table 4.3 : ChrF++ when evaluating MT systems trained with different backtranslation data mixes.

Languoid Group	# Languages	# Sentences	# Sentences after filtering
High	$\approx$ 30	$\approx$ 100M	$\approx$ 30M
Mid	$\approx$ 100	$\approx$ 150M	$\approx$ 100M
Low	$\approx$ 300	$\approx$ 100M	$\approx$ 70M
Very low	$\approx$ 1300	$\approx$ 40M	$\approx$ 20M
Zero	$\approx$ 400	$\approx$ 10M	$\approx$ 10M
All	$\approx$ 2000	$\approx$ 400M	$\approx$ 230M

Table 4.4 : Statistics about resulting backtranslation data.

Direction	En-YY					XX-En
Resource level	high	mid	low	very low	all	high	mid	low	very low	all
Baseline Data Mix	45.99	22.64	16.98	18.32	23.34	59.25	42.04	36.86	37.71	42.09
+ Mined Data	46.24	22.68	17.08	18.52	23.45	59.48	42.13	36.89	37.67	42.16

Table 4.5 : ChrF++ on FLoRes+ when evaluating MT systems continuously pretrained with and without mined data, split by whether English is the target or the source language and by the resource level of the other language.

Figure 4.1 : Steps in the creation of MeDLEy-source and MeDLEy-109. This includes (1) enumeration of grammatical features, (2) template generation including domain and source language assignment, (3) manual creation of paragraphs in 5 source languages: English, Mandarin, Spanish, Russian, and German, and (4) n-way parallelization (via English) across 8 pivot languages: English, Mandarin, Spanish, Russian, Hindi, Indonesian, Swahili, and French, resulting in MeDLEy-source. This is then (5) translated into 109 low-resource languages, each from a convenient pivot depending on the translator, resulting in MeDLEy-109.

Table 4.6 : Examples of grammatical features and their associated functions or meanings, with various language dependent forms (specific mechanisms used to express that feature).

Model	Seed Dataset	#tokens	BOUQuET		FLoRes+
Model	Seed Dataset	#tokens	En-YY	XX-En	En-YY	XX-En
Token-controlled Experiment
LLaMA	No Seed	0	8.49	14.45	10.07	16.81
	SmolDoc	215K	20.08	18.74	21.53	22.25
	SmolSent	215K	18.16	18.74	18.46	22.42
	MeDLEy	215K	19.60	20.39	20.69	23.73

Table 4.7: Token-controlled and Direct comparison: reporting number of LLaMA tokens on involved languages (Bambara, Mossi, Wolof, Yoruba, and Ganda, for both evaluation datasets) and ChrF++ numbers.

Figure 4.2 : Bouquet Language Expansion Visualization. Details on pivots that were used to translate each language show that most useful ones were fra_Latn, ind_latn, swh_latn and spa_latn.

Resource level	high	mid	low	very low	zero	total	domains
Bible	25	71	172	1289	4	1561	1
FLoRes+	24	82	73	31	3	213	3
BOUQuET	23	72	79	48	53	275	8
All benchmarks	26	89	222	1318	59	1714
All benchmarks, coarse grouping	25	83	206	1296	56	1666

Table 4.8 : Number of language varieties, grouped by resource level (as per Section 3.3 ) in each of the benchmark datasets, and the number of domains covered by them. The line “All benchmarks“ counts the union of all language codes in the three individual benchmarks, and the following line counts only unique ISO 639-3 codes, ignoring the variation in scripts, locales, or dialects.

Direction	En-YY					XX-En
Resource level	high	mid	low	very low	all	high	mid	low	very low	all
Baseline model (128K)	39.01	16.95	12.23	12.65	17.80	54.06	35.33	31.00	31.51	35.92
+ Added tokens (256K)	41.19	23.16	16.70	15.45	22.50	54.79	39.13	33.64	33.06	38.67

Table 5.1 : ChrF++ when evaluating MT systems continuously pretrained with and without our extended 256K tokenizer.

Model	BOUQuET				FLoRes+				Bible
	$\rightarrow$ en (s.)	en $\rightarrow$ (s.)	$\rightarrow$ en (p.)	en $\rightarrow$ (p.)	$\rightarrow$ en	en $\rightarrow$	$\rightarrow$ en (h)	en $\rightarrow$ (h)	$\rightarrow$ en	en $\rightarrow$	$\rightarrow$ en [sft langs]	en $\rightarrow$ [sft langs]
CP4	.725	.621	.527	.404	.735	.523	.376	.231	.683	.609	.703	.641
$\hookrightarrow$ TB (SFT)	.732	.611	.549	.404	.735	.539	.382	.233	.685	.656	.706	.685
$\hookrightarrow$ RL (DAPO)	.741	.616	.555	.403	.747	.541	.393	.233	.689	.661	.708	.685

Table 6.1 : Machine translation performance (xCOMET) after post-training. We compare the CPT model (CP4), supervised fine-tuning (SFT), and further reinforcement learning with DAPO. BOUQuET is evaluated at the sentence (s) and paragraph (p) level. We evaluate both in FLoRes+ and FLoRes-Hard (h).

System	RAG samples	ChrF++	xCOMET _both	MetricX _ref
sentence
OMT-LLaMA 8B	¡30K	31.04	0.46	10.72
$\hookrightarrow$ RAG	¡30K	31.56 (0.52)	0.47 (0.01)	10.74 (0.02)
OMT-LLaMA 8B	¿=30K	39.83	0.57	7.24
$\hookrightarrow$ RAG	¿=30K	42.13 (2.30)	0.58 (0.01)	6.70 (-0.54)
LLaMA3 8B	¡30K	21.64	0.41	13.58

Table 6.2 : Average performance metrics by system and level over 56 directions from the BOUQuET dataset. In parentheses, differences from applying RAG with the same system

Figure 7.1 : Overview of the proposed algorithm to train OMT-NLLB model.

Figure 7.2 : ChrF++ improvements (English to target translation) in languages where autoencoding data is used. Points above the diagonal indicate languages in which autoencoding data helped to improve the performance of the model.

	FLoRes+			Bible
System	ChrF++ $\uparrow$	xCOMET $\uparrow$	MetricX $\downarrow$	ChrF++ $\uparrow$	xCOMET $\uparrow$	MetricX $\downarrow$
OmniSONAR	47.54	0.65	5.66	40.61	0.52	8.79
Decoder Enhancing MT	49.58	0.65	5.51	44.85	0.52	8.61
Decoder Enhancing MT+AE	49.74	0.65	5.48	45.93	0.53	8.59
Decoder Warm-up	50.12	0.65	5.36	46.40	0.51	8.58
OMT-NLLB	50.92	0.66	5.32	46.28	0.50	8.61

Table 7.1 : Results of our proposed model on each training step of the proposed methodology compared to the original OmniSONAR

Figure 8.1 : Score Distribution of XSTS+R+P, XSTS and RSQM rnd .

Direction	XSTS+R+P	XSTS	RSQM
Czech $\rightarrow$ Hungarian	0.86	0.52	0.40
German $\rightarrow$ Croatian	0.89	0.72	0.49
English $\rightarrow$ Korean	0.67	0.54	0.52
English $\rightarrow$ Russian	0.60	0.51	0.43
English $\rightarrow$ Spanish	0.71	0.53	0.47
French $\rightarrow$ Kinshasa Lingala	0.90	0.77	0.51

Table 8.1 : Inter-annotator agreement (IAA) for different language directions, calculated with Krippendorf α using the squared distance penalty. Best results in bold.

Category	Model Version	Params	Citation Reference
Open Systems	NLLB-200	3B	( NLLB Team et al., 2024 )
	LLaMA3	3B, 8B, 70B	( AI@Meta, 2024 )
	TowerInstruct v02	7B	( Alves et al., 2024b )
	Aya-101	13B	( Üstün et al., 2024 )
	Aya Expanse	8B	( Dang et al., 2024 )
	Cohere R	7B	( Cohere et al., 2025 )

Table 8.2: Summary of MT Systems (not including small variants of each), Selection Methodology and setup and goals of Round 1 and Round 2

Panel of Figure 8 — Figure 8.2 : BLASER 3 Methodology

Protocol	Unique Directions	Examples (K)
Training
DA	46	745
ESA	9	72
MQM	12	187
SQM	15	150
XSTS	121	422

Table 8.3 : Training data statistics by protocol, and combined. Examples are in thousands.

Model	XX-En	En-YY	XX-YY	All
	(17)	(19)	(41)	(77)
Previous Works
BLASER 2	0.47	0.38	0.45	0.44
xCOMET-XL	0.51	0.45	0.40	0.43
MetricX-24	0.52	0.47	0.45	0.47
Proposed

Table 8.4 : Spearman’s

\rho

(

\uparrow

) on the Met-BOUQuET (XSTS+R+P) test set. X indicates non-English source languages, and Y indicates non-English target languages. In bold is the best among the proposed BLASER 3 and the three baselines. In parenthesis is the number of pairs in each group (note that we use a subset of Met-BOUQuET from r1 annotations, available at the time of experimentation).

	Standalone			LID-adjusted
	src	ref	both	src	ref	both
BLEU	-	0.318	-	-	0.328	-
ChrF++	-	0.414	-	-	0.404	-
METEOR	-	0.332	-	-	0.336	-
BLASER2	0.368	0.492	-	0.431	0.477	-
BLEURT	-	0.486	-	-	0.481	-

Table 8.5 : Mean per-direction Spearman’s

\rho

on the test set of Met-BOUQuET (XSTS+R+P) for several popular metrics, depending on whether they compare the translation with source, reference, or both, and whether they are adjusted with the GlotLID score. The best metric for each setting is in boldface.

Best metrics per directions group		high-high	low-high	high-low	low-low
Number of directions		34	36	32	42
Best metric signature	BLEU	0.368	0.349	0.346	0.262
	ChrF++	0.418	0.430	0.424	0.389
	METEOR	0.382	0.371	0.340	0.265
	BLASER2	0.544	0.545	0.446	0.453
	BLEURT	0.588	0.571	0.417	0.390

Table 8.6: Mean per-direction Spearman’s

\rho

on the test set of Met-BOUQuET, aggregated per group of translation directions. Top part: for each metric, we report the best correlation over the metric signatures (whether to use source, reference, and LID adjustment). Bottom part: for each signature, we report the best score over different metrics.

Source text	Translated text	Comment
Åh! Ursäkta, men ta med den snälla.	Oh! Excuse me, but take it kindly.	This is the last sentence of a dialogue where one person forgot their purse at work and said to the other person “Oh! Sorry, but please take it with you” meaning “please take my purse with you”. Not knowing this is about a purse, even a human can misunderstand the sentence, and the literal translation is very close to what the MT here produced.
En av gurkorna växte sig större.	One of the tomatoes grew bigger.	In this Swedish original, we see cucumbers, not tomatoes. According to our XSTS+R+P protocol guidelines, this should score 2 (crucial information is missing/mistranslated in the target), but the automatic metrics do not penalize in the same manner.
Att köpa en kaffe för 5 euro på Starbucks varje dag kan kosta dig 1 300 euro per år.	Buying a $5 coffee at Starbucks every day can cost you$ 1,300 a year.	Though the meaning is clear, the localization is redundant and unnecessary, and the details are changed.
Sorry man, I am going to Shirdi with my family.	Ursäkta, jag ska till Shirdi med min familj.	In this example, “man” is omitted completely and nothing conveys the informal address. That’s why this translation scored very low in human evaluation. However, the automatic metrics give a score much higher than expected.

Table 8.7: Examples where automated metrics and human judgment do not align

Partition	Languages	Size Range	Total
Train	English	12,906	60,194
	Spanish	10,716
	Other (28 langs)	887–1,476
Dev	English	894	19,488
	Spanish	758
	Other (28 langs)	414–738

Table 8.8 : Training and evaluation data statistics for OmniTOX. We use a subset of the MuTox dataset covering 30 languages across three partitions.

Model	Overall	Mean per Lang.	Range per Lang.
	ROC AUC	ROC AUC	ROC AUC
MuTox	0.787 [0.775,0.799]	0.798	0.645–0.949
OmniTOX	0.845 [0.835,0.856]	0.860	0.664–0.972
Ablations
$\hookrightarrow$ w/o Classifier Tuning (Baseline)	0.839 [0.829,0.850]	0.854	0.682–0.975
$\hookrightarrow$ w/o Multilingual Training (Baseline ZS)	0.793 [0.781,0.807]	0.821	0.623–0.974

Table 8.9 : Performance comparison on the DevTest partition (30 languages). Overall ROC AUC: computed on the aggregated test set with 95% confidence intervals via bootstrap resampling (1000 iterations). Mean per Lang.: average of individual language ROC AUCs. Range per Lang.: minimum and maximum ROC AUC across languages. Baseline uses OmniSONAR embeddings with MuTox hyperparameters; Baseline ZS is trained only on English and Spanish. Best results in bold.

Figure 8.3 : ROC AUC per language for MuTox and OmniTOX. OmniTOX outperforms MuTox on 28 of 30 languages.

System	Size(s)	Reference	Comment
OMT-LLaMA	8B, 3B, 1B	Section 6	Specialised
OMT-NLLB	3B	Section 7	Specialised
Aya-101	13B	( Üstün et al., 2024 )	General
Aya Expanse	8B	( Dang et al., 2024 )	General
EuroLLM	9B	( Martins et al., 2024 )	General
Gemma3	27B	( Team et al., 2025 )	General

Table 9.1 : Proposed and external MT systems evaluated throughout this section and classified as General or Specialised (on MT).

	XX-En						En-YY						All
Resource level	high	mid	low	v. low	zero	total	high	mid	low	v. low	zero	total	total
OMT-LLaMA 8B	65.1	53.5	38.7	27.6	21.6	40.6	60.7	45.8	30.8	18.8	12.6	32.8	36.7
OMT-NLLB	64.0	56.1	37.4	28.4	23.2	41.3	61.1	47.6	27.3	17.5	11.5	31.9	36.6
OMT-LLaMA 3B	64.4	54.6	38.8	28.0	22.3	41.0	59.4	45.0	28.7	16.8	11.3	31.3	36.2
NLLB-200	61.5	52.4	32.1	25.5	22.7	37.9	62.9	46.8	24.6	17.5	13.2	31.3	34.6
GPT-OSS 120B	64.6	49.9	33.6	25.4	24.3	38.3	63.6	43.7	26.0	16.0	13.4	30.8	34.5

Table 9.2 : Translation performance on BOUQuET (ChrF++) by the non-English language resource level.

	XX-En					En-YY					All
Resource level	high	mid	low	v. low	total	high	mid	low	v. low	total	total
OMT-NLLB	65.0	58.3	54.6	52.3	57.3	57.6	46.0	41.0	34.6	44.6	50.9
OMT-LLaMA 8B	64.6	56.4	50.6	48.3	54.6	57.4	46.5	39.8	33.8	44.2	49.4
NLLB-200	63.5	55.3	50.9	49.3	54.2	57.2	44.3	39.4	32.5	43.0	48.6
Gemma 3	64.2	52.3	46.4	48.1	51.4	58.6	40.1	31.5	26.4	38.1	44.8
TranslateGemma	62.5	51.2	45.5	47.5	50.4	55.4	39.5	30.5	27.5	37.2	43.8

Table 9.3 : Translation performance on FLoRes+ (ChrF++) by the non-English language resource level.

Metric	ChrF++				BLASER3+GlotLID
Direction	XX-En	En-YY	XX-Proxy	Proxy-YY	XX-En	En-YY	XX-Proxy	Proxy-YY
OMT-LLaMA 8B	40.6	32.8	35.8	30.0	0.47	0.32	0.40	0.32
OMT-NLLB	41.3	31.9	35.5	29.7	0.49	0.32	0.42	0.32
OMT-LLaMA 3B	41.0	31.3	34.8	28.4	0.46	0.31	0.39	0.31
GPT-OSS 120B	38.3	30.8	33.8	28.8	0.43	0.29	0.38	0.30
Gemma 3	39.4	28.9	33.7	26.7	0.44	0.26	0.37	0.27

Table 9.4 : Systems ranking on BOUQuET depending on the direction type.

Figure 9.1 : Translation quality (LID-normalized xCOMET) of the OMT-LLaMA model out of (left) and into (right) each language paired with English (horizontal axis) and non-English proxy languages (vertical axis).

Figure 9.2 : Relative performance in terms of ChrF++ (top) and XCOMET (bottom) of OMT models in understanding the longtail in the Bible domain compared to external baselines. Languages sorted by average performance of the models; curves smoothed with exponential moving average.

Figure 9.3 : A curve to predict XSTS+R+P scores from reference-based MetricX (on Met-BOUQuET, XX-En subset; to visualize XSTS+R+P distribution, a random jitter has been added to the Y axis).

Figure 9.4 : Predicted XSTS+R+P machine translation quality score for the Bible languages into English using MetricX extrapolation on the test set (languages sorted individually for each of the evaluated models) .

Figure 9.5 : Relative performance in terms of ChrF++ (top) and LID-adjusted BLASER 3 (bottom) of OMT models in generating the longtail in the Bible domain compared to external baselines.

Figure 9.6 : Interaction of model size and translation quality for the OMT models and several other open model families on the BOUQuET dataset (from and into English).

Figure 9.7 : Performance of OMT-LLaMA models on BOUQuET (test set, sentence-level) by language for XX-En translation (left) and En-YY translation (right). Languages sorted by the decreasing performance of the average of the three models; curved smoothed with sliding window.

Feature	OMT-LLaMA	OMT-NLLB
Sizes	1B, 3B, 8B	3B
Architecture	decoder-only	encoder-decoder
Understanding Languages	around 1000	almost any language
Generating Languages	around 1000	around 250
Zero/few-shot	few-shot	0-shot (source side)

Table 9.5 : List of features comparing OMT-LLaMA vs OMT-NLLB models presented in this paper.

Direction group	OMT-LLaMA	OMT-LLaMA	Baseline	Num
	Win rate	Mean score	Mean score	Directions
high-high	0.75	4.03	3.45	16
high-low	0.83	2.99	1.61	18
low-high	0.80	3.26	2.69	15
low-low	0.62	2.96	2.66	8
total	0.77	3.38	2.67	57

Table 9.6 : Average results of Met-BOUQuET Round 2 annotations: OMT-LLaMA win rate, OMT-LLaMA and baseline mean score and number of directions for each. “Win rate” is defined as the proportion of directions with the average scores for OMT-LLaMA higher than for the baseline system.

Figure 9.8 : Mean XSTS+R+P scores for each of the 57 directions of Met-Bouquet Round 2 for the OMT system (vertical axis) and the baseline system (horizontal axis).

System	Average Added Toxicity		Flagging Rate
	XX-En	En-YY	XX-En	En-YY
NLLB-200-3B	0.19	-0.20	1.10%	0.07%
Gemma-3-27b	-0.20	0.07	0.25%	0.06%
OMT-NLLB	0.02	-0.17	0.69%	0.04%
OMT-LLaMA	-0.07	0.17	0.52%	0.06%

Table 9.7 : Average Added Toxicity and Flagging Rate (%) on BOUQuET dataset for 207 languages.

Metric	ChrF++				BLASER 3 + LID
Languages	20 hardest		5 hard		20 hardest		5 hard
Direction	En-YY	XX-En	En-YY	XX-En	En-YY	XX-En	En-YY	XX-En
OMT-LLaMA + FT + RAG	29.25	36.67	46.74	53.22	0.2577	0.3427	0.4682	0.6349
OMT-LLaMA + FT	28.91	29.03	47.49	42.69	0.2547	0.3010	0.4747	0.5285
OMT-LLaMA + RAG	26.79	36.18	44.84	53.96	0.2486	0.3445	0.4595	0.6420
OMT-LLaMA	24.12	29.92	43.01	48.43	0.2178	0.3135	0.4422	0.6130

Table 10.1 : Results of extending the OMT-LLaMA and LLaMA-base models via focused fine-tuning, retrieval augmentation, or both. Reported on the dev split of BOUQuET, for 20 hard languages and 5 mid-resourced languages.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Feature	Function
Case marking	Nominative case	Marks subject of a clause
	Accusative case	Marks patient or theme
	Genitive case	Marks possession
	Dative case	Marks recipient or experiencer
	Locative or spatial case	Marks location
	Instrumental or comitative case	Marks means, tool, or companion

Appendix

Table A.1: Feature Inventory

Number

- At least 5 quantified expressions (e.g., “some”, “many”, “all”, etc.)

- At least 5 numbers (including digits and spelled out)

Gender / Noun classes

- Nouns in all genders of the language

- Personal pronouns in all genders (e.g., Spanish pronouns “nosotras/vosotras”); it includes things like past tense verbs in the feminine in Russian for first and second persons

- Animate and inanimate forms in various semantic/syntactic roles (i.e., subject, object, beneficiary, etc.)

Appendix

Table A.2: A global checklist for linguists to include over all source paragraphs for a single source language.

Paradigm	SmolSent	SmolDoc	NLLB-Seed	MeDLEy
Case	1.01	0.92	1.00	0.92
Number	0.52	0.59	0.58	0.54
Person	0.86	0.81	0.13	0.97
Tense	0.80	0.79	0.69	0.83
Aspect	0.93	1.16	0.98	1.27
Mood	0.43	0.14	0.02	0.29

Appendix

Table A.4: Entropy over paradigm distributions over features for different datasets (bold indicates highest value per paradigm).

Paradigm	Feature	SmolSent	SmolDoc	NLLB-Seed	MeDLEy
case	Accusative	0.16	0.16	0.15	0.23
	Genitive	0.35	0.21	0.34	0.15
	Nominative	0.49	0.63	0.51	0.62
number	Dual	0.00	0.00	0.00	0.00
	Plural	0.21	0.27	0.25	0.21
	Singular	0.79	0.73	0.75	0.79

Appendix

Table A.5: Feature distributions per paradigm and dataset. Cell values are proportions of that feature given the paradigm.

Paradigm	Features
Number	Singular, Plural
Formality	Formal, Informal
Tense	Present, Past, Future
Aspect	Habitual, Imperfective, Perfect, Perfective
Mood	Conditional, Imperative, Indicative, Subjunctive
Valency	Impersonal, Intransitive, Monotransitive, Ditransitive

Appendix

Table A.6: Feature types contained in each paradigm for Hindi case study.

Paradigm	MeDLEy	SmolSent	NLLB-Seed
Formality	0.60	0.07	0.00
Tense	0.79	0.67	0.64
Aspect	1.22	1.19	1.13
Valency	1.28	1.17	1.18
Negation	0.78	0.60	0.53
Number	0.43	0.41	0.46

Appendix

Table A.7: Entropy of feature paradigms for Hindi across datasets (bold indicates highest entropy per paradigm).

Feature	rus-eng	rus-eng-spa	spa-eng	spa-eng-rus
Dative	46.7	60.0	55.6	22.2
Instrumental	50.0	50.0	92.3	69.2
Past tense	96.6	89.7	94.1	91.2
Passive	88.9	100.0	81.8	72.7
Imperative	100.0	100.0	100.0	100.0
Ditransitive	91.7	91.7	68.8	68.8

Appendix

Table A.8: Percentage of transferred feature for 10 features, starting with Spanish or Russian as source languages, and for 1 or 2 hops.

Figure A.1 : Number of examples per language across seed datasets.

Figure A.2 : Token length distribution per language across seed datasets.

Figure A.3 : Total number of tokens per language across seed datasets.

Hyperparameter	NLLB	LLaMa	Notes
Epochs	10	10
Batch Size	8	1	LLaMA uses packing
Max Num Tokens	512	1024
Learning Rate	5e-5	1e-5
Optimizer	AdamW	AdamW
Learning Rate Schedule	Inverse Square Root	Cosine Annealing	With 10% of warmup steps

Appendix

Table A.9: Training Hyperparameters

Hyperparameter	NLLB	LLaMa	Notes
Decoding	Beam Search	Greedy
Beam Width	5	N/A
Max Num Tokens	512	1024
Temperature	0.0	0.0
Top-P	1.0	1.0	No Sampling
Top-K	N/A	N/A	No Sampling

Appendix

Table A.10: Inference hyperparameters

		BOUQuET						FLORES+
		en-xx			xx-en			en-xx			xx-en
Model	Seed Dataset	chrF++	MetricX	XCOMET	chrF++	MetricX	XCOMET	chrF++	MetricX	XCOMET	chrF++	MetricX	XCOMET
LLaMA	No Seed	8.49	16.02	0.22	14.45	10.44	0.24	10.07	17.16	0.18	16.81	10.86	0.20
	SmolDoc	20.08	13.89	0.34	18.74	12.66	0.35	21.53	14.91	0.24	22.25	12.96	0.27
	SmolSent	18.16	15.04	0.33	18.74	13.00	0.34	18.46	16.36	0.23	22.42	13.67	0.28
	Medley	19.60	13.97	0.33	20.39	11.94	0.33	20.69	15.48	0.24	23.73	12.63	0.27

Appendix

Table A.11 : Average translation performance when fine-tuning on token-controlled seed datasets.

Model	Seed Dataset	BOUQuET						FLORES+
		en-xx			xx-en			en-xx			xx-en
		chrF++	MetricX	XCOMET	chrF++	MetricX	XCOMET	chrF++	MetricX	XCOMET	chrF++	MetricX	XCOMET
LLaMA	No Seed	8.49	16.02	0.22	14.45	10.44	0.24	10.07	17.16	0.18	16.81	10.86	0.20
	SmolDoc (D)	25.07	11.99	0.35	23.06	11.41	0.40	25.03	12.94	0.24	25.09	11.85	0.31
	SmolSent (S)	18.69	14.69	0.33	19.47	12.85	0.34	19.70	15.98	0.24	23.87	12.99	0.29
	Medley (M)	22.30	12.66	0.34	23.18	11.35	0.37	22.43	14.33	0.24	25.71	12.00	0.29

Appendix

Table A.12 : Average translation performance when fine-tuning on seed datasets and their combination.

Model	Language	Seed Dataset	BOUQuET						FLORES+
			en-xx			xx-en			en-xx			xx-en
			chrF++	MetricX	XCOMET	chrF++	MetricX	XCOMET	chrF++	MetricX	XCOMET	chrF++	MetricX	XCOMET
LLaMA	bam_Latn	No Seed	7.49	16.67	0.22	14.24	10.74	0.25	7.06	17.51	0.17	15.36	11.65	0.22
		SmolDoc	22.56	13.03	0.34	17.62	13.30	0.33	22.83	13.85	0.25	21.24	14.12	0.26
		SmolSent	15.85	17.08	0.31	16.29	12.89	0.31	14.04	18.69	0.21	20.54	14.11	0.26
		Medley	21.34	13.50	0.32	18.08	12.24	0.30	21.05	14.43	0.25	22.27	13.66	0.24

Appendix

Table A.13: Per-language translation performance when fine-tuning on token-controlled seed datasets.

Model	Language	Seed Dataset	BOUQuET						FLORES+
			en-xx			xx-en			en-xx			xx-en
			chrF++	MetricX	XCOMET	chrF++	MetricX	XCOMET	chrF++	MetricX	XCOMET	chrF++	MetricX	XCOMET
LLaMA	bam_Latn	No Seed	7.49	16.67	0.22	14.24	10.74	0.25	7.06	17.51	0.17	15.36	11.65	0.22
		SmolDoc (D)	27.82	10.82	0.35	20.57	12.16	0.38	27.32	11.42	0.24	23.46	12.54	0.29
		SmolSent (S)	18.6	15.4	0.32	15.21	14.51	0.28	19.56	16.42	0.24	19.35	14.83	0.24
		Medley (M)	24.97	11.83	0.32	22.15	11.67	0.38	23.88	12.68	0.24	24.38	12.69	0.28

Appendix

Table A.14 : Per-language translation performance when fine-tuning on seed datasets and their combinations.

Figure A.4 : Spearman rank correlation of chrF++ with other evaluation metrics, across different dimensions.

Model	Seed Dataset	comments		conversation		instruction		narration		other misc.		reflection		social posts		web misc.
Model	Seed Dataset	xx-en	en-xx	xx-en	en-xx	xx-en	en-xx	xx-en	en-xx	xx-en	en-xx	xx-en	en-xx	xx-en	en-xx	xx-en	en-xx
LLaMA	No Seed	7.19	13.49	9.27	13.37	6.97	15.19	7.87	14.07	6.81	14.33	6.42	15.13	7.18	15.50	6.50	13.44
	SmolDoc	19.64	14.50	25.63	17.35	22.40	17.68	23.91	19.47	20.23	15.91	21.73	17.61	24.37	20.82	19.18	17.04
	SmolSent	14.75	13.66	15.97	16.00	15.87	16.62	16.44	16.55	14.80	15.74	15.44	16.31	17.39	18.38	15.68	16.55
	Medley	20.12	15.61	21.68	16.40	21.20	18.30	22.42	19.91	20.89	18.42	21.94	18.81	22.91	19.80	19.10	18.20
NLLB	No Seed	25.92	27.85	30.63	34.74	30.15	33.18	31.78	32.55	27.57	29.49	27.11	27.05	29.72	36.22	23.44	27.59

Appendix

Table A.15 : Per-domain BOUQuET evaluation results comparing models fine-tuned on token-controlled version of the seed datasets.

Model	Seed Dataset	BOUQUET						FLORES+
		en-xx			xx-en			en-xx			xx-en
		chrF++	MetricX	XCOMET	chrF++	MetricX	XCOMET	chrF++	MetricX	XCOMET	chrF++	MetricX	XCOMET
LLaMA	No Seed	6.69	14.73	0.19	13.54	10.40	0.23	7.78	15.71	0.16	15.64	11.04	0.20
	NLLB Seed	18.42	15.05	0.32	17.43	14.21	0.29	21.08	15.23	0.24	24.03	13.19	0.29
	Medley	24.92	13.20	0.33	23.17	11.29	0.37	22.19	14.40	0.24	24.35	12.49	0.28
NLLB	No Seed	25.69	12.07	0.35	33.29	10.80	0.46	25.53	13.00	0.24	33.85	11.03	0.39

Appendix

Table A.16 : Average translation performance comparing NLLB -Seed and MeDLEy .

Model	Language	Seed Dataset	BOUQUET						FLORES+
			en-xx			xx-en			en-xx			xx-en
			chrF++	MetricX	XCOMET	chrF++	MetricX	XCOMET	chrF++	MetricX	XCOMET	chrF++	MetricX	XCOMET
LLaMA	bam_Latn	No Seed	7.55	16.67	0.22	14.29	10.75	0.25	7.13	17.57	0.18	15.30	11.52	0.22
		NLLB Seed	19.34	14.54	0.32	18.17	13.86	0.30	22.97	14.32	0.25	24.29	13.16	0.27
		Medley	24.46	11.90	0.32	22.91	11.45	0.37	23.56	12.70	0.24	23.88	13.16	0.27
	dik_Latn	No Seed	4.95	12.38	0.13	12.10	10.05	0.19	7.61	13.69	0.13	15.12	10.54	0.18

Appendix

Table A.17 : Per-language translation performance comparing NLLB -Seed and MeDLEy .

Dataset	Protocol	Langs	Dir	Dir w/o Eng	Src Sent	Systems	Domains
MLQE PE	DA (z-scores)	2	1	1	995	NA	NA
	DA	8	7	0	60,788	NA	NA
	HTER	8	7	0	60,757	NA	NA
	DA & HTER	12	11	0	10,970	NA	NA
	Catastrophic Errors	5	4	0	15,261	NA	NA
IndicMT Eval	MQM	6	5	0	1,423	7	NA

Appendix

Table B.1 : Summary of key attributes of MT metrics evaluation datasets, including language coverage, source sentences, systems evaluated, and domains. Statistics are reported only for datasets with available evaluation scores.

Figure B.1: For Met-BOUQuET Round 1: Top, XSTS+R+P consensus scores histogram across English and non-English directions; Bottom, XSTS+R+P consensus averages across source and target languages.

Figure B.2: For Met-BOUQuET Round 2: Top, XSTS+R+P consensus scores histogram; Bottom, XSTS+R+P consensus averages across directions.

Task	Metric	Type	Citation
LID	glotlid	Automatic	( Kargaran et al., 2023a )
MT Metrics	Spearman’s rank correlation coefficient	Automatic
MT	BLEU	Automatic	( Papineni et al., 2002 )
	ChrF++	Automatic	( Popović, 2015 )
	METEOR	Automatic	( Banerjee and Lavie, 2005b )
	BLASER2	Automatic	( Dale and Costa-jussà, 2024a )

Appendix

Table C.1 : Automatic and human evaluation metrics/protocols used in this work.

Panel of Table 72 — Table D.1: Covered Languages for the evaluation dataset contribution in this work (275 in BOUQuET, and 120 in Met-BOUQuET), the training MeDLEy dataset and our previous and ongoing evaluation community effort of FLoRes+.

Explore similar work

Last Translation Benchmark
Sep 3, 2026Vilém Zouhar, Niyati Bafna, Mukund Choudhary +257MT Evaluation
MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages
Jul 1, 2026Maximilian Idahl, Jörg Tiedemann, Sampo Pyysalo +19Language Model Pretraining Synthetic Data Pretraining
ellamind · Leibniz University Hannover · University of Helsinki +14
LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation
Apr 20, 2026Samar M. Magdy, Fakhraddin Alwajih, Abdellah El Mekki +2Arabic NLP Machine Translation
λThe University of British Columbia · ξMinia University · γCanada Research Chair in NLP and ML

Sep 3, 2026J/K move · Enter open · S save

Last Translation Benchmark

Vilém Zouhar, Niyati Bafna, Mukund Choudhary, Maike Züfle+256

MT Evaluation

Open paper AskPDF arXiv

For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, standard benchmarks for machine translation are approaching saturation. Further, automatic translation metrics are unreliable, opaque, and vulnerable to reward-hacking. Even gold human evaluation is not problem-free, because it often lacks reproducibility, objectivity, and scalability. Overall, this prevents us from tracking progress in the field and identifying pathways for improvement. We introduce the Last Translation Benchmark, a collection of human-authored and peer-reviewed examples (texts, images, audio, videos) that break leading machine translation models. We also present a new evaluation approach: each example comes with handcrafted verification rules describing concrete failure cases on that example, therefore allowing reliable and actionable future evaluation. The Last Translation Benchmark is a live dataset that accepts ongoing contributions. The latest version is LTBv1, containing accepted contributions prior to September 1st 2026, with future releases planned as new data is continuously collected.

Abstract

Explore similar work

Last Translation Benchmark

MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages

LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation