Esperanto is a widespread constructed language, known for its regular grammar and productive word formation. Besides having substantial resources available thanks to its online community, it remains relatively underexplored in the context of modern machine translation (MT) approaches. In this work, we present the first comprehensive evaluation of open-source MT systems for Esperanto, comparing rule-based systems, encoder-decoder models, and LLMs across model sizes. We evaluate translation quality across six language directions involving English, Spanish, Catalan, and Esperanto using multiple automatic metrics as well as human evaluation. Our results show that the NLLB family achieves the best performance in all language pairs, followed closely by our trained compact models and a fine-tuned general-purpose LLM. Human evaluation confirms this trend, with NLLB translations preferred in approximately half of the comparisons, although noticeable errors remain. In line with Esperanto's tradition of openness and international collaboration, we release our code and best-performing models publicly.
Figures & tables
Figure 1: Cumulative number of ACL Anthology papers mentioning ’Esperanto’ and ’Esperanto and Machine Translation’. Early work involving Esperanto focused primarily on MT, while more recent work covers a broader range of topics.
Training Data
Used for Training
FLORES+
Language Pair
Raw
Filtered
% Removed
Marian
LLM FT
Dev
Test
en–eo
70.34M
45.39M
35.5%
5.0M
100k
997
1012
es–eo
6.21M
4.68M
24.7%
4.7M
100k
997
1012
ca–eo
1.17M
673.93k
42.7%
673k
100k
997
1012
Total
77.73M
50.74M
10.35M
300k
2991
Table 1: Parallel data statistics before and after filtering, and final training sizes per model family.
Figure 2: Prompt example for fine-tuning Llama-3.1-8B-Instruct
Figure 3: Human evaluation win rates for both translation directions.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Optimizer
Adam
Adam β
(0.9, 0.98)
Adam ϵ
1e-9
Learning rate
3e-4
LR warmup
16000 steps
LR decay strategy
epoch+stalled
Appendix
Table 5: Hyperparameters for Transformer-base
Hyperparameter
Value
Optimizer
Adam
Adam β
(0.9, 0.98)
Adam ϵ
1e-9
Learning rate
3e-4
LR warmup
16000 steps
LR decay strategy
epoch+stalled
Appendix
Table 6: Hyperparameters for Transfomer-tiny
Hyperparameter
Value
Learning Rate
5e-05
LR Scheduler Type
Linear
Warmup Ratio
0.3
Weight Decay
0.0
Per Device Train Batch Size
4
Gradient Accumulation Steps
4
Appendix
Table 7: Hyperparameters for Llama fine-tuning
Hyperparameter
Value
Optimizer
Adafactor
Learning Rate
1e-4
LR Scheduler Type
Constant
LR Warmup
1000 steps
Weight Decay
1e-3
Per Device Train Batch Size
32
Appendix
Table 8: Hyperparameters for NLLB fine-tuning
Error Type
Source
Translation
Transformer-base
Lexical
La epidemio igis la baratan registaron entrepreni tiajn rimedojn, kiel instalado de porkokaptiloj en la serioze damaĝitaj areoj, disdonado de miloj da kontraŭmoskitaj retoj kaj ŝprucado de pesticidoj.
La epidemia llevó al gobierno indio a emprender tales medios, como la instalación de trampas de cerdo en las áreas seriamente dañadas, la distribución de miles de redes anti-moscoquios y el brote de pesticidas.
Grammatical
Malgraŭ foresto de cunama minaco, loĝantoj ekpanikis kaj komencis forlasadi siajn komercojn kaj hejmojn.
A pesar de la ausencia de la amenaza tsunami, los residentes comenzaron a pánico y comenzaron a dejar sus negocios y casas.
Untranslated
Gregon konsistigas de unu ĝis tri parencaj plenkreskaj maskloj kaj multaj ĝis tridek inoj kaj idoj.
Gregon consiste de uno a tres varones adultos relacionados y muchos a treinta hembras y crías.
NLLB-200-3.3B
Mistranslation
Nekredantoj vivantaj en kristanaj landoj aŭ aliloke en la tuta mondo adoptis multajn el la tradicioj pri ĉi tiu festo.
Los incrédulos que viven en países cristianos o en otras partes del mundo han adoptado muchas de las tradiciones de esta fiesta.
Appendix
Table 12: Illustrative examples of error categories identified in the qualitative analysis for Esperanto into Spanish. Highlighted spans mark erroneous content.
Error Type
Source
Translation
Transformer-base
Grammatical
En verdad, el formato de 35 mm es algo confuso , ya que sus medidas son 36 mm de anchura por 24 mm de alto.
Fakte, la 35mm formato estas iom konfuzita , ĉar ĝiaj mezuroj estas 36 mm larĝa de 24 mm alta.
Mistranslation
Conforme surge de las imágenes infrarrojas, las variaciones de temperatura durante la noche y el día indican que, probablemente, se trate de cuevas.
Ĉar ĝi ekestiĝas de infraruĝaj bildoj, temperaturkatastrofoj dum la nokto kaj tago indikas ke ili estas verŝajne kavernoj.
Named Entities
El día de hoy se originó en el Océano Atlántico la tormenta número diez de su temporada de huracanes en ser nombrada, a la que se denominó Tormenta subtropical Jerry.
La atlantika ŝtormo numero dek el sia uragansezono estis nomita Jerry Storm.
Appendix
Table 13: Illustrative examples of error categories identified in the qualitative analysis for Spanish into Esperanto. Highlighted spans mark erroneous content.
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
High-quality machine translation (MT) can scale to hundreds of languages, setting a high bar for multilingual systems. However, compared to the world's 7,000 languages, current systems still offer only limited coverage: about 200 languages on the target side, and maybe a few hundreds more on the source side, supported due to cross-lingual transfer. And even these numbers have been hard to evaluate due to the lack of reliable benchmarks and metrics. We present Omnilingual Machine Translation (OMT), the first MT system supporting more than 1,600 languages. This scale is enabled by a comprehensive data strategy that integrates large public multilingual corpora with newly created datasets, including manually curated MeDLEY bitext. We explore two ways of specializing a Large Language model (LLM) for machine translation: as a decoder-only model (OMT-LLaMA) or as a module in an encoder-decoder architecture (OMT-NLLB). Notably, all our 1B to 8B parameter models match or exceed the MT performance of a 70B LLM baseline, revealing a clear specialization advantage and enabling strong translation quality in low-compute settings. Moreover, our evaluation of English-to-1,600 translations further shows that while baseline models can interpret undersupported languages, they frequently fail to generate them with meaningful fidelity; OMT-LLaMA models substantially expand the set of languages for which coherent generation is feasible. Additionally, OMT models improve in cross-lingual transfer, being close to solving the "understanding" part of the puzzle in MT for the 1,600 evaluated. Our leaderboard and main human-created evaluation datasets (BOUQuET and Met-BOUQuET) are dynamically evolving towards Omnilinguality and freely available.
Dialectal machine translation remains challenging due to limited data and strong linguistic variation not captured by standard benchmarks, which often assume standardized and well-edited text. We study Polish-Silesian MT using neural and rule-based systems, evaluating on SiLTT - a new Pol-Szl testset, alongside established BOUQuET and FLORES benchmarks. Results show our rule-based system is consistently strongest on SiLTT and BOUQuET datasets and that TranslateGemma fine-tuned on a curated dataset improves over strong neural baselines but does not surpass the rule-based system in dialectal settings. We release SiLTT and our best neural model to support further research.
Grzegorz Kulik, Mikołaj Pokrywka, Adam Jatowt +1
NASK National Research Institute, Poland · University of Innsbruck, Austria
LLM-based evaluators have rapidly become a standard tool for machine translation evaluation, yet the design choices that shape their behavior remain poorly understood. We present a systematic study of how instruction design, few-shot demonstrations, backbone model choice, and multi-step workflows affect span-level and ranking-based MT evaluation. We collect new expert RATE annotations on WMT24 and WMT25 English-Russian translations, with transfer validation on MQM- and ESA-annotated benchmarks across five additional language pairs. Our experiments reveal four robust findings: instruction structure shapes evaluator behavior, high-quality demonstrations matter more than quantity, frontier backbones exhibit distinct evaluator profiles (recall-, precision-, or ranking-oriented), and fusing outputs across diverse backbones outperforms single-evaluator multi-call setups. Although evaluator design exposes a large combinatorial space, several discovered design principles transfer consistently across annotation protocols and language pairs. We synthesize these findings into ORBIT, a family of LLM-based MT evaluators. Our strongest configurations achieve expert-level performance in error span detection and in Accuracy and Fluency ranking at roughly 20x lower cost than expert human annotation.
Dmitry Popov, Roman Bokhyan, Ekaterina Enikeeva +3