MT Evaluation

MT: Machine Translation

Overview

Machine translation evaluation studies how well translations preserve meaning and meet linguistic, cultural, and practical needs, helping researchers compare systems and identify failures. Approaches range from reference-based metrics such as BLEU and chrF to neural metrics such as COMET, quality estimation without references, and human assessment using error taxonomies like MQM. Benchmarks increasingly test low-resource languages, specialized domains, document context, cultural nuance, speech, and images, while LLM judges and targeted error checks offer ways to assess qualities that simple text overlap misses. A central challenge is that automatic scores can disagree with human judgments and overlook valid variation, so reliable evaluation must balance scalable metrics with context-sensitive human review.

Latest papers

CardsList

181 papers10 papers in the last four weeks, down 41% on the four weeks before. 0.1% of all new papers.

  1. Cross-cultural Adaptation of Recipes for Low-Resource Languages: A Benchmark and Multilingual Culinary Lexicon for Hausa, Yorùbá, and Igbo

    Nov 17, 2026Nuhu Ibrahim, Maryam Abdulazeez, Fatsuma Jauro +3Machine TranslationLow-Resource MT

  2. Pterodactyl: Pushing the Limits of Efficient Automated Translation Evaluation

    Nov 10, 2026Evelyn Y. Garland, Carola F. BergerMT EvaluationAutomated Evaluation

  3. Current state of LLMs for Arabic dialectal machine translation

    Nov 1, 2026Josef Jon, Rawan Bondok, Ondřej BojarDialect IdentificationMachine Translation

  4. CULTURE-CMT: Benchmarking Contextualized Social Media Translation

    Nov 1, 2026Linjuan Wu, Dingming Li, Yiwen Qiu +5Machine TranslationContext-Aware MT

  5. Retrieval-Enhanced Cultural Evaluation for Social Media Translation Training

    Nov 1, 2026Linjuan Wu, Yiwen Qiu, Xinze Lyu +11Machine TranslationMT Evaluation

  6. ImgMT: Wikipedia-based Dataset For Evaluating Text-in-Image Translation

    Oct 26, 2026Šárka Uramová, Petar Kirilov Petrov, Josef Jon +2Image Text TranslationMultimodal Dataset

  7. The Lay of the Land in Idiom-Translation Quality Estimation

    Oct 26, 2026Malik Marmonier, Benoît Sagot, Rachel BawdenMT EvaluationFigurative Language Understanding

  8. Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators

    Oct 24, 2026Dmitry Popov, Roman Bokhyan, Ekaterina Enikeeva +3MT EvaluationLLM-as-a-Judge

  9. Communicating with the Entire Population: Machine Translation of Emergency Alerts and Warnings

    Oct 23, 2026Jen Wilson, Vipasha Bansal, William D. LewisMachine TranslationDisaster Response

  10. Towards Figurative Language Understanding in Moroccan Darija: An Evaluation of Large Language Models

    Oct 18, 2026Jihad R'baiti, Youssef Hmamouche, Emmanuele Chersoni +2Figurative Language UnderstandingMachine Translation

  11. Does Machine Translation Retain Media Bias?

    Oct 14, 2026Silvia V. Gargova, Tim Menzner, Jochen L. LeidnerMachine TranslationMT Evaluation

  12. SIREN: Situation-aware Index for Response and Emergency ActioNability in Crisis Translation

    Oct 12, 2026Maria Carmen Staiano, Lifeng Han, Johanna Monti +1Machine TranslationMT Evaluation

  13. PatTransBench: an ipc-stratified, multi-dimensional benchmark for patent machine translation evaluation

    Oct 12, 2026Xiaowei Su, Xusen ChengMT Evaluation

  14. The Constitution Speaks Nheengatu: An Open MT System and Reproducible Corpus Pipeline for the Amazonian Língua Geral

    Oct 10, 2026Fabrício Carraro, Rodolfo Joel Zevallos, Rodrigo Gonçalves de Souza +2Machine TranslationLow-Resource MT

  15. Generative AI translations in high-stakes emergency messaging

    Oct 6, 2026Nune Ayvazan, Anthony Pym, Yu HaoDisaster ResponseMachine Translation

  16. Precision over Scale: A Polish-Silesian Benchmark and a Translation System Outperforming Open-Source and Commercial Models

    Oct 1, 2026Grzegorz Kulik, Mikołaj Pokrywka, Adam Jatowt +1Low-Resource MTDialectal MT

  17. TermJudge: A Document-Level Metric Judging, Not Counting, Terminology in Machine Translation Evaluation

    Sep 28, 2026Nicolas Dahan, Fran{\cc}ois Yvon, Rachel BawdenAutomated EvaluationDocument-Level MT

  18. TTLab at AlexandriaX-2026: A Fine-Tuned Surface Tagger for Arabic Machine-Translation Error-Span Detection and Classification

    Sep 24, 2026Ali Abusaleh, Bhuvanesh Verma, Alexander MehlerArabic NLPMachine Translation

  19. COILD: An Indic-Centric Parallel Corpus and Benchmark for Machine Translation Across Indian Languages

    Sep 23, 2026Kshetrimayum Boynao Singh, Nitin Kumar Mishra, Palash Pratim Dutta +22Low-Resource MTMachine Translation

  20. MICRO: Multi-Fidelity Active Search for Severe Error Discovery

    Sep 22, 2026Orlando Leone, Niclas Pokel, Pehuén Moure +2Human-in-the-Loop AnnotationMT Evaluation

  21. The Ongiini-Eval-OW Benchmark: A Concept Paper for the Planned Benchmarking of Machine Translation and Large Language Models on Oshindonga and Oshikwanyama

    Sep 22, 2026Sebastian KüpersMT EvaluationLow-Resource MT

  22. AlexandriaX 2026: The First Shared Task on Dialectal Arabic Machine Translation

    Sep 19, 2026Abdellah El Mekki, AbdelRahim A. Elmadany, Samar M. Magdy +6Context-Aware MTDialectal MT