MT Evaluation

MT: Machine Translation

Ask about this topic

Overview

Machine translation evaluation studies how well translations preserve meaning and meet linguistic, cultural, and practical needs, helping researchers compare systems and identify failures. Approaches range from reference-based metrics such as BLEU and chrF to neural metrics such as COMET, quality estimation without references, and human assessment using error taxonomies like MQM. Benchmarks increasingly test low-resource languages, specialized domains, document context, cultural nuance, speech, and images, while LLM judges and targeted error checks offer ways to assess qualities that simple text overlap misses. A central challenge is that automatic scores can disagree with human judgments and overlook valid variation, so reliable evaluation must balance scalable metrics with context-sensitive human review.

Latest papers

CardsList

173 papers16 papers in the last four weeks, down 38% on the four weeks before. 0.1% of all new papers.

  1. Nsanku: Evaluating Zero-Shot Translation Performance of LLMs for Ghanaian Languages

    May 5, 2026Stephen E. Moore, Mich-Seth Owusu, Akwasi Asare +13Low-Resource Language ProcessingOpen-Weight Language Models

    Department of Mathematics, University of Cape Coast, Cape Coast, Ghana · Ghana Natural Language Processing, Cape Coast, Cape Coast, Ghana · Khaya AI LTD Ghana

  2. VIDA: A Dataset for Visually Dependent Ambiguity in Multimodal Machine Translation

    May 3, 2026Jingheng Pan, Xintong Wang, Longyue Wang +3Large Vision-Language ModelsMultimodal MT

    Department of Informatics, Universität Hamburg · Alibaba Group

  3. Is Textual Similarity Invariant under Machine Translation? Evidence Based on the Political Manifesto Corpus

    May 1, 2026Daria Boratyn, Damian Brzyski, Albert Leśniak +5Semantic Textual SimilarityText Embedding Evaluation

    aJagiellonian Center for Quantitative Political Science, Jagiellonian University, Kraków, Poland · bAGH University, Kraków, Poland · cJagiellonian Center for Digital Humanities, Jagiellonian University, Kraków, Poland

  4. Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues

    Apr 30, 2026Muhammad Dehan Al Kautsar, Saeed Almheiri, Momina Ahsan +13Low-Resource Language ProcessingArabic NLP

    Mohamed bin Zayed University of Artificial Intelligence · IBM Research AI · American University in the Emirates

  5. Beyond Semantics: Measuring Fine-Grained Emotion Preservation in Small Language Model-Based Machine Translation

    Apr 30, 2026Dawid Wisniewski, Igor CzudySmall Language ModelsMachine Translation

    Pozna´n University of Technology

  6. Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation

    Apr 27, 2026Zekun Yuan, Yangfan Ye, Xiaocheng Feng +5LLM EvaluationCultural Knowledge in Language Models

    Harbin Institute of Technology · Peng Cheng Laboratory · Huawei Technologies Co., Ltd

  7. FairQE: Multi-Agent Framework for Mitigating Gender Bias in Translation Quality Estimation

    Apr 23, 2026Jinhee Jang, Juhwan Choi, Dongjin Lee +2Algorithmic FairnessMT Evaluation

    Chung-Ang University · AITRICS

  8. LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation

    Apr 20, 2026Samar M. Magdy, Fakhraddin Alwajih, Abdellah El Mekki +2Arabic NLPMachine Translation

    λThe University of British Columbia · ξMinia University · γCanada Research Chair in NLP and ML

  9. Beyond Reproduction: A Paired-Task Framework for Assessing LLM Comprehension and Creativity in Literary Translation

    Apr 20, 2026Ran Zhang, Steffen Eger, Arda Tezcan +3Creativity AssessmentLLM Evaluation

    University of Mannheim, Data and Web Science Group · University of Technology Nuremberg (UTN), Department Engineering · 5Natural Language Learning and Generation (NLLG) Lab +2

  10. Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains

    Apr 19, 2026Finn Schmidt, Jan Philip Wahle, Terry Ruas +1Annotator DisagreementAutomated Evaluation

    University of Göttingen, Germany

  11. The impact of postediting on AI generative translation in Yemeni context: Translating literary prose by ChatGPT

    Apr 17, 2026Nasim Al-wagieh, Mohammed Q. ShormaniHuman-AI CollaborationMachine Translation

    Ibb University

  12. Fabricator or dynamic translator?

    Apr 16, 2026Lisa Vasileva, Karin SimLLM EvaluationHallucination Detection

    Language Weaver

  13. XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics

    Apr 16, 2026Jingxuan Liu, Zhi Qu, Jin Tei +3MT Evaluation

    Nara Institute of Science and Technology, Japan. · Fudan University, China.

  14. Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss

    Apr 14, 2026Ronald Skorobogat, Ameya Prabhu, Matthias BethgeMultilingual Language Model EvaluationLanguage Model Generation Evaluation

    Tübingen AI Center, University of Tübingen

  15. Should We be Pedantic About Reasoning Errors in Machine Translation?

    Apr 10, 2026Calvin Bao, Marine CarpuatMachine TranslationMT Evaluation

    Department of Computer Science, University of Maryland, College Park, MD 20742, USA

  16. MERIT: Multilingual Expert-Reward Informed Tuning for Chinese-Centric Low-Resource Machine Translation

    Apr 6, 2026Zhixiang Lu, Chong Zhang, Chenyu Xue +4Machine TranslationLow-Resource MT

    Xi’an Jiaotong-Liverpool University

  17. Blinded Radiologist and LLM-Based Evaluation of LLM-Generated Japanese Translations of Chest CT Reports: Comparative Study

    Apr 2, 2026Yosuke Yamagishi, Atsushi Takamatsu, Yasunori Hamaguchi +4Radiology Report EvaluationMT Evaluation

    Division of Radiology and Biomedical Engineering, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan · Department of Radiology, The University of Tokyo Hospital, Tokyo, Japan · Department of Radiology, Kanazawa University Graduate School of Medical Sciences, Ishikawa, Japan +2

  18. GaelEval: Benchmarking LLM Performance for Scottish Gaelic

    Apr 2, 2026Peter Devine, William Lamb, Beatrice Alex +5Multilingual Language Model EvaluationLLM Evaluation

    University of Edinburgh · Lancaster University · University of Cardiff +2

  19. Diagnosing Translated Benchmarks: An Automated Quality Assurance Study of the EU20 Benchmark Suite

    Apr 2, 2026Klaudia Thellmann, Bernhard Stadler, Michael FärberBenchmark AuditingMT Evaluation

    TU Dresden and ScaDS.AI · InfAI e.V.

  20. Open Machine Translation for Esperanto

    Mar 31, 2026Ona de Gibert, Lluís de GibertMachine TranslationMT Evaluation

    University of Helsinki, Department of Digital Humanities · Kataluna Esperanto-Asocio (KEA) · Sennacieca Asocio Tutmonda (SAT)

  21. Translation or Recitation? Calibrating Evaluation Scores for Machine Translation of Extremely Low-Resource Languages

    Mar 26, 2026Danlu Chen, Ka Sing He, Jiahe Tian +4Low-Resource MTMT Evaluation

    UC San Diego · New York University · Durham University +3

  22. Toward domain-specific machine translation and quality estimation systems

    Mar 26, 2026Javad Pourmostafa Roshan SharamiMachine TranslationDomain Adaptation

  23. MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation

    Mar 25, 2026Gengluo Li, Chengquan Zhang, Yupu Liang +9Image Text TranslationMachine Translation

    Institute of Information Engineering, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Tencent +3

  24. Multi-Method Validation of Large Language Model Medical Translation Across High- and Low-Resource Languages

    Mar 23, 2026Chukwuebuka Anyaegbuna, Eduardo Juan Perez Guerrero, Jerry Liu +6Machine TranslationMultilingual Language Model Evaluation

  25. Span-Level Machine Translation Meta-Evaluation

    Mar 20, 2026Stefano Perrella, Eric Morales Agostinho, Hugo ZaragozaMT Evaluation

    Sapienza University of Rome · Amazon

  26. ConGA: Guidelines for Contextual Gender Annotation. A Framework for Annotating Gender in Machine Translation

    Mar 18, 2026Argentina Anna Rescigno, Eva Vanmassenhove, Johanna MontiMachine TranslationMT Evaluation

    University of Pisa · University of Naples “L’Orientale” · Tilburg University

  27. From Words to Worlds: Benchmarking Cross-Cultural Cultural Understanding in Machine Translation

    Mar 18, 2026Bangju Han, Yingqi Wang, Huang Qing +6Cross-Cultural Language Model EvaluationMachine Translation

    Xinjiang Technical Institute of Physics & Chemistry, Chinese Academy of Sciences, Urumqi, China · University of Chinese Academy of Sciences, Beijing, China (second affiliation) · Xinjiang Laboratory of Minority Speech and Language Information Processing, Urumqi, China (third affiliation)

  28. Omnilingual MT: Machine Translation for 1,600 Languages

    Mar 17, 2026Omnilingual MT Team, Belen Alastruey, Niyati Bafna +29Machine TranslationLow-Resource MT

    FAIR at Meta

  29. Bidirectional Chinese and English Passive Sentences Dataset for Machine Translation

    Mar 16, 2026Xinyue Ma, Pol Pastells, Mireia Farrús +1Machine TranslationMT Evaluation

    Centre de Llenguatge i Computacio (CLiC), Universitat de Barcelona, Spain · Institut de Recerca en Sistemes Complexos (UBICS), Universitat de Barcelona, Spain

  30. Developing an English-Efik Corpus and Machine Translation System for Digitization Inclusion

    Mar 16, 2026Offiong Bassey Edet, Mbuotidem Sunday Awak, Emmanuel Oyo-Ita +2Machine TranslationLow-Resource MT

    University of Cross River State · ML Collective · Arthur Jarvis University +1