MT Evaluation

MT: Machine Translation

Machine translation evaluation studies how well translations preserve meaning and meet linguistic, cultural, and practical needs, helping researchers compare systems and identify failures. Approaches range from reference-based metrics such as BLEU and chrF to neural metrics such as COMET, quality estimation without references, and human assessment using error taxonomies like MQM. Benchmarks increasingly test low-resource languages, specialized domains, document context, cultural nuance, speech, and images, while LLM judges and targeted error checks offer ways to assess qualities that simple text overlap misses. A central challenge is that automatic scores can disagree with human judgments and overlook valid variation, so reliable evaluation must balance scalable metrics with context-sensitive human review.

Latest papers

173 papers16 papers in the last four weeks, down 38% on the four weeks before. 0.1% of all new papers.

Towards Privacy-Preserving Machine Translation at the Inference Stage: A New Task and Benchmark
Mar 16, 2026Wei Shao, Lemao Liu, Yinqiao Li +3Privacy-Preserving ML Machine Translation
City University of Hong Kong · City University of Hong Kong Shenzhen Research Institute · Tencent AI Lab, Shenzhen
Parameter-Efficient Quality Estimation via Frozen Recursive Models
Mar 15, 2026Umar Abubacar, Roman Bauer, Diptesh KanojiaLow-Resource Language Processing MT Evaluation
NICE Research Group, University of Surrey, UK · Surrey Institute for People-Centred AI (PAI), University of Surrey, UK
Is Human Annotation Necessary? Iterative MBR Distillation for Error Span Detection in Machine Translation
Mar 13, 2026Boxuan Lyu, Haiyue Song, Zhi QuMBR Decoding LLM-Assisted Annotation
Institute of Science Tokyo · National Institute of Information and Communications Technology, Japan
Semi-Synthetic Parallel Data for Translation Quality Estimation: A Case Study of Dataset Building for an Under-Resourced Language Pair
Mar 12, 2026Assaf Siani, Anna Kernerman, Ilan KernermanMT Evaluation Synthetic Data Generation
Large Language Models as Annotators for Machine Translation Quality Estimation
Mar 11, 2026Sidi Wang, Sophie Arnoult, Amir KamranMT Evaluation LLM-Assisted Annotation
Maastricht university · Vrije Universiteit Amsterdam · Taus
IMTBench: A Multi-Scenario Cross-Modal Collaborative Evaluation Benchmark for In-Image Machine Translation
Mar 11, 2026Jiahao Lyu, Pei Fu, Zhenhang Li +7Image Text Translation Multimodal MT
Institute of Information Engineering, Chinese Academy of Science, China · MiLM Plus, Xiaomi Inc, China · School of Cyber Security, University of Chinese Academy of Science, China +3
ASCAT: An Arabic Scientific Corpus and Benchmark for Advanced Translation Evaluation
Mar 10, 2026Serry Sibaee, Khloud Al Jallad, Zineb Yousfi +4Machine Translation MT Evaluation
Prince Sultan University · SySSR · NAMAA Community +2
A Dataset for Probing Translationese Preferences in English-to-Swedish Translation
Mar 9, 2026Jenny Kunz, Anja Jarochenko, Marcel BollmannMachine Translation MT Evaluation
Department of Computer and Information Science, Linköping University
Gender Bias in MT for a Genderless Language: New Benchmarks for Basque
Mar 9, 2026Amaia Murillo, Olatz-Perez-de-Viñaspre, Naiara PerezGender Bias in Language Models Machine Translation
HiTZ Center - Ixa, University of the Basque Country UPV/EHU
Domain-Specific Quality Estimation for Machine Translation in Low-Resource Scenarios
Mar 7, 2026Namrata Patil Gurav, Akashdeep Ranu, Archchana Sindhujan +1MT Evaluation Machine Translation
Institute for People-Centred AI, University of Surrey, United Kingdom
Hindsight Quality Prediction Experiments in Multi-Candidate Human-Post-Edited Machine Translation
Mar 4, 2026Malik Marmonier, Benoît Sagot, Rachel BawdenMachine Translation MT Evaluation
Inria, Paris Center
Evaluating LLM-Based Translation of a Low-Resource Technical Language: The Medical and Philosophical Greek of Galen
Feb 27, 2026James L. Zainaldin, Cameron Pattison, Manuela Marai +2Machine Translation Low-Resource MT
Department of Classical and Mediterranean Studies, Vanderbilt University, Nashville, TN 37235, USA · Center for Hellenic Studies, Harvard University, Washington, DC 20008, USA · Department of Philosophy, Vanderbilt University, Nashville, TN 37235, USA +3
Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets
Feb 25, 2026Hanna Yukhymenko, Anton Alexandrov, Martin VechevMultilingual Language Model Evaluation MT Evaluation
INSAIT, Sofia University "St. Kliment Ohridski" · ETH Zurich
DEEP: Docker-based Execution and Evaluation Platform
Feb 23, 2026Sergio Gómez González, Miguel Domingo, Francisco CasacubertaAutomated Evaluation MT Evaluation
PRHLT Research Center - Universitat Politècnica de València · ValgrAI - Valencian Graduate School and Research Network for Artificial Intelligence
Evaluating Extremely Low-Resource Machine Translation: A Comparative Study of ChrF++ and BLEU Metrics
Feb 19, 2026Sanjeev Kumar, Preethi Jyothi, Pushpak BhattacharyyaMT Evaluation Machine Translation
Dept. of CSE, IIT Bombay, Mumbai, India
LuxMT Technical Report
Feb 17, 2026Nils RehlingerMachine Translation Low-Resource MT
University of Luxembourg / Esch-Belval, Esch-sur-Alzette, Luxembourg
GRRM: Group Relative Reward Modeling for Machine Translation
Feb 15, 2026Sen Yang, Shanbo Cheng, Lu Xu +2Reward Modeling RL for Language Models
National Key Laboratory for Novel Software Technology, Nanjing University · Singapore University of Technology and Design
Is Knowledge Distillation Actually Greener? A Case Study in Machine Translation
Feb 10, 2026Joseph Attieh, Timothee Mickus, Anne-Laure Ligozat +2Energy-Efficient ML Knowledge Distillation
University of Helsinki · Université Paris-Saclay, CNRS, LISN
Benchmarking Machine Translation on Chinese Social Media Texts
Jan 30, 2026Kaiyan Zhao, Zheyong Xie, Zhongtao Miao +3Machine Translation MT Evaluation
1The University of Tokyo · 2Xiaohongshu Inc. · 3Tsinghua University
HOMURA: Taming the Sand-Glass for Time-Constrained LLM Translation via Reinforcement Learning
Jan 15, 2026Ziang Cui, Mengran Yu, Chenyu Shi +2Multilingual Language Models RL for Language Models
Bilibili Inc., Shanghai, China
Is MT Ready for the Next Crisis or Pandemic?
Jan 15, 2026Vipasha Bansal, William D. Lewis, Nicole Mihaylova +2Machine Translation MT Evaluation
University of Washington
Evaluating Large Language Models on Urdu Idioms
Oct 20, 2025Muhammad Farmal Khan, Mousumi AkterMultilingual Language Model Evaluation Low-Resource Language Processing
Technical University Dortmund · Research Center Trustworthy Data Science and Security
AskQE: Question Answering as Automatic Evaluation for Machine Translation
Apr 15, 2025Dayeon Ki, Kevin Duh, Marine CarpuatMT Evaluation
University of Maryland · Johns Hopkins University

Mar 16, 2026J/K move · Enter open · S save

Towards Privacy-Preserving Machine Translation at the Inference Stage: A New Task and Benchmark

Wei Shao, Lemao Liu, Yinqiao Li, Guoping Huang+2

City University of Hong Kong · City University of Hong Kong Shenzhen Research Institute · Tencent AI Lab, Shenzhen

Privacy-Preserving ML Machine Translation MT Evaluation

Open paper AskPDF arXiv

Current online translation services require sending user text to cloud servers, posing a risk of privacy leakage when the text contains sensitive information. This risk hinders the application of online translation services in privacy-sensitive scenarios. One way to mitigate this risk for online translation services is introducing privacy protection mechanisms targeting the inference stage of translation models. However, compared to subfields of NLP like text classification and summarization, the machine translation research community has limited exploration of privacy protection during the inference stage. There is no clearly defined privacy protection task for the inference stage, dedicated evaluation datasets and metrics, and reference benchmark methods. The absence of these elements has seriously constrained researchers' in-depth exploration of this direction. To bridge this gap, this paper proposes a novel "Privacy-Preserving Machine Translation" (PPMT) task, aiming to protect the private information in text during the model inference stage. For this task, we constructed three benchmark test datasets, designed corresponding evaluation metrics, and proposed a series of benchmark methods as a starting point for this task. The definition of privacy is complex and diverse. Considering that named entities often contain a large amount of personal privacy and commercial secrets, we have focused our research on protecting only the named entity's privacy in the text. We expect this research work will provide a new perspective and a solid foundation for the privacy protection problem in machine translation.