LLM-based machine translation evaluation can closely match human judgments, but in practice it remains largely diagnostic, with the signals rarely translating into direct quality improvements under real production constraints. We propose a two-stage, evaluator-guided automatic post-editing framework that turns MQM-style evaluation into targeted repairs: a retrieval-augmented LLM evaluator outputs structured, span-level MQM diagnoses under an explicit edit contract, and a separate LLM post-editor applies minimal edits restricted to those diagnoses. This separation improves controllability and reduces paraphrastic drift compared to one-stage "judge-and-refine" baselines. In a systematic study involving seven LLMs spanning three model providers and seven languages, our best configuration consistently improves both COMET-22 and COMETKiwi scores over one-stage post-edit methods, while the evaluator's error spans and severities show strong agreement with human MQM annotations and human editor preferences.
Figures & tables
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Evaluator LLM
Comet
CometKiwi
No post-edit
87.19
82.65
Claude 3.5 Haiku
88.28 (+1.09)
83.97 (+1.32)
Claude Sonnet 4
89.80 (+2.61)
83.59 (+0.94)
Claude Sonnet 4.5
90.37 (+3.18)
83.10 (+0.45)
Claude Opus 4.5
90.81 (+3.62)
83.27 (+0.62)
Appendix
Table 6: Comet -22 and CometKiwi scores of y(1) upon varying the evaluator LLM
Method
Model
Comet
CometKiwi
No post-edit
Claude 3.5 Sonnet
86.39
82.66
One-stage
Claude Opus 4.5
86.51 (+0.12)
82.88 (+0.22)
(GEMBA-MQM)
Two-stage
Claude Opus 4.5 +
89.93 (+3.54)
84.50 (+1.84)
Claude Sonnet 4.5
Appendix
Table 7: Comet -22 and CometKiwi scores for y(0) and y(1) for one-stage GEMBA-MQM ( i.e. with prompt P1 ) and our two-stage framework
Stage
Mean
Median
P75
P95
Stdev
Evaluator (Claude Opus 4.5)
14.34
7.80
12.54
54.38
17.38
Post-editor (Claude Sonnet 4.5)
3.30
3.10
4.15
5.04
1.08
End-to-end (single round)
17.64
10.90
16.69
59.41
17.41
Appendix
Table 8: Latency statistics (in seconds) for each stage of the two-stage pipeline, measured over 50 segments
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
LLM-based evaluators have rapidly become a standard tool for machine translation evaluation, yet the design choices that shape their behavior remain poorly understood. We present a systematic study of how instruction design, few-shot demonstrations, backbone model choice, and multi-step workflows affect span-level and ranking-based MT evaluation. We collect new expert RATE annotations on WMT24 and WMT25 English-Russian translations, with transfer validation on MQM- and ESA-annotated benchmarks across five additional language pairs. Our experiments reveal four robust findings: instruction structure shapes evaluator behavior, high-quality demonstrations matter more than quantity, frontier backbones exhibit distinct evaluator profiles (recall-, precision-, or ranking-oriented), and fusing outputs across diverse backbones outperforms single-evaluator multi-call setups. Although evaluator design exposes a large combinatorial space, several discovered design principles transfer consistently across annotation protocols and language pairs. We synthesize these findings into ORBIT, a family of LLM-based MT evaluators. Our strongest configurations achieve expert-level performance in error span detection and in Accuracy and Fluency ranking at roughly 20x lower cost than expert human annotation.
Dmitry Popov, Roman Bokhyan, Ekaterina Enikeeva +3
As MT quality increases, interest in enhanced post-editing features such as QE-derived error highlights is growing, yet evidence for their usefulness remains limited. In this work, we explore the usefulness of LLM-derived error highlights and correction suggestions based on automatic post-editing (APE). We conduct a study where professional translators (En-Nl) post-edit translations using APE error highlights and correction suggestions and compare productivity, quality and user experience to regular PE and PE with QE-derived highlights. While no condition yielded productivity or quality gains compared to regular PE, APE highlights were better received than QE-derived highlights, and correction suggestions improved overall user experience.
Fleur V. J. van Tellingen, Gautam Ranka, Dora Žugčić +4
1Leiden University Centre for Linguistics · 2Visvesvaraya National Institute of Technology · Department of Bionanoscience, Faculty of Applied Sciences, Delft University of Technology +2
Large Language Models (LLMs) have demonstrated excellent performance on Machine Translation Quality Estimation (MTQE), yet their high inference costs make them impractical for direct application. In this work, we propose applying LLMs to generate MQM-style annotations for training a COMET model: following Fernandes et al. (2023), we reckon that segment-level annotations provide a strong rationale for LLMs and are key to good segment-level QE. We propose a simplified MQM scheme, mostly restricted to top-level categories, to guide LLM selection. We present a systematic approach for the development of a GPT-4o-based prompt, called PPbMQM (Prompt-Pattern-based-MQM). We show that the resulting annotations correlate well with human annotations and that training COMET on them leads to competitive performance on segment-level QE for Chinese-English and English-German.
Sidi Wang, Sophie Arnoult, Amir Kamran
Maastricht university · Vrije Universiteit Amsterdam · Taus