Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

Hala Almaghout; Christian Federmann; Qin Gao

cs.CLOct 7, 2026

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

Authors: Hala Almaghout, Christian Federmann, Qin Gao

Organizations: Apple

Abstract

Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human annotators. We present results on a long-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs and domains. Our results show that while LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks, both vary substantially across annotation schemes, language pairs and domains and remain unreliable for most settings. Furthermore, we identify challenges facing both human and LLM annotators: humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. Our results highlight the potential for improvement for both human and LLM annotation performance, possibly through human-LLM collaborative annotation pipelines that address the reliability issues identified in this work.

Figures & tables

Figure 1 : Inter-annotator agreement on the WMT23 MQM test set, showing Krippendorff’s

\alpha

for MQM scores, major error counts, and minor error counts.

Error category	H-H	Gem	GPT	DS	Qwen
Span overlap	.51	.43 ∗	.44 ∗	.39 ∗	.39 ∗
Character overlap	.43	.39 ∗	.31 ∗	.27 ∗	.35 ∗
Severity minor	.33	.17 ∗	.16 ∗	.13 ∗	.12 ∗
Mistranslation	.37	.35 ∗	.37	.32 ∗	.34 ∗
Severity major	.45	.43 ∗	.43 ∗	.31 ∗	.41 ∗
Awkward	.19	.13 ∗	.12 ∗	.01 ∗	.02 ∗

Table 1 : Span-annotation F1 by error category on the WMT23 MQM test set, averaged over language pairs and ordered by decreasing category frequency. H-H is agreement between human annotation rounds; the remaining columns are each LLM’s agreement with the human rounds. Gem = Gemini 2.5 Flash, GPT = GPT-OSS-120B, DS = DeepSeek-V3.1, Qwen = Qwen3.5-397B. ∗ marks a difference from H-H that is statistically significant at

p<0.05

. Categories whose human-human baseline rests on fewer than 5 source segments are omitted, since no comparison against it is estimable: Address format, Time format.

Figure 2 : Krippendorff’s $\alpha$ agreement on ESA scores for each language pair and domain on the WMT25 ESA test set. Within each domain, the columns are H-H (Human with Human), Gem-H (Gemini-2.5-Flash with Human), GPT-H (GPT-OSS-120B with Human), DS-H (DeepSeek-V3.1 with Human), Qwen-H (Qwen3.5-397B with Human). The dashed line on the colorbar marks $\alpha=0.667$ . $*$ marks cells where LLM-human agreement differs significantly from the human-human agreement of the same cell (paired cluster bootstrap on source segments, $p<0.05$ ).

Domain	#LPs	Gemini	GPT-OSS	DeepSeek	Qwen
Dialogue	2	1/0	0/1	0/1	1/0
Edu	2	0/1	0/1	1/1	0/1
Literary	12	4/6	1/9	1/7	3/6
News	14	1/6	0/11	1/11	1/7
Social	14	2/7	2/10	0/8	1/9
All domains	44	8/20	3/32	3/28	6/23

Table 2: Language pairs per domain whose LLM-human Krippendorff’s

\alpha

agreement on ESA scores differs significantly from human-human agreement, on the WMT25 ESA test set, shown separately for each LLM. The first number counts language pairs where LLM-human Krippendorff’s

\alpha

significantly exceeds human-human agreement, the second those where human-human significantly exceeds LLM-human. LPs with no significant difference in either direction appear in neither number. Gemini = Gemini 2.5 Flash, GPT-OSS = GPT-OSS-120B, DeepSeek = DeepSeek-V3.1, Qwen = Qwen3.5-397B. Benjamini–Hochberg FDR applied separately to each LLM’s comparisons,

p<0.05

Span agreement	H-H	Gem	GPT	DS	Qwen
Character overlap	.43	.17 ∗	.16 ∗	.17 ∗	.18 ∗
Span overlap	.45	.30 ∗	.31 ∗	.31 ∗	.31 ∗
Minor severity	.23	.15 ∗	.12 ∗	.16 ∗	.15 ∗
Major severity	.32	.22 ∗	.21 ∗	.21 ∗	.22 ∗
No error	.47	.37 ∗	.25 ∗	.05 ∗	.29 ∗
Missing span	.20	.15 ∗	.09 ∗	.07 ∗	.11 ∗

Table 3 : Span-annotation F1 by span-agreement measure on the WMT25 ESA test set, averaged over language pairs and domains and ordered by decreasing frequency. H-H is agreement between human annotation rounds; the remaining columns are each LLM’s agreement with the human rounds. Gem = Gemini 2.5 Flash, GPT = GPT-OSS-120B, DS = DeepSeek-V3.1, Qwen = Qwen3.5-397B. ∗ marks a difference from H-H that is statistically significant at

p<0.05

Figure 3 : Inter-annotator agreement for MQM scores on the Long-Context Web MQM test set for en_US-hi_IN and en_US-th_TH, showing Gemini 2.5 Flash’s agreement with each individual human annotation round and agreement between human annotation rounds.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4 : Inter-annotator agreement on the Long-Context Web MQM test set, showing Krippendorff’s

\alpha

for MQM scores, major error counts, and minor error counts, averaged over the language pairs of each translation direction group (en-X, X-en, X-Y).

Direction	#LPs	Gemini	GPT-OSS	DeepSeek	Qwen
en-X	19	4/2	4/4	0/16	1/4
X-en	19	1/3	2/3	0/18	0/8
X-Y	32	20/2	19/4	3/21	9/6
All	70	25/7	25/11	3/55	10/18

Appendix

Table 4: Language pairs per translation direction whose LLM-human Krippendorff’s

\alpha

on MQM scores differs significantly from human-human agreement, on the Long-Context Web MQM test set. The first number counts language pairs where LLM-human Krippendorff’s

\alpha

p<0.05

Error category	H-H	Gem	GPT	DS	Qwen
Span overlap	.36	.33 ∗	.24 ∗	.23 ∗	.28 ∗
Character overlap	.28	.25 ∗	.18 ∗	.16 ∗	.20 ∗
Severity minor	.27	.19 ∗	.15 ∗	.15 ∗	.18 ∗
Mistranslation	.29	.30 ∗	.19 ∗	.12 ∗	.21 ∗
Severity major	.20	.24 ∗	.16 ∗	.10 ∗	.17 ∗
Awkward	.13	.06 ∗	.07 ∗	.03 ∗	.07 ∗

Appendix

Table 5 : Span-annotation F1 by error category on the Long-Context Web MQM test set, averaged over language pairs and ordered by decreasing category frequency. H-H is agreement between human annotation rounds; the remaining columns are each LLM’s agreement with the human rounds. Gem = Gemini 2.5 Flash, GPT = GPT-OSS-120B, DS = DeepSeek-V3.1, Qwen = Qwen3.5-397B. ∗ marks a difference from H-H that is statistically significant at

p<0.05

. Categories whose human-human baseline rests on fewer than 5 source segments are omitted, since no comparison against it is estimable: Address, Mismatched locale, Telephone.

Figure 5 : Krippendorff’s $\alpha$ agreement on MQM scores, major and minor error counts for each of the 70 language pairs of the Long-Context Web MQM test set, grouped by translation direction (en-X, X-en, X-Y). Within each group the columns are H-H (Human with Human), Gem-H (Gemini-2.5-Flash with Human), GPT-H (GPT-OSS-120B with Human), DS-H (DeepSeek-V3.1 with Human), Qwen-H (Qwen3.5-397B with Human). The dashed line on the colorbar marks $\alpha=0.667$ . $*$ marks cells where LLM-human agreement differs significantly from the human-human agreement of the same cell (paired cluster bootstrap on source segments, $p<0.05$ ).

Explore similar work

Apr 20, 2026cs.CL

LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation

Existing MT evaluation frameworks, including automatic metrics and human evaluation schemes such as Multidimensional Quality Metrics (MQM), are largely language-agnostic. However, they often fail to capture dialect- and culture-specific errors in diglossic languages (e.g., Arabic), where translation failures stem from mismatches in language variety, content coverage, and pragmatic appropriateness rather than surface form alone.We introduce LQM: Linguistically Motivated Multidimensional Quality Metrics for MT. LQM is a hierarchical error taxonomy for diagnosing MT errors through six linguistically grounded levels: sociolinguistics, pragmatics, semantics, morphosyntax, orthography, and graphetics (Figure 1). We construct a bidirectional parallel corpus of 3,850 sentences (550 per variety) spanning seven Arabic dialects (Egyptian, Emirati, Jordanian, Mauritanian, Moroccan, Palestinian, and Yemeni), derived from conversational, culturally rich content. We evaluate six LLMs in a zero-shot setting and conduct expert span-level human annotation using LQM, producing 6,113 labeled error spans across 3,495 unique erroneous sentences, along with severity-weighted quality scores. We complement this analysis with an automatic metric (spBLEU). Though validated here on Arabic, LQM is a language-agnostic framework designed to be easily applied to or adapted for other languages. LQM annotated errors data, prompts, and annotation guidelines are publicly available at https://github.com/UBC-NLP/LQM_MT.

Samar M. Magdy, Fakhraddin Alwajih, Abdellah El Mekki +2

λThe University of British Columbia · ξMinia University · γCanada Research Chair in NLP and ML

Arabic NLP Machine Translation MT Evaluation

Ask PDF

May 24, 2026cs.CL

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation

Machine-translated benchmarks are widely used to assess the multilingual capabilities of large language models (LLMs), yet translation errors in these benchmarks remain underexplored, raising concerns about the reliability and comparability of multilingual evaluation. We address two practical gaps: (i) how well automatic MQM-style error spans from LLM judges and a span-aware QE baseline (xCOMET-XXL) match expert human span annotations on benchmark translations, and (ii) how strongly translation errors (as opposed to source-side issues in the English original) explain accuracy drops on translated benchmarks. We find that span agreement is non-trivial on naturally occurring benchmark translations, and that target-side translation errors are consistently associated with measurable, percentage-point drops in translated accuracy even after controlling for English correctness and source-side anomalies.

Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber +1

TUD Dresden University of Technology · 2InfAI e.V. · 3Amazon +1

Multilingual Language Model Evaluation MT Evaluation

Ask PDF

May 15, 2026cs.CL

CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMs

Current state-of-the-art Quality Estimation (QE) in machine translation relies on massive, proprietary LLMs, raising data privacy concerns. We demonstrate that smaller, open-source LLMs (<30B parameters) are a viable, cost-effective and privacy-preserving alternative. Using a single-pass prompting strategy, our models simultaneously generate quality scores, MQM error annotations, suggested error corrections, and full post-editions. Our analysis shows these models achieve highly competitive system-level correlations with human judgments that outperform traditional neural metrics, fine-tuned models, and human inter-annotator agreement, effectively approximating the capabilities of much larger proprietary LLMs.

Kamil Guttmann, Zofia Fraś, Artur Nowakowski +1

Laniqo, Pozna´n, Poland · Faculty of Mathematics and Computer Science, Adam Mickiewicz University, Pozna´n, Poland

Small Language Models MT Evaluation

Ask PDF