Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

Hala Almaghout; Christian Federmann; Qin Gao

cs.CLOct 7, 2026

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

Authors: Hala Almaghout, Christian Federmann, Qin Gao

Organizations: Apple

Abstract

Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human annotators. We present results on a long-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs and domains. Our results show that while LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks, both vary substantially across annotation schemes, language pairs and domains and remain unreliable for most settings. Furthermore, we identify challenges facing both human and LLM annotators: humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. Our results highlight the potential for improvement for both human and LLM annotation performance, possibly through human-LLM collaborative annotation pipelines that address the reliability issues identified in this work.

Figures & tables

Figure 1 : Inter-annotator agreement on the WMT23 MQM test set, showing Krippendorff’s

\alpha

for MQM scores, major error counts, and minor error counts.

Error category	H-H	Gem	GPT	DS	Qwen
Span overlap	.51	.43 ∗	.44 ∗	.39 ∗	.39 ∗
Character overlap	.43	.39 ∗	.31 ∗	.27 ∗	.35 ∗
Severity minor	.33	.17 ∗	.16 ∗	.13 ∗	.12 ∗
Mistranslation	.37	.35 ∗	.37	.32 ∗	.34 ∗
Severity major	.45	.43 ∗	.43 ∗	.31 ∗	.41 ∗
Awkward	.19	.13 ∗	.12 ∗	.01 ∗	.02 ∗

Table 1 : Span-annotation F1 by error category on the WMT23 MQM test set, averaged over language pairs and ordered by decreasing category frequency. H-H is agreement between human annotation rounds; the remaining columns are each LLM’s agreement with the human rounds. Gem = Gemini 2.5 Flash, GPT = GPT-OSS-120B, DS = DeepSeek-V3.1, Qwen = Qwen3.5-397B. ∗ marks a difference from H-H that is statistically significant at

p<0.05

. Categories whose human-human baseline rests on fewer than 5 source segments are omitted, since no comparison against it is estimable: Address format, Time format.

Figure 2 : Krippendorff’s $\alpha$ agreement on ESA scores for each language pair and domain on the WMT25 ESA test set. Within each domain, the columns are H-H (Human with Human), Gem-H (Gemini-2.5-Flash with Human), GPT-H (GPT-OSS-120B with Human), DS-H (DeepSeek-V3.1 with Human), Qwen-H (Qwen3.5-397B with Human). The dashed line on the colorbar marks $\alpha=0.667$ . $*$ marks cells where LLM-human agreement differs significantly from the human-human agreement of the same cell (paired cluster bootstrap on source segments, $p<0.05$ ).

Domain	#LPs	Gemini	GPT-OSS	DeepSeek	Qwen
Dialogue	2	1/0	0/1	0/1	1/0
Edu	2	0/1	0/1	1/1	0/1
Literary	12	4/6	1/9	1/7	3/6
News	14	1/6	0/11	1/11	1/7
Social	14	2/7	2/10	0/8	1/9
All domains	44	8/20	3/32	3/28	6/23

Table 2: Language pairs per domain whose LLM-human Krippendorff’s

\alpha

agreement on ESA scores differs significantly from human-human agreement, on the WMT25 ESA test set, shown separately for each LLM. The first number counts language pairs where LLM-human Krippendorff’s

\alpha

significantly exceeds human-human agreement, the second those where human-human significantly exceeds LLM-human. LPs with no significant difference in either direction appear in neither number. Gemini = Gemini 2.5 Flash, GPT-OSS = GPT-OSS-120B, DeepSeek = DeepSeek-V3.1, Qwen = Qwen3.5-397B. Benjamini–Hochberg FDR applied separately to each LLM’s comparisons,

p<0.05

Span agreement	H-H	Gem	GPT	DS	Qwen
Character overlap	.43	.17 ∗	.16 ∗	.17 ∗	.18 ∗
Span overlap	.45	.30 ∗	.31 ∗	.31 ∗	.31 ∗
Minor severity	.23	.15 ∗	.12 ∗	.16 ∗	.15 ∗
Major severity	.32	.22 ∗	.21 ∗	.21 ∗	.22 ∗
No error	.47	.37 ∗	.25 ∗	.05 ∗	.29 ∗
Missing span	.20	.15 ∗	.09 ∗	.07 ∗	.11 ∗

Table 3 : Span-annotation F1 by span-agreement measure on the WMT25 ESA test set, averaged over language pairs and domains and ordered by decreasing frequency. H-H is agreement between human annotation rounds; the remaining columns are each LLM’s agreement with the human rounds. Gem = Gemini 2.5 Flash, GPT = GPT-OSS-120B, DS = DeepSeek-V3.1, Qwen = Qwen3.5-397B. ∗ marks a difference from H-H that is statistically significant at

p<0.05

Figure 3 : Inter-annotator agreement for MQM scores on the Long-Context Web MQM test set for en_US-hi_IN and en_US-th_TH, showing Gemini 2.5 Flash’s agreement with each individual human annotation round and agreement between human annotation rounds.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4 : Inter-annotator agreement on the Long-Context Web MQM test set, showing Krippendorff’s

\alpha

for MQM scores, major error counts, and minor error counts, averaged over the language pairs of each translation direction group (en-X, X-en, X-Y).

Direction	#LPs	Gemini	GPT-OSS	DeepSeek	Qwen
en-X	19	4/2	4/4	0/16	1/4
X-en	19	1/3	2/3	0/18	0/8
X-Y	32	20/2	19/4	3/21	9/6
All	70	25/7	25/11	3/55	10/18

Appendix

Table 4: Language pairs per translation direction whose LLM-human Krippendorff’s

\alpha

on MQM scores differs significantly from human-human agreement, on the Long-Context Web MQM test set. The first number counts language pairs where LLM-human Krippendorff’s

\alpha

p<0.05

Error category	H-H	Gem	GPT	DS	Qwen
Span overlap	.36	.33 ∗	.24 ∗	.23 ∗	.28 ∗
Character overlap	.28	.25 ∗	.18 ∗	.16 ∗	.20 ∗
Severity minor	.27	.19 ∗	.15 ∗	.15 ∗	.18 ∗
Mistranslation	.29	.30 ∗	.19 ∗	.12 ∗	.21 ∗
Severity major	.20	.24 ∗	.16 ∗	.10 ∗	.17 ∗
Awkward	.13	.06 ∗	.07 ∗	.03 ∗	.07 ∗

Appendix

Table 5 : Span-annotation F1 by error category on the Long-Context Web MQM test set, averaged over language pairs and ordered by decreasing category frequency. H-H is agreement between human annotation rounds; the remaining columns are each LLM’s agreement with the human rounds. Gem = Gemini 2.5 Flash, GPT = GPT-OSS-120B, DS = DeepSeek-V3.1, Qwen = Qwen3.5-397B. ∗ marks a difference from H-H that is statistically significant at

p<0.05

. Categories whose human-human baseline rests on fewer than 5 source segments are omitted, since no comparison against it is estimable: Address, Mismatched locale, Telephone.

Figure 5 : Krippendorff’s $\alpha$ agreement on MQM scores, major and minor error counts for each of the 70 language pairs of the Long-Context Web MQM test set, grouped by translation direction (en-X, X-en, X-Y). Within each group the columns are H-H (Human with Human), Gem-H (Gemini-2.5-Flash with Human), GPT-H (GPT-OSS-120B with Human), DS-H (DeepSeek-V3.1 with Human), Qwen-H (Qwen3.5-397B with Human). The dashed line on the colorbar marks $\alpha=0.667$ . $*$ marks cells where LLM-human agreement differs significantly from the human-human agreement of the same cell (paired cluster bootstrap on source segments, $p<0.05$ ).

Explore similar work

LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation
Apr 20, 2026Samar M. Magdy, Fakhraddin Alwajih, Abdellah El Mekki +2Arabic NLP Machine Translation
λThe University of British Columbia · ξMinia University · γCanada Research Chair in NLP and ML
Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation
May 24, 2026Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber +1Multilingual Language Model Evaluation MT Evaluation
TUD Dresden University of Technology · 2InfAI e.V. · 3Amazon +1
CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMs
May 15, 2026Kamil Guttmann, Zofia Fraś, Artur Nowakowski +1Small Language Models MT Evaluation
Laniqo, Pozna´n, Poland · Faculty of Mathematics and Computer Science, Adam Mickiewicz University, Pozna´n, Poland

Apr 20, 2026, cs.CLJ/K move · Enter open · S save

LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation

Samar M. Magdy, Fakhraddin Alwajih, Abdellah El Mekki, Wesam El-Sayed+1

λThe University of British Columbia · ξMinia University · γCanada Research Chair in NLP and ML

Arabic NLP Machine Translation MT Evaluation

Open paper AskPDF arXiv

Existing MT evaluation frameworks, including automatic metrics and human evaluation schemes such as Multidimensional Quality Metrics (MQM), are largely language-agnostic. However, they often fail to capture dialect- and culture-specific errors in diglossic languages (e.g., Arabic), where translation failures stem from mismatches in language variety, content coverage, and pragmatic appropriateness rather than surface form alone.We introduce LQM: Linguistically Motivated Multidimensional Quality Metrics for MT. LQM is a hierarchical error taxonomy for diagnosing MT errors through six linguistically grounded levels: sociolinguistics, pragmatics, semantics, morphosyntax, orthography, and graphetics (Figure 1). We construct a bidirectional parallel corpus of 3,850 sentences (550 per variety) spanning seven Arabic dialects (Egyptian, Emirati, Jordanian, Mauritanian, Moroccan, Palestinian, and Yemeni), derived from conversational, culturally rich content. We evaluate six LLMs in a zero-shot setting and conduct expert span-level human annotation using LQM, producing 6,113 labeled error spans across 3,495 unique erroneous sentences, along with severity-weighted quality scores. We complement this analysis with an automatic metric (spBLEU). Though validated here on Arabic, LQM is a language-agnostic framework designed to be easily applied to or adapted for other languages. LQM annotated errors data, prompts, and annotation guidelines are publicly available at https://github.com/UBC-NLP/LQM_MT.

Abstract

Explore similar work

LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation

CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMs