Large Language Models as Annotators for Machine Translation Quality Estimation

Sidi Wang; Sophie Arnoult; Amir Kamran

cs.CLMar 11, 2026

Large Language Models as Annotators for Machine Translation Quality Estimation

Authors: Sidi Wang, Sophie Arnoult, Amir Kamran

Organizations: Maastricht university · Vrije Universiteit Amsterdam · Taus

Abstract

Large Language Models (LLMs) have demonstrated excellent performance on Machine Translation Quality Estimation (MTQE), yet their high inference costs make them impractical for direct application. In this work, we propose applying LLMs to generate MQM-style annotations for training a COMET model: following Fernandes et al. (2023), we reckon that segment-level annotations provide a strong rationale for LLMs and are key to good segment-level QE. We propose a simplified MQM scheme, mostly restricted to top-level categories, to guide LLM selection. We present a systematic approach for the development of a GPT-4o-based prompt, called PPbMQM (Prompt-Pattern-based-MQM). We show that the resulting annotations correlate well with human annotations and that training COMET on them leads to competitive performance on segment-level QE for Chinese-English and English-German.

Figures & tables

Figure 1: Example MQM annotation. Error spans are marked with error type and severity.

Question	GPT-3.5	GPT-4 Turbo	GPT-4o	LLaMA 3
1. Can you explain what machine translation quality estimation is in 100 words?	3	4	4	4
2. Could you provide an overview of the Multidimensional Quality Metrics (MQM) annotation scheme in around 100 words?	3	4	4	4
3. What are the core error categories of MQM?	2	3	4	4
4. How can I effectively evaluate the Chinese sentence and its English translation using the MQM annotation scheme? Provide a concise response within 100 words, please.	3	5	5	4
5. How would you, as a language model, annotate the translation of a Chinese source sentence into English using the MQM scheme? Please provide an example.	3	4	3	3
Total score	14	20	20	19

Table 1: Exploring LLMs’ knowledge of QE and MQM. In each API request, we presented the LLMs with one single question. Answers are scored on a scale of 1-5 (very poor-very good) by a single annotator.

Model	segments	errors ( major )	Span	Severity	Type	$\rho$	$r$
zero-shot
GPT-4 Turbo	692	1237 (684)	0.359	0.49	0.24	0.383 ∗∗	0.343 ∗∗
GPT-4o	957	3146 (1256)	0.318	0.51	0.26	0.485 ∗∗	0.382 ∗∗
LLaMA 3	885	2286 (1143)	0.296	0.49	0.16	0.486 ∗∗	0.401 ∗∗
few-shot
GPT-4o ( $m_{1-3}$ )	909	2428 (354)	0.333	0.65	0.37	0.492 ∗∗	0.376 ∗∗

Table 2: Zero-shot and few-shot span metrics and segment-level quality score correlation with Pearson

\rho

and Spearman

r

on parsable segments; In the few-shot setting, we either map errors of severity 1-3 to minor errors (

m_{1-3}

), or severity-3 errors, discarding errors of severity 1-2 (

m_{3}

). ** indicates p-value ¡ 0.001.

	errors		major/minor
	gold	PPbMQM	gold	PPbMQM
GPT4 Turbo	1.39	1.79	0.24	1.24
GPT4o	1.59	3.29	0.27	0.66
LLaMA 3	1.57	2.58	0.28	1.0

Table 3: Average error counts per segment and major/minor ratios in human and PPbMQM annotations.

	Accuracy	Style	Fluency
GPT-4 Turbo	0.55	0.27	0.28
GPT-4o	0.51	0.24	0.29
LLaMA 3	0.42	0.10	0.22

Table 4: Error-type F1 for frequent error types.

segments	quality	$\rho$		$r$		$\tau$
	scores	gold	PPbMQM	gold	PPbMQM	gold	PPbMQM
English
4806	0.0-1.0	0.470 ∗∗	0.513 ∗∗	0.410 ∗∗	0.381 ∗∗	0.290 ∗∗	0.272 ∗∗
2592	0.9-1.0	0.019	-0.157 ∗∗	0.020	-0.128 ∗∗	0.016	-0.085 ∗∗
910	0.8-0.9	0.063	-0.005	0.077 ∗	0.016	0.053 ∗	0.011
1304	$<$ 0.8	0.434 ∗∗	0.523 ∗∗	0.425 ∗∗	0.526 ∗∗	0.294 ∗∗	0.370 ∗∗

Table 5: Bucket analysis for en-de: quality score correlation to gold annotations (Pearson

\rho

, Spearman

r

and Kendall

\tau

) of COMET-QE trained on GPT-4o annotations (PPbMQM) vs human annotations (gold), overall and per bucket. p-values ¡ 0.001 are marked with **, p-values ¡ 0.05 with *.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	segments	Span	Severity	Type	$\rho$	$r$
basic prompt	380	0.307	0.46	0.23	0.452*	0.345*
prompt variant	386	0.308	0.49	0.20	0.457*	0.376*

Appendix

Table 6: Zero-shot results of the additional stability test with 400 output segments from GPT-4o: span metrics (span F1, severity F1, and error type F1) and segment-level correlation with Pearson

\rho

and Spearman

r

system_fingerprint	segments	Span	Severity	Type	$\rho$	$r$
fp_319be4768e	19155	0.276	0.47	0.29	0.399*	0.274*
fp_aa87380ac5	714	0.289	0.49	0.28	0.478*	0.239*

Appendix

Table 7: Fingerprint analysis on the PPbMQM-generated QE-training dataset: span metrics (span F1, severity F1, and error type F1) and segment-level correlation with Pearson

\rho

and Spearman

r

Annotators	Number of segments	$\rho$	$r$
rater1, rater2	702	0.287	0.274
rater1, rater3	1049	0.036	0.063
rater1, rater6	299	0.059	0.088
rater1, rater5	11	-0.407	-0.337
rater1, rater4	13	-0.143	-0.06
rater2, rater3	515	-0.005	-0.042

Appendix

Table 8: Pearson

\rho

and Spearman

r

correlations between annotators on the same segments in EbHE-WMT-MT 2022 (zh-en) dataset

MT Evaluation

Machine translation evaluation measures translation quality, error types, and real-world suitability across languages, domains, and modalities.

See all 173 papers

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
Oct 7, 2026Hala Almaghout, Christian Federmann, Qin GaoMT Evaluation LLM Reliability
Apple
CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMs
May 15, 2026Kamil Guttmann, Zofia Fraś, Artur Nowakowski +1Small Language Models MT Evaluation
Laniqo, Pozna´n, Poland · Faculty of Mathematics and Computer Science, Adam Mickiewicz University, Pozna´n, Poland
Domain-Specific Quality Estimation for Machine Translation in Low-Resource Scenarios
Mar 7, 2026Namrata Patil Gurav, Akashdeep Ranu, Archchana Sindhujan +1MT Evaluation Machine Translation
Institute for People-Centred AI, University of Surrey, United Kingdom

Oct 7, 2026J/K move · Enter open · S save

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

Hala Almaghout, Christian Federmann, Qin Gao

Apple

MT Evaluation LLM Reliability LLM-Assisted Annotation Multilingual Language Model Evaluation

Open paper AskPDF arXiv

Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human annotators. We present results on a long-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs and domains. Our results show that while LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks, both vary substantially across annotation schemes, language pairs and domains and remain unreliable for most settings. Furthermore, we identify challenges facing both human and LLM annotators: humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. Our results highlight the potential for improvement for both human and LLM annotation performance, possibly through human-LLM collaborative annotation pipelines that address the reliability issues identified in this work.

Abstract

MT Evaluation

Explore similar work

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMs

Domain-Specific Quality Estimation for Machine Translation in Low-Resource Scenarios