Machine Translation (MT) and automatic MT evaluation have improved dramatically in recent years, enabling numerous novel applications. Automatic evaluation techniques have evolved from producing scalar quality scores to precisely locating translation errors and assigning them error categories and severity levels. However, it remains unclear how to reliably measure the evaluation capabilities of auto-evaluators that do error detection, as no established technique exists in the literature. This work investigates different implementations of span-level precision, recall, and F-score, showing that seemingly similar approaches can yield substantially different rankings, and that certain widely-used techniques are unsuitable for evaluating MT error detection. We propose "match with partial overlap and partial credit" (MPP) with micro-averaging as a robust meta-evaluation strategy and release code for its use publicly. Finally, we use MPP to assess the state of the art in MT error detection.
Figures & tables
Micro
Macro
Auto-evaluator
em
mp
mpp
w 25
em
mp
mpp
w 25
Qwen3 235b
5
1
1
1
12
12
10
12
GemSpanEval.sec
3
2
2
2
4
4
4
4
Sonnet 4.5
2
5
3
6
8
11
7
9
GemSpanEval.pri
4
4
4
5
6
5
5
5
XCOMET-XXL
11
3
5
4
11
7
11
7
Table 1: Ranking of auto-evaluators on the MQM split of the WMT 2025 Automated Translation Quality Evaluation Systems Shared Task under the measures defined in Section 2 ( em , mp , and mpp ), alongside w 25, that is, the measure used at WMT25 (illustrated in Section C.2 ), both micro- and macro-averaging results. We run the auto-evaluators highlighted in gray, the others are submissions to WMT.
Figure 1: Span-level annotations for the translation x= “The quick brown fox jumps”. s^1,s^2∈S^ are hypothesis error spans, while s1,s2∈S are the ground-truth ones.
Figure 2: Performance of auto-evaluators and imprecise sentinels at varying span length, with micro (left) and macro (right) averaging, under all meta-metrics defined in Section 2 . x=0 shows the performance of the base auto-evaluators before their spans were extended.
Figure 3: Performance of auto-evaluators and low-recall sentinels at varying error removal probabilities, with micro (left) and macro (right) averaging, under all meta-metrics defined in Section 2 . x=0 shows the performance of the base auto-evaluators before any span was removed from their predictions.
Metric
P
R
F 1
Qwen3 235b
17.36
31.45
22.37
GemSpanEval-QE.sec
16.33
28.61
20.60
Claude Sonnet-4.5
21.47
18.58
19.92
GemSpanEval.pri
16.60
25.34
19.84
XCOMET-XXL.bas
16.38
21.39
18.33
AIP.sec
37.34
11.99
18.16
Table 2: mpp with micro averaging on the MQM split of WMT25. We run the auto-evaluators highlighted in grey, the others are submissions to WMT25 Lavie et al. (2025) .
Metric
en → de (2022)
en → zh (2022)
en → de (2023)
zh → en (2023)
P
R
F 1
P
R
F 1
P
R
F 1
P
R
F 1
MQM #1
42.66
44.62
43.62
29.88
25.49
27.51
38.04
35.51
36.73
40.17
39.48
39.82
MQM #2
39.22
43.56
41.28
35.95
25.68
29.96
39.02
37.47
38.23
44.57
39.82
42.06
Sonnet 4.5
32.61
29.75
31.12
34.31
21.57
26.49
38.92
27.80
32.44
48.82
33.94
40.04
Qwen3 235b
23.62
38.87
29.39
28.88
30.92
29.87
31.18
30.41
30.79
40.12
39.50
39.81
gpt-oss 120b
24.08
31.04
27.12
28.89
21.42
24.60
32.33
24.60
27.94
44.55
29.10
35.20
Table 3: Results on the en → de MQM test set from WMT 2022 Kocmi et al. (2022) ; Freitag et al. (2022b) , the en → zh MQM annotations released by Riley et al. (2024) , and the en → de and zh → en MQM test sets from WMT 2023 Kocmi et al. (2023) ; Freitag et al. (2023) . “MQM #1” and “MQM #2” are human evaluators.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Year
Language Pair
#Samples
#Errors
%Samples w/o Errors
2022
en → de
21,040
16,074
53.5%
en → zh
49,558
29,558
48.6%
2023
en → de
5,980
18,698
26.9%
zh → en
18,829
43,767
18.9%
2024
en → de
9,253
11,650
48.3%
en → es
9,345
4,077
77.4%
Appendix
Table 4: Statistics of the test sets used in this work. These test sets have been sourced from several previous works: the WMT Metrics Shared Task editions from 2022 to 2024 Freitag et al. (2022b) ; Freitag et al. (2023) ; Freitag et al. (2024) , the study from Riley et al. (2024) regarding finding replicable human evaluation techniques, the WMT25 General Machine Translation shared task Kocmi et al. (2025b) , and the WMT25 shared task on Automated Translation Evaluation Systems Lavie et al. (2025) .
Auto-evaluator
Research Paper
GemSpanEval
Juraska et al. (2025)
GemSpanEval-QE
Juraska et al. (2025)
XCOMET-XL
Guerreiro et al. (2024)
XCOMET-XXL
Guerreiro et al. (2024)
AIP (pri)
Yeom et al. (2025)
AIP (sec)
Yeom et al. (2025)
Appendix
Table 5: Research papers associated with the WMT25 submissions and baselines used in this work.
Figure 4: Given the translation x= “The quick brown fox jumps”, this example shows span-level annotations of translation quality, featuring three ground-truth error spans – s1,s2,s3∈S – and two hypothesis error spans – s^1,s^2∈S^ .
Auto-Evaluator
#E
#E/S
SL
MQM
16,297
2.21
10.7
AutoLQA41.sec
2,116
0.29
53.1
Claude Haiku 4.5
11,439
1.55
10.7
Claude Sonnet 4.5
17,584
2.38
7.3
AutoLQAESA.sec
8,549
1.16
36.5
AIP.pri
6,606
0.90
5.1
Appendix
Table 6: Number of errors (“#E”), average number of errors per sample (“#E/S”), and average span length (“SL”), for each evaluator, computed on the MQM split of the WMT 2025 data Lavie et al. (2025) .
Micro
Macro
Auto-evaluator
em
mp
mpp
w 25
em
mp
mpp
w 25
Qwen3 235b
6.99
28.33
22.37
23.31
7.21
23.75
17.45
18.33
GemSpanEval-QE.sec
7.76
25.87
20.60
22.10
14.99
27.33
22.11
22.61
Sonnet 4.5
7.98
25.05
19.92
18.37
12.28
23.84
18.87
19.11
GemSpanEval.pri
7.11
25.29
19.84
20.61
13.95
25.88
20.65
20.95
XCOMET-XXL
2.68
25.53
18.33
20.89
7.81
24.73
17.43
19.48
Appendix
Table 7: F-score of auto-evaluators on the MQM split of the WMT 2025 Automated Translation Quality Evaluation Systems shared task under em , mp , mpp , and w 25, with both micro- and macro-averaging. We run the auto-evaluators highlighted in gray; the others are submissions to WMT.
Metric
P
R
F 1
AIP.sec
17.00
14.75
15.17
AIP.pri
16.06
13.02
13.52
AutoLQA.pri
10.61
14.59
10.95
AutoLQAESA.sec
11.45
13.50
10.77
XCOMET-XXL.bas
6.80
28.45
10.55
GemSpanEval-QE.sec
5.98
33.99
9.95
Appendix
Table 8: Full results on the ESA split of WMT25 under mpp with micro-averaging.
AI-Bio Convergence Research Inst. Soongsil University · School of Software Soongsil University · Dept. of Intelligent Semiconductors Soongsil University