Document-level machine translation (MT) evaluation extends segment-level protocols by presenting full documents to annotators, on the assumption that such presentation elicits document-level judgments. We test this assumption with a counterfactual condition (MIX) in which each document combines segments drawn from different systems, preserving document-level presentation while breaking cross-segment consistency. Across 18,420 expert Englis-to-Korean annotations and 14 automatic metrics, scores, system rankings, and error annotations are statistically equivalent between coherent and incoherent documents. Perception does not explain this: shown matched passages, raters identify the coherent one as the work of a single translator in 87.3% of trials. Document presentation does change how annotators work, but that change does not reach the recorded output. What is blind is the protocol, not the annotator. The concern is not that scores fall short, but that the resources invested in document-level systems, metrics, and annotation may not be measuring what they are intended to measure.
Figures & tables
Figure 1: The three evaluation conditions. SEG presents a segment in isolation, without surrounding context. DOC presents a coherent document produced by a single system (here, System A). MIX presents a document of the same form but assembled from segments of different systems (A, B, C), preserving document-level layout while disrupting cross-segment coherence.
Figure 2: Overview of human ESA evaluation results across three conditions ( SEG , DOC , MIX ). (A) Score distributions (KDE) are nearly identical across conditions; mean and SD shown in inset. (B) Within-annotator paired differences fall within the ±3 equivalence band for all three comparisons ( Section 4.1 ); each is signed as labelled, e.g. DOC − MIX . (C) System-level mean scores: SEG vs. document-level conditions; points near y=x indicate that doc-level rankings closely follow segment-level rankings ( Section 4.1 ). (D) Error annotation rate per 1,000 target characters by severity ( Section 4.1 ). (E) Krippendorff’s α computed per document and per segment. The two units yield opposite orderings: DOC agrees most per document but least per segment. (F) Score recoverability: SEG scores predict 52% of the variance in MIX scores but only 20% of DOC scores ( n=230 matched pairs per regression).
Evaluator / Metric
n
ε
Mean diff
90% CI
Human ESA: pairwise paired TOST
DOC vs MIX
308
3.00
−1.00
[−2.05,+0.04]
DOC vs SEG
364
3.00
+0.07
[−0.83,+0.97]
SEG vs MIX
1249
3.00
−1.09
[−1.65,−0.52]
Segment-level metrics: DOC vs MIX
chr F
230
3.00
+0.0187
[−0.515,+0.552]
Table 1: Paired TOST for Section 4.1 : score equivalence across evaluators. Human ESA reports all three pairwise comparisons; metrics report only DOC vs. MIX . TOST rejects the null at p<0.001 for all rows, establishing equivalence within ±ε ( ε∈{0.05,0.5,3.0} depending on scale). The shaded row indicates a known-failure case for context truncation. The metrics with wrong direction are colored in red.
Human
chr F
x comet-xxl
CometKiwi-xxl
metricX-24-xxl ∗
metricX-24-qe-xxl ∗
gemba v2
System
SEG
DOC
MIX
DOC
MIX
DOC
MIX
DOC
MIX
DOC
MIX
DOC
MIX
DOC
MIX
Gemini-2.5-Pro
89.63 (1)
91.97 (1)
91.05 (2)
33.18 (1)
33.21 (1)
.791 (1)
.792 (1)
.845 (3)
.845 ( 2 )
3.83 (1)
3.81 (1)
3.72 (1)
3.72 (1)
87.95 (1)
87.91 (1)
REF
89.20 (1)
91.78 (1)
91.75 (1)
—
—
—
—
—
—
—
—
—
—
—
—
Wenyiil
89.34 (1)
89.11 (2)
89.63 (2)
30.72 (3)
30.79 (3)
.790 (1)
.787 (1)
.852 (2)
.851 (2)
3.97 (1)
3.97 (1)
3.64 (1)
3.66 (1)
86.74 (2)
86.71 (2)
GPT-4.1
87.45 (2)
89.16 (2)
88.90 (2)
30.35 (3)
30.25 (3)
.782 (1)
.782 (1)
.837 (3)
.836 (3)
4.30 (2)
4.27 (2)
3.98 (3)
3.97 (3)
86.18 (2)
86.22 (2)
Claude-4
86.98 (2)
86.32 (3)
88.45 ( 2 )
32.21 (2)
32.29 (2)
.735 (3)
.734 (3)
.842 (3)
.842 (3)
4.39 (2)
4.37 (2)
4.09 (3)
4.10 (3)
85.43 (3)
85.39 (3)
Table 2: System rankings under each evaluator for Section 4.1 . Cell colors indicate cluster rank (darker green = higher), derived from pairwise bootstrap tests ( B=1,000 , α=0.05 ); systems within a cluster are not significantly distinguishable. Bold values mark cluster shifts between DOC and MIX . Bottom rows report the number of clusters per condition and Kendall’s τ between DOC and MIX system rankings. ∗ (lower = better).
(a) Per-annotation error counts
Condition
n
Total
Major
SEG
5,920
2.36
0.63
DOC
5,920
2.31
0.55
MIX
5,920
2.37
0.53
(b) Paired TOST: total errors ( εerr=0.5 )
Comparison
n pairs
Mean diff (90% CI)
TOST p
Table 3: Error counts and paired TOST results for Section 4.1 . (a) Mean error counts per annotation across conditions. (b) Pairwise paired TOST on total errors with εerr=0.5 . The highlighted row marks Section 4.1 ’s primary comparison. TOST rejects the null at p<0.001 for all comparisons, establishing equivalence within ±0.5 errors per annotation.
Figure 3: Metric ↔ human ranking agreement under DOC and MIX . Each point shows Kendall’s τ between a metric’s system ranking and the human ranking, under DOC ( x -axis) and MIX ( y -axis). All metrics lie above y=x (shaded region), meaning that every metric agrees more closely with human rankings under MIX than under DOC .
Figure 4: Domain-stratified replication of Sections 4.1 , 4.1 and 4.1 on human ESA, with domains ordered by increasing context demand (news < social < literary). Throughout, the plotted paired difference is DOC − MIX . (A) Score TOST: paired mean differences on segment scores. All point estimates fall within the ±3 equivalence band. (B) System ranking agreement between DOC and MIX . The dotted line marks the conventional acceptability threshold ( τmin=0.667 ). (C) Error TOST: paired mean differences on per-annotation error counts. All estimates fall within the ±0.5 equivalence band.
Domain
SEG
DOC
MIX
Literary
67s
53s
41s
News
92s
78s
76s
Social
74s
57s
60s
Overall
76s
61s
56s
Table 4: Per-segment annotation time across conditions, by domain. The decrease from SEG to DOC and MIX suggests that context shapes annotators’ processing, regardless of whether that context is coherent or not.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
News
Social
Literary
Total
Source documents and segments
# documents
13
8
2
23
# unique segments
80
90
137
307
Mean segs/doc
6.2
11.3
68.5
13.4
System outputs
# MT systems
9
9
9
9
Appendix
Table 5: Dataset statistics by domain. Mean segs/doc is # unique segments divided by # documents, counted after the sub-document splitting described in Section A.1 . Each unique translation is evaluated under three conditions ( SEG , DOC , MIX ) and annotated by two independent annotators, yielding 18,420 total annotations. Note the unbalanced composition: the literary domain rests on 2 source documents but contributes 44.6% of all annotations.
Table 6: Pairwise preference results across three system pairs (486 judgments total; 162 per pair, 3 annotators). For each pair, we report Overall results (Raw and Normalized via domain-weighted averaging with a 50% cap) and per-domain breakdowns. Distribution bars show m1 preference , m2 preference , and tie . Sign tests apply to non-tie judgments at the overall (raw) level only.
System
Translation
Level
DOC : single system throughout
s5
gemini
…거의 같은 높이다.
Plain
s6
gemini
…손을 흔들어 주었다.
Plain
s7
gemini
…막지 않는다.
Plain
MIX : three systems
s5
gemini
…거의 같은 높이다.
Plain
Appendix
Table 7: Three consecutive segments of one social-media document. Korean marks speech level on every sentence-final verb (bolded). DOC holds a single level throughout, whereas MIX moves Plain (한다체) → Formal-polite (합쇼체) → Informal-polite (해요체) within three sentences, a register discontinuity with no counterpart in the source. Both passages were shown in the same document-level ESA interface, and MIX was not scored lower.
Figure 5: The segment- and document-level evaluation screens of Pearmut. For SEG items, raters are presented with a single segment, whereas DOC and MIX items display a full document.
TOST p -value
Comparison
Design
n
Mean diff.
90% CI
ε=3
ε=5
ε=7
DOC vs. MIX
independent
5920/5920
-0.82
[-1.25, -0.40]
< 0.001
< 0.001
< 0.001
paired
308
-1.00
[-2.05, +0.04]
< 0.001
< 0.001
< 0.001
DOC vs. SEG
independent
5920/5920
+0.55
[+0.11, +0.99]
< 0.001
< 0.001
< 0.001
paired
364
+0.07
[-0.83, +0.97]
< 0.001
< 0.001
< 0.001
SEG vs. MIX
independent
5920/5920
-1.37
[-1.79, -0.95]
< 0.001
< 0.001
< 0.001
Appendix
Table 8: TOST equivalence test results across all pairwise condition comparisons, designs (independent vs. paired), and equivalence margins ε∈{3,5,7} points on the 100-point ESA scale. For independent designs, n is reported as n1/n2 ; for paired designs, n is the number of matched pairs. All eighteen cells support equivalence at p<0.001 , demonstrating robustness to comparison choice, study design, and equivalence margin.
Krippendorff’s α
Condition
Per document
Per segment
SEG
0.149
0.172
DOC
0.186
0.155
MIX
0.128
0.225
Appendix
Table 9: Inter-annotator agreement, computed at two units. Per document treats ⟨ doc_id , system ⟩ as the item, averaging each annotator’s segment scores within it; per segment treats ⟨ doc_id , seg_id , system ⟩ as the item. The orderings invert between the two: DOC agrees most per document but least per segment. All values fall below the conventional threshold for acceptable reliability ( α≥0.667 ), consistent with known variability in scalar MT evaluation ( graham-etal-2013-continuous ; kocmi-etal-2025-findings ) .
Metric
Condition
Kendall’s τ
p
Pearson’s r
Surface
chr F
DOC
0.667
0.013
0.815
MIX
0.889
<0.001
0.946
d- bleu
DOC
0.667
0.013
0.831
MIX
0.889
<0.001
0.950
Learned ref-based
Appendix
Table 10: System-level human-metric correlations by condition ( n=9 systems). Kendall’s τ values for QE metrics ( CometKiwi-xxl , metricX-24-qe-xxl ) are not statistically significant at p<0.05 , reflecting the small system pool and narrow score distributions of these metrics. We focus on the relative pattern across conditions in our analysis.
Scale
Range
SD
Min
Max
Human evaluation
0–100
12.28
4.09
78.60
90.88
Surface metrics
chr F
0–100
9.81
3.08
23.37
33.18
d- bleu
0–100
11.66
3.64
15.52
27.18
Learned reference-based
x comet-xxl
0–1
0.076
0.032
0.716
0.792
Appendix
Table 11: System-level mean score distributions across nine systems under each metric. Range is computed as max − min of system means. Reference-free QE metrics produce the narrowest output ranges ( ∼3 – 6% of metric scale), making ranking-based human agreement noise-sensitive. Surface metrics span ranges comparable to human evaluation. ∗ metricX variants are error scores (lower = better); ranges are reported in the metric’s native scale.
Existing automatic metrics for evaluating terminological use in machine translation (MT) penalise any divergence from a fixed reference, conflating translation errors with the valid terminological variation that human translators routinely produce. We introduce TermJudge, a document-level terminology metric that assigns an interpretable verdict to every term occurrence: glossary-conforming occurrences are settled deterministically, while divergences are assessed under a two-step LLM-as-judge procedure using the full document context: the first detects and labels terminology errors; the second sorts valid document-level variations from inconsistencies. Validated against expert error annotations and document-level human MQM scores, TermJudge ranks first in both system- and segment-level meta-evaluation, ahead of glossary-conformity and quality-estimation baselines. When applied to eight systems translating academic documents, under two prompting conditions, we observe that glossary injection improves terminology translation in all paired comparisons, by removing genuine errors rather than valid variation. TermJudge is released as open-source code.
Terminology evaluation in machine translation (MT) usually assumes a single correct target form per source term. However, human translators routinely introduce variation that current metrics penalize as inconsistency. We examine how to account for this variation in document-level MT evaluation of English-French scientific translation, combining glossary-based accuracy, translation consistency, and a new cross-term variation (CTV) diagnostic measure that tests whether variation relationships are preserved across languages. Based on analyses of two parallel corpora, translated by four MT systems, we find that (1) MT systems generate less target-side variation than human translators; (2) transfer patterns strongly depend on the variation type; (3) consistency rankings vary with the choice of metric; and (4) constraining MT with a glossary improves accuracy and consistency but degrades CTV by suppressing valid variation. We argue for variation-aware evaluation that conditions consistency penalties on whether target-side variation mirrors source-side variation.
Machine Translation (MT) and automatic MT evaluation have improved dramatically in recent years, enabling numerous novel applications. Automatic evaluation techniques have evolved from producing scalar quality scores to precisely locating translation errors and assigning them error categories and severity levels. However, it remains unclear how to reliably measure the evaluation capabilities of auto-evaluators that do error detection, as no established technique exists in the literature. This work investigates different implementations of span-level precision, recall, and F-score, showing that seemingly similar approaches can yield substantially different rankings, and that certain widely-used techniques are unsuitable for evaluating MT error detection. We propose "match with partial overlap and partial credit" (MPP) with micro-averaging as a robust meta-evaluation strategy and release code for its use publicly. Finally, we use MPP to assess the state of the art in MT error detection.
Stefano Perrella, Eric Morales Agostinho, Hugo Zaragoza