The Blindness of Document-Level Translation Evaluation

Ahrii Kim; Vilém Zouhar; Chanjun Park; Seong-heum Kim

cs.CLSep 5, 2026

The Blindness of Document-Level Translation Evaluation

Authors: Ahrii Kim, Vilém Zouhar, Chanjun Park, Seong-heum Kim

Organizations: AI-Bio Convergence Research Inst. · ETH Zurich · School of Software · Dept. of Intelligent Semiconductors

Abstract

Document-level machine translation (MT) evaluation extends segment-level protocols by presenting full documents to annotators, on the assumption that such presentation elicits document-level judgments. We test this assumption with a counterfactual condition (MIX) in which each document combines segments drawn from different systems, preserving document-level presentation while breaking cross-segment consistency. Across 18,420 expert Englis-to-Korean annotations and 14 automatic metrics, scores, system rankings, and error annotations are statistically equivalent between coherent and incoherent documents. Perception does not explain this: shown matched passages, raters identify the coherent one as the work of a single translator in 87.3% of trials. Document presentation does change how annotators work, but that change does not reach the recorded output. What is blind is the protocol, not the annotator. The concern is not that scores fall short, but that the resources invested in document-level systems, metrics, and annotation may not be measuring what they are intended to measure.

Figures & tables

Figure 1: The three evaluation conditions. SEG presents a segment in isolation, without surrounding context. DOC presents a coherent document produced by a single system (here, System A). MIX presents a document of the same form but assembled from segments of different systems (A, B, C), preserving document-level layout while disrupting cross-segment coherence.

Figure 2: Overview of human ESA evaluation results across three conditions ( SEG , DOC , MIX ). (A) Score distributions (KDE) are nearly identical across conditions; mean and SD shown in inset. (B) Within-annotator paired differences fall within the

\pm 3

equivalence band for all three comparisons ( Section 4.1 ); each is signed as labelled, e.g. DOC

-

MIX . (C) System-level mean scores: SEG vs. document-level conditions; points near

y=x

indicate that doc-level rankings closely follow segment-level rankings ( Section 4.1 ). (D) Error annotation rate per 1,000 target characters by severity ( Section 4.1 ). (E) Krippendorff’s

\alpha

computed per document and per segment. The two units yield opposite orderings: DOC agrees most per document but least per segment. (F) Score recoverability: SEG scores predict

52\%

of the variance in MIX scores but only

20\%

of DOC scores (

n=230

matched pairs per regression).

Evaluator / Metric	$n$	$\varepsilon$	Mean diff	90% CI
Human ESA: pairwise paired TOST
DOC vs MIX	308	3.00	$-1.00$	$[-2.05,+0.04]$
DOC vs SEG	364	3.00	$+0.07$	$[-0.83,+0.97]$
SEG vs MIX	1249	3.00	$-1.09$	$[-1.65,-0.52]$
Segment-level metrics: DOC vs MIX
chr F	230	3.00	$+0.0187$	$[-0.515,+0.552]$

Table 1: Paired TOST for Section 4.1 : score equivalence across evaluators. Human ESA reports all three pairwise comparisons; metrics report only DOC vs. MIX . TOST rejects the null at

p<0.001

for all rows, establishing equivalence within

\pm\varepsilon

(

\varepsilon\in\{0.05,0.5,3.0\}

depending on scale). The shaded row indicates a known-failure case for context truncation. The metrics with wrong direction are colored in red.

	Human			chr F		x comet-xxl		CometKiwi-xxl		metricX-24-xxl ∗		metricX-24-qe-xxl ∗		gemba v2
System	SEG	DOC	MIX	DOC	MIX	DOC	MIX	DOC	MIX	DOC	MIX	DOC	MIX	DOC	MIX
Gemini-2.5-Pro	89.63 (1)	91.97 (1)	91.05 (2)	33.18 (1)	33.21 (1)	.791 (1)	.792 (1)	.845 (3)	.845 ( 2 )	3.83 (1)	3.81 (1)	3.72 (1)	3.72 (1)	87.95 (1)	87.91 (1)
REF	89.20 (1)	91.78 (1)	91.75 (1)	—	—	—	—	—	—	—	—	—	—	—	—
Wenyiil	89.34 (1)	89.11 (2)	89.63 (2)	30.72 (3)	30.79 (3)	.790 (1)	.787 (1)	.852 (2)	.851 (2)	3.97 (1)	3.97 (1)	3.64 (1)	3.66 (1)	86.74 (2)	86.71 (2)
GPT-4.1	87.45 (2)	89.16 (2)	88.90 (2)	30.35 (3)	30.25 (3)	.782 (1)	.782 (1)	.837 (3)	.836 (3)	4.30 (2)	4.27 (2)	3.98 (3)	3.97 (3)	86.18 (2)	86.22 (2)
Claude-4	86.98 (2)	86.32 (3)	88.45 ( 2 )	32.21 (2)	32.29 (2)	.735 (3)	.734 (3)	.842 (3)	.842 (3)	4.39 (2)	4.37 (2)	4.09 (3)	4.10 (3)	85.43 (3)	85.39 (3)

Table 2: System rankings under each evaluator for Section 4.1 . Cell colors indicate cluster rank (darker green = higher), derived from pairwise bootstrap tests (

B=1{,}000

\alpha=0.05

); systems within a cluster are not significantly distinguishable. Bold values mark cluster shifts between DOC and MIX . Bottom rows report the number of clusters per condition and Kendall’s

\tau

between DOC and MIX system rankings. ∗ (lower = better).

(a) Per-annotation error counts
Condition	$n$	Total	Major
SEG	5,920	2.36	0.63
DOC	5,920	2.31	0.55
MIX	5,920	2.37	0.53
(b) Paired TOST: total errors ( $\varepsilon_{\mathrm{err}}=0.5$ )
Comparison	$n$ pairs	Mean diff (90% CI)	TOST $p$

Table 3: Error counts and paired TOST results for Section 4.1 . (a) Mean error counts per annotation across conditions. (b) Pairwise paired TOST on total errors with

\varepsilon_{\mathrm{err}}=0.5

. The highlighted row marks Section 4.1 ’s primary comparison. TOST rejects the null at

p<0.001

for all comparisons, establishing equivalence within

\pm 0.5

errors per annotation.

Figure 3: Metric

\bm{\leftrightarrow}

human ranking agreement under DOC and MIX . Each point shows Kendall’s

\tau

between a metric’s system ranking and the human ranking, under DOC (

x

-axis) and MIX (

y

-axis). All metrics lie above

y=x

(shaded region), meaning that every metric agrees more closely with human rankings under MIX than under DOC .

Figure 4: Domain-stratified replication of Sections 4.1 , 4.1 and 4.1 on human ESA, with domains ordered by increasing context demand (news

<

social

<

literary). Throughout, the plotted paired difference is DOC

-

MIX . (A) Score TOST: paired mean differences on segment scores. All point estimates fall within the

\pm 3

equivalence band. (B) System ranking agreement between DOC and MIX . The dotted line marks the conventional acceptability threshold (

\tau_{\min}=0.667

). (C) Error TOST: paired mean differences on per-annotation error counts. All estimates fall within the

\pm 0.5

equivalence band.

Domain	SEG	DOC	MIX
Literary	67s	53s	41s
News	92s	78s	76s
Social	74s	57s	60s
Overall	76s	61s	56s

Table 4: Per-segment annotation time across conditions, by domain. The decrease from SEG to DOC and MIX suggests that context shapes annotators’ processing, regardless of whether that context is coherent or not.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Source documents and segments
	News	Social	Literary	Total
# documents	13	8	2	23
# unique segments	80	90	137	307
Mean segs/doc	6.2	11.3	68.5	13.4
System outputs
# MT systems	9	9	9	9

Appendix

Table 5: Dataset statistics by domain. Mean segs/doc is # unique segments divided by # documents, counted after the sub-document splitting described in Section A.1 . Each unique translation is evaluated under three conditions ( SEG , DOC , MIX ) and annotated by two independent annotators, yielding 18,420 total annotations. Note the unbalanced composition: the literary domain rests on 2 source documents but contributes

44.6\%

of all annotations.

Panel of Table 6 — Table 6: Pairwise preference results across three system pairs (486 judgments total; 162 per pair, 3 annotators). For each pair, we report Overall results (Raw and Normalized via domain-weighted averaging with a 50% cap) and per-domain breakdowns. Distribution bars show m1 preference , m2 preference , and tie . Sign tests apply to non-tie judgments at the overall (raw) level only.

	System	Translation	Level
DOC : single system throughout
$s_{5}$	gemini	…거의 같은 높이다.	Plain
$s_{6}$	gemini	…손을 흔들어 주었다.	Plain
$s_{7}$	gemini	…막지 않는다.	Plain
MIX : three systems
$s_{5}$	gemini	…거의 같은 높이다.	Plain

Appendix

Table 7: Three consecutive segments of one social-media document. Korean marks speech level on every sentence-final verb (bolded). DOC holds a single level throughout, whereas MIX moves Plain (한다체)

\rightarrow

Formal-polite (합쇼체)

\rightarrow

Informal-polite (해요체) within three sentences, a register discontinuity with no counterpart in the source. Both passages were shown in the same document-level ESA interface, and MIX was not scored lower.

Panel of Figure 5 — Figure 5: The segment- and document-level evaluation screens of Pearmut. For SEG items, raters are presented with a single segment, whereas DOC and MIX items display a full document.

Comparison	Design	n	Mean diff.	90% CI	$\varepsilon{=}3$	$\varepsilon{=}5$	$\varepsilon{=}7$
					TOST $p$ -value
DOC vs. MIX	independent	5920/5920	-0.82	[-1.25, -0.40]	$<$ 0.001	$<$ 0.001	$<$ 0.001
	paired	308	-1.00	[-2.05, +0.04]	$<$ 0.001	$<$ 0.001	$<$ 0.001
DOC vs. SEG	independent	5920/5920	+0.55	[+0.11, +0.99]	$<$ 0.001	$<$ 0.001	$<$ 0.001
	paired	364	+0.07	[-0.83, +0.97]	$<$ 0.001	$<$ 0.001	$<$ 0.001
SEG vs. MIX	independent	5920/5920	-1.37	[-1.79, -0.95]	$<$ 0.001	$<$ 0.001	$<$ 0.001

Appendix

Table 8: TOST equivalence test results across all pairwise condition comparisons, designs (independent vs. paired), and equivalence margins

\varepsilon\in\{3,5,7\}

points on the 100-point ESA scale. For independent designs,

n

is reported as

n_{1}/n_{2}

; for paired designs,

n

is the number of matched pairs. All eighteen cells support equivalence at

p<0.001

, demonstrating robustness to comparison choice, study design, and equivalence margin.

	Krippendorff’s $\alpha$
Condition	Per document	Per segment
SEG	0.149	0.172
DOC	0.186	0.155
MIX	0.128	0.225

Appendix

Table 9: Inter-annotator agreement, computed at two units. Per document treats

\langle

doc_id , system

\rangle

as the item, averaging each annotator’s segment scores within it; per segment treats

\langle

doc_id , seg_id , system

\rangle

as the item. The orderings invert between the two: DOC agrees most per document but least per segment. All values fall below the conventional threshold for acceptable reliability (

\alpha\geq 0.667

), consistent with known variability in scalar MT evaluation ( graham-etal-2013-continuous ; kocmi-etal-2025-findings ) .

Metric	Condition	Kendall’s $\tau$	$p$	Pearson’s $r$
Surface
chr F	DOC	$0.667$	$0.013$	$0.815$
	MIX	$0.889$	$<0.001$	$0.946$
d- bleu	DOC	$0.667$	$0.013$	$0.831$
	MIX	$0.889$	$<0.001$	$0.950$
Learned ref-based

Appendix

Table 10: System-level human-metric correlations by condition (

n=9

systems). Kendall’s

\tau

values for QE metrics ( CometKiwi-xxl , metricX-24-qe-xxl ) are not statistically significant at

p<0.05

, reflecting the small system pool and narrow score distributions of these metrics. We focus on the relative pattern across conditions in our analysis.

Surface metrics
	Scale	Range	SD	Min	Max
Human evaluation	0–100	$12.28$	$4.09$	$78.60$	$90.88$
chr F	0–100	$9.81$	$3.08$	$23.37$	$33.18$
d- bleu	0–100	$11.66$	$3.64$	$15.52$	$27.18$
Learned reference-based
x comet-xxl	0–1	$0.076$	$0.032$	$0.716$	$0.792$

Appendix

Table 11: System-level mean score distributions across nine systems under each metric. Range is computed as max

-

min of system means. Reference-free QE metrics produce the narrowest output ranges (

\sim 3

–

6\%

of metric scale), making ranking-based human agreement noise-sensitive. Surface metrics span ranges comparable to human evaluation. ∗ metricX variants are error scores (lower = better); ranges are reported in the metric’s native scale.

MT Evaluation

Machine translation evaluation measures translation quality, error types, and real-world suitability across languages, domains, and modalities.

See all 173 papers

TermJudge: A Document-Level Metric Judging, Not Counting, Terminology in Machine Translation Evaluation
Sep 28, 2026Nicolas Dahan, Fran{\cc}ois Yvon, Rachel BawdenAutomated Evaluation Document-Level MT
ISIR, ALMAnaCH · MLIA, ISIR · ALMAnaCH
Improving Term Evaluation in Machine Translation: Variation Matters
Sep 8, 2026Nicolas Dahan, Ziqian Peng, François Yvon +1Machine Translation MT Evaluation
MLIA, ISIR, ALMAnaCH · MLIA · ALMAnaCH
Span-Level Machine Translation Meta-Evaluation
Mar 20, 2026Stefano Perrella, Eric Morales Agostinho, Hugo ZaragozaMT Evaluation
Sapienza University of Rome · Amazon

Sep 28, 2026J/K move · Enter open · S save

TermJudge: A Document-Level Metric Judging, Not Counting, Terminology in Machine Translation Evaluation

Nicolas Dahan, Fran{\cc}ois Yvon, Rachel Bawden

ISIR, ALMAnaCH · MLIA, ISIR · ALMAnaCH

Automated Evaluation Document-Level MT MT Evaluation

Open paper AskPDF arXiv

Existing automatic metrics for evaluating terminological use in machine translation (MT) penalise any divergence from a fixed reference, conflating translation errors with the valid terminological variation that human translators routinely produce. We introduce TermJudge, a document-level terminology metric that assigns an interpretable verdict to every term occurrence: glossary-conforming occurrences are settled deterministically, while divergences are assessed under a two-step LLM-as-judge procedure using the full document context: the first detects and labels terminology errors; the second sorts valid document-level variations from inconsistencies. Validated against expert error annotations and document-level human MQM scores, TermJudge ranks first in both system- and segment-level meta-evaluation, ahead of glossary-conformity and quality-estimation baselines. When applied to eight systems translating academic documents, under two prompting conditions, we observe that glossary injection improves terminology translation in all paired comparisons, by removing genuine errors rather than valid variation. TermJudge is released as open-source code.

Abstract

MT Evaluation

Explore similar work

TermJudge: A Document-Level Metric Judging, Not Counting, Terminology in Machine Translation Evaluation

Improving Term Evaluation in Machine Translation: Variation Matters

Span-Level Machine Translation Meta-Evaluation