Discourse Dependency: A Continuous Criterion for Translation Difficulty

Ahrii Kim; Chanjun Park; Seong-heum Kim

cs.CLSep 4, 2026

Discourse Dependency: A Continuous Criterion for Translation Difficulty

Authors: Ahrii Kim, Chanjun Park, Seong-heum Kim

Organizations: AI-Bio Convergence Research Inst. Soongsil University · School of Software Soongsil University · Dept. of Intelligent Semiconductors Soongsil University

Abstract

Recent calls for harder machine translation benchmarks have not clarified what difficulty should mean. We argue that one meaningful and currently unmeasured axis is referential reach, the distance a segment must look back into its document to resolve the entities and pronouns it contains. We formalize this as discourse dependency (DDP), a metric-free, source-side measure computed from named entity re-mentions and pronominal coreference. Validated against gold coreference, DDP errs one-sidedly in 99.2% of segments, so a high-DDP segment is certified to require long-range context. Applying DDP to WMT24++ and WMT25 shows that both are heavily skewed toward low-DDP segments, which domain labels do not distinguish. Building on DDP, we compare five context injection strategies in an English-Korean post-editing setup, varying context size and selection. As DDP grows, no strategy keeps pace with human post-editing. On segments with DDP >= 15 raters prefer human translations, while automatic metrics register no difference. As frontier systems saturate aggregate scores, DDP shifts evaluation from how well models score to how far they can reach.

Figures & tables

Seg	Anchor	Antecedent	$d$
(1)	John	— (first mention)	0
(2)	he	John (1)	1
(3)	manager	— (first mention)	0
	him	John (1)	2
	$\ddpop_{(3)}=\max(0,2)$		2
(4)	John	John (1)	3

Table 1:

\mathsf{DDP}

computation on the example passage. Each anchor is traced to its antecedent; the segment

\mathsf{DDP}

is the maximum anchor distance (

d

). † him is chained through its most recent named entity ( John , seg 4), whose own distance from its first mention (seg 1) is accumulated:

(5{-}4)+3=4

Figure 1:

\mathsf{DDP}

score distributions across domains in WMT24++ and WMT25. (a) Mean

\mathsf{DDP}

per domain: literary texts exhibit substantially higher dependency than news and social across both benchmarks, with WMT25 showing a marked increase across all domains (

\mu

=222.6 for literary vs. 52.8 in WMT24++) due to the inclusion of long-form narrative texts and longer conversational threads. (b–c) Cumulative distribution of

\mathsf{DDP}

scores: news segments saturate quickly in both benchmarks, while literary scores spread widely, reaching

\ddpop=171

in WMT24++ and

\ddpop=482

in WMT25. Note the different x-axis ranges in (b) and (c).

Model	Params	Context	Cutoff	En-X	Reasoning	MoE	Thinking
Open-source
Gemma-4-31B ( gemma4 )	30.7B	256K $\rightarrow$ 128K	Jan 2025	✓	✓	–	✓
Qwen3.5-27B ( qwen3.5 )	27B	256K $\rightarrow$ 128K	–	✓	–	–	✓
HyperCLOVA-X-SEED-Think-32B ( navercloudhyperclovaxteam2026hyperclovax32bthink ) $\blacklozenge$	32B	128K	–	✓	✓	–	✓
DeepSeek-V4-Pro ( deepseekai2026deepseekv4 )	1.6T (A49B)	1M $\rightarrow$ 128K	–	–	✓	✓	✓
Closed

Table 2: LLMs evaluated across input configurations. Cutoff indicates the knowledge cutoff date as publicly disclosed; “–” denotes undisclosed. En-X indicates whether the model was explicitly trained on X

\in

{Ko, Zh}. Reasoning = general reasoning capability, MoE = Mixture of Experts architecture, Thinking = a chain-of-thought mode is available . It is disabled in all our runs for comparability ( Section 4 ).

\blacklozenge

Korean-specialized models. Context lengths are restricted to 128K.

Figure 2: LOWESS-smoothed preference scores and pairwise differences as a function of segment-level

\mathsf{DDP}

. Left panels show the mean preference score (higher = better) of each strategy across the

\mathsf{DDP}

spectrum. Middle panels show the preference difference between APE seq and each competing strategy (positive = APE seq preferred). Right panels show the corresponding xComet scores for comparison with human judgments.

Figure 3: LOWESS-smoothed metric scores per model as a function of segment-level $\mathsf{DDP}$ ( $n=229{,}230$ segment–model–strategy–metric observations). Models cluster into two groups across all metrics regardless of $\mathsf{DDP}$ . TER (bottom right) decreases as $\mathsf{DDP}$ grows across most models, reflecting more conservative outputs in contextually demanding segments.

Model	Overall	By $\mathsf{DDP}$ stratum (%)
	(%)	$=0$	$1$ – $4$	$5$ – $14$	$15+$
DeepSeek-V4-Pro	3.2	5.1	1.9	1.9	0.7
HCX	7.2	10.1	6.4	4.5	2.8
Gemma-4-31B	10.6	13.2	12.3	6.4	4.2
GPT-5.4	11.5	13.8	16.0	6.4	2.8
Qwen3.5-27B	13.8	17.7	13.2	10.0	7.0

Table 3: Generation failure rate ( TER

>

100) per model and

\mathsf{DDP}

stratum. Failure rates consistently decrease as

\mathsf{DDP}

increases across all models.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	spaCy ( trf )				Flair				spaCy ( sm , weak)
Domain	$\mu$	Med	Max	% $=$ 0	$\mu$	Med	Max	% $=$ 0	$\mu$	Med	Max	% $=$ 0
News	9.7	4	52	28.9	9.2	4	50	32.9	11.2	4	50	26.2
Social	3.8	0	166	62.4	4.1	0	130	63.0	4.4	0	82	60.9
Literary	52.8	44	171	13.1	52.2	33	200	16.0	44.6	32	162	15.5

Appendix

Table 4: Per-domain

\mathsf{DDP}

statistics under spaCy ( en_core_web_trf ), Flair, and a deliberately weaker extractor ( en_core_web_sm ). The Literary

>

News

>

Social ordering and the ordering of

\ddpop{=}0

shares are preserved under all three, including the weakened pipeline.

Figure 4: Segment-lveel

\mathsf{DDP}

from spaCy vs Flair on WMT24++. Each point is a unique source segment, colored by domain. Most segmetns lie on or near the identity line.

	WMT24++		WMT25
Domain	$\mu$	max	$\mu$	max
News	9.7	52	10.6	41
Social	3.8	166	21.3	186
Literary	52.8	171	222.6	482

Appendix

Table 5: Mean and maximum segment-level

\mathsf{DDP}

scores across domains in WMT24++ and WMT25.

Figure 5: LOWESS-smoothed metric scores per model as a function of segment-level $\mathsf{DDP}$ ( En–Zh , all injection strategies averaged; four models available). As in En–Ko ( Figure 3 ), metric scores remain largely flat across the $\mathsf{DDP}$ spectrum and no model shows consistent improvement or degradation as discourse dependency increases, confirming that the insensitivity of automatic metrics to $\mathsf{DDP}$ is not specific to Korean.

Domain	Docs	Segs	$\ddpop$				% $=\!0$
Domain	Docs	Segs	mean (std)	med	p95	max	% $=\!0$
News	17	149	9.7 (12.4)	4	35	52	28.9
Social	34	494	3.8 (14.3)	0	17	166	62.3
Literary	8	206	52.8 (46.9)	44	141	171	13.1
Total	59	849	16.7 (33.2)	1	101	171	44.5

Appendix

Table 6: Statistics of the WMT24++ En–Ko subset used in our experiments. %

=\!0

reports the share of segments with

\ddpop=0

, for which no prior discourse context is required. Social texts skew strongly toward self-contained segments, while literary texts span a wide range of dependencies.

Panel of Figure 6 — Figure 6: Delta vs. Failure count per $k$ . $k{=}5$ ( blue ) achieves the lowest failure count with moderate delta.

Model	Strategy	Input	Input	Output	Latency	Latency
		(tok)	(ratio)	(tok)	(sec)	(ratio)
GPT-5.4	APE seg	216	1.0 $\times$	68	–	–
	APE seq	644	3.0 $\times$	60	–	–
	APE rel	730	3.4 $\times$	62	–	–
	APE exp	347	1.6 $\times$	72	–	–
	APE full	10,688	49.5 $\times$	62	–	–

Appendix

Table 7: Inference overhead per model and curation strategy, averaged over all evaluation segments (En–Ko). Input/latency ratios are relative to APE seg (null baseline). Latency for API-based models ( GPT-5.4 , Gemini-3-flash , DeepSeek-V4-Pro ) includes network round-trip time and is not directly comparable to open-weight models (reported as –). APE full requires 40–84

\times

more input tokens than APE seg , yet provides no consistent improvement in translation quality ( Section 5 ).

ID	$\ddpop$	(a) Literary (Forever Snow)
871	0	Prologue
872	2	"Please!" Queen Eirwen [0] begged, "don’t send her [→1] away! She ’s [→1] just a child!" The Ice King [0] looked down at his [→1] wife from his [→1] throne, his [→1] royal advisor next to him [→1] . She [→2] had given birth to a girl. He [→2] scowled at the thought.
873	6	The girl [0] in question looked a lot like her [→1] mother; pale blue eyes and white hair decorated with blue strands. The Ice King [4] thought a daughter made him [→5] look weak. After all, he [→6] had only had sons up until this point.
874	11	"The only options you have are to kill the child or send it [→1] out to the wilderness. This is her [→4] punishment for your wrongdoing, Eirwen [9] ." He [→10] spat out her [→5] name as if it [→2] was poison. He [→11] always did.
876	6	The South [→2] was desperate, and their [→2] royal family [0] would do anything to stop the bloodshed. They [→2] offered their only princess to be queen of the North [→4] and bear a son to be the future king. Then, the South [→4] would join with the North [→5] to make one kingdom. The North [→6] accepted, and Princess Eirwen [0] married Prince Akull [0] .

Appendix

Figure 7:

\mathsf{DDP}

computation on three example documents from WMT24++, one per domain. Some segments are omitted for space. Green marks named-entity anchors (Rule 1 ); blue marks pronominal anchors (Rule 2 ). Superscripts indicate the distance to the antecedent in sentences, with [0] denoting first mentions; segment-level

\mathsf{DDP}

is the maximum anchor distance within the segment.

	seg_id=923 ( $\ddpop=6$ )	seg_id=924 ( $\ddpop=53$ )	seg_id=925 ( $\ddpop=54$ )
Source	They opened the door, expecting the normal sight. Paint splotches everywhere, pencils and papers scattered across the room, easels covering up every last inch of wall. Unexpectedly, there was a different sight to be found. It was all clean, and there was a red velvet background and a stool in front of it. On the stool was Crown Prince Aquilo , getting his portrait done.	“Ah, good morning, Ivory! Care to watch your future king get his portrait done?” Aquilo inquired, winking at Ivory.	“Sorry, princey,” they said, partially ignoring him and searching for the things that their silent friend wanted. “I’m just here to get art supplies.”
MT ( Gemini-1 )	그들은 평소의 광경을 예상하며 문을 열었다. 사방에 흩어진 페인트 얼룩, 방바닥에 어지럽게 널린 연필과 종이들, 벽의 모든 공간을 차지한 이젤들. 그러나 예상치 못한 광경이 눈앞에 펼쳐졌다. 방은 깨끗했고, 붉은 벨벳 배경과 그 앞에 놓인 의자가 있었다. 의자에는 아퀼로 왕세자 가 앉아 초상화를 그리는 중이었다.	“아, 좋은 아침이야, 아이보리! 네 미래의 왕이 초상화를 그리는 걸 구경할래?” 아퀼로 가 아이보리에게 윙크하며 물었다.	“미안해, 꼬마야,” 그들은 그를 반쯤 무시하며 조용한 친구가 원하는 물건들을 찾았다. “나 그냥 미술 용품 사러 온 거야.”
HPE	그는 평범한 광경을 기대하며 문을 열었다 . 곳곳에 물감이 얼룩져 있고 연필과 종이가 방 전체에 흩어져 있으며 이젤이 벽 구석구석을 뒤덮고 있을 것이라 생각했다 . 그러나 예상과는 다른 광경이 펼쳐졌다 . 모든 것이 깨끗하게 정리되어 있었고 붉은 벨벳을 배경으로 앞에 의자가 놓여 있었다 . 그 의자에는 아퀼로 왕세자 가 앉아 있었고, 그의 초상화가 그려지고 있었다 .	“아, 좋은 아침이야 , 아이보리! 네 미래 왕의 초상화가 완성되는 모습을 보고 싶나 ?” 아퀼로 가 아이보리에게 윙크하며 물었다 .	“ 죄송합니다 , 왕자님” 아이보리는 그를 약간 무시하고 카리 공주 가 원하는 것을 찾으며 말했다 . “ 저는 그저 미술용품을 구하러 온 것뿐입니다 .”
APE seg	그들은 평범한 광경을 기대하며 문을 열었습니다. 사방에 페인트 자국이 가득하고, 연필과 종이가 방 안에 흩어져 있으며, 벽에는 이젤이 가득 차 있었습니다. 하지만 예상치 못한 다른 광경이 펼쳐졌습니다. 모든 것이 깨끗하게 정리되어 있었고, 빨간 벨벳 배경과 그 앞에 놓인 의자가 있었습니다. 그 의자 위에는 아킬로 왕자 가 초상화를 그리고 있었습니다.	“아, 좋은 아침이에요, 아이보리! 미래의 왕이 초상화를 그리는 걸 보실래요?” 아퀼로 가 아이보리에게 윙크하며 물었다.	“미안해, 왕자님.” 그들은 그를 부분적으로 무시하며 말없는 친구가 원하는 것을 찾고 있었다. “그냥 미술 용품을 사러 왔어요.”
APE seq	그들은 평범한 광경을 기대하며 문을 열었습니다 . 사방에 흩어진 페인트 자국, 연필과 종이, 벽마다 세워진 이젤들. 하지만 예상과 달리 완전히 정돈된 모습이었습니다 . 빨간 벨벳 배경 앞에 의자가 놓여 있었고, 그 의자에 아쿠일로 왕세자 가 초상화를 그리고 있었습니다 .	“아, 좋은 아침이에요 , 아이보리! 내 미래의 왕이 초상화를 그리는 걸 볼래요 ” 아킬로 가 아이보리에게 윙크하며 말했다 .	“ 미안해요 , 왕자님.” 그들은 그를 일부러 외면하며 말 없는 친구가 원하는 것을 찾으려 했습니다 . “그냥 미술 용품을 찾으러 왔어요 .”
APE rel	그들은 평범한 광경을 예상하며 문을 열었습니다. 페인트 자국이 사방에 있고, 연필과 종이가 방 안에 흩어져 있으며, 이젤이 벽 전체를 덮고 있는 모습이죠. 하지만 그들의 예상과 달리 완전히 다른 광경이 펼쳐졌습니다. 모든 것이 깔끔하게 정돈되어 있었고, 빨간 벨벳 배경 앞에는 의자가 놓여 있었습니다. 그리고 그 의자 위에는 아쿠일로 왕세자 가 초상화를 그리는 중이었습니다.	“아, 좋은 아침이에요, 아이보리! 미래의 왕이 초상화를 그리는 걸 보실래요?” 아킬로 가 아이보리에게 윙크하며 물었다.	“미안해요, 왕자님.” 그들은 그를 부분적으로 무시한 채로 말없는 친구가 원하는 물건들을 찾았습니다. “그냥 미술 용품을 사러 왔어요.”

Appendix

Table 8: Qualitative comparison of five context injection strategies on three consecutive segments from a literary document ( doc_id : forever_snow_chunk_2 , seg_id : 923–925) translated by Gemini-1 and post-edited by HyperCLOVA-X-SEED-Think-32B .

\mathsf{DDP}

increases from 6 to 53 and 54 across segments. Highlights focus on NER anchors and pronoun resolution: correct NER mention or pronoun (consistent with HPE ), incorrect or inconsistent NER transliteration, correct pronoun resolution, incorrect pronoun resolution, source pronoun requiring discourse-level resolution. Bold blue and bold black indicate formal and informal register respectively; inconsistency within a single translation reflects the failure to maintain discourse-level formality, a pattern observed across all injection strategies.

Panel of Figure 8 — Figure 8: Prompt template used for APE exp construction.

Panel of Figure 9 — Figure 9: Prompt templates for each APE condition. The system prompt (blue) and user-side context fields differ across conditions: APE seg provides no context; APE seq and APE rel supply surrounding source and target segments; APE exp provides a document-level summary; and APE full passes the full source and draft translation document.

Panel of Figure 10 — Figure 10: LOWESS-smoothed automatic metric scores as a function of segment-level $\mathsf{DDP}$ ( En–Ko , all six models averaged). Arrows indicate the preferred direction ( $\uparrow$ higher is better, $\downarrow$ lower is better).

Panel of Figure 11 — Figure 11: LOWESS-smoothed automatic metric scores as a function of segment-level $\mathsf{DDP}$ ( En–Zh , four models averaged). Arrows indicate the preferred direction ( $\uparrow$ higher is better, $\downarrow$ lower is better).

MT Evaluation

Machine translation evaluation measures translation quality, error types, and real-world suitability across languages, domains, and modalities.

See all 166 papers

Jun 15, 2026

How Far Can Machine Translation Quality Take You? Extrinsic Discourse Evaluation in Goal-Oriented Setups

Existing machine translation (MT) metrics and discourse-focused evaluations primarily assess translation quality intrinsically, without measuring the downstream consequences of translation errors. In this work, we focus on extrinsic discourse evaluation of machine translation under two distinct regimes: static and interactive. Under the static regime, we propose an entity counting task as a probe of referential consistency in discourse. We show that high intrinsic MT quality does not reliably predict downstream discourse success and strong MT systems still produce referential inconsistencies. For the interactive regime, we study the goal-oriented multi-agent Welfare Diplomacy game as a probe of long-horizon communication and coordination. We find that interaction-specific translation failures impact downstream coordination. Our results highlight goal-oriented environments as a viable framework for discourse-sensitive extrinsic MT evaluation.

Wafaa Mohammed, Kata Naszadi, Vlad Niculae

Language Technology Lab, University of Amsterdam

MT Evaluation

Ask PDF

Apr 19, 2026

Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains

Automatic evaluation metrics are central to the development of machine translation systems, yet their robustness under domain shift remains unclear. Most metrics are developed on the Workshop on Machine Translation (WMT) benchmarks, raising concerns about their robustness to unseen domains. Prior studies that analyze unseen domains vary translation systems, annotators, or evaluation conditions, confounding domain effects with human annotation noise. To address these biases, we introduce a systematic multi-annotator Cross-Domain Error-Span-Annotation dataset (CD-ESA), comprising 18.8k human error span annotations across three language pairs, where we fix annotators within each language pair and evaluate translations of the same six translation systems across one seen news domain and two unseen technical domains. Using this dataset, we first find that automatic metrics appear surprisingly robust to domain-shifts at the segment level (up to 0.69 agreement), but this robustness largely disappears once we account for human label variation. Averaging annotations increases inter-annotator agreement by up to +0.11. Metrics struggle on the unseen chemical domain compared to humans (inter-annotator agreement of 0.78-0.83 vs. 0.96). We recommend comparing metric-human agreement against inter-annotator agreement, rather than comparing raw metric-human agreement alone, when evaluating across different domains.

Finn Schmidt, Jan Philip Wahle, Terry Ruas +1

University of Göttingen, Germany

Annotator Disagreement Automated Evaluation MT Evaluation

Ask PDF

Sep 3, 2026

Last Translation Benchmark

For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, standard benchmarks for machine translation are approaching saturation. Further, automatic translation metrics are unreliable, opaque, and vulnerable to reward-hacking. Even gold human evaluation is not problem-free, because it often lacks reproducibility, objectivity, and scalability. Overall, this prevents us from tracking progress in the field and identifying pathways for improvement. We introduce the Last Translation Benchmark, a collection of human-authored and peer-reviewed examples (texts, images, audio, videos) that break leading machine translation models. We also present a new evaluation approach: each example comes with handcrafted verification rules describing concrete failure cases on that example, therefore allowing reliable and actionable future evaluation. The Last Translation Benchmark is a live dataset that accepts ongoing contributions. The latest version is LTBv1, containing accepted contributions prior to September 1st 2026, with future releases planned as new data is continuously collected.

Vilém Zouhar, Niyati Bafna, Mukund Choudhary +257

Multimodal MT MT Evaluation

Ask PDF

Abstract

MT Evaluation

Explore similar work

How Far Can Machine Translation Quality Take You? Extrinsic Discourse Evaluation in Goal-Oriented Setups

Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains

Last Translation Benchmark