In the Blind: Building Pseudo-References for MT Evaluation

Diptesh Kanojia; Chi-kiu Lo; Archchana Sindhujan; Samuel Larkin; Greg Hanneman; Alon Lavie

cs.CLSep 11, 2026

In the Blind: Building Pseudo-References for MT Evaluation

Authors: Diptesh Kanojia, Chi-kiu Lo, Archchana Sindhujan, Samuel Larkin, Greg Hanneman, Alon Lavie

Organizations: University of Surrey · National Research Council Canada · Unaffiliated · Carnegie Mellon University

Abstract

The WMT26 General MT task evaluates systems on 10 language pairs that have no human references (neither translated from scratch nor post-edited from MT output by humans). We describe how we built the pseudo-references for these pairs and six other language pairs (in which some forms of human references are available): seven models translate the 3,277 official documents under up to five prompt conditions, giving a total of 26 system-prompt combinations; then three reference-free quality estimation (QE) models score every candidate; and a per-document selector picks one translation, which GPT-5.5 post-edits where needed. Working without references exposed a failure mode of QE-guided selection: the metrics rank fluent output in the wrong language above correct translations. Adding a confidence-scaled language identification penalty to the score fusion drives the wrong-language count to zero, and the resulting selector still scores better on MetricX than the rank-fusion baseline it replaces. Since no references were available for these pairs while we were building them, we calibrate every selection decision on last year's WMT25 human judgments. The human evaluation, released after construction, shows the cost of getting selection wrong: our references stand with the strongest participating systems when the selector kept a frontier-model candidate, and fall up to 17 ESA points below them when it did not. We release the selection method and the provenance of every reference (https://github.com/surrey-nlp/PseudoRef)

Figures & tables

Panel of Figure 1 — Figure 1: The pseudo-reference pipeline.

Pair	Docs	Domain	Docs
eng $\rightarrow$ bel, deu, ekk hye, ind, isl,	198	spoken	2,090
jpn, kaz, kor, rus, tha, ukr,		social	737
zho-Hans, zho-Hant		news	259
ces $\rightarrow$ deu	315	software	98
zho-Hans $\rightarrow$ jpn	190	edu	63
		factcheck	30

Table 1: Documents by language pair and by domain.

Model	Prompt conditions	Cells
Closed
GPT-5.5	P1–P3, P4 $\times 2$	5
Gemini-3.1-Pro	P1–P3, P4 $\times 2$	5
Open-weight
Aya-Expanse-32B	P1–P3, P4 $\times 2$	5
Tower-Plus-9B	P1–P3, P4 $\times 2$	5

Table 2: Candidate-generation: A cell denotes one model–prompt configuration.

Pair	N	RF	XR
eng $\rightarrow$ kaz	198	54	36
eng $\rightarrow$ bel	198	63	29
eng $\rightarrow$ hye	198	14	1
zho-Hans $\rightarrow$ jpn	190	4	1
eng $\rightarrow$ ekk	198	3	2
eng $\rightarrow$ tha	198	3	1

Table 3: Wrong-language documents per selector: langid prediction differs from the expected target. Rank-Fusion, Cross-reference, APE5 post-edited (APE-tight), C6 the language-aware selector of Section 4.1 . † eng

\rightarrow

ind counts are an ind/msa identifier artefact.

Selector	MX $\downarrow$	CK $\uparrow$	xC $\uparrow$	WL $\downarrow$	Crit $\downarrow$
Rank-fusion	4.068	0.670	0.697	154	5,516
APE5 (post-edit all)	4.515	0.644	0.678	19	5,595
C6: MX + span tiebreak	4.005	0.654	0.691	0	4,129
C2: MX only	4.004	0.655	0.690	0	5,196
C5: hard gate	4.067	0.663	0.698	0	4,450
C: 0.70/0.15/0.15	4.071	0.662	0.698	0	4,431

Table 4: Selector variants over all 3,277 documents. No variant is best on every measure; C6 is chosen for the MetricX near-optimum at zero wrong-language. MX: MetricX mean (lower better); CK: CometKiwi; xC: xCOMET; WL: wrong-language documents; Crit: xCOMET critical spans. Weight triples are MetricX/CometKiwi/xCOMET in Eq. 1 .

Pair	ESA	MX $\downarrow$	CK $\uparrow$	xC $\uparrow$
ces $\rightarrow$ deu	98.7	3.98	0.550	0.446
eng $\rightarrow$ ekk	88.9	8.26	0.697	0.564
eng $\rightarrow$ isl	83.4	7.57	0.647	0.520
eng $\rightarrow$ jpn	98.4	6.11	0.681	0.513
eng $\rightarrow$ rus	95.1	5.98	0.637	0.582
eng $\rightarrow$ ukr	95.6	6.36	0.626	0.540

Table 5: What human-quality translation scores: mean human ESA and our three metrics on the human-best WMT25 candidates (1,409 paragraphs, 7 pairs).

Metric	$r$ top	$\tau$ sys	Pairwise %	Fitted $w$
MetricX-24	0.495	0.29	58.3	0.51
CometKiwi-XL	0.071	0.17	56.6	0.42
xCOMET-XL	0.166	0.12	55.9	$-$ 0.07
3-metric fusion	–	0.26	–	–

Table 6: Agreement between each QE metric and WMT25 human judgment, computed on WMT25 data.

r

top: Pearson against ESA on human-best candidates (

n{=}1{,}409

);

\tau

sys: mean per-pair Kendall

\tau

of system rankings (37 systems, 7 pairs,

n{=}25{,}824

); Pairwise: segment-level pairwise accuracy; Fitted

w

: normalised OLS weight.

Selector	Total spans	MQM-wt	Spans/doc	Spans/1k	%crit	%clean	Wrong-lang	MetricX $\downarrow$
Rank fusion	29,328	121,836	8.95	11.1	13	14	154	4.068
Cross-reference	29,751	125,667	9.08	11.2	13	14	87	4.187
Best open system	27,788	113,294	8.48	10.3	14	16	116	4.508
Best closed system	29,392	116,242	8.97	11.3	12	17	24	4.488
Pooled, 3 metrics	27,725	109,619	8.46	10.4	10	16	131	4.067
APE-tight, all documents	30,414	129,457	9.28	11.4	15	16	19	4.515

Table 7: All selections over the 3,277 documents. Rank fusion is Borda over MetricX-24 and CometKiwi-XL; cross-reference is cross-system MBR; the pooled selector rank-fuses over the six prior selections rather than over the 26; per-language APE uses APE-spans on the CJK pairs and APE-tight elsewhere; C6 is the selector of Section 4.1 . Span inventory: total; MQM-weighted

1{\cdot}

minor

+5{\cdot}

major

+10{\cdot}

critical; per document; per 1k characters; percent of documents with a critical span; percent with no spans.

Domain	Docs	MX	C6-MX	WL
spoken	2,090	1.940	1.994	127
social	737	8.483	8.127	15
news	259	8.986	8.813	4
software	98	2.589	2.587	6
edu	63	4.463	4.345	2
factcheck	30	5.428	5.290	0

Table 8: Per-domain comparison of rank fusion and C6.

Cell	#tokens	emission (kgCO2e)
p1	3.13M	0.06
p2	5.17M	0.09
p3	9.78M	0.17
p4-gemini	8.43M	0.15
p4-qwen	8.57M	0.15
APE-scores	7.44M	0.13

Table 9: Total number of input, reasoning, output tokens and an estimation of the carbon footprint of each GPT-5.5 cell

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Cell	Total	Usable	Dominant failure
GPT-5.5	p1	3,277	3,277	none
	p2	3,277	3,277	none
	p3	3,277	3,277	none
	p4-qwen	2,706	2,706	none
	p4-gemini	2,706	2,702	malformed - 4
Gemini-3.1	p1	3,277	3,276	malformed - 1

Appendix

Table 10: Panel usability per cell. HY-MT’s wrong-script output concentrates on en–hy (188) and en–kk (187), scripts it cannot produce. Aya’s analysis prompts copy the English reasoning block on hard pairs. EuroLLM fails non-Latin targets.

	Input (#1)	APE-scores	APE-spans	APE-tight
Language (langid, wrong docs)	154	21	24	19
Structure fidelity	0.987	0.995	0.995	0.996
Content anchors	0.863	0.839	0.834	0.911
Fluency	0.971	0.971	0.971	0.971
Median similarity to input	–	0.590	0.562	0.571
Documents byte-identical to input	–	15	3	47

Appendix

Table 11: The three post-editing runs over all 3,277 documents. APE-scores sees QE scores, APE-spans adds xCOMET spans, APE-tight is the tightened prompt. Structure fidelity, content-anchor preservation, and fluency are the heuristic measures of Appendix G . The language row for the runs uses the same langid audit as Table 3 computed on the APE outputs directly (run counts include the en–id artefact).

Pair	TransGemma	HY-MT	GPT5 p1	GPT5 p4q	Gemini p1	Gemini p4q	Aya p1	Tower p1	EuroLLM p1
ces $\rightarrow$ deu	4.07	4.42	4.44	2.69	4.52	2.73	4.47	4.61	8.69
eng $\rightarrow$ bel	5.32	5.51	6.29	5.60	6.05	5.81	12.12	8.50	12.59
eng $\rightarrow$ deu	2.54	3.00	3.31	2.62	3.21	2.89	3.10	3.15	3.35
eng $\rightarrow$ ekk	6.14	6.47	6.97	6.41	7.06	6.76	15.31	14.49	7.53
eng $\rightarrow$ hye	5.28	6.36	5.33	4.41	4.90	4.65	19.01	17.78	18.76
eng $\rightarrow$ ind	3.27	3.57	4.41	3.68	4.08	3.77	4.36	6.13	8.50

Appendix

Table 12: Candidate-level MetricX means (lower is better) per pair for nine of the 26 cells: the two native cells, the P1 and P4-qwen cells of the two closed models, and the P1 cells of Aya, Tower-Plus and EuroLLM, their best condition. Bold: best of the shown cells, except that the P4-qwen columns are computed on that model’s 2,706-document multimodal subset and are not comparable with the rest.

Candidate	MX $\downarrow$	CK	xC	langid ( $p$ )	$F$	$F-\lambda p$
eurollm-p3	1.65	0.772	0.945	ru (1.00)	0.82	$-2.18$
gemini-p4q	2.58	0.720	0.890	kk (1.00)	0.71	0.71

Appendix

Table 13: The EuroLLM candidate is fluent Russian and best on all three metrics, so no reweighting would have rejected it; the language penalty demotes it below every correct-Kazakh candidate.

Domain	Docs	Segs	Seg/doc	seg ESA	doc ESA
software	72	1,825	25.4	86.8	86.3
news	179	2,130	11.9	82.6	82.2
factchecking	6	32	5.3	85.2	85.5
social	509	2,130	4.2	68.5	71.9
edu	10	33	3.3	81.8	77.9
speech	1,516	1,516	1.0	63.2	63.2

Appendix

Table 14: How many annotation items each document contributes, by domain, and the mean human ESA of our references under the two aggregation conventions. The spoken domain is one item per document and drives the difference between the two final rows.

Unit	Comparison	n	ours	human	gap (95% CI)
vs. Human (from scratch)
seg	all	1,363	78.0	83.7	$-$ 5.7 [ $-$ 7.0, $-$ 4.5]
seg	excl. speech	1,096	80.2	83.8	$-$ 3.6 [ $-$ 5.0, $-$ 2.3]
seg	speech only	267	69.2	83.4	$-$ 14.3 [ $-$ 17.5, $-$ 11.0]
doc	all	414	71.9	83.2	$-$ 11.3 [ $-$ 13.8, $-$ 9.0]
doc	excl. speech	147	77.0	82.8	$-$ 5.9 [ $-$ 8.6, $-$ 3.2]

Appendix

Table 15: Our references against the human systems, paired on identical segments (seg) or identical documents (doc), mean ESA. Intervals are percentile bootstrap, 2,000 resamples. The speech-only rows are identical across units because every spoken document is one segment. Human systems appear in ces

\rightarrow

deu, eng

\rightarrow

deu, eng

\rightarrow

ekk, eng

\rightarrow

isl, eng

\rightarrow

jpn and zho-Hans

\rightarrow

jpn.

Domain	Segs	ESA	Median	% $<$ 50
software	1,825	86.8	95	5.5
factchecking	32	85.2	92	6.2
news	2,130	82.6	90	7.7
edu	33	81.8	90	6.1
social	2,130	68.5	80	20.2
speech	1,516	63.2	70	29.7

Appendix

Table 16: Human ESA of our references by domain, segment-averaged. The document-averaged column is given in Table 14 .

all domains
		gap against participating system
Group	Ours	Gem.	TRIVE	Wayf.	Lumen	GPT	UvA
frontier, 4,912	81.4	$-$ 1.4	$-$ 1.7	$-$ 1.9	$-$ 2.1	$+$ 0.9	$+$ 0.1
open MT, 2,541	65.2	$-$ 17.3	$-$ 16.5	$-$ 16.1	$-$ 15.2	$-$ 14.4	$-$ 14.4
spoken documents
frontier, 584	80.1	$+$ 0.4	$+$ 0.5	$+$ 0.9	$+$ 3.5	$+$ 1.7	$+$ 1.1

Appendix

Table 17: Mean human ESA of our references by the family of the selected candidate, and the paired gap against each participating system on identical segments. In the frontier rows every gap except Lumen on spoken documents has a 95% interval containing zero or close to it; in the open rows every interval excludes zero by a wide margin. Widths are given in the released verification output.

Pair	Docs	Segs	ESA	% open	gap (95% CI)	doc gap
eng $\rightarrow$ zho-Hans	25	70	66.0	66	$-$ 11.3 [ $-$ 16.7, $-$ 5.9]	$-$ 12.8
eng $\rightarrow$ kor	51	167	69.9	27	$-$ 11.1 [ $-$ 15.3, $-$ 7.0]	$-$ 26.5
eng $\rightarrow$ jpn	198	691	79.0	55	$-$ 9.5 [ $-$ 11.3, $-$ 8.0]	$-$ 14.2
eng $\rightarrow$ deu	68	173	59.7	61	$-$ 9.3 [ $-$ 12.6, $-$ 6.0]	$-$ 13.5
eng $\rightarrow$ rus	189	664	72.4	48	$-$ 9.2 [ $-$ 11.2, $-$ 7.5]	$-$ 17.5
eng $\rightarrow$ bel	136	435	67.5	17	$-$ 9.0 [ $-$ 11.4, $-$ 6.8]	$-$ 13.5

Appendix

Table 18: Per-pair difficulty, ordered by the segment-level gap. ESA is our own segment-averaged score; % open is the share of annotated segments on which the selector kept a translation-specialised open model; gap is our ESA minus the mean of Gemini 3.1 Pro, TRIVE, Wayfinder, Lumen, GPT-5.5 and UvA-MT on identical segments, with a percentile bootstrap interval over 2,000 resamples; doc gap is the same quantity computed on document means. Segments are counted where at least four of the six anchors were annotated, which is 7,644 of our 7,666.

Family	Docs	ESA	MX all	MX sel.
TranslateGemma	564	54.1	4.63	3.29
GPT-5.5	578	79.0	5.06	5.12
Gemini-3.1-Pro	506	80.9	5.11	4.98
HY-MT1.5	377	52.8	5.40	2.60
Tower-Plus	122	63.9	9.51	3.26
Aya-Expanse	89	67.1	12.53	4.56

Appendix

Table 19: Human ESA of our references by producing family, document-averaged, against mean MetricX (lower is better) computed two ways: over every candidate that family produced for these documents (MX all), and over only the documents where that family won selection (MX sel.). The second column must not be used to rank families.

MT Evaluation

Machine translation evaluation measures translation quality, error types, and real-world suitability across languages, domains, and modalities.

See all 173 papers

FACET at WMT 2026 Automated Translation Quality Evaluation Task
Sep 8, 2026Ahrii Kim, Chanjun Park, Seong-heum KimMT Evaluation Machine Translation
AI-Bio Convergence Research Inst. Soongsil University · School of Software Soongsil University · Dept. of Intelligent Semiconductors Soongsil University
Source-Free MT Evaluation Is Not MT Evaluation
Aug 21, 2026Baban Gain, Ramakrishna Appicharla, Asif EkbalMT Evaluation Reference-Free Evaluation
Indian Institute of Technology Patna · Shiv Nadar University, Chennai
AskQE: Question Answering as Automatic Evaluation for Machine Translation
Apr 15, 2025Dayeon Ki, Kevin Duh, Marine CarpuatMT Evaluation
University of Maryland · Johns Hopkins University

Sep 8, 2026J/K move · Enter open · S save

FACET at WMT 2026 Automated Translation Quality Evaluation Task

Ahrii Kim, Chanjun Park, Seong-heum Kim

AI-Bio Convergence Research Inst. Soongsil University · School of Software Soongsil University · Dept. of Intelligent Semiconductors Soongsil University

MT Evaluation Machine Translation

Open paper AskPDF arXiv

Different error types in machine translation require different evidence. Whether meaning is preserved can be judged only against the source, while whether the target is well-formed, or whether it names one entity consistently, can be judged from the target alone. We present FACET, our reference-free submission to the WMT26 Automated Translation Quality Evaluation Task, which decomposes evaluation into Fluency, Accuracy, and Consistency passes and gives each pass only the context its error type requires. A single fixed model is prompted three times, and the merged error spans yield the three task outputs, error spans, quality scores, and error-free labels, with no trained components. We also submit FACET-C, which omits the Consistency pass. Without gold labels, we characterize the predictions of FACET. Its system rankings place post-edited human translation first, and the Consistency pass changes about a tenth of segment scores while leaving the ranking nearly unchanged.

Abstract

MT Evaluation

Explore similar work

FACET at WMT 2026 Automated Translation Quality Evaluation Task

Source-Free MT Evaluation Is Not MT Evaluation

AskQE: Question Answering as Automatic Evaluation for Machine Translation