Reference-based metrics remain the standard choice in machine translation evaluation, partly because quality estimation methods often correlate less well with human judgments. As a result, source-free, reference-based evaluation has become the practical norm, even though it is unfaithful to the definition of translation adequacy and unfair to systems whose outputs preserve the source meaning while differing from the reference. This paper argues that adequacy must be judged with respect to the source. A reference is only one possible rendering of the source and may introduce bias, under-specification, or errors. We further argue that source-reference-hypothesis evaluation is fair only when the judge treats the reference as auxiliary evidence rather than as the primary standard. Otherwise, even source-aware evaluation can reduce adequacy to preference towards reference. We show the existing hybrid metrics are highly reliant on reference compared to source. Our argument is not that all automatic MT metrics fail to use the source. Rather, we argue that any evaluation protocol that removes the source, or allows the reference to dominate the source, is structurally incomplete for adequacy evaluation. However, existing MT papers generally prefer reference-based metrics and use QE metrics only when reference is unavailable. We therefore call for QE to be reframed as a primary approach to source-grounded adequacy evaluation, rather than as a fallback motivated by missing references. We further call for hybrid metrics whose designs explicitly prioritize source--hypothesis faithfulness while using references only as complementary evidence.
Figures & tables
Figure 1: Paired source- and reference-corruption sensitivities for COMET. Positive sensitivity denotes a reduction in score after corruption. Each pair corresponds to the same hypothesis. Reference corruption produces a substantially larger score reduction for almost every example.
Mean COMET score
Sensitivity
Language pair
Original
Source replaced
Reference replaced
Δsrc
Δref
Ref/Src
Ref. dominates
Into English
de-en
0.8898
0.8459
0.4280
0.0438
0.4618
10.53×
100.0%
es-en
0.8992
0.8527
0.3959
0.0465
0.5033
10.83×
99.8%
fr-en
0.8957
0.8538
0.4206
0.0418
0.4751
11.36×
100.0%
it-en
0.8846
0.8561
0.5027
0.0285
0.3819
13.41×
99.8%
Table 1: COMET scores and corruption sensitivities by language pair. Original is the mean score for the consistent source, good translation, and reference. Source replaced and Reference replaced report the mean scores after replacing the corresponding input. Ref/Src is the ratio of mean reference sensitivity to mean source sensitivity. Ref. dominates is the percentage of examples for which reference sensitivity exceeds source sensitivity.
Figure 2: Paired source- and reference-corruption sensitivities for XCOMET-XXL. Positive sensitivity denotes a reduction in score after corruption. Each pair corresponds to the same hypothesis. Reference corruption produces a larger score reduction for the great majority of examples.
Mean XCOMET-XXL score
Sensitivity
Language pair
Original
Source replaced
Reference replaced
Δsrc
Δref
Ref/Src
Ref. dominates
Into English
de-en
0.8660
0.4468
0.2364
0.4191
0.6295
1.50×
97.4%
es-en
0.9115
0.4398
0.2305
0.4717
0.6810
1.44×
99.0%
fr-en
0.8130
0.4373
0.2205
0.3758
0.5925
1.58×
96.6%
it-en
0.7035
0.4443
0.2197
0.2592
0.4837
1.87×
98.2%
Table 2: XCOMET-XXL scores and corruption sensitivities by language pair. Original is the mean score for the consistent source, good translation, and reference. Source replaced and Reference replaced report the mean scores after replacing the corresponding input. Ref/Src is the ratio of mean reference sensitivity to mean source sensitivity. Ref. dominates is the percentage of examples for which reference sensitivity exceeds source sensitivity. Each direction contains 500 examples except en-ja , with 430, and ru-en , with 472.
Figure 3: Paired source- and reference-corruption sensitivities for MetricX-24-Hybrid-XL. Positive sensitivity denotes an increase in score after corruption because lower MetricX scores indicate better translation quality. Reference corruption has the larger effect overall, but the two sensitivity distributions overlap substantially.
Mean MetricX score
Sensitivity
Language pair
Original
Source replaced
Reference replaced
Δsrc
Δref
Ref/Src
Ref. dominates
Into English
de-en
3.2048
9.7065
16.7304
6.5017
13.5257
2.08×
99.4%
es-en
2.9215
13.4021
16.8966
10.4805
13.9750
1.33×
91.0%
fr-en
3.5234
12.1798
16.8541
8.6564
13.3307
1.54×
94.0%
it-en
3.8907
8.3780
15.6120
4.4873
11.7213
2.61×
98.6%
Table 3: MetricX-24-Hybrid-XL scores and corruption sensitivities by language pair. Lower MetricX scores are better. Original is the mean score for the consistent source, good translation, and reference. Source replaced and Reference replaced report the mean scores after replacing the corresponding input. Ref/Src is the ratio of mean reference sensitivity to mean source sensitivity. Ref. dominates is the percentage of examples for which reference sensitivity exceeds source sensitivity.
Terminology evaluation in machine translation (MT) usually assumes a single correct target form per source term. However, human translators routinely introduce variation that current metrics penalize as inconsistency. We examine how to account for this variation in document-level MT evaluation of English-French scientific translation, combining glossary-based accuracy, translation consistency, and a new cross-term variation (CTV) diagnostic measure that tests whether variation relationships are preserved across languages. Based on analyses of two parallel corpora, translated by four MT systems, we find that (1) MT systems generate less target-side variation than human translators; (2) transfer patterns strongly depend on the variation type; (3) consistency rankings vary with the choice of metric; and (4) constraining MT with a glossary improves accuracy and consistency but degrades CTV by suppressing valid variation. We argue for variation-aware evaluation that conditions consistency penalties on whether target-side variation mirrors source-side variation.
The WMT26 General MT task evaluates systems on 10 language pairs that have no human references (neither translated from scratch nor post-edited from MT output by humans). We describe how we built the pseudo-references for these pairs and six other language pairs (in which some forms of human references are available): seven models translate the 3,277 official documents under up to five prompt conditions, giving a total of 26 system-prompt combinations; then three reference-free quality estimation (QE) models score every candidate; and a per-document selector picks one translation, which GPT-5.5 post-edits where needed. Working without references exposed a failure mode of QE-guided selection: the metrics rank fluent output in the wrong language above correct translations. Adding a confidence-scaled language identification penalty to the score fusion drives the wrong-language count to zero, and the resulting selector still scores better on MetricX than the rank-fusion baseline it replaces. Since no references were available for these pairs while we were building them, we calibrate every selection decision on last year's WMT25 human judgments. The human evaluation, released after construction, shows the cost of getting selection wrong: our references stand with the strongest participating systems when the selector kept a frontier-model candidate, and fall up to 17 ESA points below them when it did not. We release the selection method and the provenance of every reference (https://github.com/surrey-nlp/PseudoRef)
We propose Dynamic Meta-Metrics (DMM), a framework for machine translation evaluation that learns source-sentence conditioned combinations of existing metrics. Rather than relying on a single static ensemble or language-specific weighting, DMM adapts the metric combination based on properties of the source segment. We study hard conditioning, which fits an interpretable combiner per cluster, and an exploratory soft-conditioned extension whose weights vary continuously with source-cluster responsibilities. We evaluate DMM on the WMT Metrics Shared Task data across multiple language pairs using pairwise agreement measures at the system and segment levels. Across settings, MLP-based combinations outperform linear and Gaussian process-based ensembles, and introducing soft conditioning yields gains over linear models.
Luke Zhang, Justin Vasselli, Aditya Khan +2
University of Toronto, Canada · Nara Institute of Science and Technology, Japan · Ontario Tech University, Canada