Rethinking Round-Trip Translation for Machine Translation Evaluation
Authors: Terry Yue Zhuo, Qiongkai Xu, Xuanli He, Trevor Cohn
Organizations: Monash University, Clayton, VIC, Australia · The University of Melbourne, Carlton, VIC, Australia · University College London, London, United Kingdom
Automatic evaluation on low-resource language translation suffers from a deficiency of parallel corpora. Round-trip translation could be served as a clever and straightforward technique to alleviate the requirement of the parallel evaluation corpus. However, there was an observation of obscure correlations between the evaluation scores by forward and round-trip translations in the era of statistical machine translation (SMT). In this paper, we report the surprising finding that round-trip translation can be used for automatic evaluation without the references. Firstly, our revisit on the round-trip translation in SMT evaluation unveils that its long-standing misunderstanding is essentially caused by copying mechanism. After removing copying mechanism in SMT, round-trip translation scores can appropriately reflect the forward translation performance. Then, we demonstrate the rectification is overdue as round-trip translation could benefit multiple machine translation evaluation tasks. To be more specific, round-trip translation could be used i) to predict corresponding forward translation scores; ii) to improve the performance of the recently advanced quality estimation model; and iii) to identify adversarial competitors in shared tasks via cross-system verification.
Figures & tables
Figure 1: Given the corpus DA in Language A , we are able to acquire the round-trip translation (RT) results DA′ and forward translation (FT) results DB via machine translation. A question was raised by machine translation community two decades ago, “Can RT results be used to estimate FT performance?”. The answers to the question appeared to be controversial , where most researchers tend to be against the use of round-trip translation due to the obscure correlations between them. Our work gives a clear and positive answer to the usefulness of RT, based on extensive experiments.
Figure 2: The comparison of the forward translation (FT) and round-trip translation (RT) performance of two translation systems, System 1 and System 2 are based on Statistical Machine Translation (SMT) and Neural Machine Translation (NMT), respectively. The conflict conclusions by FT Scores (System 1 < System 2) and RT Scores (System 1 > System 2) are attributed to the translation of the underlined words, ‘reclassified’ and ‘Biotech’.
Lang. Pair
K. τ↑
P. r↑
w/ cp
w/o cp
w/ cp
w/o cp
de-en
-0.11
0.20
-0.90
1.00
en-de
-0.40
0.60
-1.00
1.00
cs-en
-0.20
0.30
-0.99
0.99
en-cs
-0.40
0.60
-0.90
0.99
fr-en
0.20
0.60
-1.00
1.00
Table 1: Comparison between RT-Score and FT-Score on two groups of systems with copying (w/ cp) and without copying (w/o cp) unknown words using Kendall’s τ and Pearson’s r on six language pairs.
Lang. Pair
Avg. Copy (%)
SMT
NMT
de-en
17.39
9.28
en-de
21.47
9.54
Table 2: Comparison of word copy frequency between SMT and NMT on two language pairs. We calculate average percentage of copy (Avg. Copy) per sentence. The details of selected Moses system are reported in Appendix C .
Figure 3: The 33 languages in FLORES-AE33 are separated into two categories, Seen ( ) includes the languages used in both training and testing , and Unseen ( ) is composed of the languages only used in testing . contains 7 High-resource (H.), 7 Medium-resource (M.) and 6 Low-resource (L.) languages, while involves 9 Medium-resource (M.) and 4 Low-resource (L.) languages. These two sets are used to construct three types of language pairs for test. Type I and Type III target on translation among and language pairs, respectively. Type II targets on translation between and . The test setting with more is usually more challenging, i.e., Type I < Type II < Type III .
MT System
Trans. Metric
Type I
MAE ↓
RMSE ↓
P. r↑
mBART50-m2m
BLEU
1.80
2.70
0.94
spBLEU
2.13
2.99
0.94
chrF
3.51
4.53
0.96
BERTScore
4.98
7.07
0.88
M2M-100-base
BLEU
3.86
5.82
0.95
Table 3: The results of predicted FT-Score s of mBART50-m2m , M2M-100-base and Google-Trans on Type I test set based on different translation evaluation metrics (Trans. Metric). *MAE: Mean Absolute Error, RMSE: Root Mean Square Error, P. r : Pearson’s r .
MT System
Trans. Metric
Type II
Type III
MAE ↓
RMSE ↓
P. r↑
MAE ↓
RMSE ↓
P. r↑
mBART50-m2m
BLEU
1.36
1.97
0.93
0.81
0.95
0.96
spBLEU
1.61
2.19
0.93
1.20
1.38
0.94
chrF
3.80
4.89
0.95
3.04
3.89
0.95
BERTScore
4.67
6.38
0.88
5.08
6.88
0.87
M2M-100-base
BLEU
3.10
4.16
0.95
2.99
3.76
0.94
Table 4: The results of predicted FT-Score s of mBART50-m2m (a seen MT system) and M2M-100-base (an unseen MT system) on Type II and Type III (with unseen languages) test sets based on different translation evaluation metrics (Trans. Metric).
MAE ↓
P. r↑
H.
M.
L.
H.
M.
L.
mBART50-m2m
H.
3.17
2.90
2.70
0.94
0.94
0.77
M.
1.51
1.37
1.77
0.97
0.85
0.22
L
1.22
1.27
1.16
0.97
0.87
0.78
M2M-100-base
H.
8.72
5.41
3.50
0.51
0.80
0.67
M.
4.86
4.01
2.93
0.86
0.90
0.69
Table 5: The results of predicted FT-Score s of mBART50-m2m and M2M-100-base on nine sets of language pairs, categorized by different scale of the resources, High (H.), Medium (M.) and Low (L.). The three categories in rows are source languages, and the ones in columns are target languages. We report Mean Absolute Error (MAE) and Pearson’s r .
Lang. Pair
A↻B
B↻A
A↻B & B↻A
MAE ↓
RMSE ↓
K. τ↑
P. r↑
MAE ↓
RMSE ↓
K. τ↑
P. r↑
MAE ↓
RMSE ↓
K. τ↑
P. r↑
cs-en
4.01
4.34
0.20
0.45
8.92
9.08
0.60
0.91
8.53
8.71
0.60
0.88
de-en
13.23
13.26
0.80
0.95
1.69
1.77
0.80
0.95
1.26
1.38
0.80
0.96
de-fr
10.45
10.53
1.00
0.99
1.72
2.05
0.80
0.97
1.59
1.93
1.00
0.97
en-cs
6.96
7.49
0.20
0.25
1.39
1.79
0.60
0.94
1.25
1.80
0.60
0.95
en-de
2.96
4.00
0.40
0.59
2.29
2.70
1.00
0.92
2.75
3.12
1.00
0.93
Table 6: The results of our predictors on ranking the selected MT systems on WMT2020-News shared tasks.
QE model
zh-en
en-de
K. τ↑
P. r↑
K. τ↑
P. r↑
RTT - BLEU
15.17
21.76
11.83
19.71
RTT - spBLEU
13.55
18.30
11.49
19.00
RTT - chrF
15.52
21.74
13.57
22.93
RTT - BERTScore
15.70
21.96
25.89
44.10
RTT -ALL
15.90
22.36
26.02
44.33
Table 7: Comparisons of RT-Score for QE. RTT -ALL refers to the combination of RTT - BLEU , RTT - spBLEU and RTT - chrF . COMET-QE-DA + RTT -ALL incorporates both COMET-QE-DA and all RT-Score s.
# Sys.
Method
No Adversary
One adversarial SMT
Two adversarial SMTs
K. τ↑
P. r↑
Hit@1 ↑
Avg. Rank ↓
K. τ↑
P. r↑
Hit@2 ↑
Avg. Rank ↓
K. τ↑
P. r↑
3
Sing-Check
0.07
0.17
0.50
2.00
0.33
0.51
0.00
4.75
-0.15
-0.30
X-Check
0.47
0.43
1.00
1.00
0.33
0.98
1.00
1.50
0.55
0.98
4
Sing-Check
0.33
0.37
0.25
2.75
0.40
0.39
0.00
5.75
-0.03
-0.33
X-Check
0.57
0.81
1.00
1.00
0.60
0.97
1.00
1.50
0.70
0.98
5
Sing-Check
0.48
0.58
0.25
3.25
0.30
0.25
0.00
6.75
-0.05
-0.40
Table 8: Results of the competition between 3 to 5 honest competitors, with a combination of additional adversarial competing systems (No Adversary, One adversarial SMT ( X=0.1 ) w/ copy, Two adversarial SMTs ( X=0.1 and X=0.5 ) w/ copy). We measure the identifiability of the adversarial MT systems by Hit@ K , where K is decided by the number of adversarial systems. We also report the average ranking (Avg. Rank.) of the adversarial systems, and correlation scores, Kendall’s τ and Pearson’s r .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Table 9: The statistics of FLORES-AE33 . 20 languages are used in both training and test ( ), the other 13 languages are used in test only ( ).
Figure 4: The first row is the correlations between RT-ScoreA↻BM and FT-ScoreA→BM on mBART50-m2m using (a) BLEU , (b) spBLEU , (c) chrF and (d) BERTScore . The second row is the correlations between RT-ScoreB↻AM and FT-ScoreA→BM on mBART50-m2m using (e) BLEU , (f) spBLEU , (g) chrF and (h) BERTScore . All experiments with overall Pearson’s r .
Figure 5: The first row is the correlations between RT-ScoreA↻BM and FT-ScoreA→BM on M2M-100-base using (a) BLEU , (b) spBLEU ;, (c) chrF and (d) BERTScore . The second row is the correlations between RT-ScoreB↻AM and FT-ScoreA→BM on M2M-100-base using (e) BLEU , (f) spBLEU , (g) chrF and (h) BERTScore . All experiments with overall Pearson’s r .
MT System
Comparison
BLEU
spBLEU
chrF
BERTScore
mBART50-m2m
A→B vs. A↻B
0.78
0.86
0.63
0.53
A→B vs. B↻A
0.94
0.94
0.96
0.88
M2M-100-base
A→B vs. A↻B
0.83
0.93
0.87
0.53
A→B vs. B↻A
0.95
0.96
0.96
0.90
Appendix
Table 10: Pearson’s r between \textscFT−ScoreA→BM and \textscRT−ScoreM (both A↻B and B↻A ) using different automatic evaluation metrics M on Type I test set of FLORES-AE33 .
MT System
Self-Trans Feature
Type I
Type II
Type III
MAE ↓
RMSE ↓
r↑
MAE ↓
RMSE ↓
r↑
MAE ↓
RMSE ↓
r↑
mBART50-m2m
spBLEU (basic model)
2.13
2.99
0.94
1.61
2.19
0.93
1.20
1.38
0.94
+ Max-4 Count
2.01
2.92
0.94
1.54
2.15
0.94
1.12
1.34
0.94
+ Ref Length
2.07
2.96
0.94
1.61
2.21
0.93
1.17
1.45
0.94
+ Max-4 Count & Ref Length
2.00
2.92
0.94
1.53
2.16
0.94
1.08
1.33
0.95
M2M-100-base
spBLEU (basic model)
3.97
5.72
0.96
3.24
4.18
0.96
3.18
3.88
0.95
Appendix
Table 11: The results of using auxiliary features to spBLEU for training predictors. We test the performance of mBART50-m2m and M2M-100-base cross language pairs in Type I , Type II and Type III of FLORES-AE33 .
Langauge Pair
MAE ↓
RMSE ↓
K. τ↑
P. r↑
de-fr
2.21
2.67
1.00
0.98
en-ta
0.88
0.98
1.00
0.99
zh-en
1.69
2.37
1.00
0.99
Average
1.59
2.01
1.00
0.99
Appendix
Table 12: Results of prediction and ranking on translation quality of WMT2020-News synthetic data for three language pairs.
Langauge Pair
B↻A
A↻B & B↻A
MAE ↓
RMSE ↓
K. τ↑
P. r↑
MAE ↓
RMSE ↓
K.τ↑
P. r↑
de-en
10.96
11.06
0.80
0.75
10.15
10.21
0.80
0.76
en-de
5.41
5.69
0.80
0.63
5.94
6.06
0.80
0.63
en-es
6.42
7.95
0.80
0.82
6.31
7.42
0.80
0.83
en-fr
4.03
6.27
0.40
0.19
3.68
5.86
0.40
0.20
en-it
6.13
6.92
0.40
0.56
5.94
6.58
0.40
0.57
Appendix
Table 13: Results of our predictors on ranking the selected MT systems on WMT2020-Bio shared tasks.
Machine Translation (MT) and automatic MT evaluation have improved dramatically in recent years, enabling numerous novel applications. Automatic evaluation techniques have evolved from producing scalar quality scores to precisely locating translation errors and assigning them error categories and severity levels. However, it remains unclear how to reliably measure the evaluation capabilities of auto-evaluators that do error detection, as no established technique exists in the literature. This work investigates different implementations of span-level precision, recall, and F-score, showing that seemingly similar approaches can yield substantially different rankings, and that certain widely-used techniques are unsuitable for evaluating MT error detection. We propose "match with partial overlap and partial credit" (MPP) with micro-averaging as a robust meta-evaluation strategy and release code for its use publicly. Finally, we use MPP to assess the state of the art in MT error detection.
Stefano Perrella, Eric Morales Agostinho, Hugo Zaragoza
As machine translation systems approach human-level quality, traditional evaluation methodologies struggle to detect subtle translation errors. We critically examine limitations in current gold-standard approaches (MQM and ESA), including inconsistencies from variable annotator expertise, excessive categorization complexity, coarse severity granularity, accuracy bias over fluency, and time constraints. To address this issue, we introduce a high-quality dataset consisting of human evaluations for English-Russian translations from WMT24, created by professional linguists. We show that expert assessments without time pressure yield substantially different results from standard evaluations. To enable consistent and rich annotation by these experts, we developed the RATE (Refined Assessment for Translation Evaluation) protocol. RATE provides a streamlined error taxonomy, expanded severity ratings, and multidimensional scoring balancing accuracy and fluency, facilitating deeper analysis of MT outputs. Our analysis, powered by this expert dataset, reveals that state-of-the-art MT systems may have surpassed human translations in accuracy while still lagging in fluency - a critical distinction obscured by existing accuracy-biased metrics. Our findings highlight that advancing MT evaluation requires not only better protocols but crucially, high-quality annotations from skilled linguists.
Terminology evaluation in machine translation (MT) usually assumes a single correct target form per source term. However, human translators routinely introduce variation that current metrics penalize as inconsistency. We examine how to account for this variation in document-level MT evaluation of English-French scientific translation, combining glossary-based accuracy, translation consistency, and a new cross-term variation (CTV) diagnostic measure that tests whether variation relationships are preserved across languages. Based on analyses of two parallel corpora, translated by four MT systems, we find that (1) MT systems generate less target-side variation than human translators; (2) transfer patterns strongly depend on the variation type; (3) consistency rankings vary with the choice of metric; and (4) constraining MT with a glossary improves accuracy and consistency but degrades CTV by suppressing valid variation. We argue for variation-aware evaluation that conditions consistency penalties on whether target-side variation mirrors source-side variation.