Rethinking Round-Trip Translation for Machine Translation Evaluation
Authors: Terry Yue Zhuo, Qiongkai Xu, Xuanli He, Trevor Cohn
Organizations: Monash University, Clayton, VIC, Australia · The University of Melbourne, Carlton, VIC, Australia · University College London, London, United Kingdom
Automatic evaluation on low-resource language translation suffers from a deficiency of parallel corpora. Round-trip translation could be served as a clever and straightforward technique to alleviate the requirement of the parallel evaluation corpus. However, there was an observation of obscure correlations between the evaluation scores by forward and round-trip translations in the era of statistical machine translation (SMT). In this paper, we report the surprising finding that round-trip translation can be used for automatic evaluation without the references. Firstly, our revisit on the round-trip translation in SMT evaluation unveils that its long-standing misunderstanding is essentially caused by copying mechanism. After removing copying mechanism in SMT, round-trip translation scores can appropriately reflect the forward translation performance. Then, we demonstrate the rectification is overdue as round-trip translation could benefit multiple machine translation evaluation tasks. To be more specific, round-trip translation could be used i) to predict corresponding forward translation scores; ii) to improve the performance of the recently advanced quality estimation model; and iii) to identify adversarial competitors in shared tasks via cross-system verification.
Figures & tables
Figure 1: Given the corpus DA in Language A , we are able to acquire the round-trip translation (RT) results DA′ and forward translation (FT) results DB via machine translation. A question was raised by machine translation community two decades ago, “Can RT results be used to estimate FT performance?”. The answers to the question appeared to be controversial , where most researchers tend to be against the use of round-trip translation due to the obscure correlations between them. Our work gives a clear and positive answer to the usefulness of RT, based on extensive experiments.
Figure 2: The comparison of the forward translation (FT) and round-trip translation (RT) performance of two translation systems, System 1 and System 2 are based on Statistical Machine Translation (SMT) and Neural Machine Translation (NMT), respectively. The conflict conclusions by FT Scores (System 1 < System 2) and RT Scores (System 1 > System 2) are attributed to the translation of the underlined words, ‘reclassified’ and ‘Biotech’.
Lang. Pair
K. τ↑
P. r↑
w/ cp
w/o cp
w/ cp
w/o cp
de-en
-0.11
0.20
-0.90
1.00
en-de
-0.40
0.60
-1.00
1.00
cs-en
-0.20
0.30
-0.99
0.99
en-cs
-0.40
0.60
-0.90
0.99
fr-en
0.20
0.60
-1.00
1.00
Table 1: Comparison between RT-Score and FT-Score on two groups of systems with copying (w/ cp) and without copying (w/o cp) unknown words using Kendall’s τ and Pearson’s r on six language pairs.
Lang. Pair
Avg. Copy (%)
SMT
NMT
de-en
17.39
9.28
en-de
21.47
9.54
Table 2: Comparison of word copy frequency between SMT and NMT on two language pairs. We calculate average percentage of copy (Avg. Copy) per sentence. The details of selected Moses system are reported in Appendix C .
Figure 3: The 33 languages in FLORES-AE33 are separated into two categories, Seen ( ) includes the languages used in both training and testing , and Unseen ( ) is composed of the languages only used in testing . contains 7 High-resource (H.), 7 Medium-resource (M.) and 6 Low-resource (L.) languages, while involves 9 Medium-resource (M.) and 4 Low-resource (L.) languages. These two sets are used to construct three types of language pairs for test. Type I and Type III target on translation among and language pairs, respectively. Type II targets on translation between and . The test setting with more is usually more challenging, i.e., Type I < Type II < Type III .
MT System
Trans. Metric
Type I
MAE ↓
RMSE ↓
P. r↑
mBART50-m2m
BLEU
1.80
2.70
0.94
spBLEU
2.13
2.99
0.94
chrF
3.51
4.53
0.96
BERTScore
4.98
7.07
0.88
M2M-100-base
BLEU
3.86
5.82
0.95
Table 3: The results of predicted FT-Score s of mBART50-m2m , M2M-100-base and Google-Trans on Type I test set based on different translation evaluation metrics (Trans. Metric). *MAE: Mean Absolute Error, RMSE: Root Mean Square Error, P. r : Pearson’s r .
MT System
Trans. Metric
Type II
Type III
MAE ↓
RMSE ↓
P. r↑
MAE ↓
RMSE ↓
P. r↑
mBART50-m2m
BLEU
1.36
1.97
0.93
0.81
0.95
0.96
spBLEU
1.61
2.19
0.93
1.20
1.38
0.94
chrF
3.80
4.89
0.95
3.04
3.89
0.95
BERTScore
4.67
6.38
0.88
5.08
6.88
0.87
M2M-100-base
BLEU
3.10
4.16
0.95
2.99
3.76
0.94
Table 4: The results of predicted FT-Score s of mBART50-m2m (a seen MT system) and M2M-100-base (an unseen MT system) on Type II and Type III (with unseen languages) test sets based on different translation evaluation metrics (Trans. Metric).
MAE ↓
P. r↑
H.
M.
L.
H.
M.
L.
mBART50-m2m
H.
3.17
2.90
2.70
0.94
0.94
0.77
M.
1.51
1.37
1.77
0.97
0.85
0.22
L
1.22
1.27
1.16
0.97
0.87
0.78
M2M-100-base
H.
8.72
5.41
3.50
0.51
0.80
0.67
M.
4.86
4.01
2.93
0.86
0.90
0.69
Table 5: The results of predicted FT-Score s of mBART50-m2m and M2M-100-base on nine sets of language pairs, categorized by different scale of the resources, High (H.), Medium (M.) and Low (L.). The three categories in rows are source languages, and the ones in columns are target languages. We report Mean Absolute Error (MAE) and Pearson’s r .
Lang. Pair
A↻B
B↻A
A↻B & B↻A
MAE ↓
RMSE ↓
K. τ↑
P. r↑
MAE ↓
RMSE ↓
K. τ↑
P. r↑
MAE ↓
RMSE ↓
K. τ↑
P. r↑
cs-en
4.01
4.34
0.20
0.45
8.92
9.08
0.60
0.91
8.53
8.71
0.60
0.88
de-en
13.23
13.26
0.80
0.95
1.69
1.77
0.80
0.95
1.26
1.38
0.80
0.96
de-fr
10.45
10.53
1.00
0.99
1.72
2.05
0.80
0.97
1.59
1.93
1.00
0.97
en-cs
6.96
7.49
0.20
0.25
1.39
1.79
0.60
0.94
1.25
1.80
0.60
0.95
en-de
2.96
4.00
0.40
0.59
2.29
2.70
1.00
0.92
2.75
3.12
1.00
0.93
Table 6: The results of our predictors on ranking the selected MT systems on WMT2020-News shared tasks.
QE model
zh-en
en-de
K. τ↑
P. r↑
K. τ↑
P. r↑
RTT - BLEU
15.17
21.76
11.83
19.71
RTT - spBLEU
13.55
18.30
11.49
19.00
RTT - chrF
15.52
21.74
13.57
22.93
RTT - BERTScore
15.70
21.96
25.89
44.10
RTT -ALL
15.90
22.36
26.02
44.33
Table 7: Comparisons of RT-Score for QE. RTT -ALL refers to the combination of RTT - BLEU , RTT - spBLEU and RTT - chrF . COMET-QE-DA + RTT -ALL incorporates both COMET-QE-DA and all RT-Score s.
# Sys.
Method
No Adversary
One adversarial SMT
Two adversarial SMTs
K. τ↑
P. r↑
Hit@1 ↑
Avg. Rank ↓
K. τ↑
P. r↑
Hit@2 ↑
Avg. Rank ↓
K. τ↑
P. r↑
3
Sing-Check
0.07
0.17
0.50
2.00
0.33
0.51
0.00
4.75
-0.15
-0.30
X-Check
0.47
0.43
1.00
1.00
0.33
0.98
1.00
1.50
0.55
0.98
4
Sing-Check
0.33
0.37
0.25
2.75
0.40
0.39
0.00
5.75
-0.03
-0.33
X-Check
0.57
0.81
1.00
1.00
0.60
0.97
1.00
1.50
0.70
0.98
5
Sing-Check
0.48
0.58
0.25
3.25
0.30
0.25
0.00
6.75
-0.05
-0.40
Table 8: Results of the competition between 3 to 5 honest competitors, with a combination of additional adversarial competing systems (No Adversary, One adversarial SMT ( X=0.1 ) w/ copy, Two adversarial SMTs ( X=0.1 and X=0.5 ) w/ copy). We measure the identifiability of the adversarial MT systems by Hit@ K , where K is decided by the number of adversarial systems. We also report the average ranking (Avg. Rank.) of the adversarial systems, and correlation scores, Kendall’s τ and Pearson’s r .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Table 9: The statistics of FLORES-AE33 . 20 languages are used in both training and test ( ), the other 13 languages are used in test only ( ).
Figure 4: The first row is the correlations between RT-ScoreA↻BM and FT-ScoreA→BM on mBART50-m2m using (a) BLEU , (b) spBLEU , (c) chrF and (d) BERTScore . The second row is the correlations between RT-ScoreB↻AM and FT-ScoreA→BM on mBART50-m2m using (e) BLEU , (f) spBLEU , (g) chrF and (h) BERTScore . All experiments with overall Pearson’s r .
Figure 5: The first row is the correlations between RT-ScoreA↻BM and FT-ScoreA→BM on M2M-100-base using (a) BLEU , (b) spBLEU ;, (c) chrF and (d) BERTScore . The second row is the correlations between RT-ScoreB↻AM and FT-ScoreA→BM on M2M-100-base using (e) BLEU , (f) spBLEU , (g) chrF and (h) BERTScore . All experiments with overall Pearson’s r .
MT System
Comparison
BLEU
spBLEU
chrF
BERTScore
mBART50-m2m
A→B vs. A↻B
0.78
0.86
0.63
0.53
A→B vs. B↻A
0.94
0.94
0.96
0.88
M2M-100-base
A→B vs. A↻B
0.83
0.93
0.87
0.53
A→B vs. B↻A
0.95
0.96
0.96
0.90
Appendix
Table 10: Pearson’s r between \textscFT−ScoreA→BM and \textscRT−ScoreM (both A↻B and B↻A ) using different automatic evaluation metrics M on Type I test set of FLORES-AE33 .
MT System
Self-Trans Feature
Type I
Type II
Type III
MAE ↓
RMSE ↓
r↑
MAE ↓
RMSE ↓
r↑
MAE ↓
RMSE ↓
r↑
mBART50-m2m
spBLEU (basic model)
2.13
2.99
0.94
1.61
2.19
0.93
1.20
1.38
0.94
+ Max-4 Count
2.01
2.92
0.94
1.54
2.15
0.94
1.12
1.34
0.94
+ Ref Length
2.07
2.96
0.94
1.61
2.21
0.93
1.17
1.45
0.94
+ Max-4 Count & Ref Length
2.00
2.92
0.94
1.53
2.16
0.94
1.08
1.33
0.95
M2M-100-base
spBLEU (basic model)
3.97
5.72
0.96
3.24
4.18
0.96
3.18
3.88
0.95
Appendix
Table 11: The results of using auxiliary features to spBLEU for training predictors. We test the performance of mBART50-m2m and M2M-100-base cross language pairs in Type I , Type II and Type III of FLORES-AE33 .
Langauge Pair
MAE ↓
RMSE ↓
K. τ↑
P. r↑
de-fr
2.21
2.67
1.00
0.98
en-ta
0.88
0.98
1.00
0.99
zh-en
1.69
2.37
1.00
0.99
Average
1.59
2.01
1.00
0.99
Appendix
Table 12: Results of prediction and ranking on translation quality of WMT2020-News synthetic data for three language pairs.
Langauge Pair
B↻A
A↻B & B↻A
MAE ↓
RMSE ↓
K. τ↑
P. r↑
MAE ↓
RMSE ↓
K.τ↑
P. r↑
de-en
10.96
11.06
0.80
0.75
10.15
10.21
0.80
0.76
en-de
5.41
5.69
0.80
0.63
5.94
6.06
0.80
0.63
en-es
6.42
7.95
0.80
0.82
6.31
7.42
0.80
0.83
en-fr
4.03
6.27
0.40
0.19
3.68
5.86
0.40
0.20
en-it
6.13
6.92
0.40
0.56
5.94
6.58
0.40
0.57
Appendix
Table 13: Results of our predictors on ranking the selected MT systems on WMT2020-Bio shared tasks.