Minimum Bayes Risk (MBR) decoding is a text generation technique that has been shown to improve the quality of machine translations, but is expensive, even if a sampling-based approximation is used. Besides requiring a large number of sampled sequences, it requires the pairwise calculation of a utility metric, which has quadratic complexity. In this paper, we propose to approximate pairwise metric scores with scores calculated against aggregated reference representations. This changes the complexity of utility estimation from O(n2) to O(n), while empirically preserving most of the quality gains of MBR decoding. We release our source code at https://github.com/ZurichNLP/mbr
Figures & tables
Figure 1: How accurately do MBR efficiency methods approximate standard MBR? In this validation experiment on newstest21 , we gradually increase efficiency by using fewer references for pairwise utility estimation – either by subsampling the references (N-by-S; Eikema and Aziz, 2022 ) or by aggregating their representations using partial aggregation (Section 3.3 ). We report top-20 accuracy, which describes how often an efficiency method ranks the correct hypothesis (as selected by standard MBR) among the top 20 hypotheses. An efficiency method with a high top-20 accuracy could be used for pruning the number of hypotheses to 20 before standard MBR is applied.
en–de
de–en
en–ru
ru–en
Avg.
Time (utility / total)
Beam search (size 5)
76.16
72.56
68.50
75.47
73.17
- / 0 0.2 s
Epsilon sampling ( ϵ=0.02 )
73.39
69.70
65.79
72.13
70.25
- / 0 0.2 s
MBR with ChrF metric
–standard MBR
76.03
72.73
69.52
75.51
73.44
0 0.4 s / 0 5.2 s
–reference aggregation
75.95
72.79
69.46
75.45
73.41
0 0.1 s / 0 4.9 s
–aggregate-to-fine MBR
76.02
72.80
69.54
75.47
73.46
0 0.2 s / 0 5.0 s
Table 1: Test results on newstest22 , using BLEURT-20 for automatic evaluation. We use 1024 samples/references for MBR. In the last column, we report the average time needed for translating a segment, measuring (a) the time needed for utility estimation only, and (b) the total, end-to-end time needed for translation. Underline : no significant BLEURT difference to standard MBR; bold : significantly better than standard MBR (bootstrap test, p<0.05 ).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
# Segments
# Samples per segment
# Unique samples per segment
newstest21 en–de
1002
1024
874.2
newstest21 de–en
1000
1024
716.9
newstest21 en–ru
1002
1024
896.7
newstest21 ru–en
1000
1024
727.3
newstest22 en–de
2037
1024
697.5
newstest22 de–en
1984
1024
671.4
Appendix
Table 2: Statistics for the datasets used in this paper. We sample 1024 hypotheses per source segment using epsilon sampling and find that most of the samples are unique.
ChrF
Cometinho
COMET -22
xCOMET-XL
BLEURT-20
Beam search (size 5)
58.6
56.0
84.3
92.2
73.2
Epsilon sampling ( ϵ=0.02 )
52.6
45.3
81.9
89.4
70.3
MBR with ChrF metric
–standard MBR
59.8
58.3
84.5
91.8
73.4
–reference aggregation
59.8
58.2
84.5
91.7
73.4
–aggregate-to-fine MBR
59.8
58.3
84.5
91.8
73.5
Appendix
Table 3: Extended results on newstest22 with 1024 samples/references for MBR. In this table, we include Cometinho Rei et al. (2022b) as utility metric, which is a distilled COMET model. Furthermore, as an additional evaluation metric, we report xCOMET-XL Guerreiro et al. (2023) . We average the evaluation scores across the four translation directions. Underline : no significant difference to standard MBR; bold : significantly better than standard MBR (bootstrap test, p<0.05 ).
en–de
de–en
en–ru
ru–en
Avg.
Time (utility / total)
Beam search (size 5)
76.16
72.56
68.50
75.47
73.17
- / 0 0.2 s
Epsilon sampling ( ϵ=0.02 )
73.39
69.70
65.79
72.13
70.25
- / 0 0.2 s
MBR with ChrF metric
–standard MBR
75.90
72.66
69.27
75.60
73.36
0 0.1 s / 0 1.4 s
–reference aggregation
75.83
72.69
69.19
75.53
73.31
<0.1 s / 0 1.3 s
–aggregate-to-fine MBR
75.90
72.67
69.29
75.58
73.36
<0.1 s / 0 1.3 s
Appendix
Table 4: Version of Table 1 that uses 256 samples/references for MBR.
Beam size
en–de
de–en
en–ru
ru–en
Avg.
5
76.16
72.56
68.50
75.47
73.17
10
76.20
72.57
67.92
75.51
73.05
15
76.19
72.53
68.10
75.48
73.08
20
76.18
72.54
67.84
75.49
73.01
25
76.19
72.50
67.82
75.46
72.99
Appendix
Table 5: Increasing the beam size to values larger than 5 does not tend to improve translation quality of beam search on newstest22 in terms of BLEURT-20 .
Figure 2: Top-1 accuracy of MBR efficiency methods on newstest21 , analogous to Figure 1 .
Figure 3: Accuracy of MBR efficiency methods on newstest21 when using the Cometinho model Rei et al. (2022b) as utility metric.
Minimum Bayes Risk (MBR) decoding yields more robust and higher-quality text generation than maximum a posteriori (MAP) decoding by selecting hypotheses that maximize expected utility over sampled pseudo-references. However, there exists a discrepancy in the design: hypothesis selection calculates expected utility scores conditioned on given pseudo-references, while commonly used evaluation metrics, e.g., BLEU and COMET, are asymmetric. Therefore, it is important to consider both hypothesis-to-reference and reference-to-hypothesis directional effects. In this study, we introduce a noisy channel decomposition of MBR decoding that naturally incorporates bidirectional effects to account for these asymmetries. We decompose MBR decoding into four interacting components: hypothesis-to-reference likelihood, reference-to-hypothesis likelihood, hypothesis prior, and reference prior. This decomposition provides a unified interpretation of existing MBR variants and enables metric- and task-specific interpretability by isolating the contribution of each channel. Our comprehensive analysis reveals that channel-wise contributions exhibit distinct characteristics across metrics while remaining consistent across tasks, and suggests that appropriate channel weighting may lead to improvements over original MBR decoding.
Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe
Nara Institute of Science and Technology, Nara, Japan.
Minimum Bayes Risk (MBR) decoding enables high-quality text generation by selecting the hypothesis that maximizes a utility metric over sampled pseudo-references. However, it is highly susceptible to metric overfitting: it can irregularly inflate the chosen utility metric at the direct expense of other unoptimized evaluation metrics. To mitigate this, we introduce SVD-MBR, which frames the pairwise utility matrix as a noisy information signal. By computing a low-rank approximation via Singular Value Decomposition (SVD) and retaining only the top-k components, we effectively decouple true consensus from metric noise. Experiments demonstrate that SVD-MBR successfully regularizes decoding, yielding substantial gains across a range of generalized metrics. Furthermore, we reveal that this denoising is metric-dependent: neural metrics encode a robust low-rank consensus ideal for SVD, whereas surface-level metrics struggle to separate signal from metric noise.
Riza Setiawan Soetedjo, Yusuke Sakai, Hidetaka Kamigaito +2
Nara Institute of Science and Technology (NAIST) · The University of Tokyo
Non-autoregressive (NAR) decoding generates output tokens in parallel, making speech recognition faster than autoregressive decoding, which generates them sequentially from left to right. However, the recognition performance is degraded because NAR decoding cannot resolve uncertainty by conditioning on previously generated tokens. To address this issue, we propose a novel NAR decoding framework based on minimum Bayes' risk (MBR) decoding, termed NAR-MBR decoding, that maximizes the expected utility calculated from samples drawn from the output probability of an NAR model rather than maximizing the output probability. Notably, by leveraging the nature of NAR models, multiple samples are obtained efficiently with a single forward computation. Our experiments across LibriSpeech, Switchboard, AMI, and web presentation corpus demonstrated that our NAR-MBR decoding outperformed previous NAR decoding and ran faster than AR decoding.