When is more search useful? An Empirical Study of Search Utility in Quality-Aware Decoding for Machine Translation

Javier Garcia Gilabert; Rico Sennrich; André F. T. Martins

cs.CLOct 23, 2026

When is more search useful? An Empirical Study of Search Utility in Quality-Aware Decoding for Machine Translation

Authors: Javier Garcia Gilabert, Rico Sennrich, André F. T. Martins

Organizations: Department of Computational Linguistics, University of Zurich · Instituto Superior Técnico, Universidade de Lisboa · Instituto de Telecomunicações · ELLIS Unit Lisbon

Abstract

Quality-Aware Decoding (QAD) improves machine translation through test-time scaling, generating multiple candidate translations and selecting the final output with a quality-aware decoder. Usually the same candidate budget N is spent on every source sentence, assuming that all inputs require the same amount of search. We show that this assumption does not hold and that the quality gain obtained from additional candidates varies across sentences. We study search utility through three quantities: total scaling gain, how much quality a sentence can gain from a much larger candidate pool; marginal scaling gain, the expected gain from generating one additional batch of candidates; and the sampling saturation point, the candidate budget required to realize most of the total scaling gain. We investigate two sources of information for predicting these quantities: signals available before candidate-set expansion, and statistics computed from the candidate pool accumulated during search. We find that sentence-level signals correlate with total scaling gain but are less informative about the sampling saturation point. In contrast, properties of the current candidate pool provide useful signals of marginal scaling gain. Finally, we show that these signals can identify sentences that obtain larger gains when additional search is selectively applied.

Figures & tables

		Qwen3-8B				Qwen3-4B				Qwen3-1.7B				TowerPlus-2B
		MBR		Reranking		MBR		Reranking		MBR		Reranking		MBR		Reranking
		$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$
Source-only	PreComet diff.	.41	.26	.26	.10	.40	.28	.29	.13	.29	.30	.18	.17	.34	.28	.26	.11
	PreComet div.	.35	.28	.19	.09	.39	.30	.26	.13	.34	.29	.24	.18	.20	.25	.18	.10
	Sentinel 24	.37	.32	.21	.12	.36	.34	.25	.16	.23	.34	.13	.21	.25	.30	.21	.13
	Sentinel 25	.36	.30	.21	.10	.36	.31	.26	.15	.23	.32	.13	.18	.28	.30	.19	.11

Table 1: Spearman correlation of source- and generation-time predictors with scaling gain (

\rho_{G}

) and the sampling saturation point (

\rho_{N^{\text{.95}}}

) under MBR and reranking. Values are averaged across all language pairs.

Figure 1: Fraction of sentences that have not yet achieved

95\%

of their total scaling gain (vertical axis), as a function of candidate budget

n

(horizontal axis) for TowerPlus-2B averaged across language pairs.

Figure 2: Spearman correlation with marginal scaling gain

\Delta_{i}^{d}(n,n)

(vertical axis) as a function of candidate budget

n

(horizontal axis), averaged across language pairs.

Figure 3: Final dataset-level translation quality (vertical axis) as a function of the fraction

p

of sentences selected (horizontal axis) from

N_{\mathrm{low}}=8

N_{\mathrm{high}}=256

candidates. Sentences not selected remain at the initial budget.

Figure 4: Advantage over random selection in average marginal scaling gain (vertical axis) as a function of the fraction

p

of selected sentences (horizontal axis).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Fraction of sentences that have not yet achieved

95\%

of their total scaling gain (vertical axis), as a function of candidate budget

n

(horizontal axis).

Qwen3-8B
		en-es				en-de				en-ru
		MBR		Reranking		MBR		Reranking		MBR		Reranking
		$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$
Source-only	PreComet diff.	0.43	0.26	0.29	0.13	0.41	0.24	0.27	0.10	0.40	0.28	0.22	0.07
	PreComet div.	0.37	0.31	0.19	0.09	0.37	0.28	0.25	0.07	0.30	0.25	0.12	0.11
	Sentinel 24	0.42	0.36	0.26	0.14	0.34	0.30	0.21	0.14	0.34	0.30	0.15	0.08

Appendix

Table 2: Predictor correlations with total scaling gain

\rho_{G}

and the sampling saturation point

\rho_{N^{.95}}

across models and language pairs.

	MBR Comet-22		MBR xComet
	$\rho_{G}^{\textsc{Comet-22}}$	$\rho_{G}^{\textsc{xComet}}$	$\rho_{G}^{\textsc{Comet-22}}$	$\rho_{G}^{\textsc{xComet}}$
Sentinel 24	0.20	0.22	0.06	0.41
Sentinel 25	0.20	0.20	0.06	0.39
PreComet diff.	0.24	0.20	0.10	0.35
PreComet div.	0.31	0.27	0.19	0.41
SrcLen	0.15	0.33	0.07	0.49

Appendix

Table 3: Spearman correlation between each predictor and total scaling gain

G_{i}^{d}

for Qwen3-1.7B averaged across language pairs with

N_{\max}=128

. Columns vary the metric used as the MBR utility and the held-out metric used to compute total scaling gain. The first and fourth columns correspond to decoder–evaluator aligned settings, while the remaining columns use different metrics for decoding and evaluation.

Figure 6: Final dataset-level translation quality (vertical axis) as a function of the fraction

p

of sentences selected (horizontal axis) from

N_{\mathrm{low}}=8

N_{\mathrm{high}}=128

candidates. Sentences not selected remain at the initial budget.

Figure 7: Final dataset-level translation quality (vertical axis) as a function of the fraction

p

of sentences selected (horizontal axis) from

N_{\mathrm{low}}=8

N_{\mathrm{high}}=512

candidates. Sentences not selected remain at the initial budget.

Reranking
	en-es			en-de			en-ru
	$p=0.1$	$p=0.5$	$p=1$	$p=0.1$	$p=0.5$	$p=1$	$p=0.1$	$p=0.5$	$p=1$
Oracle	$0.08$ $\pm 0.00$	$0.03$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.08$ $\pm 0.00$	$0.03$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.10$ $\pm 0.00$	$0.04$ $\pm 0.00$	$0.01$ $\pm 0.00$
[.4pt/2pt] TopkEnt	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$
LexDiv	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$
ObjStab	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.02$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$

Appendix

Table 4: Average selected-set gain in Comet-22 for Qwen3-8B . Results are reported separately for each language pair and decoding method for expansions from

N_{\mathrm{low}}=8

N_{\mathrm{high}}=256

. Values show the mean

\pm

sample standard deviation over 20 resampling trials.

p

denotes the fraction of sentences selected for expansion.

Reranking
	en-es			en-de			en-ru
	$p=0.1$	$p=0.5$	$p=1$	$p=0.1$	$p=0.5$	$p=1$	$p=0.1$	$p=0.5$	$p=1$
Oracle	$0.10$ $\pm 0.00$	$0.04$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.12$ $\pm 0.01$	$0.05$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.13$ $\pm 0.01$	$0.06$ $\pm 0.00$	$0.02$ $\pm 0.00$
[.4pt/2pt] TopkEnt	$0.02$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.02$ $\pm 0.01$	$0.02$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.03$ $\pm 0.01$	$0.03$ $\pm 0.00$	$0.02$ $\pm 0.00$
LexDiv	$0.02$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.02$ $\pm 0.01$	$0.02$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.03$ $\pm 0.01$	$0.02$ $\pm 0.00$	$0.02$ $\pm 0.00$
ObjStab	$0.02$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.02$ $\pm 0.01$	$0.02$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.02$ $\pm 0.01$	$0.02$ $\pm 0.00$	$0.02$ $\pm 0.00$

Appendix

Table 5: Average selected-set gain in Comet-22 for Qwen3-4B . Results are reported separately for each language pair and decoding method for expansions from

N_{\mathrm{low}}=8

N_{\mathrm{high}}=256

. Values show the mean

\pm

sample standard deviation over 20 resampling trials.

p

denotes the fraction of sentences selected for expansion.

Reranking
	en-es			en-de			en-ru
	$p=0.1$	$p=0.5$	$p=1$	$p=0.1$	$p=0.5$	$p=1$	$p=0.1$	$p=0.5$	$p=1$
Oracle	$0.13$ $\pm 0.01$	$0.05$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.17$ $\pm 0.01$	$0.08$ $\pm 0.00$	$0.03$ $\pm 0.00$	$0.18$ $\pm 0.01$	$0.09$ $\pm 0.00$	$0.03$ $\pm 0.00$
[.4pt/2pt] TopkEnt	$0.04$ $\pm 0.01$	$0.03$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.04$ $\pm 0.01$	$0.04$ $\pm 0.00$	$0.03$ $\pm 0.00$	$0.05$ $\pm 0.01$	$0.04$ $\pm 0.00$	$0.03$ $\pm 0.00$
LexDiv	$0.03$ $\pm 0.01$	$0.03$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.04$ $\pm 0.01$	$0.04$ $\pm 0.00$	$0.03$ $\pm 0.00$	$0.05$ $\pm 0.01$	$0.04$ $\pm 0.00$	$0.03$ $\pm 0.00$
ObjStab	$0.03$ $\pm 0.01$	$0.02$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.03$ $\pm 0.01$	$0.04$ $\pm 0.00$	$0.03$ $\pm 0.00$	$0.03$ $\pm 0.01$	$0.03$ $\pm 0.00$	$0.03$ $\pm 0.00$

Appendix

Table 6: Average selected-set gain in Comet-22 for Qwen3-1.7B . Results are reported separately for each language pair and decoding method for expansions from

N_{\mathrm{low}}=8

N_{\mathrm{high}}=256

. Values show the mean

\pm

sample standard deviation over 20 resampling trials.

p

denotes the fraction of sentences selected for expansion.

Reranking
	en-es			en-de			en-ru
	$p=0.1$	$p=0.5$	$p=1$	$p=0.1$	$p=0.5$	$p=1$	$p=0.1$	$p=0.5$	$p=1$
Oracle	$0.09$ $\pm 0.01$	$0.03$ $\pm 0.00$	$0.00$ $\pm 0.00$	$0.09$ $\pm 0.00$	$0.03$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.11$ $\pm 0.01$	$0.04$ $\pm 0.00$	$0.01$ $\pm 0.00$
[.4pt/2pt] TopkEnt	$0.00$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.00$ $\pm 0.00$	$0.00$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.02$ $\pm 0.01$	$0.02$ $\pm 0.00$	$0.01$ $\pm 0.00$
LexDiv	$0.00$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.00$ $\pm 0.00$	$0.00$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.02$ $\pm 0.01$	$0.02$ $\pm 0.00$	$0.01$ $\pm 0.00$
ObjStab	$0.01$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.00$ $\pm 0.00$	$0.01$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.02$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$

Appendix

Table 7: Average selected-set gain in Comet-22 for TowerPlus-2B . Results are reported separately for each language pair and decoding method for expansions from

N_{\mathrm{low}}=8

N_{\mathrm{high}}=256

. Values show the mean

\pm

sample standard deviation over 20 resampling trials.

p

denotes the fraction of sentences selected for expansion.

Machine Translation

Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.

See all 267 papers

Quit While You're Ahead: Quit for Efficient Candidate Generation in Machine Translation Reranking
Sep 1, 2026Guangyu Chen, Boxuan Lyu, Hidetaka Kamigaito +2Early Stopping Machine Translation
Institute of Science Tokyo · Nara Institute of Science and Technology
CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMs
May 15, 2026Kamil Guttmann, Zofia Fraś, Artur Nowakowski +1Small Language Models MT Evaluation
Laniqo, Pozna´n, Poland · Faculty of Mathematics and Computer Science, Adam Mickiewicz University, Pozna´n, Poland
AskQE: Question Answering as Automatic Evaluation for Machine Translation
Apr 15, 2025Dayeon Ki, Kevin Duh, Marine CarpuatMT Evaluation
University of Maryland · Johns Hopkins University

Sep 1, 2026J/K move · Enter open · S save

Quit While You're Ahead: Quit for Efficient Candidate Generation in Machine Translation Reranking

Guangyu Chen, Boxuan Lyu, Hidetaka Kamigaito, Kotaro Funakoshi+1

Institute of Science Tokyo · Nara Institute of Science and Technology

Early Stopping Machine Translation

Open paper AskPDF arXiv

Reranking methods, such as Minimum Bayes Risk (MBR) decoding and Quality Estimation (QE) reranking, have been widely used in modern neural machine translation (NMT) to select an output from a set of candidate hypotheses. However, the performance gains come at the cost of high inference latency. Existing acceleration methods target MBR decoding and reduce only the reranking computation, leaving QE reranking unaddressed and candidate generation---which can be the larger computational bottleneck---largely untouched. In this work, we propose Quit (Quantifying Uncertainty for Incremental Termination), a novel early-stopping strategy for the entire generation--reranking pipeline. Quit treats candidate generation as a sequential decision-making process under uncertainty. It incrementally generates and reranks candidates, stopping when the best reranking score stabilizes. Comprehensive experiments with three NMT models across 19 language pairs show that Quit achieves end-to-end speedups of

1.47

2.66\times

for MBR decoding and

3.43

4.12\times

for QE reranking while preserving automatic metric scores.

Abstract

Machine Translation

Explore similar work

Quit While You're Ahead: Quit for Efficient Candidate Generation in Machine Translation Reranking

CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMs

AskQE: Question Answering as Automatic Evaluation for Machine Translation