When is more search useful? An Empirical Study of Search Utility in Quality-Aware Decoding for Machine Translation

Javier Garcia Gilabert; Rico Sennrich; André F. T. Martins

cs.CLOct 23, 2026

When is more search useful? An Empirical Study of Search Utility in Quality-Aware Decoding for Machine Translation

Authors: Javier Garcia Gilabert, Rico Sennrich, André F. T. Martins

Organizations: Department of Computational Linguistics, University of Zurich · Instituto Superior Técnico, Universidade de Lisboa · Instituto de Telecomunicações · ELLIS Unit Lisbon

Abstract

Quality-Aware Decoding (QAD) improves machine translation through test-time scaling, generating multiple candidate translations and selecting the final output with a quality-aware decoder. Usually the same candidate budget N is spent on every source sentence, assuming that all inputs require the same amount of search. We show that this assumption does not hold and that the quality gain obtained from additional candidates varies across sentences. We study search utility through three quantities: total scaling gain, how much quality a sentence can gain from a much larger candidate pool; marginal scaling gain, the expected gain from generating one additional batch of candidates; and the sampling saturation point, the candidate budget required to realize most of the total scaling gain. We investigate two sources of information for predicting these quantities: signals available before candidate-set expansion, and statistics computed from the candidate pool accumulated during search. We find that sentence-level signals correlate with total scaling gain but are less informative about the sampling saturation point. In contrast, properties of the current candidate pool provide useful signals of marginal scaling gain. Finally, we show that these signals can identify sentences that obtain larger gains when additional search is selectively applied.

Figures & tables

		Qwen3-8B				Qwen3-4B				Qwen3-1.7B				TowerPlus-2B
		MBR		Reranking		MBR		Reranking		MBR		Reranking		MBR		Reranking
		$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$
Source-only	PreComet diff.	.41	.26	.26	.10	.40	.28	.29	.13	.29	.30	.18	.17	.34	.28	.26	.11
	PreComet div.	.35	.28	.19	.09	.39	.30	.26	.13	.34	.29	.24	.18	.20	.25	.18	.10
	Sentinel 24	.37	.32	.21	.12	.36	.34	.25	.16	.23	.34	.13	.21	.25	.30	.21	.13
	Sentinel 25	.36	.30	.21	.10	.36	.31	.26	.15	.23	.32	.13	.18	.28	.30	.19	.11

Table 1: Spearman correlation of source- and generation-time predictors with scaling gain (

\rho_{G}

) and the sampling saturation point (

\rho_{N^{\text{.95}}}

) under MBR and reranking. Values are averaged across all language pairs.

Figure 1: Fraction of sentences that have not yet achieved

95\%

of their total scaling gain (vertical axis), as a function of candidate budget

n

(horizontal axis) for TowerPlus-2B averaged across language pairs.

Figure 2: Spearman correlation with marginal scaling gain

\Delta_{i}^{d}(n,n)

(vertical axis) as a function of candidate budget

n

(horizontal axis), averaged across language pairs.

Figure 3: Final dataset-level translation quality (vertical axis) as a function of the fraction

p

of sentences selected (horizontal axis) from

N_{\mathrm{low}}=8

N_{\mathrm{high}}=256

candidates. Sentences not selected remain at the initial budget.

Figure 4: Advantage over random selection in average marginal scaling gain (vertical axis) as a function of the fraction

p

of selected sentences (horizontal axis).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Fraction of sentences that have not yet achieved

95\%

of their total scaling gain (vertical axis), as a function of candidate budget

n

(horizontal axis).

Qwen3-8B
		en-es				en-de				en-ru
		MBR		Reranking		MBR		Reranking		MBR		Reranking
		$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$	$\rho_{G}$	$\rho_{N^{\text{.95}}}$
Source-only	PreComet diff.	0.43	0.26	0.29	0.13	0.41	0.24	0.27	0.10	0.40	0.28	0.22	0.07
	PreComet div.	0.37	0.31	0.19	0.09	0.37	0.28	0.25	0.07	0.30	0.25	0.12	0.11
	Sentinel 24	0.42	0.36	0.26	0.14	0.34	0.30	0.21	0.14	0.34	0.30	0.15	0.08

Appendix

Table 2: Predictor correlations with total scaling gain

\rho_{G}

and the sampling saturation point

\rho_{N^{.95}}

across models and language pairs.

	MBR Comet-22		MBR xComet
	$\rho_{G}^{\textsc{Comet-22}}$	$\rho_{G}^{\textsc{xComet}}$	$\rho_{G}^{\textsc{Comet-22}}$	$\rho_{G}^{\textsc{xComet}}$
Sentinel 24	0.20	0.22	0.06	0.41
Sentinel 25	0.20	0.20	0.06	0.39
PreComet diff.	0.24	0.20	0.10	0.35
PreComet div.	0.31	0.27	0.19	0.41
SrcLen	0.15	0.33	0.07	0.49

Appendix

Table 3: Spearman correlation between each predictor and total scaling gain

G_{i}^{d}

for Qwen3-1.7B averaged across language pairs with

N_{\max}=128

. Columns vary the metric used as the MBR utility and the held-out metric used to compute total scaling gain. The first and fourth columns correspond to decoder–evaluator aligned settings, while the remaining columns use different metrics for decoding and evaluation.

Figure 6: Final dataset-level translation quality (vertical axis) as a function of the fraction

p

of sentences selected (horizontal axis) from

N_{\mathrm{low}}=8

N_{\mathrm{high}}=128

candidates. Sentences not selected remain at the initial budget.

Figure 7: Final dataset-level translation quality (vertical axis) as a function of the fraction

p

of sentences selected (horizontal axis) from

N_{\mathrm{low}}=8

N_{\mathrm{high}}=512

candidates. Sentences not selected remain at the initial budget.

Reranking
	en-es			en-de			en-ru
	$p=0.1$	$p=0.5$	$p=1$	$p=0.1$	$p=0.5$	$p=1$	$p=0.1$	$p=0.5$	$p=1$
Oracle	$0.08$ $\pm 0.00$	$0.03$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.08$ $\pm 0.00$	$0.03$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.10$ $\pm 0.00$	$0.04$ $\pm 0.00$	$0.01$ $\pm 0.00$
[.4pt/2pt] TopkEnt	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$
LexDiv	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$
ObjStab	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.02$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$

Appendix

Table 4: Average selected-set gain in Comet-22 for Qwen3-8B . Results are reported separately for each language pair and decoding method for expansions from

N_{\mathrm{low}}=8

N_{\mathrm{high}}=256

. Values show the mean

\pm

sample standard deviation over 20 resampling trials.

p

denotes the fraction of sentences selected for expansion.

Reranking
	en-es			en-de			en-ru
	$p=0.1$	$p=0.5$	$p=1$	$p=0.1$	$p=0.5$	$p=1$	$p=0.1$	$p=0.5$	$p=1$
Oracle	$0.10$ $\pm 0.00$	$0.04$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.12$ $\pm 0.01$	$0.05$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.13$ $\pm 0.01$	$0.06$ $\pm 0.00$	$0.02$ $\pm 0.00$
[.4pt/2pt] TopkEnt	$0.02$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.02$ $\pm 0.01$	$0.02$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.03$ $\pm 0.01$	$0.03$ $\pm 0.00$	$0.02$ $\pm 0.00$
LexDiv	$0.02$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.02$ $\pm 0.01$	$0.02$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.03$ $\pm 0.01$	$0.02$ $\pm 0.00$	$0.02$ $\pm 0.00$
ObjStab	$0.02$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.02$ $\pm 0.01$	$0.02$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.02$ $\pm 0.01$	$0.02$ $\pm 0.00$	$0.02$ $\pm 0.00$

Appendix

Table 5: Average selected-set gain in Comet-22 for Qwen3-4B . Results are reported separately for each language pair and decoding method for expansions from

N_{\mathrm{low}}=8

N_{\mathrm{high}}=256

. Values show the mean

\pm

sample standard deviation over 20 resampling trials.

p

denotes the fraction of sentences selected for expansion.

Reranking
	en-es			en-de			en-ru
	$p=0.1$	$p=0.5$	$p=1$	$p=0.1$	$p=0.5$	$p=1$	$p=0.1$	$p=0.5$	$p=1$
Oracle	$0.13$ $\pm 0.01$	$0.05$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.17$ $\pm 0.01$	$0.08$ $\pm 0.00$	$0.03$ $\pm 0.00$	$0.18$ $\pm 0.01$	$0.09$ $\pm 0.00$	$0.03$ $\pm 0.00$
[.4pt/2pt] TopkEnt	$0.04$ $\pm 0.01$	$0.03$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.04$ $\pm 0.01$	$0.04$ $\pm 0.00$	$0.03$ $\pm 0.00$	$0.05$ $\pm 0.01$	$0.04$ $\pm 0.00$	$0.03$ $\pm 0.00$
LexDiv	$0.03$ $\pm 0.01$	$0.03$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.04$ $\pm 0.01$	$0.04$ $\pm 0.00$	$0.03$ $\pm 0.00$	$0.05$ $\pm 0.01$	$0.04$ $\pm 0.00$	$0.03$ $\pm 0.00$
ObjStab	$0.03$ $\pm 0.01$	$0.02$ $\pm 0.00$	$0.02$ $\pm 0.00$	$0.03$ $\pm 0.01$	$0.04$ $\pm 0.00$	$0.03$ $\pm 0.00$	$0.03$ $\pm 0.01$	$0.03$ $\pm 0.00$	$0.03$ $\pm 0.00$

Appendix

Table 6: Average selected-set gain in Comet-22 for Qwen3-1.7B . Results are reported separately for each language pair and decoding method for expansions from

N_{\mathrm{low}}=8

N_{\mathrm{high}}=256

. Values show the mean

\pm

sample standard deviation over 20 resampling trials.

p

denotes the fraction of sentences selected for expansion.

Reranking
	en-es			en-de			en-ru
	$p=0.1$	$p=0.5$	$p=1$	$p=0.1$	$p=0.5$	$p=1$	$p=0.1$	$p=0.5$	$p=1$
Oracle	$0.09$ $\pm 0.01$	$0.03$ $\pm 0.00$	$0.00$ $\pm 0.00$	$0.09$ $\pm 0.00$	$0.03$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.11$ $\pm 0.01$	$0.04$ $\pm 0.00$	$0.01$ $\pm 0.00$
[.4pt/2pt] TopkEnt	$0.00$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.00$ $\pm 0.00$	$0.00$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.02$ $\pm 0.01$	$0.02$ $\pm 0.00$	$0.01$ $\pm 0.00$
LexDiv	$0.00$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.00$ $\pm 0.00$	$0.00$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.02$ $\pm 0.01$	$0.02$ $\pm 0.00$	$0.01$ $\pm 0.00$
ObjStab	$0.01$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.00$ $\pm 0.00$	$0.01$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$	$0.02$ $\pm 0.01$	$0.01$ $\pm 0.00$	$0.01$ $\pm 0.00$

Appendix

Table 7: Average selected-set gain in Comet-22 for TowerPlus-2B . Results are reported separately for each language pair and decoding method for expansions from

N_{\mathrm{low}}=8

N_{\mathrm{high}}=256

. Values show the mean

\pm

sample standard deviation over 20 resampling trials.

p

denotes the fraction of sentences selected for expansion.

Machine Translation

Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.

See all 267 papers

Sep 1, 2026

Quit While You're Ahead: Quit for Efficient Candidate Generation in Machine Translation Reranking

Reranking methods, such as Minimum Bayes Risk (MBR) decoding and Quality Estimation (QE) reranking, have been widely used in modern neural machine translation (NMT) to select an output from a set of candidate hypotheses. However, the performance gains come at the cost of high inference latency. Existing acceleration methods target MBR decoding and reduce only the reranking computation, leaving QE reranking unaddressed and candidate generation---which can be the larger computational bottleneck---largely untouched. In this work, we propose Quit (Quantifying Uncertainty for Incremental Termination), a novel early-stopping strategy for the entire generation--reranking pipeline. Quit treats candidate generation as a sequential decision-making process under uncertainty. It incrementally generates and reranks candidates, stopping when the best reranking score stabilizes. Comprehensive experiments with three NMT models across 19 language pairs show that Quit achieves end-to-end speedups of

1.47

2.66\times

for MBR decoding and

3.43

4.12\times

for QE reranking while preserving automatic metric scores.

Guangyu Chen, Boxuan Lyu, Hidetaka Kamigaito +2

Institute of Science Tokyo · Nara Institute of Science and Technology

Early Stopping Machine Translation

Ask PDF

May 15, 2026

CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMs

Current state-of-the-art Quality Estimation (QE) in machine translation relies on massive, proprietary LLMs, raising data privacy concerns. We demonstrate that smaller, open-source LLMs (<30B parameters) are a viable, cost-effective and privacy-preserving alternative. Using a single-pass prompting strategy, our models simultaneously generate quality scores, MQM error annotations, suggested error corrections, and full post-editions. Our analysis shows these models achieve highly competitive system-level correlations with human judgments that outperform traditional neural metrics, fine-tuned models, and human inter-annotator agreement, effectively approximating the capabilities of much larger proprietary LLMs.

Kamil Guttmann, Zofia Fraś, Artur Nowakowski +1

Laniqo, Pozna´n, Poland · Faculty of Mathematics and Computer Science, Adam Mickiewicz University, Pozna´n, Poland

Small Language Models MT Evaluation

Ask PDF

Apr 15, 2025

AskQE: Question Answering as Automatic Evaluation for Machine Translation

How can a monolingual English speaker determine whether an automatic translation in French is good enough to be shared? Existing MT error detection and quality estimation (QE) techniques do not address this practical scenario. We introduce AskQE, a question generation and answering framework designed to detect critical MT errors and provide actionable feedback, helping users decide whether to accept or reject MT outputs even without the knowledge of the target language. Using ContraTICO, a dataset of contrastive synthetic MT errors in the COVID-19 domain, we explore design choices for AskQE and develop an optimized version relying on LLaMA-3 70B and entailed facts to guide question generation. We evaluate the resulting system on the BioMQM dataset of naturally occurring MT errors, where AskQE has higher Kendall's Tau correlation and decision accuracy with human ratings compared to other QE metrics.

Dayeon Ki, Kevin Duh, Marine Carpuat

University of Maryland · Johns Hopkins University

MT Evaluation

Ask PDF

Abstract

Machine Translation

Explore similar work

Quit While You're Ahead: Quit for Efficient Candidate Generation in Machine Translation Reranking

CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMs

AskQE: Question Answering as Automatic Evaluation for Machine Translation