UvA-MT's Participation in the WMT26 General Translation Shared Task

Di Wu; Sergey Troshin; Wafaa Mohammed; Seth Aycock; Evgeniia Tokarchuk; Vlad Niculae; Christof Monz

cs.CLOct 20, 2026

UvA-MT's Participation in the WMT26 General Translation Shared Task

Authors: Di Wu, Sergey Troshin, Wafaa Mohammed, Seth Aycock, Evgeniia Tokarchuk, Vlad Niculae, Christof Monz

Organizations: Language Technology Lab, University of Amsterdam

Abstract

This paper presents UvA-MT's submission to the WMT 2026 General Machine Translation shared task, competing in the unconstrained track across all 23 translation directions. This year, we fully leverage the test-time methods of Large Language Models (LLMs) for machine translation. Specifically, (1) we use sequential sampling with a refinement prompt to generate a pool of translation candidates for each source segment, a strategy shown to outperform traditional parallel sampling; and (2) we employ LLMs as pairwise judges to select the best candidate via a round-robin voting mechanism, with offline evaluation on the WMT25 benchmark showing a very clear improvement over point-wise best-of-N selection. Combined, these two strategies form our WMT26 submission.

Figures & tables

Figure 1: An overview of our pipeline: (left) sequential sampling to generate a list of candidates, see Section 2.1 for details; (right) contrastive round-robin voting with a LLM-as-a-Judge to select the best sample. Note that, to eliminate positional bias in LLM-based pairwise judging, each pair of samples is evaluated twice with their order swapped. The final score is computed by combining the scores that a hypothesis achieves in both directions, see Section 2.2 for details.

Figure 2: Sequential + reranking demonstrates superior quality@budget trade-off, compared to the parallel sampling + reranking, and sequential sampling (self-refinement), following Wu et al. (2026) . Here we report the quality of the methods on WMT24++ with the gpt-4o-mini-2024-07-18 ; metrics aggregated over the 6 languages.

Rank	Type	System	$\tau_{a}$	$\rho$	Best@1	Best@2	Tie	Disagree
1	judge	Gemini-3.5-Flash (reasoning)	23.1	30.9	17.7	32.8	12.8	12.6
2	judge	GPT-5.4 (reasoning)	22.6	30.6	22.2	36.9	17.2	16.9
3	judge	Gemini-3.5-Flash (direct)	22.4	31.1	20.2	33.8	13.9	13.5
4	judge	GPT-5.4 (direct)	20.8	28.7	19.7	32.3	18.4	18.0
5	judge	GPT-4o-mini (direct)	16.7	25.9	21.7	33.8	31.6	31.3
6	judge	GPT-4o-mini (reasoning)	12.4	23.7	13.6	28.8	53.7	53.5

Table 1: EN-ZH results (

n=198

documents), reported as percentages.

\tau_{a}

is the per-pair-verdict Kendall, which is the coefficient comparable across the judge/metric boundary and is used for system ranking. The best value among judge models is shown in bold. For Tie and Disagree, lower is better.

Rank	Type	System	$\tau_{a}$	$\rho$	Best@1	Best@2	Tie	Disagree
1	judge	GPT-5.4 (direct)	30.8	42.5	36.0	51.5	17.2	16.8
2	judge	Gemini-3.5-Flash (reasoning)	30.8	41.7	37.0	52.0	11.9	11.6
3	judge	Gemini-3.5-Flash (direct)	30.6	41.6	36.0	52.0	10.6	10.3
4	judge	GPT-5.4 (reasoning)	29.7	40.1	35.5	54.5	16.8	16.5
5	judge	GPT-4o-mini (direct)	22.4	34.2	29.5	41.0	30.5	30.2
6	judge	GPT-4o-mini (reasoning)	19.8	33.4	28.0	43.0	43.7	43.4

Table 2: EN-RU results (

n=200

documents), reported as percentages.

\tau_{a}

is the per-pair-verdict Kendall, which is the coefficient comparable across the judge–metric boundary and is used for system ranking. The best value among judge models is shown in bold. For Tie and Disagree, lower is better.

Machine Translation

Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.

See all 267 papers

Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators
Oct 24, 2026Dmitry Popov, Roman Bokhyan, Ekaterina Enikeeva +3MT Evaluation LLM-as-a-Judge
Yandex · HSE University
Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation
Apr 27, 2026Zekun Yuan, Yangfan Ye, Xiaocheng Feng +5LLM Evaluation Cultural Knowledge in Language Models
Harbin Institute of Technology · Peng Cheng Laboratory · Huawei Technologies Co., Ltd
Reasoning about In-Context Samples for Machine-Translation
Aug 27, 2026Maxime Bouthors, Josep Crego, François YvonMachine Translation In-Context Learning
SYSTRAN by ChapsVision, 5 rue Feydeau, F-75002 Paris, France · Sorbonne Université, CNRS, ISIR, F-75005 Paris, France

Oct 24, 2026J/K move · Enter open · S save

Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators

Dmitry Popov, Roman Bokhyan, Ekaterina Enikeeva, Artem Mekhraliev+2

Yandex · HSE University

MT Evaluation LLM-as-a-Judge

Open paper AskPDF arXiv

LLM-based evaluators have rapidly become a standard tool for machine translation evaluation, yet the design choices that shape their behavior remain poorly understood. We present a systematic study of how instruction design, few-shot demonstrations, backbone model choice, and multi-step workflows affect span-level and ranking-based MT evaluation. We collect new expert RATE annotations on WMT24 and WMT25 English-Russian translations, with transfer validation on MQM- and ESA-annotated benchmarks across five additional language pairs. Our experiments reveal four robust findings: instruction structure shapes evaluator behavior, high-quality demonstrations matter more than quantity, frontier backbones exhibit distinct evaluator profiles (recall-, precision-, or ranking-oriented), and fusing outputs across diverse backbones outperforms single-evaluator multi-call setups. Although evaluator design exposes a large combinatorial space, several discovered design principles transfer consistently across annotation protocols and language pairs. We synthesize these findings into ORBIT, a family of LLM-based MT evaluators. Our strongest configurations achieve expert-level performance in error span detection and in Accuracy and Fluency ranking at roughly 20x lower cost than expert human annotation.

Abstract

Machine Translation

Explore similar work

Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators

Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation

Reasoning about In-Context Samples for Machine-Translation