Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators

Dmitry Popov; Roman Bokhyan; Ekaterina Enikeeva; Artem Mekhraliev; Stepan Vysotsky; Nikolay Karpachev

cs.CLOct 24, 2026

Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators

Authors: Dmitry Popov, Roman Bokhyan, Ekaterina Enikeeva, Artem Mekhraliev, Stepan Vysotsky, Nikolay Karpachev

Organizations: Yandex · HSE University

Abstract

LLM-based evaluators have rapidly become a standard tool for machine translation evaluation, yet the design choices that shape their behavior remain poorly understood. We present a systematic study of how instruction design, few-shot demonstrations, backbone model choice, and multi-step workflows affect span-level and ranking-based MT evaluation. We collect new expert RATE annotations on WMT24 and WMT25 English-Russian translations, with transfer validation on MQM- and ESA-annotated benchmarks across five additional language pairs. Our experiments reveal four robust findings: instruction structure shapes evaluator behavior, high-quality demonstrations matter more than quantity, frontier backbones exhibit distinct evaluator profiles (recall-, precision-, or ranking-oriented), and fusing outputs across diverse backbones outperforms single-evaluator multi-call setups. Although evaluator design exposes a large combinatorial space, several discovered design principles transfer consistently across annotation protocols and language pairs. We synthesize these findings into ORBIT, a family of LLM-based MT evaluators. Our strongest configurations achieve expert-level performance in error span detection and in Accuracy and Fluency ranking at roughly 20x lower cost than expert human annotation.

Figures & tables

	Span-level					Ranking
System	R	P	F1	R 4+	P 4+	MQM
Human E	46.8	44.3	45.5	65.3	65.3	75.3
Human A-SbS	39.3	47.1	42.8	55.7	63.2	68.0
Human A-PW	31.8	46.4	37.7	49.8	57.1	65.0
Human WMT	12.3	61.9	20.6	26.6	62.8	42.7
GEMBA-MQM	24.6	43.0	31.3	45.0	51.0	58.7

Table 1: Overview of human baselines, automatic baselines, and our configurations. Prompt-based baselines and Orbit-SC use GPT-5.4. Orbit-MC is a multi-call setup that fuses three evaluator backbones with an LLM fuser.

	Span-level					Ranking
System	R	P	F1	R 4+	P 4+	MQM	Acc	FL
Human E	46.8	44.3	45.5	65.3	65.3	75.3	67.3	70.0
GEMBA-MQM	19.9	42.1	27.0	39.6	51.1	60.7	–	–
GEMBA-ESA	20.8	39.6	27.3	39.3	53.0	60.7	–	–
MQM-AE	22.2	53.8	31.4	42.9	64.4	63.0	–	–
ESA-AE	26.3	54.5	35.5	47.5	61.9	59.3	–	–

Table 2: Effect of instruction design (zero-shot, GPT-5.2 backbone). GEMBA systems use their original few-shot demonstrations. All other systems are zero-shot.

	Span-level			Ranking
Few-shot source	F1	R 4+	P 4+	MQM	Acc	FL
RATE-AE full+ex (0-shot)	39.3	52.9	66.3	56.7	60.3	65.3
WMT	37.0 ${}_{[\text{36.4},\text{37.7}]}$	50.1 ${}_{[\text{49.8},\text{50.4}]}$	69.6 ${}_{[\text{69.2},\text{70.0}]}$	61.1 ${}_{[\text{60.7},\text{62.0}]}$	64.5 ${}_{[\text{63.2},\text{66.0}]}$	61.5 ${}_{[\text{60.7},\text{62.7}]}$
RATE-WMT24	38.8 ${}_{[\text{37.5},\text{40.3}]}$	52.8 ${}_{[\text{51.0},\text{55.1}]}$	67.9 ${}_{[\text{66.3},\text{70.3}]}$	61.6 ${}_{[\text{61.0},\text{62.3}]}$	64.1 ${}_{[\text{63.0},\text{65.3}]}$	63.9 ${}_{[\text{63.7},\text{64.0}]}$
Ours	41.2 ${}_{[\text{40.9},\text{41.7}]}$	56.8 ${}_{[\text{56.0},\text{57.5}]}$	68.1 ${}_{[\text{67.8},\text{68.7}]}$	62.4 ${}_{[\text{59.0},\text{67.3}]}$	63.4 ${}_{[\text{63.0},\text{64.3}]}$	65.3 ${}_{[\text{64.3},\text{67.0}]}$

Table 3: Effect of few-shot annotation quality. Each value is the mean over 3 runs. Brackets show the corresponding minimum and maximum.

Figure 1: Effect of few-shot example count (

k

) across span-level and ranking metrics. Each condition uses 5 random draws. Boxes show IQR, points show individual runs.

Coverage-based selection (varying $k$ ), with RATE-AE full+ex
	Span-level					Ranking
Few-shot config	R	P	F1	R 4+	P 4+	MQM	Acc	FL
RATE-AE full+ex (0-shot)	30.9	53.8	39.3	52.9	66.3	56.7	60.3	65.3
$k{=}2$	39.8	51.1	44.8	63.4	67.8	64.0	62.7	69.3
$k{=}3$	35.8	53.4	42.9	59.8	66.5	60.3	62.6	64.6
$k{=}4$	37.8	53.2	44.2	61.6	68.1	63.7	65.7	65.3

Table 4: Effect of coverage-based few-shot selection. Upper block: varying

k

with RATE-AE full+ex . Lower block:

k{=}2

with varying instruction.

	Span-level					Ranking
Model	R	P	F1	R 4+	P 4+	MQM	Acc	FL
Human E	46.8	44.3	45.5	65.3	65.3	75.3	67.3	70.0
GPT-4.1	51.7	45.6	48.5	72.7	61.7	61.0	59.0	57.0
GPT-5.2	42.4	49.9	45.9	64.7	65.4	65.0	62.3	66.0
GPT-5.4	51.5	46.2	48.7	74.7	63.5	67.0	65.0	65.0
Claude Sonnet 4.6	45.8	45.6	45.7	67.1	63.5	64.0	59.5	62.9

Table 5: Model comparison using Orbit-SC across frontier models.

		Span-level					Ranking
Backbone	Format	R	P	F1	R 4+	P 4+	MQM	Acc	FL
GPT-5.4	PW	51.5	46.2	48.7	74.7	63.5	67.0	65.0	65.0
	SbS	48.7	48.8	48.8	73.5	64.2	68.3	66.7	70.3
Claude Opus 4.6	PW	47.8	46.9	47.3	72.2	68.3	65.7	68.3	70.7
	SbS	42.1	50.5	45.9	66.8	71.9	65.7	71.0	73.3
Gemini 3 Flash	PW	38.9	49.1	43.4	62.2	66.8	68.0	59.3	67.0

Table 6: Point-wise (PW) Orbit-SC and its side-by-side (SbS) adaptation.

Single-call
	Span-level					Ranking
System	R	P	F1	R 4+	P 4+	MQM	Acc	FL
Human E	46.8	44.3	45.5	65.3	65.3	75.3	67.3	70.0
Orbit-SC (GPT-5.4)	51.5	46.2	48.7	74.7	63.5	67.0	65.0	65.0
Multi-call: dimension split
Sem/non-sem concatenated	54.8	37.6	44.6	76.9	52.0	69.7	64.1	63.7

Table 7: Multi-call strategies. C = Claude Opus 4.6, G = GPT-5.4, Gem = Gemini 3.1 Pro. Fused = LLM-merged report.

Evaluator setup	Format	USD/10k seg.
Human, one annotator pass
Professional expert RATE	SbS	$48,920
Assessor RATE	SbS	$6,500
Assessor RATE	PW	$6,865
ORBIT-SC, varying backbone and format
Gemini 3 Flash	PW	$20

Table 8: API cost and estimated cost of one human annotator pass per 10,000 evaluated segments.

	en-de MQM					zh-en MQM					cs-de ESA					en-it ESA					en-zh ESA
System	R	P	F1	R 4+	MQM	R	P	F1	R 4+	MQM	R	P	F1	R 4+	MQM	R	P	F1	R 4+	MQM	R	P	F1	R 4+	MQM
Human 2	31.8	40.7	35.7	48.7	60.7	44.5	41.5	42.9	63.2	68.7	21.8	36.0	27.2	29.3	45.3	25.5	20.3	22.6	37.2	48.0	15.5	18.1	16.7	17.8	41.3
GEMBA-MQM	29.9	35.5	32.5	43.2	60.0	29.3	44.3	35.2	46.6	63.8	17.0	22.4	19.3	23.8	46.7	17.8	10.5	13.2	29.8	43.3	33.2	12.8	18.5	43.9	50.7
MQM-AE	30.1	45.5	36.2	46.2	56.9	31.6	51.7	39.3	52.0	58.0	29.1	29.5	29.3	38.4	32.0	43.8	14.7	22.0	60.3	50.0	38.8	14.4	21.0	46.7	42.0
ESA-AE	29.8	46.8	36.4	45.2	59.8	33.2	60.6	42.9	55.7	62.2	37.7	27.9	32.1	45.4	52.0	45.1	15.1	22.7	63.6	46.0	42.1	14.6	21.7	46.7	36.0
UMB-v5	32.5	42.6	36.9	48.0	61.3	37.7	49.1	42.6	58.3	63.1	23.5	28.9	25.9	32.7	50.0	39.9	13.6	20.3	60.3	50.0	35.5	13.1	19.1	44.9	43.3

Table 9: Multilingual transfer on WMT23 MQM (en-de, zh-en) and WMT25 ESA (cs-de, en-it, en-zh) data. DS = direction-specific demonstrations: two few-shot examples in the target translation direction.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Segments
Literary	166
News	146
Speech	146
Social	142
Total	600

Appendix

Table 10: Domain distribution of source–translation pairs.

Translation system	Segments
Gemini 2.5	69
Claude-3.5	55
Dubformer	55
GPT-4	53
ONLINE-G	53
Unbabel-Tower70B	53

Appendix

Table 11: Translation system distribution.

Severity	Combined	Annotator A	Annotator B
1 (trivial)	0.65	0.37	0.29
2 (minor)	4.16	2.16	2.00
3 (minor)	3.60	1.83	1.77
4 (major)	1.71	0.85	0.86
5 (critical)	0.53	0.28	0.25
Total	10.64	5.48	5.22

Appendix

Table 12: Mean error spans per segment by severity. “Combined” aggregates two independent expert annotations. Per-annotator columns show individual outputs.

Category	Spans / segment
Fluency	6.06
Mistranslation	2.34
Undertranslation	0.61
Overtranslation	0.44
Grammar	0.32
Other	0.31

Appendix

Table 13: Mean error spans per segment by RATE category, aggregated across the two expert gold annotations.

	$R$			$P$			$F_{1}$
System	MP	MPP	LLM	MP	MPP	LLM	MP	MPP	LLM
Human E	46.8	41.9	54.1	44.3	38.6	52.1	45.5	40.2	53.1
Human A-SbS	39.3	34.8	47.6	47.1	40.8	55.2	42.8	37.5	51.1
Orbit-SC GPT-5.4	51.5	49.9	55.7	46.2	26.7	51.2	48.7	34.8	53.4
Orbit-SC Claude	47.8	45.9	57.1	46.9	30.6	51.4	47.3	36.8	54.1
Orbit-SC Gemini	45.0	42.8	47.6	51.2	36.9	56.7	47.9	39.6	51.7

Appendix

Table 14: Span-level metrics under three span-matching formulations: MP with

\tau{=}1

(primary), MPP ( Perrella et al., 2026 ) (partial overlap with partial credit), and an LLM reconciliation proxy using GPT-5.4. All metrics are micro-averaged.

	Spans / segment			Severity-restricted metrics
Model	Non-trivial	Major	Critical	R 4+	P 4+	$R_{4+}^{\bullet}$	$P_{4+}^{\bullet}$
Human E	5.28	1.01	0.28	65.3	65.3	54.0	52.5
GPT-4.1	5.67	0.93	0.28	72.7	61.7	53.1	56.1
GPT-5.2	4.26	0.90	0.27	64.8	67.5	45.4	59.1
GPT-5.4	5.57	1.55	0.49	74.7	63.5	61.3	55.7
Claude Sonnet 4.6	5.02	1.10	0.33	67.1	63.5	56.8	53.7

Appendix

Table 15: Extended model comparison using Orbit-SC (RATE-AE brief with coverage-based

k{=}2

demonstrations). Non-trivial, Major, and Critical report mean predicted spans per segment at severity

\geq 2

\geq 4

, and

=5

respectively.

R_{4+}

measures recovery of gold major spans regardless of predicted severity, while

P_{4+}

uses predictions assigned severity

\geq 4

. The bullet variants use predicted severity

\geq 3

, providing one-level tolerance around the major threshold.

Table 16: Matched design comparison across two backbones. Orbit-SC uses RATE-AE brief with curated coverage (

k{=}2

	Spans / segment			Mean scores
Annotator	All	Major	Crit.	Acc	FL
Human E1 (baseline)	5.60	1.00	0.28	76.6	74.8
Human E1 + post-hint	7.04	1.11	0.30	72.3	72.1
$\Delta$ within-subject	+1.44	+0.11	+0.02	$-$ 4.3	$-$ 2.7
Human E2 + pre-hint	6.00	1.03	0.26	74.1	73.0
Orbit-MC (raw hint)	8.09	0.78	0.24	60.2	59.7

Appendix

Table 17: Effect of Orbit-MC hints on expert annotation (held-out 600-segment set).

MT Evaluation

Machine translation evaluation measures translation quality, error types, and real-world suitability across languages, domains, and modalities.

See all 173 papers

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
Oct 7, 2026Hala Almaghout, Christian Federmann, Qin GaoMT Evaluation LLM Reliability
Apple
Diagnose, Then Repair: A Two-Stage MQM-Guided Post-Editing Framework for Domain-Specific Machine Translation
Sep 19, 2026Ji Hun Wang, Siyu WuMachine Translation MT Evaluation
Amazon
Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation
May 24, 2026Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber +1Multilingual Language Model Evaluation MT Evaluation
TUD Dresden University of Technology · 2InfAI e.V. · 3Amazon +1

Oct 7, 2026J/K move · Enter open · S save

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

Hala Almaghout, Christian Federmann, Qin Gao

Apple

MT Evaluation LLM Reliability LLM-Assisted Annotation Multilingual Language Model Evaluation

Open paper AskPDF arXiv

Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human annotators. We present results on a long-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs and domains. Our results show that while LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks, both vary substantially across annotation schemes, language pairs and domains and remain unreliable for most settings. Furthermore, we identify challenges facing both human and LLM annotators: humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. Our results highlight the potential for improvement for both human and LLM annotation performance, possibly through human-LLM collaborative annotation pipelines that address the reliability issues identified in this work.

Abstract

MT Evaluation

Explore similar work

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

Diagnose, Then Repair: A Two-Stage MQM-Guided Post-Editing Framework for Domain-Specific Machine Translation

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation