FACET at WMT 2026 Automated Translation Quality Evaluation Task
Authors: Ahrii Kim, Chanjun Park, Seong-heum Kim
Organizations: AI-Bio Convergence Research Inst. Soongsil University · School of Software Soongsil University · Dept. of Intelligent Semiconductors Soongsil University
Different error types in machine translation require different evidence. Whether meaning is preserved can be judged only against the source, while whether the target is well-formed, or whether it names one entity consistently, can be judged from the target alone. We present FACET, our reference-free submission to the WMT26 Automated Translation Quality Evaluation Task, which decomposes evaluation into Fluency, Accuracy, and Consistency passes and gives each pass only the context its error type requires. A single fixed model is prompted three times, and the merged error spans yield the three task outputs, error spans, quality scores, and error-free labels, with no trained components. We also submit FACET-C, which omits the Consistency pass. Without gold labels, we characterize the predictions of FACET. Its system rankings place post-edited human translation first, and the Consistency pass changes about a tenth of segment scores while leaving the ranking nearly unchanged.
Figures & tables
Figure 1: facet evaluates a translation with three independent prompting passes under asymmetric input . Accuracy reads the source and the target, while Fluency and Consistency read the target only. Predictions are merged and scored per segment. facet uses all three passes, and facet − c drops Consistency (dashed).
System
cs-de
en-de
cs-vi
en-is
en-et
zh-ja
en-se
en-zh
en-th
en-cs
en-kk
en-ar
en-ru
cs-uk
en-lij
en-uk
en-lld
en-hy
en-be
en-id
en-zh-TW
avg
Human (postediting)
1
2
1
–
–
1
–
–
–
1
–
2
–
–
–
–
–
–
–
–
–
1.3
Human (from scratch)
15
–
15
8
9
–
11
–
–
–
–
–
–
–
20
–
15
–
–
–
–
13.3
Wayfinder
5
3
5
1
2
5
5
2
3
10
1
1
3
2
2
1
11
5
3
20
4
4.5
Lumen
9
7
3
2
4
8
3
4
1
5
4
9
6
1
1
4
4
2
1
11
6
4.5
GPT 5.5
2
6
2
3
1
3
7
10
2
4
6
3
8
3
11
6
3
3
2
3
9
4.6
UvA-MT
3
4
8
7
5
2
2
1
4
8
3
7
5
7
3
5
7
4
5
9
5
5.0
Table 1: facet system rankings across the 21 language pairs. Each cell is the rank of a system within that language pair (1 = best), darker is better, and -- marks a pair the system did not enter. Systems below the rule are ordered by mean rank. Human (postediting) attains the best mean rank, while Human (from scratch) sits mid-table.
Figure 2: A full document evaluated by facet , with all detected errors highlighted. Fluency and Accuracy act within a segment, from the target alone and from the source–target pair. Consistency acts across the document. The facility named Bürstenfabrik in Segment 1 becomes Schleiferei in Segment 2, a divergence that no single-segment judgment can detect.
How can a monolingual English speaker determine whether an automatic translation in French is good enough to be shared? Existing MT error detection and quality estimation (QE) techniques do not address this practical scenario. We introduce AskQE, a question generation and answering framework designed to detect critical MT errors and provide actionable feedback, helping users decide whether to accept or reject MT outputs even without the knowledge of the target language. Using ContraTICO, a dataset of contrastive synthetic MT errors in the COVID-19 domain, we explore design choices for AskQE and develop an optimized version relying on LLaMA-3 70B and entailed facts to guide question generation. We evaluate the resulting system on the BioMQM dataset of naturally occurring MT errors, where AskQE has higher Kendall's Tau correlation and decision accuracy with human ratings compared to other QE metrics.
Machine Translation (MT) and automatic MT evaluation have improved dramatically in recent years, enabling numerous novel applications. Automatic evaluation techniques have evolved from producing scalar quality scores to precisely locating translation errors and assigning them error categories and severity levels. However, it remains unclear how to reliably measure the evaluation capabilities of auto-evaluators that do error detection, as no established technique exists in the literature. This work investigates different implementations of span-level precision, recall, and F-score, showing that seemingly similar approaches can yield substantially different rankings, and that certain widely-used techniques are unsuitable for evaluating MT error detection. We propose "match with partial overlap and partial credit" (MPP) with micro-averaging as a robust meta-evaluation strategy and release code for its use publicly. Finally, we use MPP to assess the state of the art in MT error detection.
Stefano Perrella, Eric Morales Agostinho, Hugo Zaragoza
Current state-of-the-art Quality Estimation (QE) in machine translation relies on massive, proprietary LLMs, raising data privacy concerns. We demonstrate that smaller, open-source LLMs (<30B parameters) are a viable, cost-effective and privacy-preserving alternative. Using a single-pass prompting strategy, our models simultaneously generate quality scores, MQM error annotations, suggested error corrections, and full post-editions. Our analysis shows these models achieve highly competitive system-level correlations with human judgments that outperform traditional neural metrics, fine-tuned models, and human inter-annotator agreement, effectively approximating the capabilities of much larger proprietary LLMs.
Kamil Guttmann, Zofia Fraś, Artur Nowakowski +1
Laniqo, Pozna´n, Poland · Faculty of Mathematics and Computer Science, Adam Mickiewicz University, Pozna´n, Poland