Mind the Gap: Exposing LLM Translation Blind Spots Using the AlphaMWE Multilingual Parallel Corpus

Lifeng Han; Jiahui Liang; Anna Latusek; Karim El Haff; Amal Haddad Haddad; Josua Höfgen; Kilian Evang; Min Ma

cs.CLSep 6, 2026

Mind the Gap: Exposing LLM Translation Blind Spots Using the AlphaMWE Multilingual Parallel Corpus

Authors: Lifeng Han, Jiahui Liang, Anna Latusek, Karim El Haff, Amal Haddad Haddad, Josua Höfgen, Kilian Evang, Min Ma,

Organizations: Leiden University · Leiden University Medical Centre · Institute of Computer Science PAS, Warsaw · Faculty of Computer and Information Science, University of Ljubljana ·

Abstract

LLMs' performance on machine translation (MT) tasks is often dependent on the data availability in the specific domains and language pairs that they are trained upon. To examine if Multiword Expressions (MWEs) still set a bottleneck for LLMs regarding language understanding and translation, we report the system performances from the WMT2026 Test Suites shared task, for which we used the publicly available multilingual parallel corpus AlphaMWE as the test suites. We received 31 MT systems' outputs covering English to Chinese (zh), Polish (pl), German (de), Arabic (ar) including Modern Standard Arabic (MSA) and two dialectal ones (Egyptian and Tunisian Arabic). We carried out automatic evaluations using BLEU, ChrF, BERT-score to select the Top3 systems per language pair, followed up with human evaluations on the selected systems. Our findings show that: figurative/MWE phenomena remain challenging; automatic metrics sometimes disagree; human evaluation uncovers language-specific errors hidden by aggregate scores; human evaluation re-ranks the top-3 systems.

Figures & tables

Panel of Figure 1 — Figure 1: Overall methodology for AlphaMWE test-suite construction, automatic evaluation of WMT 2026 system outputs, top 3 system selection per language pair, and HOPE-based human evaluation followed by quantitative and qualitative analyses.

System	en $\rightarrow$ aeb	en $\rightarrow$ ar	en $\rightarrow$ arz	en $\rightarrow$ de	en $\rightarrow$ pl	en $\rightarrow$ zh
	(150)	(150)	(150)	(747)	(597)	(747)
CUNI-AR	✓	✓	✓
CUNI-EdUKate				✓
CUNI-MH-v3				✓
Cohere CAT+	✓	✓	✓	✓	✓	✓
Command A+	✓	✓	✓	✓	✓	✓

Table 1: Coverage of MT systems that returned translations for the AlphaMWE test suites submitted to the WMT 2026 Test Suites Shared Task. Numbers in parentheses indicate the number of evaluation segments in each language pair. English–Polish contains 597 segments because the ad subset was unavailable, while the Arabic test suites each contain the 150-segment aa subset only.

Language Pair	System	BLEU	chrF++	BERT	Avg. Rank	Selection Rank
en $\rightarrow$ aeb	TRIVE	1	1	1	1.000	1
	CUNI-AR	2	2	2	2.000	2
	DeepSeek V4 Pro	4	4	4	4.000	3
en $\rightarrow$ ar	GoogleTranslate	1	1	1	1.000	1
	Gemma 4–4B	2	2	2	2.000	2
	TRIVE	3	3	3	3.000	3

Table 2: Top 3 MT systems selected for human evaluation on each AlphaMWE language pair. Systems were selected according to the average of their SacreBLEU, chrF++, and BERTScore F1 rankings. The rank standard deviation (Rank SD) measures the agreement among the three automatic metrics; lower values indicate greater agreement but were not used as a selection criterion. Competition ranking was adopted for the final selection rank.

Language Pair	MT System	IMP	RAM	TRM	UGR	MIS	STL	PRF	PRN	SEGS
en $\rightarrow$ zh	SCIR-TG-MT	0	0	2	1	41	2	6	0	52
	GoogleTranslate	0	0	3	1	79	0	13	2	98
	Tower 9B	0	2	13	4	113	2	27	6	167
en $\rightarrow$ pl	DeepSeek V4 Pro	27	0	24	5	40	34	6	0	136
	TRIVE	21	0	36	2	22	23	3	16	123
	Dubformer	35	0	26	5	28	30	9	0	133

Table 3: Summed HOPE error penalties for the three systems selected for human evaluation on each language pair. Each value is the total penalty assigned to an error category over 150 AlphaMWE segments. Lower scores indicate fewer or less severe translation errors. Segs is the sum of the eight error-category penalties.

Figure 2: Visualisation of Human Evaluations on Four Language Pairs.

Figure 6

Figure 7

Figure 8

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Pair	Code	Language	Description
en $\rightarrow$ ar	ar	Modern Standard Arabic (MSA)	Standard written Arabic
en $\rightarrow$ arz	arz	Egyptian Arabic	Egyptian Arabic dialect
en $\rightarrow$ aeb	aeb	Tunisian Arabic	Tunisian Arabic dialect
en $\rightarrow$ de	de	German	Standard German
en $\rightarrow$ pl	pl	Polish	Polish
en $\rightarrow$ zh	zh	Chinese	Simplified Chinese

Appendix

Table 4: Language identifiers used throughout this paper and in the WMT 2026 Test Suites Shared Task.

Segs totals
	A1	A2
DeepSeek V4 Pro	116	136
TRIVE	68	123
Dubformer	89	133
best–worst spread	48	13
Annotation profile

Appendix

Table 5: Cross-annotator comparison for English–Polish over 150 segments

\times

3 MT systems (450 MT outputs). Segs totals are summed error penalties; lower is better. A1 and A2 denote the two independent annotators.

Segs totals
	A1	A2
VoxNexus_V1	42	77
DeepSeek V4 Pro	145	131
TRIVE	140	97
best–worst spread	103	54
Annotation profile

Appendix

Table 6: Cross-annotator comparison for English–German over 150 segments

\times

3 MT systems (450 MT outputs). Segs totals are summed error penalties; lower is better. A1 and A2 denote the two independent annotators.

System	en $\rightarrow$ aeb	en $\rightarrow$ ar	en $\rightarrow$ arz	en $\rightarrow$ de	en $\rightarrow$ pl	en $\rightarrow$ zh
CUNI-AR	8.46	30.02	9.03	–	–	–
CUNI-EdUKate	–	–	–	40.52	–	–
CUNI-MH-v3	–	–	–	41.38	–	–
Cohere CAT+	3.04	23.97	3.89	39.16	29.79	32.88
Command A+	2.69	21.88	2.35	36.03	26.89	40.04
DeepSeek V4 Pro	7.35	28.75	9.90	50.00	40.69	37.94

Appendix

Table 7: SacreBLEU scores for the submitted AlphaMWE test suites. Higher scores are better.

System	en $\rightarrow$ aeb	en $\rightarrow$ ar	en $\rightarrow$ arz	en $\rightarrow$ de	en $\rightarrow$ pl	en $\rightarrow$ zh
CUNI-AR	34.50	59.72	39.58	–	–	–
CUNI-EdUKate	–	–	–	63.07	–	–
CUNI-MH-v3	–	–	–	63.41	–	–
Cohere CAT+	13.75	54.41	28.32	62.97	56.02	23.80
Command A+	17.68	52.65	23.78	60.54	53.47	27.17
DeepSeek V4 Pro	32.76	58.67	39.36	69.83	63.40	27.93

Appendix

Table 8: chrF++ scores for the submitted AlphaMWE test suites. Higher scores are better.

System	en $\rightarrow$ aeb	en $\rightarrow$ ar	en $\rightarrow$ arz	en $\rightarrow$ de	en $\rightarrow$ pl	en $\rightarrow$ zh
CUNI-AR	0.7779	0.8935	0.7868	–	–	–
CUNI-EdUKate	–	–	–	0.8877	–	–
CUNI-MH-v3	–	–	–	0.8906	–	–
Cohere CAT+	0.6850	0.8756	0.7349	0.8867	0.8598	0.8382
Command A+	0.6784	0.8683	0.7049	0.8757	0.8561	0.8529
DeepSeek V4 Pro	0.7624	0.8914	0.7810	0.9061	0.8906	0.8587

Appendix

Table 9: BERTScore F1 scores for the submitted AlphaMWE test suites. Higher scores are better.

Figure 6: Top 3 MT systems selected for human evaluation: ranking and original scores

System	en $\rightarrow$ aeb	en $\rightarrow$ ar	en $\rightarrow$ arz	en $\rightarrow$ de	en $\rightarrow$ pl	en $\rightarrow$ zh
CUNI-AR	2	4	8	–	–	–
CUNI-EdUKate	–	–	–	17	–	–
CUNI-MH-v3	–	–	–	16	–	–
Cohere CAT+	16	12	18	20	15	25
Command A+	17	19	21	23	20	15
DeepSeek V4 Pro	4	5	3	2	1	18

Appendix

Table 10: System rankings based on SacreBLEU for the AlphaMWE test suites. Rank 1 denotes the best-performing system.

System	en $\rightarrow$ aeb	en $\rightarrow$ ar	en $\rightarrow$ arz	en $\rightarrow$ de	en $\rightarrow$ pl	en $\rightarrow$ zh
CUNI-AR	2	4	6	–	–	–
CUNI-EdUKate	–	–	–	19	–	–
CUNI-MH-v3	–	–	–	17	–	–
Cohere CAT+	18	13	17	20	13	25
Command A+	16	16	21	23	19	17
DeepSeek V4 Pro	4	5	7	2	1	15

Appendix

Table 11: System rankings based on chrF++ for the AlphaMWE test suites. Rank 1 denotes the best-performing system.

System	en $\rightarrow$ aeb	en $\rightarrow$ ar	en $\rightarrow$ arz	en $\rightarrow$ de	en $\rightarrow$ pl	en $\rightarrow$ zh
CUNI-AR	2	4	2	–	–	–
CUNI-EdUKate	–	–	–	17	–	–
CUNI-MH-v3	–	–	–	16	–	–
Cohere CAT+	16	14	17	19	17	23
Command A+	17	16	23	24	18	16
DeepSeek V4 Pro	4	5	6	2	1	11

Appendix

Table 12: System rankings based on BERTScore F1 for the AlphaMWE test suites. Rank 1 denotes the best-performing system.

Machine Translation

Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.

See all 267 papers

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation
May 24, 2026Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber +1Multilingual Language Model Evaluation MT Evaluation
TUD Dresden University of Technology · 2InfAI e.V. · 3Amazon +1
LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation
Apr 20, 2026Samar M. Magdy, Fakhraddin Alwajih, Abdellah El Mekki +2Arabic NLP Machine Translation
λThe University of British Columbia · ξMinia University · γCanada Research Chair in NLP and ML
Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators
Oct 24, 2026Dmitry Popov, Roman Bokhyan, Ekaterina Enikeeva +3MT Evaluation LLM-as-a-Judge
Yandex · HSE University

May 24, 2026J/K move · Enter open · S save

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation

Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber, Jens Lehmann

TUD Dresden University of Technology · 2InfAI e.V. · 3Amazon+1

Multilingual Language Model Evaluation MT Evaluation

Open paper AskPDF arXiv

Machine-translated benchmarks are widely used to assess the multilingual capabilities of large language models (LLMs), yet translation errors in these benchmarks remain underexplored, raising concerns about the reliability and comparability of multilingual evaluation. We address two practical gaps: (i) how well automatic MQM-style error spans from LLM judges and a span-aware QE baseline (xCOMET-XXL) match expert human span annotations on benchmark translations, and (ii) how strongly translation errors (as opposed to source-side issues in the English original) explain accuracy drops on translated benchmarks. We find that span agreement is non-trivial on naturally occurring benchmark translations, and that target-side translation errors are consistently associated with measurable, percentage-point drops in translated accuracy even after controlling for English correctness and source-side anomalies.

Abstract

Machine Translation

Explore similar work

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation

LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation

Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators