Evaluating Multilingual Sentence Embeddings for Translation Error Detection:An English--Greek Contrastive Study

Eleftherios Kalogeros; Athanasios Ntalakas; Manolis Gergatsoulis; Paschalis Nikolaou; Sotiria-Lito Alexaki

cs.CLAug 28, 2026

Evaluating Multilingual Sentence Embeddings for Translation Error Detection:An English--Greek Contrastive Study

Authors: Eleftherios Kalogeros, Athanasios Ntalakas, Manolis Gergatsoulis, Paschalis Nikolaou, Sotiria-Lito Alexaki

Organizations: Laboratory on Digital Libraries and Electronic Publishing, Department of Information Science, Ionian University, Ioannou Theotoki 72, 49100 Corfu, Greece · Centre for Literary Translation and Book Policy, Department of Foreign Languages, Translation and Interpreting, Ionian University, Tsirigoti Sq., 49100 Corfu, Greece

Abstract

Multilingual sentence embeddings are increasingly used to estimate semantic similarity across languages, yet their sensitivity to fine-grained translation errors remains insufficiently understood. This study investigates whether general-purpose multilingual embedding models can distinguish correct English-Greek translations from minimally modified erroneous alternatives. A contrastive dataset was developed from FLORES+ sentence-aligned reference translations and reviewed by two translation experts. It contains 1,850 examples across ten core and five exploratory error categories, covering factual, lexical-semantic, grammatical, relational, referential, and discourse-level phenomena. Five multilingual sentence-embedding models (BGE-M3, Multilingual E5, Multilingual MPNet, LaBSE, and Jina Embeddings v3) were evaluated using cosine similarity between each English source sentence and its correct and erroneous Greek translations. A reference-free COMETKiwi model was also evaluated as an MT quality-estimation baseline. Performance was assessed through contrastive accuracy and score margins for category-specific sensitivity. BGE-M3 achieved the highest accuracy among embedding models at 89.30 percent, while COMETKiwi achieved 94.49 percent. Embedding models detected explicit factual and lexical changes more reliably than tense-and-aspect and pronoun-coreference errors. COMETKiwi improved performance on several difficult categories, including tense and aspect, pronoun and coreference, and semantic-role errors, but showed lower sensitivity to date-and-time errors and underperformed the embedding models on numbers. The results show complementary error-sensitivity profiles: multilingual sentence embeddings provide useful semantic adequacy signals but are better suited as components of broader translation-evaluation frameworks than as standalone metrics.

Figures & tables

Model	Checkpoint	Parameters	Max. Tokens
BGE-M3	BAAI/bge-m3	560M	8192
Multilingual E5	intfloat/ multilingual-e5-large	560M	512
Multilingual MPNet	sentence-transformers/ paraphrase-multilingual-mpnet-base-v2	278M	128
LaBSE	sentence-transformers/LaBSE	471M	256
Jina Embeddings v3	jinaai/jina-embeddings-v3	570M	8192

Table 1: Embedding models and configurations used in the experiments.

Category	Type	Meaning	$\mathcal{N}$
Negation	Core	Changes the polarity or truth conditions of the sentence.	150
Number	Core	Changes a numerical value in the translation.	150
Named entity	Core	Replaces a person, place, organization, or other named entity.	150
Antonym	Core	Replaces a word with one expressing an opposite meaning.	150
Omission	Core	Removes information present in the source sentence.	150
Unsupported addition	Core	Introduces information that is absent from the source.	150

Table 2: Error categories used in the English–Greek contrastive dataset.

\mathcal{N}

denotes the number of contrastive examples in each category.

Statistic	Value
Total contrastive examples	1,850
Unique FLORES+ sentence pairs	1,212
Core error categories	10
Exploratory error categories	5
Core-category examples	1,500
Exploratory-category examples	350

Table 3: Summary statistics of the generated contrastive dataset.

Model	Acc. (%)	Mean $\Delta$	Median $\Delta$	Incorrect	Ties
BGE-M3	89.30	0.0271	0.0155	197	1
Multilingual E5	81.35	0.0095	0.0053	344	1
Multilingual MPNet	88.05	0.0387	0.0180	220	1
LaBSE	87.73	0.0277	0.0198	226	1
Jina Embeddings v3	87.73	0.0363	0.0213	226	1

Table 4: Overall contrastive performance of the evaluated models. Incorrect denotes cases in which the erroneous translation received a higher similarity score than the correct translation; ties denote cases in which both translations received the same similarity score following the numerical tolerance defined in Section 4 .

Figure 1: Overall contrastive accuracy of the five multilingual sentence-embedding models.

Category	BGE-M3	Mult. E5	Mult. MPNet	LaBSE	Jina Emb. v3
Negation	92.67	95.33	90.00	93.33	94.00
Number	96.67	98.00	99.33	98.00	96.67
Named entity	99.33	93.33	99.33	95.33	98.00
Antonym	86.67	80.00	89.33	92.00	85.33
Omission	84.00	81.33	92.67	92.67	86.00
Unsupported addition	95.33	91.33	90.67	97.33	91.33

Table 5: Contrastive accuracy (%) by error category.

Category	BGE-M3	Mult. E5	Mult. MPNet	LaBSE	Jina Emb. v3
Negation	0.0412	0.0149	0.0666	0.0233	0.0483
Number	0.0238	0.0128	0.0456	0.0481	0.0523
Named entity	0.0696	0.0190	0.1129	0.0560	0.0881
Antonym	0.0217	0.0073	0.0315	0.0186	0.0285
Omission	0.0226	0.0077	0.0411	0.0375	0.0361
Unsupported addition	0.0175	0.0055	0.0206	0.0345	0.0205

Table 6: Mean contrastive cosine-similarity margin by error category.

Category	$\mathcal{N}$	Acc. (%)	Mean $\Delta_{\mathrm{CK}}$
Negation	150	99.33	0.1051
Number	150	89.33	0.0148
Named entity	150	98.67	0.1130
Antonym	150	96.00	0.0400
Omission	150	99.33	0.1198
Unsupported addition	150	100.00	0.1103

Table 7: Contrastive performance of COMETKiwi by error category. The margin denotes the COMETKiwi score for the correct translation minus that for the erroneous translation and is interpreted only within the COMETKiwi scoring scale.

MT Evaluation

Machine translation evaluation measures translation quality, error types, and real-world suitability across languages, domains, and modalities.

See all 173 papers

Jul 20, 2026

Multilingual Sentence Embeddings for Linguistic-Integrated Reliability Audit

Multilingual assessment systems commonly rely on translation for scoring and quality-control processes. We evaluate whether multilingual sentence embeddings can replace translated English input for Linguistic-Integrated Reliability Auditing (LiRA) across 11 PIRLS constructed-response items and three embedding models. Native-language embeddings reproduced translation-based reliability estimates closely while recovering responses excluded after translation failure, with no meaningful change in reliability.

Ummugul Bezirhan, Ji Yoon Jung, Matthias von Davier

Boston College Chestnut Hill, Massachusetts, USA

Sentence Embeddings Educational Assessment Text Embedding Evaluation

Ask PDF

May 24, 2026

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation

Machine-translated benchmarks are widely used to assess the multilingual capabilities of large language models (LLMs), yet translation errors in these benchmarks remain underexplored, raising concerns about the reliability and comparability of multilingual evaluation. We address two practical gaps: (i) how well automatic MQM-style error spans from LLM judges and a span-aware QE baseline (xCOMET-XXL) match expert human span annotations on benchmark translations, and (ii) how strongly translation errors (as opposed to source-side issues in the English original) explain accuracy drops on translated benchmarks. We find that span agreement is non-trivial on naturally occurring benchmark translations, and that target-side translation errors are consistently associated with measurable, percentage-point drops in translated accuracy even after controlling for English correctness and source-side anomalies.

Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber +1

TUD Dresden University of Technology · 2InfAI e.V. · 3Amazon +1

Multilingual Language Model Evaluation MT Evaluation

Ask PDF

Apr 20, 2026

LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation

Existing MT evaluation frameworks, including automatic metrics and human evaluation schemes such as Multidimensional Quality Metrics (MQM), are largely language-agnostic. However, they often fail to capture dialect- and culture-specific errors in diglossic languages (e.g., Arabic), where translation failures stem from mismatches in language variety, content coverage, and pragmatic appropriateness rather than surface form alone.We introduce LQM: Linguistically Motivated Multidimensional Quality Metrics for MT. LQM is a hierarchical error taxonomy for diagnosing MT errors through six linguistically grounded levels: sociolinguistics, pragmatics, semantics, morphosyntax, orthography, and graphetics (Figure 1). We construct a bidirectional parallel corpus of 3,850 sentences (550 per variety) spanning seven Arabic dialects (Egyptian, Emirati, Jordanian, Mauritanian, Moroccan, Palestinian, and Yemeni), derived from conversational, culturally rich content. We evaluate six LLMs in a zero-shot setting and conduct expert span-level human annotation using LQM, producing 6,113 labeled error spans across 3,495 unique erroneous sentences, along with severity-weighted quality scores. We complement this analysis with an automatic metric (spBLEU). Though validated here on Arabic, LQM is a language-agnostic framework designed to be easily applied to or adapted for other languages. LQM annotated errors data, prompts, and annotation guidelines are publicly available at https://github.com/UBC-NLP/LQM_MT.

Samar M. Magdy, Fakhraddin Alwajih, Abdellah El Mekki +2

λThe University of British Columbia · ξMinia University · γCanada Research Chair in NLP and ML

Arabic NLP Machine Translation MT Evaluation

Ask PDF

Abstract

MT Evaluation

Explore similar work

Multilingual Sentence Embeddings for Linguistic-Integrated Reliability Audit

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation

LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation