LuxMT Technical Report

Nils Rehlinger

cs.CLFeb 17, 2026

LuxMT Technical Report

Authors: Nils Rehlinger

Organizations: University of Luxembourg / Esch-Belval, Esch-sur-Alzette, Luxembourg

Abstract

We introduce LuxMT, a machine translation system based on Gemma 3 27B and fine-tuned for translation from Luxembourgish (LB) into French (FR) and English (EN). To assess translation performance, we construct a novel benchmark covering LB-FR, LB-EN, and LB-FR using human-translated data from Luci, a tourist magazine about Luxembourg. Training data stems from LuxAlign, a parallel corpus of multilingual Luxembourgish news articles, and LB parliamentary transcripts augmented with Google Translate. We filter the data using LuxEmbedder, LB sentence embeddings, to remove low-equivalence segment-pairs. Overall, LuxMT's results suggest strong improvements over the Gemma 3 baseline, even for translating LB to German (DE), despite the training data not containing any DE. We also explore LuxEmbedder's potential to be used as a quality estimation metric and find strong correlations with other reference-based metrics. However, we call for further research to fully assess the metric's utility and advise using it with caution.

Figures & tables

System	LE $\uparrow$	BS $\uparrow$	B-20 $\uparrow$	xC XL $\uparrow$
Gemma 3	93.9	93.6	74.0	89.7
Aya Expanse	90.0	91.7	67.0	82.2
Command R	92.5	91.3	66.4	81.6
Llama 3.1	93.0	93.0	71.0	87.3
Llama 4	93.5	92.9	70.9	86.7
Mistral S	90.4	91.6	65.8	80.4

Table 1: Automatic evaluation results assessing various popular out-of-the-box local LLMs on the Luci benchmark for LB

\rightarrow

EN (

N=500

), LB

\rightarrow

FR (

N=500

), and LB

\rightarrow

DE (

N=500

). The results are averaged across the three language pairs. Metrics include LuxEmbedder (LE), BERTScore (BS), BLEURT-20 (B-20), and xCOMET XL (xC XL ).

\uparrow

indicates that higher scores are better. The highest scores are highlighted in bold .

System	BS $\uparrow$	B-20 $\uparrow$
Gemma 3	93.8	77.2
LE .90	93.6	77.6
LE .95	94.0	78.1
LE .99	94.4	78.8

Table 2: Automatic evaluation results assessing various LuxEmbedder (LE) filter thresholds, using BERTScore (BS) and BLEURT-20 (B-20). Results are based on average system scores for LB

\rightarrow

FR (

N=400

) and LB

\rightarrow

EN (

N=233

) on Luci segment pairs.

Epochs	BS $\uparrow$	B-20 $\uparrow$
1	94.4	78.8
2	94.4	78.5
3	94.2	78.4

Table 3: Automatic evaluation results assessing fine-tuning Gemma 3 for 1-3 epochs. Results are based on average system scores for LB

\rightarrow

FR (

N=400

) and LB

\rightarrow

EN (

N=233

) on Luci segment pairs.

Panel of Figure 1 — Figure 1: Data mixture for MT fine-tuning. LB $\rightarrow$ FR total: 32k; LB $\rightarrow$ EN total: 22.5k; Total: 54.5k.

	LE $\uparrow$	BS $\uparrow$	B-20 $\uparrow$	xC XL $\uparrow$
$\Delta$ LB $\rightarrow$ FR	0.7*	0.2*	1.3*	1.9*
$\Delta$ LB $\rightarrow$ EN	0.8*	0.8*	0.9*	1.2*
$\Delta$ LB $\rightarrow$ DE	0.2	0.2*	1.4*	0.6*

Table 4: System score deltas (

\Delta

) on the Luci benchmark. Asterisk (*) signifies statistical significance (

p<0.05

) using paired bootstrap sampling with 95% confidence interval.

\Delta=\textsc{LuxMT}-\textsc{Gemma 3}

Human accuracy estimates: > 99% ; > 90% ; > 66% ; 33%–66% ; < 33% ; < 10% ; < 1%

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

	LE $\uparrow$	BS $\uparrow$	B-20 $\uparrow$	xC XL $\uparrow$	BL $\uparrow$	cF2 $\uparrow$	TER $\downarrow$
LB $\rightarrow$ FR
BL: Gemma 3 27B IT Q8	92.7	95.2	68.5	85.1	33.5	61.2	56.1
Aya Expanse 32B Q8	87.7*	94.0*	57.7*	72.8*	26.3*	54.9*	64.3*
Command R 35B Q8 (08-2024)	87.5*	94.0*	58.0*	73.3*	24.6*	54.2*	65.7*
Llama 3.1 70B IT Q5 K M	91.4*	94.6*	63.7*	80.8*	27.7*	57.2*	60.5*
Llama 4 16x17B	92.4	94.7*	64.1*	80.3*	29.8*	57.9*	59.7*

Appendix

Table 5: BL = baseline. Automatic evaluation results using LuxEmbedder (LE), BERTScore (BS), BLEURT-20 (B-20), xCOMET XL (xC XL ), BLEU (BL), chrF2 (cF2), and Translation Edit Rate (TER) on the Luci benchmark. The best scores are in bold. Asterisks denote statistically significant differences from the Gemma 3 (BS) baseline (

p<0.05

Temp	BS $\uparrow$	B-20 $\uparrow$	BL $\uparrow$	cF2 $\uparrow$	TER $\downarrow$
0.1	94.3	78.7	44.7	68.8	43.6
0.2	94.3	78.6	44.6	68.8	44.2
0.3	94.4	78.5	44.6	68.6	43.7
0.4	94.2	78.3	44.0	68.2	44.4
0.5	94.3	78.5	43.8	68.3	44.8
0.6	94.3	78.5	43.9	68.6	44.4

Appendix

Table 6: Automatic evaluation scores across different temperature settings. The best scores are in bold.

System	BS $\uparrow$	B-20 $\uparrow$	BL $\uparrow$	cF2 $\uparrow$	TER $\downarrow$
Gemma 3	93.8	77.2	42.5	67.8	46.3
LE .90	93.6	77.6	40.4	65.6	48.2
LE .95	94.0	78.1	42.0	66.5	46.3
LE .99	94.4	78.8	44.7	69.1	43.6

Appendix

Table 7: Automatic evaluation results assessing various LuxEmbedder (LE) filter thresholds, using BERTScore (BS), BLEURT-20 (B-20), BLEU (BL), chrF2 (cF2), and Translation Edit Rate (TER). For all metrics, higher scores are better (

\uparrow

), except for TER where lower is better (

\downarrow

). Results are based on average scores for LB

\rightarrow

FR (

N=400

) and LB

\rightarrow

EN (

N=233

) on Luci segment pairs.

Epochs	BS $\uparrow$	B-20 $\uparrow$	BL $\uparrow$	cF2 $\uparrow$	TER $\downarrow$
1	94.4	78.8	44.7	69.1	43.6
2	94.4	78.5	44.1	68.4	44.4
3	94.2	78.4	43.6	68.1	44.7

Appendix

Table 8: Automatic evaluation results assessing fine-tuning Gemma 3 for 1-3 epochs. Results are based on average system scores for LB

\rightarrow

FR (

N=400

) and LB

\rightarrow

EN (

N=233

) on Luci segment pairs.

	LE $\uparrow$	BS $\uparrow$	B-20 $\uparrow$	xC XL $\uparrow$	BL $\uparrow$	cF2 $\uparrow$	TER $\downarrow$
$\Delta$ LB $\rightarrow$ FR	0.7*	0.2*	1.3*	1.9*	0.6	0.4	-1.6*
$\Delta$ LB $\rightarrow$ EN	0.8*	0.8*	0.9*	1.2*	1.2*	1.3*	-2.3*
$\Delta$ LB $\rightarrow$ DE	0.2	0.2*	1.4*	0.6*	0.9*	0.3	-0.7

Appendix

Table 9: System score deltas (

\Delta

) on the Luci benchmark. Asterisk (*) signifies statistical significance (

p<0.05

) using paired bootstrap sampling with 95% confidence interval.

\Delta=\textsc{LuxMT}-\textsc{Gemma 3}

Human accuracy estimates: > 99% ; > 90% ; > 66% ; 33%–66% ; < 33% ; < 10% ; < 1%

	Spearman’s $\rho$	Kendall’s $\tau$
LB $\rightarrow$ FR
BERTScore	0.9910*	0.9759*
BLEURT-20	0.8929*	0.8095*
xCOMET XL :	0.8214*	0.6190*
BLEU	1.0000*	1.0000*
chrF2	1.0000*	1.0000*

Appendix

Table 10: Correlation of LuxEmbedder and other automatic metrics on system-level performance scores of Gemma 3 , Aya Expanse , Command R , Llama 3.1 , Llama 4 , Mistral S , and Phi 4 on the Luci benchmark (

N=500

for each language pair). Asterisk (*) denotes statistical significance (

p<0.05

) and the highest correlations are marked in bold.

Machine Translation

Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.

See all 267 papers

LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation
Apr 20, 2026Samar M. Magdy, Fakhraddin Alwajih, Abdellah El Mekki +2Arabic NLP Machine Translation
λThe University of British Columbia · ξMinia University · γCanada Research Chair in NLP and ML
TermJudge: A Document-Level Metric Judging, Not Counting, Terminology in Machine Translation Evaluation
Sep 28, 2026Nicolas Dahan, Fran{\cc}ois Yvon, Rachel BawdenAutomated Evaluation Document-Level MT
ISIR, ALMAnaCH · MLIA, ISIR · ALMAnaCH
Omnilingual MT: Machine Translation for 1,600 Languages
Mar 17, 2026Omnilingual MT Team, Belen Alastruey, Niyati Bafna +29Machine Translation Low-Resource MT
FAIR at Meta

Apr 20, 2026J/K move · Enter open · S save

LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation

Samar M. Magdy, Fakhraddin Alwajih, Abdellah El Mekki, Wesam El-Sayed+1

λThe University of British Columbia · ξMinia University · γCanada Research Chair in NLP and ML

Arabic NLP Machine Translation MT Evaluation

Open paper AskPDF arXiv

Existing MT evaluation frameworks, including automatic metrics and human evaluation schemes such as Multidimensional Quality Metrics (MQM), are largely language-agnostic. However, they often fail to capture dialect- and culture-specific errors in diglossic languages (e.g., Arabic), where translation failures stem from mismatches in language variety, content coverage, and pragmatic appropriateness rather than surface form alone.We introduce LQM: Linguistically Motivated Multidimensional Quality Metrics for MT. LQM is a hierarchical error taxonomy for diagnosing MT errors through six linguistically grounded levels: sociolinguistics, pragmatics, semantics, morphosyntax, orthography, and graphetics (Figure 1). We construct a bidirectional parallel corpus of 3,850 sentences (550 per variety) spanning seven Arabic dialects (Egyptian, Emirati, Jordanian, Mauritanian, Moroccan, Palestinian, and Yemeni), derived from conversational, culturally rich content. We evaluate six LLMs in a zero-shot setting and conduct expert span-level human annotation using LQM, producing 6,113 labeled error spans across 3,495 unique erroneous sentences, along with severity-weighted quality scores. We complement this analysis with an automatic metric (spBLEU). Though validated here on Arabic, LQM is a language-agnostic framework designed to be easily applied to or adapted for other languages. LQM annotated errors data, prompts, and annotation guidelines are publicly available at https://github.com/UBC-NLP/LQM_MT.

Abstract

Machine Translation

Explore similar work

LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation

TermJudge: A Document-Level Metric Judging, Not Counting, Terminology in Machine Translation Evaluation

Omnilingual MT: Machine Translation for 1,600 Languages