TTLab at AlexandriaX-2026: A Fine-Tuned Surface Tagger for Arabic Machine-Translation Error-Span Detection and Classification

Ali Abusaleh; Bhuvanesh Verma; Alexander Mehler

cs.CLSep 24, 2026

TTLab at AlexandriaX-2026: A Fine-Tuned Surface Tagger for Arabic Machine-Translation Error-Span Detection and Classification

Authors: Ali Abusaleh, Bhuvanesh Verma, Alexander Mehler

Organizations: Text Technology Lab (TTLab), Goethe University Frankfurt

Abstract

We present TTLab's submission to the AlexandriaX-2026 Subtask~3 on Arabic MT error span detection and classification. Our system frames the task as token-level classification over surface forms, preserving character offsets to ensure exact alignment with the evaluation metric. To handle severe label imbalance, we employ a focal loss with class weighting and dialect-specific decoding thresholds. Among six Arabic pre-trained encoders, MARBERTv2 achieves the best overall performance of 40.8 and 40.91 on the development and test set, respectively, ranking $\nth{3}$ out of all participating teams. While our system localizes error spans effectively, classification of rare error types remains challenging, highlighting the need for data augmentation for tail categories. The code is available at ${[\faGithub~ TTLab at AlexandriaX-2026](https://github.com/ENTAILab/arabic-dialectal-mt-error-span-detection)$

Figures & tables

Split / property		Value
Sizes	Train sentences	1,125
	Dev sentences	138
	Train error spans	1,997
	Errors / sentence (mean)	1.78
Directions	ENG_EGY	263
	ENG_MAU	244

Table 1: AlexandriaX Subtask 3 training set statistics. Percentages are computed over all error spans.

Panel of Figure 1 — Figure 1: Architecture of the error span detection and classification system. The raw Arabic MT output is tokenized with character offsets (bottom row) and encoded by a fine-tuned Arabic encoder. A linear head classifies each sub-word token; contiguous non- O tokens of the same category are merged and mapped to character spans via the stored offsets. The decoding block applies dialect-specific confidence thresholds and light post-processing.

System	Backbone	OOF	Dev
Fine-tuned surface tagger
Surface tagger	NileChat-3B	28.2	29.2
Surface tagger	AraBERTv2	34.8	34.4
Surface tagger	SaudiBERT	36.7	35.1
Surface tagger	CAMeLBERT-DA	36.7	35.8
Surface tagger	MARBERTv2 (QuranQA c.e.)	38.9	38.5

Table 2: Results on AlexandriaX Subtask 3. We report

(\text{Overlap-span }F_{1}+\text{Error-Category micro-}F_{1})/2

(in percent); the official leaderboard instead combines overlap-span

F_{1}

and error-class

F_{1}

(\text{Overlap}+\text{Class})/2

, each macro-averaged across the five dialect directions (Exact-Match

F_{1}

is reported by the leaderboard separately as a diagnostic and does not enter the ranking score), so our development scores here are a pooled/micro proxy and are not directly comparable. OOF is five-fold cross-validation on the training set; Dev is the official 138-sentence development set. MARBERTv2 achieves the best performance on both splits and is our submitted system.

Direction	Exact	Overlap	Class
ENG_EGY	14	50	47
ENG_MOR	17	47	32
ENG_MAU	7	39	44
ENG_PAL	16	47	28
ENG_UAE	10	49	42
Macro	13	46	38

Table 3: Official Codabench test-set scores (%) per dialect direction for the revised focal, no-collapse MARBERTv2 system. Overall Score

(\text{Overlap}_{\text{macro}}+\text{Class}_{\text{macro}})/2=\mathbf{42.00}

. Exact-Match is a diagnostic and does not enter the ranking score.

Config	OOF ${}_{\text{pooled}}$	OOF ${}_{\text{nested}}$	Dev	Dev macro- $F_{1}^{\text{cat}}$
Focal + collapse (submitted)	39.4	35.4	38.9	16.3
Cross-entropy + collapse	39.1	34.6	38.8	14.5
Focal + no collapse	39.3	35.9	40.1	18.2

Table 4: Ablations on the final MARBERTv2 tagger (5-fold OOF, retrained per configuration). OOF

{}_{\text{pooled}}

matches the procedure in Table 2 ; OOF

{}_{\text{nested}}

tunes per-dialect thresholds on the other four folds only. Dev macro-

F_{1}^{\text{cat}}

is the true per-category macro-

F_{1}

(over all six original categories) on the official dev set, using overlap-based span matching.

Category	Supp.	P	R	$F_{1}$
sociolinguistics	1157	31.7	51.3	39.2
semantics	485	21.0	27.0	23.6
morphosyntax	164	5.8	4.9	5.3
orthography	106	5.2	4.7	4.9
pragmatics	83	0.0	0.0	0.0
graphetics	2	0.0	0.0	0.0

Table 5: Per-category precision/recall/

F_{1}

(%) of the submitted (focal, collapsed) MARBERTv2 tagger, on 5-fold OOF predictions over the training set. Support is the number of gold spans of that category. Orthography is collapsed into other at train time, so its score here reflects the fixed other

\to

orthography fallback rather than category-specific discrimination; morphosyntax is modeled explicitly and its low score is a genuine gap (see Table 4 for the un-collapsed alternative).

Explore similar work

LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation
Apr 20, 2026Samar M. Magdy, Fakhraddin Alwajih, Abdellah El Mekki +2Machine Translation Quality Machine Translation
λThe University of British Columbia · ξMinia University · γCanada Research Chair in NLP and ML
DialectSentEval 2026: Arabic Dialect Sentiment Analysis and Swapping Shared Task
Oct 5, 2026Saad Ezzini, Shadi Abudalfa, Maram Alharbi +6Arabic Natural Language Processing
King Fahd University of Petroleum and Minerals, KSA · Onaizah Colleges, KSA · University College of Applied Sciences, Palestine +5
TTLab at StanceEval-2026: A Cloze-Style Prompting Approach for Arabic-Language Stance Detection (CLASP-Ar)
Sep 24, 2026Bhuvanesh Verma, Ali Abusaleh, Alexander MehlerCross-Topic Arabic Stance Detection Natural Language Processing
Text Technology Lab (TTLab), Goethe University Frankfurt

Apr 20, 2026, cs.CLJ/K move · Enter open · S save

LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation

Samar M. Magdy, Fakhraddin Alwajih, Abdellah El Mekki, Wesam El-Sayed+1

λThe University of British Columbia · ξMinia University · γCanada Research Chair in NLP and ML

Machine Translation Quality Machine Translation Multilingual Benchmark Linguistics

Open paper AskPDF arXiv

Existing MT evaluation frameworks, including automatic metrics and human evaluation schemes such as Multidimensional Quality Metrics (MQM), are largely language-agnostic. However, they often fail to capture dialect- and culture-specific errors in diglossic languages (e.g., Arabic), where translation failures stem from mismatches in language variety, content coverage, and pragmatic appropriateness rather than surface form alone.We introduce LQM: Linguistically Motivated Multidimensional Quality Metrics for MT. LQM is a hierarchical error taxonomy for diagnosing MT errors through six linguistically grounded levels: sociolinguistics, pragmatics, semantics, morphosyntax, orthography, and graphetics (Figure 1). We construct a bidirectional parallel corpus of 3,850 sentences (550 per variety) spanning seven Arabic dialects (Egyptian, Emirati, Jordanian, Mauritanian, Moroccan, Palestinian, and Yemeni), derived from conversational, culturally rich content. We evaluate six LLMs in a zero-shot setting and conduct expert span-level human annotation using LQM, producing 6,113 labeled error spans across 3,495 unique erroneous sentences, along with severity-weighted quality scores. We complement this analysis with an automatic metric (spBLEU). Though validated here on Arabic, LQM is a language-agnostic framework designed to be easily applied to or adapted for other languages. LQM annotated errors data, prompts, and annotation guidelines are publicly available at https://github.com/UBC-NLP/LQM_MT.

Abstract

Explore similar work

LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation

DialectSentEval 2026: Arabic Dialect Sentiment Analysis and Swapping Shared Task

TTLab at StanceEval-2026: A Cloze-Style Prompting Approach for Arabic-Language Stance Detection (CLASP-Ar)