A Parallel Multi-Dialect Arabic Dependency Treebank
Authors: Noor Abo Mokh, Mohamed Adel, Jamila El-Gizuli, Go Inoue, Ahmed Elshabrawy, Nizar Habash
Organizations: Mohamed bin Zayed University of Artificial Intelligence · Computational Approaches to Modeling Language Lab, New York University Abu Dhabi
Dependency parsing resource development for Arabic has largely focused on Modern Standard Arabic, leaving dialectal Arabic comparatively under-resourced. We introduce MADARTB, the first large-scale parallel dependency treebank spanning five city dialects: Beirut, Cairo, Doha, Rabat and Tunis, and extending an existing parallel MSA version. In total MADARTB consists of 5,000 sentences annotated using the CATiB dependency scheme. We analyze syntactic similarity across all city dialects and MSA. Although global construction profiles exhibit high similarity across varieties, local syntactic structures frequently diverge, revealing substantial structural variation across dialects. We further establish a cross-dialect dependency parsing comparison using a supervised parser and four LLMs. The supervised parser achieves 87.5% LAS, while the best LLM setup trails at 85.7% LAS across all six varieties on the test set. We release our annotations and models publicly.
Figures & tables
Figure 1: CATiB-style Habash et al. (2009) parallel dependency annotations in MSA and five city dialects. English gloss: ‘Which bus goes to the city center?’. Transliterations in the Buckwalter Scheme Buckwalter (2002) .
Train
Development
Test
Total
Variety
Sent.
Words
Tokens
Sent.
Words
Tokens
Sent.
Words
Tokens
Sent.
Words
Tokens
Beirut
700
3,642
5,843
150
803
1,312
150
840
1,307
1,000
5,285
8,462
Cairo
700
3,987
5,973
150
898
1,342
150
899
1,328
1,000
5,784
8,643
Doha
700
3,625
5,369
150
802
1,204
150
809
1,195
1,000
5,236
7,768
Rabat
700
4,125
6,185
150
900
1,367
150
925
1,373
1,000
5,950
8,925
Tunis
700
3,851
5,845
150
844
1,322
150
864
1,302
1,000
5,559
8,469
Table 1: Numbers of sentences, words, and tokens in the train, development, and test splits. The six varieties are parallel at the sentence level.
Table 3
Model
Training Data / Prompting
Beirut
Cairo
Doha
Rabat
Tunis
MSA
Dialects
All
Supervised
MadarTB Beirut
81.1
77.8
78.7
73.0
75.0
72.7
77.1
76.3
Supervised
MadarTB Cairo
73.2
82.0
82.2
68.0
74.8
75.6
75.9
75.8
Supervised
MadarTB Doha
73.4
78.9
85.4
70.2
76.5
73.7
76.7
76.1
Supervised
MadarTB Rabat
66.5
72.9
76.2
83.9
75.1
70.1
75.0
74.1
Supervised
MadarTB Tunis
66.0
73.7
77.4
71.0
83.4
71.9
74.2
73.8
Supervised
MadarTB MSA
64.6
67.2
70.8
66.6
67.9
83.2
67.4
70.3
Table 4: Development-set parsing results (LAS) for supervised parsers and prompted LLMs. All Supervised rows use the CAMeLBERT-Mix Biaffine model and projective multi-root decoding; their configurations differ only in training data. All Treebanks denotes PATB, CamelTB, ARZTB, Gumar, and MadarTB . Bold indicates the best result overall, and underlining indicates the best LLM result.
Model
Training Data / Prompting
Beirut
Cairo
Doha
Rabat
Tunis
MSA
Dialects
All
Supervised
All treebanks
86.9
87.4
90.7
85.4
87.9
86.9
87.6
87.5
Gemini
10-shot
85.8
83.3
86.9
83.8
88.1
86.4
85.5
85.7
GPT
10-shot
84.5
81.9
84.5
82.4
85.0
85.1
83.6
83.9
Fanar
10-shot
67.6
70.0
72.3
72.6
72.5
73.3
71.0
71.4
Llama
10-shot
63.4
66.9
70.2
67.5
69.0
69.4
67.3
67.7
Table 5: Test-set parsing results (LAS) for the best supervised configuration and the best setup for each LLM, selected on development data. “All treebanks” denotes PATB, CamelTB, ARZTB, Gumar, and the proposed MadarTB . Bold indicates the best result overall, and underlining indicates the best LLM result. The complete test matrix appears in Appendix Table 10 .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
System
Parameters
Role and access
CAMeLBERT-Mix biaffine parser
110,936,282
Locally trained supervised parser used for the reported All Treebanks results; the count includes the CAMeLBERT-Mix encoder and SuPar parsing head.
CAMeLBERT-Mix (SimAlign)
109.1M
Frozen pretrained encoder used to obtain contextual token representations; SimAlign requires no task-specific training.
CAMeLBERT-Mix (Awesome-Align)
132.7M
Locally fine-tuned masked-language-model checkpoint, including its alignment-training heads.
Fanar-2-27B-Instruct
27B
Open-weight model served locally with vLLM.
Llama-4-Scout
109B total; 17B active
Open-weight mixture-of-experts model served locally with vLLM.
Gemini; GPT
Not disclosed
Proprietary models accessed through their providers’ Batch APIs.
Appendix
Table 6: Model sizes and access methods.
Parameter
Value
Parameter
Value
Parser
Biaffine
Epochs
20
BERT layers mixed
4
BERT pooling
Mean
Arc MLP
500
Relation MLP
100
Encoder dropout
0.1
MLP dropout
0.33
Learning rate
5×10−5
BERT LR rate
10
Batch token budget
5,000
Update steps
4
Appendix
Table 7: Hyperparameters for the reported CAMeLBERT-Mix supervised parsers.
System/stage
Configuration
SimAlign
Frozen CAMeLBERT-Mix encoder; BPE tokenization; layer 8; intersection of the bidirectional argmax alignments. No alignment-model training.
Awesome-Align training
CAMeLBERT-Mix initialization; MLM, TLM, full bidirectional TLM, SO, and PSI objectives; contrastive objective disabled; 5 epochs (1,645 optimizer updates); batch size 64; gradient accumulation 1; maximum block length 128; learning rate 2×10−5 ; weight decay 0.01; 165 warmup updates; maximum gradient norm 1.0; seed 42; one A100 GPU with TF32.
Both directions for every pair; 5 IBM Model 1, 5 HMM, 3 Model 3, and 3 Model 4 EM iterations; Models 2 and 5 disabled; 50 word classes; 4 threads per direction and 8 concurrent pair jobs. Directional links were combined with grow-diag-final-and.
Appendix
Table 8: Training and extraction settings for the alignment experiments.
System
Time (min)
Hardware/access
API cost (USD)
Gemini-3.6-Flash
41.7
Hosted Batch API
$30.92
GPT-5.6-Sol
29.3
Hosted Batch API
$35.31
Fanar-2-27B-Instruct
14.0
4 × A100 80GB
–
Llama-4-Scout
27.4
4 × A100 80GB
–
All Treebanks parser
0.21
1 × A100 80GB
–
Appendix
Table 9: Inference runtime on the evaluated data and hosted API cost. The parser measurement uses projective multi-root decoding and excludes model loading.
Model
Training Data / Prompting
Beirut
Cairo
Doha
Rabat
Tunis
MSA
Dialects
All
Supervised
MadarTB Beirut
79.6
76.1
80.0
69.3
75.1
69.8
75.9
74.8
Supervised
MadarTB Cairo
70.9
80.8
78.7
66.2
77.3
72.4
74.7
74.3
Supervised
MadarTB Doha
71.2
77.0
82.9
66.9
75.8
74.3
74.5
74.5
Supervised
MadarTB Rabat
63.7
72.1
73.1
79.4
76.3
70.2
73.0
72.5
Supervised
MadarTB Tunis
64.5
72.3
74.9
69.6
83.0
71.5
72.8
72.6
Supervised
MadarTB MSA
59.8
68.9
69.4
62.9
68.0
82.5
65.7
68.7
Appendix
Table 10: test-set parsing results (LAS) for supervised parsers and prompted LLMs. All Supervised rows use the CAMeLBERT-Mix Biaffine model and projective multi-root decoding; their configurations differ only in training data. All Treebanks denotes PATB, CamelTB, ARZTB, Gumar, and MadarTB . Bold indicates the best result overall, and underlining indicates the best LLM result.
Claim
System A
System B
Set
Δ LAS
95% CI
Development set
Dialect pooling
MadarTB Dialects
MadarTB Beirut
Dialects
9.9
[8.6, 11.2]
MadarTB Dialects
MadarTB Cairo
Dialects
11.1
[9.8, 12.4]
MadarTB Dialects
MadarTB Doha
Dialects
10.3
[9.0, 11.6]
MadarTB Dialects
MadarTB Rabat
Dialects
12.0
[10.5, 13.5]
MadarTB Dialects
MadarTB Tunis
Dialects
12.7
[11.2, 14.3]
Appendix
Table 11: Paired sentence-ID-clustered percentile bootstrap intervals for principal development- and test-set LAS comparisons (10,000 replicates per comparison). Δ LAS is System A minus System B in absolute percentage points. Within each split, every replicate resamples all 150 underlying sentence IDs with replacement and retains every relevant parallel variety realization.
Model
Training Data
Decoding
Gumar
ARZTB
CamelTB
PATB
MadarTB All
CamelParser2.0
PATB + CamelTB
Single-root
72.7
73.2
87.5
91.3
76.0
CAMeLParser-Dialects
PATB + CamelTB + ARZTB + Gumar
Single-root
80.3
84.2
87.2
90.8
81.2
Ours
All Treebanks
Single-root
80.4
84.1
87.5
90.7
86.5
Ours
All Treebanks
Multi-root
88.0
84.7
89.2
91.2
87.5
Appendix
Table 12: Test-set LAS on established Arabic dependency treebanks and MadarTB . The published baselines are CamelParser2.0 ( Elshabrawy et al., 2023 ) and CAMeLParser-Dialects ( Elshabrawy et al., 2026 ) . “All Treebanks” denotes PATB, CamelTB, ARZTB, Gumar, and MadarTB . Bold indicates the best result for each test set.
Corpus size
Lexicon and length
Tree structure
Roots
Variety
Sent.
Tokens
Unique
Sent. len.
Max depth
Dep. dist.
Roots/sent.
Multi-root (%)
Beirut
1,000
8,462
1,948
8.46
3.03
1.97
1.21
16.9
Cairo
1,000
8,643
2,005
8.64
3.27
2.01
1.14
10.9
Doha
1,000
7,768
1,846
7.77
2.97
1.94
1.20
16.7
Rabat
1,000
8,925
2,089
8.93
3.26
2.02
1.19
15.7
Tunis
1,000
8,469
2,020
8.47
3.21
2.01
1.19
16.3
Appendix
Table 13: Descriptive statistics for the six parallel treebanks. Unique forms are distinct token forms. Mean maximum depth assigns roots depth zero. Mean dependency distance is the absolute difference between the token positions of a non-root dependent and its head, averaged first within each sentence and then across sentences. Roots/sent. is the mean number of tokens with HEAD=0 per sentence; multi-root is the percentage of sentences containing more than one such token.
POS
Beirut
Cairo
Doha
Rabat
Tunis
MSA
All
NOM
50.5
53.3
49.9
50.8
50.2
49.2
50.7
PNX
17.1
17.3
19.2
17.0
17.7
15.6
17.3
PRT
19.0
16.2
13.8
16.7
14.5
22.1
17.2
VRB
11.2
10.8
14.5
13.4
15.4
11.0
12.7
PROP
2.2
2.4
2.6
2.0
2.1
1.7
2.1
VRB-PASS
0.0
0.0
0.0
0.0
0.0
0.5
0.1
Appendix
Table 14: POS distributions (%) over all data splits.
Relation
Beirut
Cairo
Doha
Rabat
Tunis
MSA
All
MOD
45.1
44.4
42.9
49.5
44.7
43.8
45.1
OBJ
21.4
19.8
21.3
19.9
22.9
21.3
21.1
—
15.1
13.9
16.5
14.2
14.8
13.8
14.7
IDF
10.3
11.3
9.5
8.7
9.9
11.4
10.2
SBJ
6.9
8.9
8.5
6.4
6.6
7.4
7.4
PRD
0.8
1.3
0.9
0.7
0.8
1.9
1.1
Appendix
Table 15: Dependency-relation distributions (%) over all data splits.
Variety pair
All tokens
Aligned token pairs
Head-aligned token pairs
Pair
Aligned
Token POS
Head also
Token+head POS
Relation
Direction
Complete
(%)
match (%)
aligned (%)
match (%)
match (%)
match (%)
match (%)
Beirut
Cairo
85.6
90.8
67.1
86.4
93.4
98.3
81.6
Beirut
Doha
87.3
89.7
71.4
79.2
93.9
99.2
76.0
Beirut
MSA
82.5
87.3
58.6
75.3
90.7
99.1
70.7
Beirut
Rabat
82.8
88.0
65.6
74.6
92.6
99.0
70.8
Appendix
Table 16: Alignment-level structural agreement from the selected Awesome-Align links by unordered variety pair; all values are percentages. The columns follow the successive evaluation subsets shown in the header. Aligned is the mean of the two varieties’ token-coverage percentages. For aligned token pairs, we report token POS match and the percentage whose heads also align. Both POS requires agreement in token and head POS. The final four measures are evaluated on this head-aligned subset; complete match requires both POS, relation, and direction to all match.
Manual evaluation data
Sentence pairs
Source tokens
Target tokens
Manual links
Complete annotated sample
300
2,545
2,615
2,172
Aligner
Predicted links
Precision
Recall
F 1
SimAlign
2,036
91.9
86.1
88.9
Awesome-Align
2,227
91.1
93.4
92.2
MGIZA++
2,215
79.0
80.6
79.8
Appendix
Table 17: Manual one-to-one alignment data and system evaluation. The upper block describes the complete sample annotated by two annotators; the lower block reports each system’s predicted links and micro-averaged precision, recall, and F 1 . These metrics compare each predicted link set with the manually annotated non-null links.
Dialectal Arabic (DA) remains under-resourced compared to Modern Standard Arabic (MSA), particularly for machine reading comprehension (MRC) and question answering (QA). Existing Arabic QA benchmarks primarily focus on formal written MSA or multiple-choice QA, with limited coverage of naturally spoken dialects. Here, we aim to bridge this gap. We introduce EDRAC, the first large-scale benchmark for dialectal Arabic machine reading comprehension (MRC) and generative QA, covering five major dialects: Egyptian, Moroccan, Emirati, Syrian, and Saudi Arabic. EDRAC contains 499 passages derived from naturally occurring spoken interactions and 4,977 corresponding QA pairs generated through a human--LLM collaborative pipeline combining iterative generation, LLM-as-a-judge evaluation, and human verification. We benchmark Arabic-centric and multilingual LLMs on EDRAC using lexical and semantic metrics. Our results reveal substantial gaps between semantic answer quality and dialectal fidelity, highlighting the limitations of existing evaluation metrics for dialectal Arabic generation. EDRAC provides a realistic and challenging MRC benchmark for future research on dialectal Arabic NLP.
Noor Abo Mokh, Kirill Chirkunov, Teresa Lynn +15
Mohamed bin Zayed University of Artificial Intelligence · New York University Abu Dhabi · IBM Research AI +1
Arabic dialects have long been under-represented in Natural Language Processing (NLP) research due to their non-standardization and high variability, which pose challenges for computational modeling. Recent advances in the field, such as Large Language Models (LLMs), offer promising avenues to address this gap by enabling Arabic to be modeled as a pluricentric language rather than a monolithic system. This paper presents Aladdin-FTI, our submission to the AMIYA shared task. The proposed system is designed to both generate and translate dialectal Arabic (DA). Specifically, the model supports text generation in Moroccan, Egyptian, Palestinian, Syrian, and Saudi dialects, as well as bidirectional translation between these dialects, Modern Standard Arabic (MSA), and English. The code and trained model are publicly available.
Jonathan Mutal, Perla Al Almaoui, Simon Hengchen +1
Faculté de traduction et d’interprétation, Université de Genève
LLMs perform strongly across NLP, but their ability to produce explicit grammatical analyses remains unclear. Arabic provides a challenging testbed due to its rich morphology and orthographic ambiguity, which create strong morphology-syntax interactions. We present a unified evaluation of LLMs on Arabic morphosyntactic tagging and dependency parsing, covering pre-tokenized, raw-text, and cascaded settings. We compare zero-shot prompting with retrieval-based in-context learning. Relevant demonstrations substantially improve performance. The strongest LLMs approach supervised tagging and parsing systems; however, they require substantial annotated data for demonstration retrieval and considerable computational resources. We make all code and data used in this paper publicly available.
Mohamed Adel, Bashar Alhafni, Nizar Habash
Computational Approaches to Modeling Language Lab 1New York University Abu Dhabi · 2Mohamed bin Zayed University of Artificial Intelligence