Large language models (LLMs) often default to Modern Standard Arabic (MSA) when generating Arabic, even when prompted with dialectal Arabic. A natural explanation is that their internal representations are dominated by MSA. We test this hypothesis by adapting the language-dominance framework of Shani and Basirat (2025) (https://doi.org/10.18653/v1/2025.blackboxnlp-1.7) to 26 Arabic varieties. Across layers and model families, we find no evidence that MSA acts as a dominant internal representation for Arabic dialects. Instead, dialect representations form a dense and highly overlapping space: normalized mutual information drops sharply compared to patterns reported for more distinct languages. Moreover, the strongest separability effects are not confined to intermediate layers, but can shift toward later layers depending on the architecture. These findings challenge a common interpretation of MSA-biased generation: output preference does not necessarily reveal internal representational dominance. Analyses of multilingual and dialectal LLMs should therefore distinguish generation bias from the geometry of internal representations.
Figures & tables
Figure 1: Layer-wise NMI values for cross-lingual vs. cross-dialectal representations for mGPT (left) and BLOOM (right). Dialectal NMI is lower and remains low over a wider middle-layer range. Cross-lingual representations were computed using Parallel Universal Dependencies (PUD) treebanks ( Zeman et al., 2017 ) (following Shani and Basirat (2025) ), cross-dialectal representations were computed using the MADAR corpus ( Bouamor et al., 2018 ) .
Figure 2: Confusion matrices for dialect clustering in mGPT (left) and BLOOM (right) grouped by country. Clear diagonals show that dialect identity is preserved despite low separability.
Figure 3: Token-level dominance statistics. Percentage of tokens with Λ<1 . A mostly flatline percentage of tokens are being assigned to competing dialects.
Figure 4: Layer-wise NMI values for Meta-Llama-3.1-8B-Instruct. The model maintains relatively stable dialect separability across middle and later layers.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Supervised linear probe accuracy across layers for BLOOM. Accuracy ranges from 0.19 to 0.25.
Figure 8: Supervised linear probe accuracy across layers for mGPT. Accuracy ranges from 0.14 to 0.21.
Figure 9: Dialect dominance scores across layers for mGPT (top) and BLOOM (bottom). Dominance remains weak across layers.
Figure 10: Complete dialectal confusion matrices for mGPT (top) and BLOOM (bottom).
Figure 11: Replicated NMI curves for mGPT (top) and BLOOM (bottom).
Figure 12: Replicated confusion matrices for mGPT (left) and BLOOM (right).
Figure 13: Replicated layer-wise dominance scores for mGPT (top) and BLOOM (bottom).
Figure 14: Replicated token-level dominance statistics for mGPT (left) and BLOOM (right).
Figure 15: Replicated token-level Λ histograms for mGPT (left) and BLOOM (right).
Dialectal data are scarce relative to Modern Standard Arabic (MSA), causing Arabic LLMs to overproduce MSA and struggle with dialectally accurate generation. This raises a fundamental interpretability question about where and how dialectal features are encoded within model internals and whether these representations can improve dialect generation without fine-tuning. We study two inference-time approaches as interpretability probes and control mechanisms. First, neuron-level analysis identifies sparse populations that encode dialect-specific features and tests whether amplifying or suppressing them steers model outputs toward target dialects. Second, vector steering extracts dialect-specific activation directions and injects them during inference, motivated by feature entanglement at the neuron level. We find that these neurons are real but only partially explanatory. They occupy under 1% of MLP dimensions but span only 5% to 21% of the residual dialect direction. This limited coverage is causally consequential. Neuron steering reinforces dialect in some varieties when the prompt is already dialectal but cannot induce it from MSA prompts, whereas vector steering succeeds in both settings. Arabic dialects are therefore steerable mainly through distributed rather than localized representations
Kareem Elozeiri, Mervat Abassy, Omar Kallas +4
1Mohamed bin Zayed University of Artificial Intelligence · 2Qatar Computing Research Institute, Hamad Bin Khalifa University · 3Tohoku University +1
Previous work on Egyptian Arabic in NLP has focused largely on the prestigious Cairene Egyptian Arabic (CEA) dialect, resulting in a lack of representation for the less prestigious Sa'idi Egyptian Arabic (SEA) dialect both in LLM and resource development. Does this lack of representation influence an LLM's view of the acceptability of SEA (upstream), and does an upstream bias against SEA lead to worse performance (downstream)? We investigate the upstream effect of SEA dialectal features on LLM preferences in a Targeted Syntactic Evaluation (TSE) task which reveals a significant bias against SEA across multiple LLMs. We then analyze the effect of these same features on downstream model performance on MMLU benchmarks and show that models experience a degradation in performance when presented with SEA. This work highlights the need for further exploration on how sub-dialectal variation impacts language technologies.
There is a significant gap in evaluating cultural reasoning in LLMs using conversational datasets that capture culturally rich and dialectal contexts. Most Arabic benchmarks focus on short text snippets in Modern Standard Arabic (MSA), overlooking the cultural nuances that naturally arise in dialogues. To address this gap, we introduce ArabCulture-Dialogue, a culturally grounded conversational dataset covering 13 Arabic-speaking countries, in both MSA and each country's respective dialect, spanning 12 daily-life topics and 54 fine-grained subtopics. We utilize the dataset to form three benchmarking tasks: (i) multiple-choice cultural reasoning, (ii) machine translation between MSA and dialects, and (iii) dialect-steering generation. Our experiments indicate that the performance gap between MSA and Arabic dialects still exists, whereby the models perform worse on all three tasks in the dialectal setup, compared to the MSA one.
Muhammad Dehan Al Kautsar, Saeed Almheiri, Momina Ahsan +13
Mohamed bin Zayed University of Artificial Intelligence · IBM Research AI · American University in the Emirates