Some Dialects Are More Equal Than Others: Non-Prestigious Arabic Dialectal Bias in LLMs

Mai Mohamed Eida; Ryan Dolan; Paul de Nijs; Jonathan Dunn

cs.CLSep 21, 2026

Some Dialects Are More Equal Than Others: Non-Prestigious Arabic Dialectal Bias in LLMs

Authors: Mai Mohamed Eida, Ryan Dolan, Paul de Nijs, Jonathan Dunn

Abstract

Previous work on Egyptian Arabic in NLP has focused largely on the prestigious Cairene Egyptian Arabic (CEA) dialect, resulting in a lack of representation for the less prestigious Sa'idi Egyptian Arabic (SEA) dialect both in LLM and resource development. Does this lack of representation influence an LLM's view of the acceptability of SEA (upstream), and does an upstream bias against SEA lead to worse performance (downstream)? We investigate the upstream effect of SEA dialectal features on LLM preferences in a Targeted Syntactic Evaluation (TSE) task which reveals a significant bias against SEA across multiple LLMs. We then analyze the effect of these same features on downstream model performance on MMLU benchmarks and show that models experience a degradation in performance when presented with SEA. This work highlights the need for further exploration on how sub-dialectal variation impacts language technologies.

Explore similar work

Side-by-side Comparison Amplifies Dialect Bias in Language Models
May 23, 2026Kritee Kondapally, Claire J. Smerdon, Pooja C. Patel +5Large Language Model Bias Dialects
Can Dialects Be Steered Like Languages? Sparse Neurons and Distributed Directions in Arabic LLMs
Jul 4, 2026Kareem Elozeiri, Mervat Abassy, Omar Kallas +4Dialects Arabic Natural Language Processing
Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models
Aug 30, 2026Ghassan Al-Sumaidaee, Sajjad Abdoli, Ahmed Rashad +1Arabic Fluency

May 23, 2026 · cs.CLJ/K move · Enter open · S save

Side-by-side Comparison Amplifies Dialect Bias in Language Models

Kritee Kondapally, Claire J. Smerdon, Pooja C. Patel, Ogheneyoma Akoni+4

University of Southern California, USA

Language models (LMs) can exhibit biases based on variations in their dialects, even in the absence of a dialect label, a behavior known as covert dialect bias. In this work, we quantify covert dialect bias in online discourse by evaluating how LMs associate stereotypical traits (derived from social psychology research on racial bias) with intent-equivalent tweets in Standard American English (SAE) and African-American Vernacular English (AAVE). While prior work shows that LMs associate more negative stereotypes with AAVE when evaluating tweets in isolation, we are surprised to find that this bias is significantly exacerbated when SAE / AAVE tweet pairs are compared side by side, a setting that more closely reflects high-impact decision making contexts in which models are used to rank candidates. The bias only worsens when dialect labels are explicitly specified. This is striking, given the extensive efforts from commercial developers to mitigate bias in their LMs. Encouragingly, we show that counterfactual fairness finetuning can mitigate covert dialect bias for some stereotypical traits, reducing average disparities when evaluating tweets in isolation, however, these improvements do not consistently hold across traits when evaluating SAE / AAVE tweets side by side. Our findings show that existing evaluation settings for covert dialect bias may underestimate its severity, specifically in contrastive settings. Additionally, overt dialect bias remains pronounced even after safety aligned finetuning, indicating that it remains an unresolved problem, and motivates the need for more robust evaluation and mitigation frameworks.

Large Language Model Bias Dialects Stereotype Mitigation Linguistics African Languages Comparison

Open paper Ask about it PDF

Abstract

Explore similar work

Side-by-side Comparison Amplifies Dialect Bias in Language Models

Can Dialects Be Steered Like Languages? Sparse Neurons and Distributed Directions in Arabic LLMs

Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models