Judgements of sexism are subjective and may vary across communities, yet sexism detection datasets usually reduce annotator judgements to a single majority label, obscuring disagreement and how models behave on contested cases. We introduce MuDAS, a dataset of YouTube comments in Saudi, Egyptian and Moroccan Arabic, each labelled by three native speakers of its dialect, with the individual annotations preserved. We use it to evaluate Arabic encoders fine-tuned with hard-label, soft-label and perspectivist objectives, GPT-5.6 Terra, and four Arabic-centric large language models. Model errors are more frequent on comments with annotator disagreement. Despite relatively close Macro-F1 scores, GPT-5.6 Terra and the fine-tuned encoders exhibit contrasting error patterns: GPT-5.6 Terra, even with the annotators' definition of sexism, over-classifies gender-related content such as religious advice on modesty, whereas the encoders miss implicit and prescriptive sexism. The Arabic-centric models also over-predict sexism and remain below the encoders. Disagreement-aware learning preserves Macro-F1, but soft labels improve alignment with annotators on contested comments at the expense of unanimous ones, and neither paradigm reliably identifies contested comments. Lower performance on Egyptian Arabic, clearest for the large language models, is not fully explained by the examined measures of test-set composition and dialectness. Majority-label scores can therefore hide contrasting error patterns that matter for content moderation.
Figures & tables
Dataset
Source
Dialect(s)
Size
Ann./item
Indiv.
Let-Mi [ 30 ]
Twitter
Levantine
6,550
3
No
ArMI [ 29 ]
Twitter
Gulf, Egyptian, Levantine
9,833
n/r
No
Guellil et al. [21]
YouTube
Algerian
3,798
3
No
ArMIS [ 6 ]
Twitter
Mixed
964
3
Yes
Alhayan et al. [5]
X
Not specified (Saudi context)
7,487
1
No
MuDAS (ours)
YouTube
Saudi, Egyptian, Moroccan
6,731
3
Yes
Table 1: Existing Arabic sexism and misogyny datasets. Size : number of annotated items; Ann./item : annotators per item; Indiv. : whether the individual annotations are kept, so that disagreement can be studied. In MuDAS, each dialect is labelled by three native speakers of that dialect. Guellil et al. [21] kept only the comments on which all three annotators agreed, and Alhayan et al. [5] had each tweet labelled by one of six annotators. n/r: not reported.
Figure 1: Construction of MuDAS. Bar widths are proportional to the number of comments at each stage; the orange segment marks the comments predicted as sexist by the pre-screening model, which made up 41.9% of the sampled comments.
Saudi
Egyptian
Moroccan
All
Videos
25
24
19
68
Comments (final)
2,257
2,149
2,325
6,731
Sexist (majority label)
42.7%
34.5%
49.2%
42.3%
Annotators labelling the comment sexist ( s )
0/3
37.8%
50.4%
37.1%
41.6%
1/3
19.5%
15.1%
13.6%
16.1%
Table 2: Statistics of the final dataset. s is the proportion of the three annotators who labelled a comment sexist; unanimous comments are those with s=0 or s=1 , and contested comments those with s∈{31,32} . Agreement is computed on the sexism label.
Figure 2: Annotator agreement in each dialect. (a) Distribution of the number of annotators, out of three, who labelled a comment sexist. (b) Proportion of comments labelled sexist by each annotator (dots) and by the majority label (vertical line); the grey bar spans the three annotators of each dialect.
Saudi
Egyptian
Moroccan
Mixed
Training pool
1,805
1,719
1,860
5,384
Training (encoders)
1,534
1,461
1,581
4,576
Validation (encoders)
271
258
279
808
Test
452
430
465
1,347
Table 3: Number of comments in each split. The mixed-dialect split is the union of the three dialect splits.
Macro-F1 ↑
MD ↓
Model
Paradigm
SA
EG
MA
Mix
SA
EG
MA
Mix
Majority class
–
0.364
0.396
0.337
0.366
–
–
–
–
Constant median
–
–
–
–
–
0.727
0.730
0.776
0.745
TF-IDF + LR
Hard
0.686
0.563
0.728
0.702
–
–
–
–
Encoders
MARBERT
Hard
0.750
0.745
0.743
0.759
0.518
0.515
0.566
0.516
Table 4: Classification (Macro-F1) and distribution alignment (MD) on each in-dialect test set (SA, EG, MA) and on the mixed-dialect test set (Mix). The encoders in the Mix column were trained on the pooled data. Encoder scores are means over three seeds; GPT-5.6 Terra scores are means over three runs. LLMs use the English few-shot prompt. The best score in each column is in bold. † Between 98.7% and 99.6% valid replies. ‡ Computed only on the 389 of the 1,347 comments (29%) with a valid reply, which include fewer contested comments than the full test set (26% against 31%), so this score is not comparable with the rest of the column; on the same 389 comments, the hard-label prompt obtained a Macro-F1 of 0.691. n/a: the perspectivist prompts were not run on the mixed set. The mean rows average the models above; they include the scores marked † . § Mean over four LLMs, excluding Fanar ( ‡ ).
Model
Paradigm
SA
EG
MA
Mix
Disagreement F1 (predicted contested share p^ )
All contested
–
0.506 (0.34)
0.399 (0.25)
0.498 (0.33)
0.470 (0.31)
MARBERT
Soft
0.500 (0.50)
0.385 (0.35)
0.453 (0.39)
0.489 (0.49)
Persp.
0.405 (0.28)
0.335 (0.21)
0.422 (0.33)
0.423 (0.29)
AraBERT
Soft
0.513 (0.45)
0.416 (0.56)
0.486 (0.41)
0.455 (0.37)
Persp.
0.401 (0.26)
0.354 (0.23)
0.419 (0.29)
0.356 (0.21)
Table 5: Identification of contested comments. Top: Disagreement F1, with the predicted contested share p^ in parentheses; the reference row marks every comment as contested and gives the observed contested share. Scores above this reference are underlined. Bottom: Disagreement AUROC, using s^(1−s^) as the score; for GPT-5.6 Terra, s^ is averaged over the three runs, and the hard-label prompt is not shown, since the spread of its three runs reflects sampling variation rather than predicted disagreement.
Figure 3: Share of comments labelled sexist by the hard-label decision at each level of annotator agreement, on the mixed-dialect test set. The solid grey line is the majority label; the dotted line is the expected share for a single annotator drawn at random from the three. Encoder decisions threshold the mean of the three seeds’ predicted probabilities at 0.5; the LLMs use the English hard-label prompt, few-shot unless marked otherwise.
Figure 4: Difference in MD between each disagreement-aware encoder and the corresponding hard-label encoder, on the contested and the unanimous comments, with 95% bootstrap confidence intervals (2,000 paired resamples, stratified by dialect). Negative values mean closer alignment with the annotators. Filled markers: the interval excludes zero. Values are computed on the seed-averaged predictions.
Category
Majority
All four
GPT FP
Encoder FN
Total
Religious or modesty prescription
not sexist
2
35
–
38
Neutral debate on roles, money or law
not sexist
4
22
–
31
Attack on one specific woman
not sexist
11
12
–
26
Derogatory generalisation about women
not sexist
11
8
–
19
Counter-speech
not sexist
2
5
–
14
Criticism of men
not sexist
1
1
–
5
Table 6: Categories of the 245 coded errors on the mixed-dialect test set. Majority : majority label of the comments in each category. All four : misclassified by AraBERT, MARBERT and both GPT-5.6 Terra prompts; GPT FP : false positives of both GPT-5.6 Terra prompts only; Encoder FN : false negatives of both encoders only. Total also includes the two smaller reverse groups. Not shown: other (11 comments), opposition to feminism with a majority label of not sexist (3) and a doubtful positive (1).
Comment (translation)
Category
Error
يا اختي الحجاب فرض علينا كمسلمات مش اختياري (“Sister, the hijab is an obligation for us as Muslim women, not a choice”; Egyptian, s=0 )
Religious or modesty prescription
GPT FP
لازم احد يساعد في البيت اذا كانت الزوجه تكرف دايم وتدرس متى تفضى لزوجها ؟ (“Someone has to help at home if the wife is always working hard and studying; when will she have time for her husband?”; Saudi, s=0 )
Neutral debate on roles, money or law
GPT FP
صح واجب الرجل و واجب الزوجة الطاعة وتربية الابناء (“Right, the man’s duty, and the wife’s duty is obedience and raising the children”; Saudi, s=32 )
Prescriptive or benevolent gender roles
Encoder FN
اقسم بالله كلامك صح انا صيدلاني وغالبية الادوية النفسية تستهلكها العوانس والمطلقات (“I swear you are right; I am a pharmacist, and most psychiatric medication is taken by spinsters and divorced women”; Egyptian, s=1 )
Generalisation within a longer comment
Encoder FN
Table 7: Examples of coded errors, with the dialect and the proportion s of annotators who labelled the comment sexist.
Figure 5: Gap of the Egyptian test set (Egyptian Macro-F1 minus the mean of the Saudi and Moroccan values), observed and expected from the composition of the test sets, with 95% bootstrap confidence intervals (2,000 resamples of each dialect test set). Filled markers: the interval excludes zero. GPT-5.6 Terra is shown with both the few-shot and the zero-shot prompt.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Theme
Saudi
Egyptian
Moroccan
Total
Marriage and divorce
7
5
7
19
Work and independence
2
1
2
5
Appearance and behaviour
3
4
4
11
Roles and relationships
12
10
5
27
Controversial issues
1
4
1
6
Total
25
24
19
68
Appendix
Table A.1: Number of selected videos by theme and dialect.
Theme
Arabic keyword
English translation
Marriage and divorce
العزوف عن الزواج / عنوسة
Avoidance of marriage / spinsterhood
الخلع وحقوق المرأة في الطلاق
Khul’ (divorce initiated by the wife) and women’s rights in divorce
النفقة وحقوق المطلقة
Alimony and the rights of divorced women
تعدد الزوجات
Polygamy
الولاية في الزواج
Male guardianship in marriage
فارق السن في الزواج
Age gap in marriage
Appendix
Table A.2: The 32 search keywords used to search for videos, grouped into five themes.
Type
Saudi
Egyptian
Moroccan
Translation
Stereotyping and objectification
أغلب الحريم تفكيرهم سطحي
أغلب الستات تفكيرهم سطحي
أغلب العيالات تفكيرهم سطحي
Most women think superficially
Insult and belittling
الحريم ما عندهم عقل
الستات مفيش عندهم عقل
العيالات ما عندهمش عقل
Women have no brains
Dominance
مكانها البيت
مكانها البيت
بلاصتها فالدار
Her place is in the home
Violence and harassment
المفروض أحد يضربها عشان تتأدب
المفروض حد يضربها عشان تتأدب
خاص شي واحد يضربها باش تتأدب
Someone should beat her to teach her manners
Endorsement and justification
هو ماله ذنب، هي اللي استفزته
هو ماله ذنب، هي اللي استفزته
هو ما عندوش ذنب، هي اللي حرّكاتو
It is not his fault; she provoked him
Doubt and victim blaming
لو تحترمين نفسك ما صار لك شي
لو بتحترمي نفسك مكانش حصل لك حاجة
لو كنتي كتحترمي راسك ما كان وقع ليك والو
If you respected yourself, nothing would have happened to you
Appendix
Table C.1: Examples of the eight types of sexism in the annotation guidelines, in each dialect.
Category
Saudi
Egyptian
Moroccan
Translation
Praise of women
ما شاء الله عليها، امرأة ناجحة
ما شاء الله عليها، ست ناجحة
ما شاء الله عليها، مرا ناجحة
Mashallah, she is a successful woman
Neutral text
الوزيرة أعلنت القرار الجديد
الوزيرة أعلنت القرار الجديد
الوزيرة أعلنت القرار الجديد
The (female) minister announced the new decision
Criticism of a specific woman, unrelated to her gender
مو متفق مع رأي الوزيرة
مش متفق مع رأي الوزيرة
ماشي متفق مع رأي الوزيرة
I do not agree with the minister’s view
A woman’s own account of mistreatment
أنا أشتغل ونتقاسم المصاريف، ومع ذلك يعاملني بطريقة سيئة
أنا بشتغل وبنتقاسم المصاريف، ومع ذلك بيعاملني بطريقة مهينة
أنا كنخدم وكنقسمو المصاريف، ومع هادشي كيعاملني بطريقة مهينة
I work and we share the expenses, yet he treats me badly
Criticism of discrimination or oppressive practices
مو لأنها مو محجبة يعني تستاهل التحرش
مش لأنها مش محجبة يبقى تستاهل التحرش
ماشي حيت ماحجباش يعني كتستاهل التحرش
Not wearing a hijab does not mean she deserves to be harassed
Appendix
Table C.2: Examples of the five categories of non-sexist comments in the annotation guidelines, in each dialect.
Macro-F1
MD (all)
MD (contested)
MD (unanimous)
Model
Test
Soft − hard
MARBERT
SA
− 0.010 [ − 0.041, 0.021]
0.004 [ − 0.027, 0.034]
− 0.169 [ − 0.215, − 0.127]
0.092 [0.054, 0.129]
EG
− 0.019 [ − 0.056, 0.016]
− 0.030 [ − 0.059, − 0.001]
0.067 [0.009, 0.124]
− 0.062 [ − 0.095, − 0.030]
MA
0.019 [ − 0.014, 0.052]
− 0.048 [ − 0.079, − 0.017]
− 0.122 [ − 0.167, − 0.081]
− 0.011 [ − 0.049, 0.030]
Mix
0.011 [ − 0.007, 0.029]
− 0.007 [ − 0.028, 0.014]
− 0.258 [ − 0.286, − 0.229]
0.104 [0.080, 0.128]
Appendix
Table E.1: Differences between paradigms with 95% bootstrap confidence intervals (2,000 paired resamples, stratified by dialect). Each cell gives the disagreement-aware model minus the hard-label model; a negative MD difference means closer alignment with the annotators. All values are computed on the seed-averaged predictions, so they differ from the differences between the seed means in Table 4 . Intervals that exclude zero are in bold.
Figure F.1: Effect of the prompt language (English, EN; Arabic, AR) and of few-shot examples on the LLMs, on each test set. Lines connect the zero-shot and few-shot results of the same language. The grey band spans the reference values of the four test sets: the majority class for Macro-F1 and the constant median of s for MD. GPT-5.6 Terra scores are means over three runs.
Macro-F1 ↑
MD ↓
English
Arabic
English
Arabic
Model
Paradigm
Zero
Few
Zero
Few
Zero
Few
Zero
Few
GPT-5.6 Terra
Hard
0.774
0.785
0.774
0.781
0.511
0.493
0.507
0.504
Soft
0.785
0.803
0.779
0.792
0.463
0.415
0.481
0.430
Persp.
–
0.798
–
0.785
–
0.427
–
0.464
ALLaM
Hard
0.594
0.668
0.559
0.613
0.820
0.693
0.872
0.801
Appendix
Table F.1: Effect of prompt language and of few-shot examples on the LLMs. Scores are averaged over the three in-dialect test sets. The perspectivist prompts are few-shot only. GPT-5.6 Terra scores are means over three runs. † Fewer than 100% valid replies on at least one test set.
Prompt
Definition
Macro-F1
Precision
Recall
FP
FN
Ratio
Zero-shot
Gender-neutral
0.747
0.649
0.875
269.7
71.3
1.35
Annotators’ (main)
0.776
0.682
0.885
235.3
65.3
1.30
Few-shot
Gender-neutral
0.780
0.682
0.897
238.0
58.7
1.31
Annotators’ (main)
0.799
0.710
0.888
206.3
64.0
1.25
Appendix
Table G.1: GPT-5.6 Terra with the gender-neutral definition of sexism in the prompt and with the definition given to the annotators, used in the rest of the paper, on the mixed-dialect test set (English hard-label prompts). Precision and recall refer to the sexist class. Ratio : number of comments predicted as sexist divided by the number whose majority label is sexist. The table reports means over three runs; the bootstrap confidence intervals in the text are based on the majority vote over runs.
Macro-F1
Err. con. − una.
Ratio
Model
All
No gold
All
No gold
All
No gold
MARBERT
0.768
0.757
0.189
0.186
0.94
0.95
AraBERT
0.785
0.775
0.198
0.193
0.95
0.95
GPT-5.6 Terra (few)
0.808
0.797
0.205
0.199
1.24
1.27
GPT-5.6 Terra (zero)
0.784
0.774
0.170
0.165
1.28
1.31
ALLaM
0.692
0.681
0.134
0.126
1.15
1.17
Appendix
Table H.1: Results on the mixed-dialect test set with all 1,347 comments and without the 90 gold items (1,257 comments), for the hard-label decisions used in the error analysis (Section 6 ). Err. con. − una. : error rate on contested minus error rate on unanimous comments. Ratio : comments predicted as sexist divided by comments whose majority label is sexist. The encoders threshold the mean of the seeds’ probabilities, and GPT-5.6 Terra takes the majority vote over its three runs, so Macro-F1 differs slightly from Table 4 .
Saudi
Egyptian
Moroccan
Model
A1
A2
A3
A1
A2
A3
A1
A2
A3
Labelled sexist (%)
46.9
46.2
36.9
34.2
35.3
30.2
34.8
53.3
54.0
MARBERT
72.6
73.7
75.4
79.1
77.9
73.3
69.9
71.6
72.7
AraBERT
74.3
75.4
72.3
76.7
77.0
71.9
75.5
76.3
75.3
GPT-5.6 Terra
72.8
80.1
73.5
75.3
74.2
66.3
73.1
83.0
84.5
ALLaM
66.8
69.2
66.6
60.0
60.7
56.5
66.7
69.7
71.6
Appendix
Table I.1: Agreement (%) between the hard-label decision of each model and the label of each annotator on the in-dialect test sets. The first row gives the share of test comments that each annotator labelled sexist. Encoder decisions threshold the mean of the seeds’ probabilities; GPT-5.6 Terra takes the majority vote over its three runs with the English few-shot prompt; the other LLMs use the English few-shot hard-label prompt.
We present ArabicDialectSafety, a human-curated Arabic safety dataset of 25,071 prompts covering six Arabic varieties: Modern Standard Arabic, Syrian, Egyptian, Algerian, Palestinian, and Moroccan. The dataset is annotated with dialect labels and seven fine-grained harm categories. We introduce a dual-task evaluation framework for binary safe/unsafe detection and granular harm classification across dialects. Benchmarking seven supervised and generative models, we find that fine-tuned MARBERTv2 achieves the strongest performance, with Macro-F1 scores of 0.95 for binary classification and 0.90 for granular classification, substantially outperforming prompted frontier LLMs, including Arabic-specialized models. Our analyses show that dialect conditioning is most effective when integrated at the representation level, while significant performance gaps remain for low-resource Maghrebi dialects. We further evaluate seven frontier LLMs as response generators on harmful dialectal Arabic prompts and observe unsafe generation rates below 5 percent across models. We release the dataset and code upon acceptance to support future research on dialect-aware Arabic safety evaluation. Warning: This paper contains examples of harmful and potentially offensive content included solely for research purposes.
Large language models are increasingly deployed in Arabic-speaking markets, yet standard benchmarks overwhelmingly reward Modern Standard Arabic (MSA) fluency while leaving dialectal and culturally grounded competence unmeasured. This gap is consequential: everyday Arabic is largely dialectal, and dialect encodes social meaning that MSA-centric evaluation cannot capture. We present a rubric-based benchmark for the Saudi dialect, comprising 31 expert-authored prompts spanning idiomatic, pragmatic, lexical, and culturally-embedded phenomena, each paired with an expert-established ground truth. Our methodology separates evaluation into a model-agnostic phase, in which atomic, MECE positive criteria are derived solely from the ground truth, and a model-specific phase, in which four state-of-the-art systems -- Claude Opus 5, Gemini 3.7, GPT-5.6, and Kimi K3 -- are scored against those criteria and penalised for errors they actively introduce. Across 124 model-prompt evaluations we catalogue 466 error instances under a nine-category taxonomy. The four systems cluster within a narrow macro-average band (42.7%-53.1%), with no model exceeding 55% and every model recording at least one negative-scoring prompt, confirming that Saudi dialectal competence remains broadly unsolved. Notably, Ambiguous Framing is the dominant failure mode (37.3% of errors) while outright Hallucination accounts for only 11.2%, indicating that models fail less by stating falsehoods than by distorting register and flattening pragmatic nuance. We further observe a consistency-versus-ceiling trade-off and model-distinctive error signatures. We release the full prompt set, ground truths, and scored rubrics to support reproducible dialectal evaluation.
Ghassan Al-Sumaidaee, Sajjad Abdoli, Ahmed Rashad +1
Research on gender-related online content is often operationalized through harm-centered tasks that collapse non-harmful gender-related and unrelated content, while existing datasets remain concentrated in high-resource languages and Global North settings. To bridge these gaps, we introduce GENDIS, a dataset of 9,600 multilingual, human-annotated YouTube comments from India, Kenya, and Nigeria (3,200 per country), annotated using a two-level framework that first identifies whether gender is referenced and then classifies gender-related content as negative, positive, or neutral. This separation provides a denominator for gender-related discourse independent of harm, enabling both prevalence estimation and characterization of its evaluative range. In the uniform random stratum, 23.8-35.0% of comments are gender-related, and among those with a resolved stance, 82% are neutral or positive. Across models, gender-reference detection is substantially easier than stance classification, especially for positive and neutral speech. Lightly supervised classifiers match frontier zero-shot large language models at substantially lower cost and outperform generic toxicity detection, while cross-country transfer remains limited. GENDIS provides a locally grounded resource for measuring when gender is invoked online and how that reference is framed.
Luise Koch, Manuel Tonneau, Niyati Malhotra +5
World Bank · Technical University of Munich · University of Oxford +2