What Majority Labels Hide: Annotator Disagreement and Model Errors in Multi-Dialect Arabic Sexism Detection
Organizations: School of Electronic Engineering and Computer Science, Queen Mary University of London, Mile End Road, London, E1 4NS, United Kingdom
Abstract
Judgements of sexism are subjective and may vary across communities, yet sexism detection datasets usually reduce annotator judgements to a single majority label, obscuring disagreement and how models behave on contested cases. We introduce MuDAS, a dataset of YouTube comments in Saudi, Egyptian and Moroccan Arabic, each labelled by three native speakers of its dialect, with the individual annotations preserved. We use it to evaluate Arabic encoders fine-tuned with hard-label, soft-label and perspectivist objectives, GPT-5.6 Terra, and four Arabic-centric large language models. Model errors are more frequent on comments with annotator disagreement. Despite relatively close Macro-F1 scores, GPT-5.6 Terra and the fine-tuned encoders exhibit contrasting error patterns: GPT-5.6 Terra, even with the annotators' definition of sexism, over-classifies gender-related content such as religious advice on modesty, whereas the encoders miss implicit and prescriptive sexism. The Arabic-centric models also over-predict sexism and remain below the encoders. Disagreement-aware learning preserves Macro-F1, but soft labels improve alignment with annotators on contested comments at the expense of unanimous ones, and neither paradigm reliably identifies contested comments. Lower performance on Egyptian Arabic, clearest for the large language models, is not fully explained by the examined measures of test-set composition and dialectness. Majority-label scores can therefore hide contrasting error patterns that matter for content moderation.
Figures & tables
| Dataset | Source | Dialect(s) | Size | Ann./item | Indiv. |
| Let-Mi [ 30 ] | Levantine | 6,550 | 3 | No | |
| ArMI [ 29 ] | Gulf, Egyptian, Levantine | 9,833 | n/r | No | |
| Guellil et al. [21] | YouTube | Algerian | 3,798 | 3 | No |
| ArMIS [ 6 ] | Mixed | 964 | 3 | Yes | |
| Alhayan et al. [5] | X | Not specified (Saudi context) | 7,487 | 1 | No |
| MuDAS (ours) | YouTube | Saudi, Egyptian, Moroccan | 6,731 | 3 | Yes |
| Saudi | Egyptian | Moroccan | All | |
| Videos | 25 | 24 | 19 | 68 |
| Comments (final) | 2,257 | 2,149 | 2,325 | 6,731 |
| Sexist (majority label) | 42.7% | 34.5% | 49.2% | 42.3% |
| Annotators labelling the comment sexist ( ) | ||||
| 0/3 | 37.8% | 50.4% | 37.1% | 41.6% |
| 1/3 | 19.5% | 15.1% | 13.6% | 16.1% |
| Saudi | Egyptian | Moroccan | Mixed | |
| Training pool | 1,805 | 1,719 | 1,860 | 5,384 |
| Training (encoders) | 1,534 | 1,461 | 1,581 | 4,576 |
| Validation (encoders) | 271 | 258 | 279 | 808 |
| Test | 452 | 430 | 465 | 1,347 |
| Macro-F1 | MD | ||||||||
| Model | Paradigm | SA | EG | MA | Mix | SA | EG | MA | Mix |
| Majority class | – | 0.364 | 0.396 | 0.337 | 0.366 | – | – | – | – |
| Constant median | – | – | – | – | – | 0.727 | 0.730 | 0.776 | 0.745 |
| TF-IDF + LR | Hard | 0.686 | 0.563 | 0.728 | 0.702 | – | – | – | – |
| Encoders | |||||||||
| MARBERT | Hard | 0.750 | 0.745 | 0.743 | 0.759 | 0.518 | 0.515 | 0.566 | 0.516 |
| Model | Paradigm | SA | EG | MA | Mix |
| Disagreement F1 (predicted contested share ) | |||||
| All contested | – | 0.506 (0.34) | 0.399 (0.25) | 0.498 (0.33) | 0.470 (0.31) |
| MARBERT | Soft | 0.500 (0.50) | 0.385 (0.35) | 0.453 (0.39) | 0.489 (0.49) |
| Persp. | 0.405 (0.28) | 0.335 (0.21) | 0.422 (0.33) | 0.423 (0.29) | |
| AraBERT | Soft | 0.513 (0.45) | 0.416 (0.56) | 0.486 (0.41) | 0.455 (0.37) |
| Persp. | 0.401 (0.26) | 0.354 (0.23) | 0.419 (0.29) | 0.356 (0.21) | |
| Category | Majority | All four | GPT FP | Encoder FN | Total |
| Religious or modesty prescription | not sexist | 2 | 35 | – | 38 |
| Neutral debate on roles, money or law | not sexist | 4 | 22 | – | 31 |
| Attack on one specific woman | not sexist | 11 | 12 | – | 26 |
| Derogatory generalisation about women | not sexist | 11 | 8 | – | 19 |
| Counter-speech | not sexist | 2 | 5 | – | 14 |
| Criticism of men | not sexist | 1 | 1 | – | 5 |
| Comment (translation) | Category | Error |
| يا اختي الحجاب فرض علينا كمسلمات مش اختياري (“Sister, the hijab is an obligation for us as Muslim women, not a choice”; Egyptian, ) | Religious or modesty prescription | GPT FP |
| لازم احد يساعد في البيت اذا كانت الزوجه تكرف دايم وتدرس متى تفضى لزوجها ؟ (“Someone has to help at home if the wife is always working hard and studying; when will she have time for her husband?”; Saudi, ) | Neutral debate on roles, money or law | GPT FP |
| صح واجب الرجل و واجب الزوجة الطاعة وتربية الابناء (“Right, the man’s duty, and the wife’s duty is obedience and raising the children”; Saudi, ) | Prescriptive or benevolent gender roles | Encoder FN |
| اقسم بالله كلامك صح انا صيدلاني وغالبية الادوية النفسية تستهلكها العوانس والمطلقات (“I swear you are right; I am a pharmacist, and most psychiatric medication is taken by spinsters and divorced women”; Egyptian, ) | Generalisation within a longer comment | Encoder FN |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Theme | Saudi | Egyptian | Moroccan | Total |
| Marriage and divorce | 7 | 5 | 7 | 19 |
| Work and independence | 2 | 1 | 2 | 5 |
| Appearance and behaviour | 3 | 4 | 4 | 11 |
| Roles and relationships | 12 | 10 | 5 | 27 |
| Controversial issues | 1 | 4 | 1 | 6 |
| Total | 25 | 24 | 19 | 68 |
| Theme | Arabic keyword | English translation |
| Marriage and divorce | العزوف عن الزواج / عنوسة | Avoidance of marriage / spinsterhood |
| الخلع وحقوق المرأة في الطلاق | Khul’ (divorce initiated by the wife) and women’s rights in divorce | |
| النفقة وحقوق المطلقة | Alimony and the rights of divorced women | |
| تعدد الزوجات | Polygamy | |
| الولاية في الزواج | Male guardianship in marriage | |
| فارق السن في الزواج | Age gap in marriage |
| Type | Saudi | Egyptian | Moroccan | Translation |
| Stereotyping and objectification | أغلب الحريم تفكيرهم سطحي | أغلب الستات تفكيرهم سطحي | أغلب العيالات تفكيرهم سطحي | Most women think superficially |
| Insult and belittling | الحريم ما عندهم عقل | الستات مفيش عندهم عقل | العيالات ما عندهمش عقل | Women have no brains |
| Dominance | مكانها البيت | مكانها البيت | بلاصتها فالدار | Her place is in the home |
| Violence and harassment | المفروض أحد يضربها عشان تتأدب | المفروض حد يضربها عشان تتأدب | خاص شي واحد يضربها باش تتأدب | Someone should beat her to teach her manners |
| Endorsement and justification | هو ماله ذنب، هي اللي استفزته | هو ماله ذنب، هي اللي استفزته | هو ما عندوش ذنب، هي اللي حرّكاتو | It is not his fault; she provoked him |
| Doubt and victim blaming | لو تحترمين نفسك ما صار لك شي | لو بتحترمي نفسك مكانش حصل لك حاجة | لو كنتي كتحترمي راسك ما كان وقع ليك والو | If you respected yourself, nothing would have happened to you |
| Category | Saudi | Egyptian | Moroccan | Translation |
| Praise of women | ما شاء الله عليها، امرأة ناجحة | ما شاء الله عليها، ست ناجحة | ما شاء الله عليها، مرا ناجحة | Mashallah, she is a successful woman |
| Neutral text | الوزيرة أعلنت القرار الجديد | الوزيرة أعلنت القرار الجديد | الوزيرة أعلنت القرار الجديد | The (female) minister announced the new decision |
| Criticism of a specific woman, unrelated to her gender | مو متفق مع رأي الوزيرة | مش متفق مع رأي الوزيرة | ماشي متفق مع رأي الوزيرة | I do not agree with the minister’s view |
| A woman’s own account of mistreatment | أنا أشتغل ونتقاسم المصاريف، ومع ذلك يعاملني بطريقة سيئة | أنا بشتغل وبنتقاسم المصاريف، ومع ذلك بيعاملني بطريقة مهينة | أنا كنخدم وكنقسمو المصاريف، ومع هادشي كيعاملني بطريقة مهينة | I work and we share the expenses, yet he treats me badly |
| Criticism of discrimination or oppressive practices | مو لأنها مو محجبة يعني تستاهل التحرش | مش لأنها مش محجبة يبقى تستاهل التحرش | ماشي حيت ماحجباش يعني كتستاهل التحرش | Not wearing a hijab does not mean she deserves to be harassed |
| Macro-F1 | MD (all) | MD (contested) | MD (unanimous) | ||
| Model | Test | ||||
| Soft hard | |||||
| MARBERT | SA | 0.010 [ 0.041, 0.021] | 0.004 [ 0.027, 0.034] | 0.169 [ 0.215, 0.127] | 0.092 [0.054, 0.129] |
| EG | 0.019 [ 0.056, 0.016] | 0.030 [ 0.059, 0.001] | 0.067 [0.009, 0.124] | 0.062 [ 0.095, 0.030] | |
| MA | 0.019 [ 0.014, 0.052] | 0.048 [ 0.079, 0.017] | 0.122 [ 0.167, 0.081] | 0.011 [ 0.049, 0.030] | |
| Mix | 0.011 [ 0.007, 0.029] | 0.007 [ 0.028, 0.014] | 0.258 [ 0.286, 0.229] | 0.104 [0.080, 0.128] | |
| Macro-F1 | MD | ||||||||
| English | Arabic | English | Arabic | ||||||
| Model | Paradigm | Zero | Few | Zero | Few | Zero | Few | Zero | Few |
| GPT-5.6 Terra | Hard | 0.774 | 0.785 | 0.774 | 0.781 | 0.511 | 0.493 | 0.507 | 0.504 |
| Soft | 0.785 | 0.803 | 0.779 | 0.792 | 0.463 | 0.415 | 0.481 | 0.430 | |
| Persp. | – | 0.798 | – | 0.785 | – | 0.427 | – | 0.464 | |
| ALLaM | Hard | 0.594 | 0.668 | 0.559 | 0.613 | 0.820 | 0.693 | 0.872 | 0.801 |
| Prompt | Definition | Macro-F1 | Precision | Recall | FP | FN | Ratio |
| Zero-shot | Gender-neutral | 0.747 | 0.649 | 0.875 | 269.7 | 71.3 | 1.35 |
| Annotators’ (main) | 0.776 | 0.682 | 0.885 | 235.3 | 65.3 | 1.30 | |
| Few-shot | Gender-neutral | 0.780 | 0.682 | 0.897 | 238.0 | 58.7 | 1.31 |
| Annotators’ (main) | 0.799 | 0.710 | 0.888 | 206.3 | 64.0 | 1.25 |
| Macro-F1 | Err. con. una. | Ratio | ||||
| Model | All | No gold | All | No gold | All | No gold |
| MARBERT | 0.768 | 0.757 | 0.189 | 0.186 | 0.94 | 0.95 |
| AraBERT | 0.785 | 0.775 | 0.198 | 0.193 | 0.95 | 0.95 |
| GPT-5.6 Terra (few) | 0.808 | 0.797 | 0.205 | 0.199 | 1.24 | 1.27 |
| GPT-5.6 Terra (zero) | 0.784 | 0.774 | 0.170 | 0.165 | 1.28 | 1.31 |
| ALLaM | 0.692 | 0.681 | 0.134 | 0.126 | 1.15 | 1.17 |
| Saudi | Egyptian | Moroccan | |||||||
| Model | A1 | A2 | A3 | A1 | A2 | A3 | A1 | A2 | A3 |
| Labelled sexist (%) | 46.9 | 46.2 | 36.9 | 34.2 | 35.3 | 30.2 | 34.8 | 53.3 | 54.0 |
| MARBERT | 72.6 | 73.7 | 75.4 | 79.1 | 77.9 | 73.3 | 69.9 | 71.6 | 72.7 |
| AraBERT | 74.3 | 75.4 | 72.3 | 76.7 | 77.0 | 71.9 | 75.5 | 76.3 | 75.3 |
| GPT-5.6 Terra | 72.8 | 80.1 | 73.5 | 75.3 | 74.2 | 66.3 | 73.1 | 83.0 | 84.5 |
| ALLaM | 66.8 | 69.2 | 66.6 | 60.0 | 60.7 | 56.5 | 66.7 | 69.7 | 71.6 |
Annotator Disagreement
Annotator disagreement studies why human labels differ, how to measure and model that variation, and how it should shape AI training and evaluation.
See all 102 papers