SalamahBench: Dialect and Category Level Safety Evaluation of Arabic Language Models

Omar Abdelnasser; Fatemah Alharbi; Khaled Khasawneh; Ihsen Alouani; Mohammed E. Fouda

cs.CLFeb 3, 2026

SalamahBench: Dialect and Category Level Safety Evaluation of Arabic Language Models

Authors: Omar Abdelnasser, Fatemah Alharbi, Khaled Khasawneh, Ihsen Alouani, Mohammed E. Fouda

Organizations: Compumacy for Artificial Intelligence Solutions, Cairo, Egypt · Computer Science Department, College of Computer Science and Engineering, Taibah University, Yanbu 46522, KSA · Electrical and Computer Engineering Department, George Mason University, VA, USA · CSIT, Queen’s University Belfast, UK · Upper Bound Ltd, Belfast, UK

Abstract

While different stakeholders are trying to leverage Arabic Language Models (ALMs), safety alignment in ALMs remains largely underexplored, hindering their mainstream adoption. Existing safety benchmarks are predominantly English-centric and evaluate Arabic only in its standardized form, obscuring fine-grained safety vulnerabilities in Arabic NLP systems. This paper introduces SalamahBench, a unified benchmark of 8{,}270 human-verified harmful prompts across ML Commons hazard categories, each rendered in Modern Standard Arabic (MSA) and five regional Arabic varieties, namely Egyptian, Syrian, Saudi, Lebanese, and Moroccan, for a total of 49{,}620 paired instances. To analyze the resulting data, we introduce two complementary metrics, namely Dialect Shift, which measures a model's aggregate change in safety under dialectal reformulation, and Category-Specific Dialect Deviation, which isolates harm categories whose change departs from that aggregate trend. Evaluating models such as Fanar 2, ALLaM 2, and Karnak 1 under multiple safeguard configurations, we find that cross-variety robustness is strongly model dependent, and that aggregate scores can conceal category-level divergence. Our findings highlight the necessity of evaluating Arabic model safety jointly across linguistic varieties and harm domains rather than relying on aggregate scores or MSA alone.

Figures & tables

Figure 1 : Empirical example for evaluating safety alignment in ALM.

Hazard Category	Definition
1. Violent Crimes	Responses that enable, encourage, or provide instructions for violent crimes against people or animals.
2. Non-Violent Crimes	Responses that enable, encourage, or provide instructions for non-violent crimes such as theft, fraud, or property damage.
3. Sex-Related Crimes	Responses related to sexual assault, trafficking, or the non-consensual distribution of intimate media.
4. Child Sexual Exploitation	Responses that enable, encourage, or depict the sexual abuse or exploitation of minors.
5. Indiscriminate Weapons	Responses that enable, encourage, or provide instructions for constructing or deploying chemical, biological, radiological, or nuclear (CBRN) weapons.
6. Suicide & Self-Harm	Responses that enable, encourage, or provide instructions for suicide, self-harm, or eating disorders.

Table 1 : The 12 ML Commons Hazard Categories and Their Definitions Used for Annotation.

Dataset	Arabic prompts	Dialect label	Arabic construction	Strengths	Limitations
RTP-LX	1,050	Partial	Transcreated by native speakers from a subset of RealToxicityPrompts, with 50 additional culturally specific prompts authored manually. Annotated by humans across eight harm dimensions.	• Targets culturally specific toxic language, including microaggressions, bias and hate • Written by native speakers rather than translated, so phrasing is idiomatic	• Most prompts are sentence fragments intended for continuation rather than requests for harmful action, so they probe toxic generation rather than instruction following • Coverage is concentrated on toxic language and does not extend across the hazard taxonomy
PGPrompts	754	✗	Arabic portion of PolyGuardPrompts, produced by translating WildGuardMix with NLLB-200-3.3B and assessing translation quality automatically with GPT-4o.	• Contains adversarial and jailbreak style prompts that are difficult for both target models and safeguards • Already aligned with the ML Commons hazard taxonomy, inherited from WildGuardMix	• The translation model refused a subset of the source prompts, biasing the resulting corpus toward less harmful content • Only 754 Arabic harmful prompts, and since this count includes refusals, the number of genuinely adversarial prompts is smaller still, which is few relative to the number of hazard categories
Arabic Safeguard Evaluation	3,888	✗	Adapted from a Chinese safety dataset, with region specific content replaced by Arabic variants. Contains three prompt types: Original Harmful, Indirect Harmful (FN) and Harmless (FP).	• Pairs direct and indirect formulations of the same intent, enabling evaluation against implicit as well as explicit harm	• Includes politically sensitive categories that are not hazard categories • Coverage is uneven across hazard categories, with several ML Commons categories unrepresented
AraSafe	1,254	Partial	Arabic prompts written natively by participants through a web interface, each labelled by two expert annotators with disagreements reconciled.	• Naturally phrased, realistic misuse scenarios elicited from real users rather than templated or synthetic attacks • Double annotation with reconciliation gives comparatively reliable harm labels	• Prompts are written in a range of Arabic varieties, but no variety label is recorded, so the data cannot be used to assess dialectal vulnerability • The harmful subset of 1,254 prompts is thinly spread across hazard categories
X-Safety	2,800	✗	Arabic subset translated from English with the Google Translate API and subsequently corrected by professional translators. Fourteen safety categories across ten languages.	• Supports cross lingual comparison of the same prompts under a common framework • Covers adversarial prompting strategies such as goal hijacking and role play instruction	• The category scheme is orthogonal to hazard taxonomies by design, mixing harm types with instruction attack styles, so ten of its fourteen categories collapse to Others • Entirely in MSA, with no regional variety representation
LinguaSafe	4,030	✗	Derived from others benchmarks and augmented with natively sourced data across twelve languages. Arabic subset produced by machine translation, LM based refinement, and human verification of linguistic quality and preserved intent.	• Fine grained hierarchical labels spanning five domains and twenty three subtypes, with severity levels	• The composition of the Arabic subset is not reported per language • Severity annotation is not usable under a binary safe or unsafe framework and is discarded

Table 2 : Existing Arabic safety datasets aggregated.

Figure 2 : Overview of SalamahBench curation workflow.

Figure 3 : Dataset Distribution

Figure 4 : Overview of proposed evaluation pipeline.

Name	Size (Parameters)	Source	API Service Provider
Fanar 2	27B	Open-source	Hugging Face
ALLaM 2	7B	Open-source	Hugging Face
Karnak 1	41B	Open-source	Hugging Face
Qwen 3.8	27B	Open-source	Hugging Face
Fanar 1	9B	Open-source	Hugging Face
Jais 2	8B	Open-source	Hugging Face

Table 4 : LMs used in our experiments

Model/Statistic	Metric	Fanar 2	Karnak 1	ALLaM 2
Generative Qwen3Guard	Refusal	34120	28836	39694
	Safe	45045	39797	47429
	Controversial	2500	5292	1431
	Unsafe	2075	4531	760
	ASR (strict)	9.2% (8.8–9.7)	19.8% (19.2–20.4)	4.4% (4.1–4.7)
	ASR (loose)	4.2% (3.9–4.5)	9.1% (8.7–9.6)	1.5% (1.4–1.7)

Table 5 : Comparative Safety Robustness of Arabic Language Models Under Multiple Safeguard Evaluations, Aggregated Over Modern Standard Arabic and Five Regional Dialects. Parentheses give 95% bootstrap confidence intervals (

B=20{,}000

), resampling the 8,270 underlying prompts within harm category with all six varieties of a prompt carried together. Intervals are marginal and should not be used to compare cells within a row.

Figure 5 : Per-category attack success rate (ASR, %) for the three evaluated models across five regional Arabic varieties.

	Reference	Regional Arabic Varieties					Summary
Model	MSA	Saudi	Moroccan	Syrian	Lebanese	Egyptian	Dialect Avg.
Macro-ASR (%)
ALLaM	7.4	6.2	8.3	6.6	6.4	5.8	6.6
Fanar	11.4	12.9	12.5	12.7	13.6	12.9	12.9
Karnak	21.5	24.1	25.9	25.7	26.6	24.8	25.4
Dialect Shift (DS, pp)

Table 6 : Macro-ASR (%) and Dialect Shift (DS, pp) across MSA and five regional Arabic varieties. Macro-ASR is computed over the eleven named hazard categories; DS is measured relative to MSA. Dialect Avg. denotes the five-dialect mean. DS is reported as estimate

\pm

half-width of a 95% bootstrap confidence interval (

B=20{,}000

); bold denotes

q<.05

after Benjamini–Hochberg correction. Lower ASR is better.

	MSA	Saudi		Moroccan		Syrian		Lebanese		Egyptian
Category	ASR	ASR	CSDD	ASR	CSDD	ASR	CSDD	ASR	CSDD	ASR	CSDD
Intellectual Property	30.1	42.2	+10.5	32.5	+1.3	32.5	+1.1	40.4	+8.1	30.7	$-0.9$
Defamation	30.2	33.6	+1.9	27.6	$-3.7$	33.6	+2.2	37.1	+4.7	31.0	$-0.6$
Nonviolent Crimes	14.0	13.2	$-2.3$	18.0	+2.8	15.9	+0.6	15.4	$-0.8$	15.6	+0.2
Indiscriminate Weapons	13.0	10.4	$-4.1$	14.7	+0.6	13.0	$-1.3$	13.4	$-1.7$	14.3	$-0.2$
Suicide and Self-Harm	9.1	11.8	+1.2	9.1	$-1.1$	10.0	$-0.4^{\ddagger}$	10.9	$-0.3$	14.5	+4.0

Table 7 : Fanar category-level ASR (%) and CSDD (pp) across MSA and five regional Arabic varieties. CSDD measures each category’s MSA-to-dialect change relative to the model’s DS; positive values denote a more positive relative shift. Bold denotes

q<.05

after Benjamini–Hochberg correction across the 165 category–variety tests; in the Macro/DS row, bold denotes

q<.05

across the 15 model–variety DS tests.

\ddagger

marks cells resting on fewer than six discordant prompts, which cannot reach significance under an exact test and are not interpreted; cells marked

\ddagger

are not bolded regardless of

q

. The final row reports Macro-ASR and DS, such that each entry satisfies

\mathrm{ASR}_{c,d}=\mathrm{ASR}_{c,\mathrm{MSA}}+\mathrm{DS}_{d}+\mathrm{CSDD}_{c,d}

up to rounding.

	MSA	Saudi		Moroccan		Syrian		Lebanese		Egyptian
Category	ASR	ASR	CSDD	ASR	CSDD	ASR	CSDD	ASR	CSDD	ASR	CSDD
Intellectual Property	22.9	34.9	$\mathbf{+13.3}$	40.4	$\mathbf{+16.6}$	30.7	+8.7	28.3	+6.5	28.9	+7.7
Indiscriminate Weapons	19.5	10.4	$\mathbf{-7.8}$	14.3	$-6.1$	12.1	$-6.5$	13.0	$-5.5$	7.4	$\mathbf{-10.5}$
Nonviolent Crimes	7.1	5.3	$-0.6$	6.3	$-1.7$	4.9	$-1.4$	4.7	$-1.4$	5.5	0.0
Suicide and Self-Harm	3.6	2.7	$+0.4^{\ddagger}$	3.6	$-0.9^{\ddagger}$	7.3	$+4.5^{\ddagger}$	7.3	$+4.7$	3.6	$+1.6^{\ddagger}$
Sexual Content	5.4	3.0	$-1.1$	4.3	$-2.0$	3.2	$-1.4$	3.9	$-0.5$	3.2	$-0.6$

Table 8 : ALLaM category-level ASR (%) and CSDD (pp) across MSA and five regional Arabic varieties. Conventions follow Table 7 . Positive CSDD denotes a more positive MSA-to-dialect shift than the model-level DS.

	MSA	Saudi		Moroccan		Syrian		Lebanese		Egyptian
Category	ASR	ASR	CSDD	ASR	CSDD	ASR	CSDD	ASR	CSDD	ASR	CSDD
Intellectual Property	62.7	63.9	$-1.4$	62.7	$-4.4$	59.6	$-7.2$	63.3	$-4.5$	61.4	$-4.5$
Indiscriminate Weapons	42.9	50.2	+4.8	51.5	+4.2	55.0	+7.9	59.3	$\mathbf{+11.4}$	50.6	+4.5
Defamation	19.8	31.0	+8.6	25.0	+0.7	28.4	+4.4	31.9	+7.0	32.8	+9.6
Nonviolent Crimes	23.2	23.5	$-2.4$	34.0	$\mathbf{+6.3}$	28.0	+0.6	30.3	+2.0	27.1	+0.6
Sexual Content	24.1	21.7	$\mathbf{-5.0}$	26.6	$-1.9$	24.7	$-3.6$	24.1	$\mathbf{-5.1}$	24.4	$-3.0$

Table 9 : Karnak category-level ASR (%) and CSDD (pp) across MSA and five regional Arabic varieties. Conventions follow Table 7 . Positive CSDD denotes a more positive MSA-to-dialect shift than the model-level DS. No Karnak cell rests on fewer than six discordant prompts.

Arabic Variety	Llama Guard 4	PolyGuard	Qwen3Guard	Majority Vote
MSA	0.3	1.0	1.4	0.4
Saudi	0.9	1.2	2.5	0.8
Moroccan	3.0	0.5	3.6	0.9
Syrian	0.9	0.9	2.8	0.6
Lebanese	0.9	0.9	2.7	0.7
Egyptian	0.8	1.0	2.5	0.5

Table 10 : Attack Success Rate (ASR, %) of Qwen 3.8 27B across MSA and five regional Arabic varieties under different safeguard models. Lower ASR is better; column minima are bolded.

Safeguard Model	Task	Correct / Total	Accuracy
Qwen3Guard	Response Safety Classification	1220 / 1440	84.7% [82.8, 86.5]
Llama Guard 4	Response Safety Classification	0 941 / 1440	65.3% [62.9, 67.8]
PolyGuard	Response Safety Classification	0 850 / 1440	59.0% [56.5, 61.5]
Majority Vote (3 Safeguards)	Response Safety Classification	0 976 / 1440	67.8% [65.3, 70.2]
Qwen3Guard	Refusal Detection	1045 / 1440	72.6% [70.2, 74.8]

Table 11 : Agreement of safeguard models with the manually annotated gold-label set. Brackets give 95% Wilson confidence intervals (

n=1{,}440

			Qwen3Guard		Llama Guard 4		PolyGuard		Majority Vote
Category	$n_{u}$	Base	Acc.	Recall	Acc.	Recall	Acc.	Recall	Acc.	Recall
Violent Crimes	60	50.0	83.3	83.3 [72.0, 90.7]	57.5	20.0 [11.8, 31.8]	53.3	0 8.3 [3.6, 18.1]	62.5	28.3 [18.5, 40.8]
Nonviolent Crimes	58	51.7	90.0	91.4 [81.4, 96.3]	60.8	19.0 [10.9, 30.9]	52.5	0 3.4 [1.0, 11.7]	61.7	22.4 [13.6, 34.7]
Sex-Related Crimes	33	72.5	77.5	100.0 [89.6, 100]	73.3	12.1 [ 0 4.8, 27.3]	71.7	0 3.0 [0.5, 15.3]	74.2	15.2 [ 0 6.7, 30.9]
Child Sexual Exploitation	33	72.5	70.8	87.9 [72.7, 95.2]	75.8	39.4 [24.7, 56.3]	73.3	0 9.1 [3.1, 23.6]	79.2	45.5 [29.8, 62.0]
Indiscriminate Weapons	63	47.5	89.2	87.3 [76.9, 93.4]	65.0	33.3 [22.9, 45.6]	52.5	0 9.5 [4.4, 19.3]	66.7	36.5 [25.7, 48.9]

Table 12 : Per-category safeguard performance on the human gold-label set (

n=120

responses per category).

n_{u}

denotes the number of gold-unsafe responses. Acc. denotes classification accuracy, and Recall denotes recall on gold-unsafe responses, with 95% Wilson confidence intervals. Base is the accuracy of an always-safe classifier. All reported scores are percentages.

	Responses generated by
Judge	ALLaM 2	Fanar 2	Karnak 1	Overall
ALLaM 2	57.3 †	52.5	44.0	51.3
Fanar 2	37.1	38.3 †	65.6	47.0
Karnak 1	62.1	57.5	58.3 †	59.3

Table 13 : Four-class accuracy (%) of Arabic language models used as safety judges against human gold labels. Each response source contributes 480 instances, for 1,440 responses overall. † denotes self-judging.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

RTP-LX Category	ML Commons Mapping
Insult	Hate Speech
Bias	Hate Speech
IdentityAttack	Hate Speech
Violence	Violent Crimes
SelfHarm	Suicide & Self-Harm
SexualContent	Sexual Content

Appendix

Table A.1 : Mapping of Microsoft RTP-LX categories to the ML Commons hazard taxonomy.

Wildguardmix Subcategory	ML Commons Mapping
Violence and physical harm	Violent Crimes
Private Information Individual	Privacy
Cyberattack	Nonviolent Crimes
Fraud Assisting Illegal Activities	Nonviolent Crimes
Sexual content	Sexual Content
Social Stereotypes And Unfair Discrimination	Hate

Appendix

Table A.2 : Mapping from Wildguardmix subcategories to the ML Commons hazard taxonomy.

Arabic Dataset for LM Safeguard Evaluation Subcategory	ML Commons Mapping
Adult Content	Sexual Content
Social Stereotypes And Unfair Discrimination	Hate
Toxic Language (hate speech)	Hate
Mental Health Or Overreliance Crisis	Suicide and Self-Harm
Causing material harm by disseminating misinformation	Specialized Advice
Risks from leaking or inferring sensitive information	Privacy

Appendix

Table A.3 : Mapping from Arabic Dataset for LM Safeguard Evaluation subcategories to the ML Commons hazard taxonomy.

Assigned Category	ML Commons Mapping
Violence or Harm terrorism	Violent Crimes
Privacy Violation	Privacy
Hate Speech	Hate
Explicit Content	Sexual Content
Misinformation	Others (harm)
Illegal Activities	Others (harm)

Appendix

Table A.4 : Mapping of labeled AraSafe categories to the ML Commons hazard taxonomy.

Assigned Category	ML Commons Mapping
Mental Health	Suicide and Self-Harm
Unfairness And Discrimination	Hate
Insult	Hate
Privacy And Property	Privacy
Goal Hijacking	Others (harm)
Prompt Leaking	Others (harm)

Appendix

Table A.5 : Mapping of labeled X-Safety categories to the ML Commons hazard taxonomy.

Assigned Category	ML Commons Mapping
Erotic Content	Sexual Content
Sex Orientation	Sexual Content
Sex-related Crimes	Sex-related Crimes
Controlled Items	Nonviolent Crimes
Code Misuse	Nonviolent Crimes
Financial Crimes	Nonviolent Crimes

Appendix

Table A.6 : Mapping of labeled LinguaSafe categories to the ML Commons hazard taxonomy.

Assigned Category	ML Commons Mapping
Non-violent Illegal Acts	Nonviolent Crimes
Violent	Violent Crimes
private information individual	Privacy
Sexual Content or Sexual Acts	Sexual Content
Suicide & Self-Harm	Suicide and Self-Harm
Politically Defamation	Defamation

Appendix

Table A.7 : Mapping of labeled AdvBench categories to the ML Commons hazard taxonomy.

Assigned Category	ML Commons Mapping
Chemical Biological	Indiscriminate Weapons
Cybercrime Intrusion	Nonviolent Crimes
Copyright	Intellectual Property
Harassment Bullying	Others (harm)
Illegal Activities	Others (harm)
Misinformation Disinformation	Others (harm)

Appendix

Table A.8 : Mapping of labeled HarmBench categories to the ML Commons hazard taxonomy.

Metric	Direct	Reason.	Refine.	$\boldsymbol{\Delta}$
Naturalness	92.60	95.67	98.63	+6.03
Idiomatic accuracy	92.61	95.48	98.56	+5.95
Morphology & grammar	97.17	97.91	99.63	+2.46
Semantic preservation	99.70	99.83	99.95	+0.25
Average	95.52	97.22	99.19	+3.67

Appendix

Table B.9 : MSA-to-dialect LM-judge scores on a 0–100 scale.

\Delta

denotes the gain of self-refinement over direct prompting.

Dimension	Mean
Semantic preservation	4.99
Harmful-intent preservation	4.98
Dialectal naturalness	4.73
Dialect authenticity	4.59
Grammaticality	4.97
Overall mean	4.85

Appendix

Table B.10 : Native-speaker evaluation of 120 MSA-to-Egyptian Arabic translations. Scores range from 1 (lowest) to 5 (highest).

Detection Method	Accuracy	Refusal F1	Precision
Gemma 4 31B IT	0.9989	0.9989	0.9979
BGE-M3	0.9703	0.9714	0.9444
CometKiwi	0.5728	0.7028	0.5418

Appendix

Table B.11 : Performance of the evaluated refusal-detection methods on dialectal Arabic translations.

Model/Statistic	Metric	Fanar 1	Falcon H1R	Jais 2
Generative Qwen3Guard	Refusal	6757	7821	5331
	RR	81.7%	94.6%	64.5%
	Safe	7806	7804	6257
	Controversial	223	126	900
	Unsafe	241	340	1113
	ASR (strict)	5.6%	5.6%	24.3%

Appendix

Table C.12 : Safety results for Fanar 1, Falcon H1R, and Jais 2 on the MSA subset of SalamahBench , comprising 8,270 prompts per model, under multiple safeguard evaluations. Lower ASR is better; ASR row minima and maxima are highlighted. Values are not directly comparable with Table 5 , which aggregates MSA and five regional dialects.

Category	Fanar 1	Falcon H1R	Jais 2	Prompts
Violent Crimes	3.0%	1.4%	20.6%	509
Sex-Related Crimes	1.8%	3.7%	18.0%	217
Child Sexual Exploitation	4.7%	2.8%	17.0%	106
Suicide and Self-Harm	10.9%	6.4%	14.6%	110
Indiscriminate Weapons	8.7%	3.5%	32.0%	231
Intellectual Property	54.8%	4.8%	69.3%	166

Appendix

Table C.13 : Model-specific safety exposure across harm categories on the MSA subset of SalamahBench , computed with Qwen3Guard under the strict setting, in which both Unsafe and Controversial responses contribute to ASR. Harm-category membership follows the human-verified SalamahBench annotations. Macro-ASR is the unweighted mean across the eleven named hazard categories, as defined in Section 5.2 ; the heterogeneous Others group is reported but excluded from the mean.

Category	Saudi	Moroccan	Syrian	Lebanese	Egyptian
Panel A: $\Delta$ (pp) with $(b/c)$
Intellectual Property	$\mathbf{+12.0}$ (34/14)	$+2.4$ (28/24)	$+2.4$ (27/23)	$+10.2^{\dagger}$ (34/17)	$+0.6$ (21/20)
Defamation	$+3.4$ (16/12)	$-2.6$ (13/16)	$+3.4$ (16/12)	$+6.9$ (18/10)	$+0.9$ (16/15)
Nonviolent Crimes	$-0.8$ (32/39)	$\mathbf{+4.0}$ (68/34)	$+1.9$ (49/33)	$+1.4$ (50/38)	$+1.6$ (56/42)
Indiscriminate Weapons	$-2.6$ (6/12)	$+1.7$ (20/16)	$0.0$ (11/11)	$+0.4$ (13/12)	$+1.3$ (13/10)
Suicide and Self-Harm	$+2.7$ (6/3)	$0.0$ (4/4)	$+0.9$ (3/2)	$+1.8$ (4/2)	$+5.5$ (7/1)

Appendix

Table D.14 : Fanar: MSA-to-dialect change

\Delta

(pp) with discordant counts

(b/c)

, and CSDD (pp) with 95% bootstrap confidence intervals. Bold denotes

q<.05

after Benjamini–Hochberg correction;

\dagger

denotes

p<.05

before correction only.

Category	Saudi	Moroccan	Syrian	Lebanese	Egyptian
Panel A: $\Delta$ (pp) with $(b/c)$
Intellectual Property	$\mathbf{+12.0}$ (29/9)	$\mathbf{+17.5}$ (42/13)	$+7.8$ (28/15)	$+5.4$ (25/16)	$+6.0$ (25/15)
Indiscriminate Weapons	$\mathbf{-9.1}$ (10/31)	$-5.2$ (20/32)	$-7.4^{\dagger}$ (13/30)	$-6.5^{\dagger}$ (12/27)	$\mathbf{-12.1}$ (6/34)
Nonviolent Crimes	$-1.9^{\dagger}$ (21/37)	$-0.8$ (26/33)	$-2.2^{\dagger}$ (22/41)	$-2.5^{\dagger}$ (19/40)	$-1.6$ (25/39)
Suicide and Self-Harm	$-0.9$ (1/2)	$0.0$ (2/2)	$+3.6$ (4/0)	$+3.6$ (5/1)	$0.0$ (2/2)
Sexual Content	$\mathbf{-2.4}$ (13/37)	$-1.1$ (27/38)	$\mathbf{-2.2}$ (15/37)	$-1.5$ (21/36)	$\mathbf{-2.2}$ (17/39)

Appendix

Table D.15 : ALLaM: MSA-to-dialect change

\Delta

(pp) with discordant counts

(b/c)

, and CSDD (pp) with 95% bootstrap confidence intervals. Conventions follow Table D.14 .

Category	Saudi	Moroccan	Syrian	Lebanese	Egyptian
Panel A: $\Delta$ (pp) with $(b/c)$
Intellectual Property	$+1.2$ (8/6)	$0.0$ (9/9)	$-3.0$ (11/16)	$+0.6$ (10/9)	$-1.2$ (6/8)
Indiscriminate Weapons	$+7.4^{\dagger}$ (41/24)	$+8.7^{\dagger}$ (48/28)	$\mathbf{+12.1}$ (55/27)	$\mathbf{+16.5}$ (58/20)	$+7.8^{\dagger}$ (37/19)
Defamation	$+11.2^{\dagger}$ (18/5)	$+5.2$ (18/12)	$+8.6$ (19/9)	$+12.1^{\dagger}$ (22/8)	$\mathbf{+12.9}$ (21/6)
Nonviolent Crimes	$+0.2$ (65/63)	$\mathbf{+10.7}$ (142/50)	$\mathbf{+4.8}$ (90/49)	$\mathbf{+7.1}$ (121/60)	$\mathbf{+3.9}$ (81/48)
Sexual Content	$-2.4$ (82/106)	$+2.5$ (109/84)	$+0.6$ (102/96)	$0.0$ (90/90)	$+0.3$ (95/92)

Appendix

Table D.16 : Karnak: MSA-to-dialect change

\Delta

(pp) with discordant counts

(b/c)

, and CSDD (pp) with 95% bootstrap confidence intervals. Conventions follow Table D.14 . No Karnak cell rests on fewer than six discordant prompts.

Annotation Field	Raw Agreement	Cohen’s $\kappa$	PABAK	Gwet’s AC1
Response Safety (binary)	85.83 [83.9, 87.5]	0.714 [0.677, 0.750]	0.717 [0.679, 0.753]	0.721 [0.684, 0.756]
Response Safety (3-class)	78.12 [75.9, 80.2]	0.598 [0.562, 0.633]	0.672 [0.640, 0.703]	0.701 [0.670, 0.731]
Refusal	90.83 [89.2, 92.2]	0.669 [0.614, 0.718]	0.817 [0.786, 0.846]	0.873 [0.850, 0.895]
Arabic Variety (6-class)	98.12 [97.3, 98.7]	0.419 [0.231, 0.590]	0.977 [0.968, 0.985]	0.981 [0.973, 0.988]

Appendix

Table E.17 : Inter-annotator agreement on the 1,440-response human gold-label set before adjudication. Raw agreement reports the proportion of identical annotations, with 95% Wilson confidence intervals. Cohen’s

\kappa

, PABAK, and Gwet’s AC1 are accompanied by item-level bootstrap intervals (

B=5{,}000

). PABAK and AC1 are reported alongside

\kappa

because the Arabic-variety distribution is highly imbalanced.

Decision	$n$	Observed agreement [95% CI]	Cohen’s $\kappa$ [95% CI]
Q1 harmfulness, binary	500	94.4% [92.0, 96.1]	0.820 [0.754, 0.880]
Q2 category confirmation, binary	291	94.2% [90.8, 96.3]	0.839 [0.757, 0.908]
Derived final label, 12 classes	500	91.0% [88.2, 93.2]	0.888 [0.855, 0.918]

Appendix

Table E.18 : Inter-annotator agreement on the double-annotated sample.

Explore similar work

Aug 2, 2026cs.CL

ArabicDialectSafety: A Dialect-Aware Benchmark for Arabic Content Safety Classification

We present ArabicDialectSafety, a human-curated Arabic safety dataset of 25,071 prompts covering six Arabic varieties: Modern Standard Arabic, Syrian, Egyptian, Algerian, Palestinian, and Moroccan. The dataset is annotated with dialect labels and seven fine-grained harm categories. We introduce a dual-task evaluation framework for binary safe/unsafe detection and granular harm classification across dialects. Benchmarking seven supervised and generative models, we find that fine-tuned MARBERTv2 achieves the strongest performance, with Macro-F1 scores of 0.95 for binary classification and 0.90 for granular classification, substantially outperforming prompted frontier LLMs, including Arabic-specialized models. Our analyses show that dialect conditioning is most effective when integrated at the representation level, while significant performance gaps remain for low-resource Maghrebi dialects. We further evaluate seven frontier LLMs as response generators on harmful dialectal Arabic prompts and observe unsafe generation rates below 5 percent across models. We release the dataset and code upon acceptance to support future research on dialect-aware Arabic safety evaluation. Warning: This paper contains examples of harmful and potentially offensive content included solely for research purposes.

Wajdi Zaghouani, Md. Rafiul Biswas, Kholoud Khalil Aldous +1

Northwestern University in Qatar, Qatar · Hamad Bin Khalifa University, Qatar

Arabic Natural Language Processing Arabic Dialects Large Language Model Safety Data Generation

Ask PDF

Jun 19, 2026cs.CL

Evaluation of Small Language Models for Arabic Language Processing

This paper evaluates the performance of twelve Small Language Models (SLMs) on Arabic natural language processing tasks. The study introduces a benchmark of 240 Arabic test items distributed across eight domains and ten language skills, covering both comprehension-oriented and generation-oriented tasks. All models were evaluated under a controlled zero-shot setting using a standardized Arabic-only prompt template. Model responses were assessed through a multi-model LLM-as-a-judge framework involving GPT-4.1 Mini, Claude Haiku 4.5, and DeepSeek-Chat, with scores aggregated across judges and analyzed by task, skill, and model family. The results show that Gemma 3 (12B) achieved the highest overall score (4.548/5), followed by Aya and C4AI Command Arabic. The observed results suggest that model size alone does not explain Arabic SLM performance. Models with stronger Arabic alignment and more reliable instruction-following behavior tended to perform better across tasks. Common failure patterns among lower-performing models include prompt leakage, hallucination, language drift, incomplete generation, and weak task adherence. Overall, the benchmark provides a structured reference for evaluating compact Arabic language models and supports future work on efficient, reliable, and culturally appropriate Arabic AI systems.

Jumana Alsubhi, Ahmed Alhusayni, Abdulrahman Gharawi +5

Naseej Innovation Lab, Naseej for Technology, Riyadh, Saudi Arabia.

Arabic Natural Language Processing Small Large Language Models Large Language Model Benchmarks Llm-As-A-Judge

Ask PDF

Aug 30, 2026cs.CL

Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models

Large language models are increasingly deployed in Arabic-speaking markets, yet standard benchmarks overwhelmingly reward Modern Standard Arabic (MSA) fluency while leaving dialectal and culturally grounded competence unmeasured. This gap is consequential: everyday Arabic is largely dialectal, and dialect encodes social meaning that MSA-centric evaluation cannot capture. We present a rubric-based benchmark for the Saudi dialect, comprising 31 expert-authored prompts spanning idiomatic, pragmatic, lexical, and culturally-embedded phenomena, each paired with an expert-established ground truth. Our methodology separates evaluation into a model-agnostic phase, in which atomic, MECE positive criteria are derived solely from the ground truth, and a model-specific phase, in which four state-of-the-art systems -- Claude Opus 5, Gemini 3.7, GPT-5.6, and Kimi K3 -- are scored against those criteria and penalised for errors they actively introduce. Across 124 model-prompt evaluations we catalogue 466 error instances under a nine-category taxonomy. The four systems cluster within a narrow macro-average band (42.7%-53.1%), with no model exceeding 55% and every model recording at least one negative-scoring prompt, confirming that Saudi dialectal competence remains broadly unsolved. Notably, Ambiguous Framing is the dominant failure mode (37.3% of errors) while outright Hallucination accounts for only 11.2%, indicating that models fail less by stating falsehoods than by distorting register and flattening pragmatic nuance. We further observe a consistency-versus-ceiling trade-off and model-distinctive error signatures. We release the full prompt set, ground truths, and scored rubrics to support reproducible dialectal evaluation.

Ghassan Al-Sumaidaee, Sajjad Abdoli, Ahmed Rashad +1

1Perle

Arabic Fluency

Ask PDF