ALPS: A Diagnostic Challenge Set for Arabic Linguistic & Pragmatic Reasoning

Hussein S. Al-Olimat; Ahmad Alshareef

cs.CLFeb 19, 2026

ALPS: A Diagnostic Challenge Set for Arabic Linguistic & Pragmatic Reasoning

Authors: Hussein S. Al-Olimat, Ahmad Alshareef

Organizations: Independent Researchers

Abstract

While recent Arabic NLP benchmarks focus on scale, they often rely on synthetic or translated data which may benefit from deeper linguistic verification. We introduce ALPS (Arabic Linguistic & Pragmatic Suite), a native, expert-curated diagnostic challenge set probing Deep Semantics and Pragmatics, capabilities that complement specialized large-scale benchmarks. While broad-coverage benchmarks prioritize scale and multi-task coverage, ALPS targets the depth of linguistic understanding through 531 rigorously crafted questions across 15 tasks and 47 subtasks. We developed the dataset with deep expertise in Arabic linguistics, guaranteeing cultural authenticity and eliminating translation artifacts. Evaluating 23 diverse models (commercial, open-source, and Arabic-native) against a single-pass human performance (avg. 84.6% accuracy) and an expert-adjudicated oracle (99.2%), we reveal a critical dissociation: models achieve high fluency but fail on fundamental morpho-syntactic dependencies, with elevated error rates on morpho-syntactic dependencies (36.5% across diacritics-reliant tasks) compared to compositional semantics. While top commercial models (Gemini-3-flash at 94.2%) surpass the average single human, a substantial gap persists between commercial giants and Arabic-native models, with the best Arabic-specific model (Jais-2-70B at 83.6%) approaching but not matching human performance.

Figures & tables

Model	Semantics								Pragmatics				Sem. & Prag.			AVG
Model	WSD	SRL	Lex	Ref	Qnt	Tmp	Mod	Cmp	Gen	Act	Imp	Pre	Txt	Dis	Met	AVG
Single-Pass Human Performance
Annotator 1 (PhD)	100.0	94.4	91.4	92.6	91.2	79.4	85.0	95.0	90.0	75.0	85.0	100.0	89.7	86.1	92.7	90.8
Annotator 2 (MA)	98.0	94.4	81.0	88.9	85.3	79.4	80.0	100.0	80.0	80.0	95.0	80.0	94.9	83.3	92.7	88.1
Annotator 3 (PhD)	88.2	83.3	86.2	87.0	82.4	85.3	90.0	95.0	76.7	65.0	90.0	80.0	84.6	91.7	85.4	85.1
Annotator 4 (BA)	72.5	68.5	70.7	81.5	55.9	67.6	65.0	85.0	76.7	75.0	75.0	75.0	74.4	80.6	90.2	74.2

Table 1: Detailed performance breakdown by subarea 2 2 2 WSD : Word Sense Disambiguation, SRL : Semantic Role Labeling, Lex : Lexical Semantics, Ref : Sense & Reference, Qnt : Quantifying Reference, Tmp : Temporal/Spatial, Mod : Modality, Cmp : Compositional Semantics, Gen : General Pragmatics, Act : Speech Acts, Imp : Implicature, Pre : Presupposition, Txt : Text Structure, Dis : Discourse Analysis, Met : Metaphor. . Underlined values indicate ceiling performance.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Area	Subarea Task & Definition	Sub-task	N
Semantics	Word Sense Disambiguation (WSD) / فك اللبس الدلالي Determining the correct meaning of a word in context Al-Suyūṭī (2003) .	Particles / الأدوات	13
		Preposition Meanings / معاني حروف الجر	10
		Disambiguation / فك اللبس الدلالي	7
		Pronoun Resolution / عود الضمير	3
		Case Markers / قرينة الحركة الإعرابية	3
		Ellipsis / الحذف	3

Appendix

Table A1: Complete ALPS taxonomy with task definitions, sub-tasks, and item counts.

WSD / فك اللبس الدلالي	SRL / وصف الدور الدلالي	Lexical Sem. / دلالات معجمية
معنى كلمةِ ”حِذاء” في جملةِ: ”من الأدبِ أن يأكُلَ الابنُ حذاءَ أبيهِ” هو: (1) جانِب (2) خلفَ (3) نَعْل (4) طعامَ	في جملة ”أكرمِ الناسَ أحمدُ” فإنَّ الدور الدلالي لـ (الناسَ) و(أحمدُ): (1) (الناسَ) مفعول به و(أحمد) مُنادى (2) (الناس) مفعولٌ به، و(أحمدُ) فاعل (3) (الناس) مبتدأ و(أحمد) مفعول به (4) (الناس) مبتدأ، و(أحمد) خبر	”فلانٌ عينٌ في قومه”، معنى ”عين” هنا: (1) صاحب مكانة (2) أعور (3) ضعيف (4) محبوب
Sense & Ref. / المعنى والمرجع	Quantifying Ref. / الدلالة الكمية	Temporal/Spatial / الزمانية والمكانية
معنى (العِقد) في جملة: ”للأسف وقع من ليلى العِقد وانتثر على الأرض”، هو : (1) مئة سنة. (2) ميثاق بين طرفين (3) القلادة (4) عشر سنوات	الإخوة لأمّ واحدة وآباء شتّى يسمون: (1) بنو الأخياف (2) بنو العلات (3) الأشقاء (4) لا شيء مما ذكر	في جملة ”أقوم للصلاة فجرًا”، في أي وقت أستيقظ؟ (1) بعد غروب الشمس (2) بعد الظهر (3) قبل الغروب (4) قبل شروق الشمس
Modality / الجهة	Compositional Sem. / الدلالات التركيبية	General Pragmatics / التداولية
في قولنا: ”ما ظنُّك بفلان”، المقصود بـ(الظنّ) هنا: (1) متأكد (2) الاعتقاد (3) غير متيقن (4) متيقن	”عينه كريمة” تركيبٌ يدل على أنه: (1) لا شيء مما ذُكر (2) عينه قويّة (3) عينه كريمة (4) لا يرى بإحدى عينيه	”هل تعاطيتَ عصيرَ البرتقال؟” لا نقبل هذا التركيب تداوليًّا لأنَّ: (1) التعاطي يكون للمواد المخدّرة والمدمن عليها (2) العصير غير مفيد (3) البرتقال لا يمكن عصره (4) البرتقال من المواد المخدّرة
Speech Acts / الأفعال الكلامية	Implicature / التلميح	Presupposition / افتراض مسبق

Appendix

Table A2: ALPS Task Examples: One example per task across all 15 subareas. These 15 items are removed from the released benchmark and used only for illustration in the paper.

Area	N	Ann. 1		Ann. 2		Ann. 3		Ann. 4
		Err	Acc	Err	Acc	Err	Acc	Err	Acc
Word Sense Disambiguation	51	0	100.0%	1	98.0%	6	88.2%	14	72.5%
Semantic Role Labeling	54	3	94.4%	3	94.4%	9	83.3%	17	68.5%
Pragmatics (General)	30	3	90.0%	6	80.0%	7	76.7%	7	76.7%
Speech Act Theory	20	5	75.0%	4	80.0%	7	65.0%	5	75.0%
Implicature	20	3	85.0%	1	95.0%	2	90.0%	5	75.0%

Appendix

Table A3: Human expert baseline performance on ALPS. Annotators include two with doctoral degrees (Ann. 1, 3), one with a master’s degree (Ann. 2), and one with a bachelor’s degree (Ann. 4) in Arabic language/linguistics. Accuracy ranges from 74.2% to 90.8%, averaging 84.6%.

Metric	Scope	Value
Human Accuracy	Average (Micro-Avg)	84.6%
	Majority Vote	83.1%
	Oracle (Upper Bound)	99.2%
Consistency Check	Raw Avg. Pairwise Agr.	79.8%
	Cohen’s $\kappa$ (All)	$0.24\pm 0.05$
	Cohen’s $\kappa$ (Core*)	$0.26\pm 0.04$

Appendix

Table A4: Human Baseline Summary. *Core Team: annotators with graduate degrees (2 PhD, 1 MA). “Majority Vote” shows hypothetical accuracy if voting were used; actual ground truth was established through iterative expert adjudication.

Task	N	Avg Acc.
Compositional Semantics	20	93.8%
WSD	51	89.7%
Metaphor & Figurative Lang.	41	90.2%
Context (Sense & Ref.)	54	87.5%
Text & Discourse Structure	39	85.9%
Implicature	20	86.2%

Appendix

Table A5: Macro-average human accuracies sorted by difficulty. Tasks requiring contextual inference (Speech Acts, Temporal/Spatial) are hardest.

Component	Content
System Message (Arabic)	أنت خبير لغوي وباحث متخصص في تحليل النصوص العربية والمنطق والاستنتاج. سأعرض عليك أسئلة متعددة الخيارات تتسم بالتعقيد اللغوي وتتطلب قدرات عالية في الاستدلال، الاسترجاع، والتحليل الدقيق للمعاني الضمنية. مهمتك: 1. اقرأ السؤال والخيارات بعناية فائقة وحلل الفروقات الدقيقة بينها. 2. حدد الإجابة الأكثر دقة وصحة بناءً على السياق والمنطق. 3. المخرج النهائي: اكتب رقم الخيار الصحيح فقط (مثل: 1، 2، 3، 4) دون أي شرح أو نص إضافي.
System Message (English Translation)	You are a linguistic expert and researcher specializing in Arabic text analysis, logic, and deduction. I will present you with multiple-choice questions characterized by linguistic complexity that require high capabilities in reasoning, retrieval, and precise analysis of implicit meanings. Your task: 1. Read the question and options very carefully and analyze the subtle differences between them. 2. Determine the most accurate and correct answer based on context and logic. 3. Final Output: Write only the number of the correct option (e.g., 1, 2, 3, 4) without any explanation or additional text.
User Prompt Template (Arabic)	السؤال: {نص السؤال} الخيارات: 1. {الخيار الأول} 2. {الخيار الثاني} …
User Prompt Template (English Translation)	Question: {question} Options: 1. {option 1} 2. {option 2} …

Appendix

Table A6: The standardized zero-shot prompt used for all models. The prompt consists of a system message establishing the persona and a user prompt containing the question and options.

Category	Model	Description & Rationale
Commercial	Gemini-3-flash-preview	Dec 2025 (Google). Latest Gemini with strong multilingual support and robust Arabic capabilities.
	Gemini-3-pro-preview	Dec 2025 (Google). Pro variant with enhanced reasoning for complex linguistic tasks.
	Claude-Opus-4.5	Dec 2025 (Anthropic). Excels at nuance, pragmatics, and linguistic subtlety critical for implicature.
	Gemini-2.5-pro	Nov 2025 (Google). Strong multilingual model with thinking capabilities.
	GPT-5.2	Dec 2025 (OpenAI). Current gold standard for general reasoning and instruction following.
	Gemini-2.0-flash-lite	2024 (Google). Efficient variant for cost-effective evaluation.

Appendix

Table A7: Complete list of 23 models evaluated on ALPS, organized by category. We select models that are state-of-the-art, Arabic-specific, or efficient baselines. Underlined values in Table 1 indicate the ceiling performance achieved by any model or human annotator.

Model	Category	Platform	Temp.	Max Tokens
Commercial Models
Gemini-3-flash-preview	Commercial	Google AI API	1.0	512
Gemini-3-pro-preview	Commercial	Google AI API	1.0	512
Claude-Opus-4-5-20251101	Commercial	Anthropic API	0	64
Gemini-2.5-pro	Commercial	Google AI API	0	256
GPT-5.2-2025-12-11	Commercial	OpenAI API	0	16

Appendix

Table A8: Model configurations and inference platforms used for ALPS evaluation.

Rank	Sub-Task	Err%	N	Diff.
1	Verbal Clues	68.3	3	$\blacksquare$
2	Case Markers	66.7	3	$\blacksquare$
3	Collocations	61.4	5	$\blacksquare$
4	Connectives	49.2	3	$\blacksquare$
5	Semantic Fields	46.7	8	$\blacksquare$
6	Disambiguation	44.0	8	$\blacksquare$

Appendix

Table A9: Complete sub-task difficulty ranking by error rate. Difficulty:

\blacksquare

Hardest (

>

60Note: Sub-tasks with small

N

(e.g., Case Markers

N=3

, Verbal Clues

N=3

) serve as diagnostic probes; their error rates indicate qualitative patterns rather than statistically robust estimates.

Figure 1: The Syntax-Pragmatics Inversion. Scatter plot comparing morpho-syntactic competence (SRL accuracy) vs. pragmatic competence (Implicature accuracy). SRL (Semantic Role Labeling) serves as a proxy for morpho-syntactic understanding because it requires correct interpretation of case markers and diacritics. Models achieve near-perfect Implicature scores (100% for six commercial models) while showing high variance on SRL (38–96%), suggesting pragmatics may be solved via pattern matching on conversational structures. Note the distinct cluster of Arabic-native models (e.g., SILMA-9B, Jais-2-70B) that achieve competitive pragmatic scores despite variable syntactic understanding, highlighting the “fluency trap.”

Arabic NLP

Arabic natural language processing develops methods and resources for understanding and generating Arabic across dialects, domains, and media.

See all 104 papers

Apr 30, 2026

Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues

There is a significant gap in evaluating cultural reasoning in LLMs using conversational datasets that capture culturally rich and dialectal contexts. Most Arabic benchmarks focus on short text snippets in Modern Standard Arabic (MSA), overlooking the cultural nuances that naturally arise in dialogues. To address this gap, we introduce ArabCulture-Dialogue, a culturally grounded conversational dataset covering 13 Arabic-speaking countries, in both MSA and each country's respective dialect, spanning 12 daily-life topics and 54 fine-grained subtopics. We utilize the dataset to form three benchmarking tasks: (i) multiple-choice cultural reasoning, (ii) machine translation between MSA and dialects, and (iii) dialect-steering generation. Our experiments indicate that the performance gap between MSA and Arabic dialects still exists, whereby the models perform worse on all three tasks in the dialectal setup, compared to the MSA one.

Muhammad Dehan Al Kautsar, Saeed Almheiri, Momina Ahsan +13

Mohamed bin Zayed University of Artificial Intelligence · IBM Research AI · American University in the Emirates

Low-Resource Language Processing Arabic NLP Cultural Knowledge in Language Models Multilingual Language Model Evaluation Text Generation MT Evaluation+1

Ask PDF

Jun 19, 2026

Evaluation of Small Language Models for Arabic Language Processing

This paper evaluates the performance of twelve Small Language Models (SLMs) on Arabic natural language processing tasks. The study introduces a benchmark of 240 Arabic test items distributed across eight domains and ten language skills, covering both comprehension-oriented and generation-oriented tasks. All models were evaluated under a controlled zero-shot setting using a standardized Arabic-only prompt template. Model responses were assessed through a multi-model LLM-as-a-judge framework involving GPT-4.1 Mini, Claude Haiku 4.5, and DeepSeek-Chat, with scores aggregated across judges and analyzed by task, skill, and model family. The results show that Gemma 3 (12B) achieved the highest overall score (4.548/5), followed by Aya and C4AI Command Arabic. The observed results suggest that model size alone does not explain Arabic SLM performance. Models with stronger Arabic alignment and more reliable instruction-following behavior tended to perform better across tasks. Common failure patterns among lower-performing models include prompt leakage, hallucination, language drift, incomplete generation, and weak task adherence. Overall, the benchmark provides a structured reference for evaluating compact Arabic language models and supports future work on efficient, reliable, and culturally appropriate Arabic AI systems.

Jumana Alsubhi, Ahmed Alhusayni, Abdulrahman Gharawi +5

Naseej Innovation Lab, Naseej for Technology, Riyadh, Saudi Arabia.

Arabic NLP Small Language Models Arabic Language Models

Ask PDF

Feb 3, 2026

SalamahBench: Dialect and Category Level Safety Evaluation of Arabic Language Models

While different stakeholders are trying to leverage Arabic Language Models (ALMs), safety alignment in ALMs remains largely underexplored, hindering their mainstream adoption. Existing safety benchmarks are predominantly English-centric and evaluate Arabic only in its standardized form, obscuring fine-grained safety vulnerabilities in Arabic NLP systems. This paper introduces SalamahBench, a unified benchmark of 8{,}270 human-verified harmful prompts across ML Commons hazard categories, each rendered in Modern Standard Arabic (MSA) and five regional Arabic varieties, namely Egyptian, Syrian, Saudi, Lebanese, and Moroccan, for a total of 49{,}620 paired instances. To analyze the resulting data, we introduce two complementary metrics, namely Dialect Shift, which measures a model's aggregate change in safety under dialectal reformulation, and Category-Specific Dialect Deviation, which isolates harm categories whose change departs from that aggregate trend. Evaluating models such as Fanar 2, ALLaM 2, and Karnak 1 under multiple safeguard configurations, we find that cross-variety robustness is strongly model dependent, and that aggregate scores can conceal category-level divergence. Our findings highlight the necessity of evaluating Arabic model safety jointly across linguistic varieties and harm domains rather than relying on aggregate scores or MSA alone.

Omar Abdelnasser, Fatemah Alharbi, Khaled Khasawneh +2

Compumacy for Artificial Intelligence Solutions, Cairo, Egypt · Computer Science Department, College of Computer Science and Engineering, Taibah University, Yanbu 46522, KSA · Electrical and Computer Engineering Department, George Mason University, VA, USA +2

LLM Safety Benchmarks Multilingual Language Model Evaluation Arabic NLP Arabic Language Models Language Model Safety Evaluation Language Modeling+2

Ask PDF

Abstract

Arabic NLP

Explore similar work

Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues

Evaluation of Small Language Models for Arabic Language Processing

SalamahBench: Dialect and Category Level Safety Evaluation of Arabic Language Models