cs.CLSep 18, 2026

QuranicMMLU: A Cognitively-Aware Benchmark for Evaluating Generative AI Solutions on Quranic Linguistic Knowledge

Authors: Rawan El Ghali, Umm Kulsoom, Anas Madkoor, Dima Faris Alsaudi, Roaa Abdelmagid, Roaa Ibrahim, Raghad Mousa, Hamza Aljaji, 

Organizations: Qatar Computing Research Institute, Hamad Bin Khalifa University · Carnegie Mellon University Qatar · University of Doha for Science and Technology · Qatar University · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We introduce QuranicMMLU, a benchmark for evaluating generative AI on Quranic Arabic across multiple dimensions of linguistic complexity. Existing Quranic benchmarks center on general question answering and semantic retrieval, without probing specific linguistic competencies or stratifying by cognitive demand and verse difficulty. We construct a five-pillar Quranic taxonomy spanning Phonology, Morphology, Syntax, Semantics, and Pragmatics, with 31 leaves covering phenomena from tajwīd and root-and-pattern morphology to occasions of revelation and inter-surah coherence. For each leaf we generate questions stratified by Bloom's cognitive level and verse perplexity, then have LLM as a judge to independently answer and score every item and route the annotations to manual review. The resulting dataset comprises 980 human-reviewed questions, each issued in both open-ended and multiple-choice form. We benchmark 12 systems on these items and find that the Islamic-specialized model leads, yet every system scores higher on multiple-choice accuracy (average 84%) than open-ended answer quality (average 60%): the two rankings agree closely (Kendall's τ=0.73), but multiple-choice scoring hides failures that surface only once answer choices are removed. QuranicMMLU thus offers a rigorous, linguistically grounded framework for evaluating Arabic NLP in the Quranic domain.

Figures & tables

Figure 1: Overview of QuranicMMLU: taxonomy-driven question generation, LLM annotation and human review, then model evaluation with LLM-as-judge scoring. LLM1 is Claude (generator); LLM2 and LLM3 are GPT and Gemini (annotators).
Figure 2: The QuranicMMLU taxonomy: 5 linguistic pillars, 12 categories, and 31 leaves. Leaf codes (e.g., P1 , Sm8 ) index the specific phenomena evaluated by the benchmark.
Figure 3: Dataset composition: (a) Bloom-level distribution per pillar and (b) verse-perplexity class per pillar. The overall Bloom-level distribution across all 980 items is given in Figure 5 .
Figure 4: Per-model performance across the five linguistic pillars and the five Bloom levels, for (a) MCQ accuracy and (b) open-ended Gemini score. Models are ordered by overall MCQ accuracy. Phonology is the weakest pillar for nearly every system, especially in the open-ended setting.
ModelMCQ Acc. (%)Open (%)
  mean ±\pm SD
Ansari96.493.2 ±\pm 21.8
gemma-4-31B93.382.5 ±\pm 28.8
Qwen3.6-27B91.862.1 ±\pm 37.0
gemma-4-26B-A4B91.275.7 ±\pm 32.8
Qwen3.5-35B-A3B89.762.1 ±\pm 35.8
Table 1: Overall benchmarking results: multiple-choice accuracy and open-ended Gemini score, both as percentages, reported as mean ±\pm standard deviation across items and sorted by MCQ accuracy. Per-pillar and per-Bloom breakdowns are in Figure 4 .
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Overall Bloom-level distribution across all 980 items.
Figure 6: We stratify benchmark questions across five cognitive levels from Bloom’s revised taxonomy (Remember through Evaluate), excluding the Create level, as synthesizing novel Quranic content would violate theological and scholarly principles in Islamic tradition.
Figure 7: Hierarchical prompt-composition pipeline used to generate each benchmark question.
Figure 8: Question-quality rubric used to guide human review.
Figure 9: The custom web interface used for human review and adjudication.
Figure 10: (continued) Human review and adjudication interface.
Figure 11: (continued) Human review and adjudication interface.
Figure 12: LLM-judge scoring prompt used to grade open-ended answers.
More in this topic

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Explore similar work

CardsList

  1. IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledge

    Mar 24, 2026Ali Abdelaal, Mohammed Nader Al Haffar, Mahmoud Fawzi +1LLM EvaluationCultural Knowledge in Language Models

    The University of Edinburgh

  2. UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding

    Jun 5, 2026Ahmer Tabassum, Sarfraz Ahmad, Hasan Iqbal +3Multilingual Language Model EvaluationMultilingual QA

    Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)

  3. Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation

    Apr 3, 2026Leen AlQadi, Ahmed Alzubaidi, Mohammed Alyafeai +6LLM EvaluationBenchmark Validity

    Technology Innovation Institute, Abu Dhabi, UAE