QuranicMMLU: A Cognitively-Aware Benchmark for Evaluating Generative AI Solutions on Quranic Linguistic Knowledge
Authors: Rawan El Ghali, Umm Kulsoom, Anas Madkoor, Dima Faris Alsaudi, Roaa Abdelmagid, Roaa Ibrahim, Raghad Mousa, Hamza Aljaji, Abdullah Khanafer, Abdallah Alkanani, Salah Feras Alali, Rawan Khaled Mohamed, Ehsaneddin Asgari
Organizations: Qatar Computing Research Institute, Hamad Bin Khalifa University · Carnegie Mellon University Qatar · University of Doha for Science and Technology · Qatar University · University of Birmingham · American University of Beirut
We introduce QuranicMMLU, a benchmark for evaluating generative AI on Quranic Arabic across multiple dimensions of linguistic complexity. Existing Quranic benchmarks center on general question answering and semantic retrieval, without probing specific linguistic competencies or stratifying by cognitive demand and verse difficulty. We construct a five-pillar Quranic taxonomy spanning Phonology, Morphology, Syntax, Semantics, and Pragmatics, with 31 leaves covering phenomena from tajwīd and root-and-pattern morphology to occasions of revelation and inter-surah coherence. For each leaf we generate questions stratified by Bloom's cognitive level and verse perplexity, then have LLM as a judge to independently answer and score every item and route the annotations to manual review. The resulting dataset comprises 980 human-reviewed questions, each issued in both open-ended and multiple-choice form. We benchmark 12 systems on these items and find that the Islamic-specialized model leads, yet every system scores higher on multiple-choice accuracy (average 84%) than open-ended answer quality (average 60%): the two rankings agree closely (Kendall's τ=0.73), but multiple-choice scoring hides failures that surface only once answer choices are removed. QuranicMMLU thus offers a rigorous, linguistically grounded framework for evaluating Arabic NLP in the Quranic domain.
Figures & tables
Figure 1: Overview of QuranicMMLU: taxonomy-driven question generation, LLM annotation and human review, then model evaluation with LLM-as-judge scoring. LLM1 is Claude (generator); LLM2 and LLM3 are GPT and Gemini (annotators).
Figure 2: The QuranicMMLU taxonomy: 5 linguistic pillars, 12 categories, and 31 leaves. Leaf codes (e.g., P1 , Sm8 ) index the specific phenomena evaluated by the benchmark.
Figure 3: Dataset composition: (a) Bloom-level distribution per pillar and (b) verse-perplexity class per pillar. The overall Bloom-level distribution across all 980 items is given in Figure 5 .
Figure 4: Per-model performance across the five linguistic pillars and the five Bloom levels, for (a) MCQ accuracy and (b) open-ended Gemini score. Models are ordered by overall MCQ accuracy. Phonology is the weakest pillar for nearly every system, especially in the open-ended setting.
Model
MCQ Acc. (%)
Open (%)
mean ± SD
Ansari
96.4
93.2 ± 21.8
gemma-4-31B
93.3
82.5 ± 28.8
Qwen3.6-27B
91.8
62.1 ± 37.0
gemma-4-26B-A4B
91.2
75.7 ± 32.8
Qwen3.5-35B-A3B
89.7
62.1 ± 35.8
Table 1: Overall benchmarking results: multiple-choice accuracy and open-ended Gemini score, both as percentages, reported as mean ± standard deviation across items and sorted by MCQ accuracy. Per-pillar and per-Bloom breakdowns are in Figure 4 .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Overall Bloom-level distribution across all 980 items.
Figure 6: We stratify benchmark questions across five cognitive levels from Bloom’s revised taxonomy (Remember through Evaluate), excluding the Create level, as synthesizing novel Quranic content would violate theological and scholarly principles in Islamic tradition.
Figure 7: Hierarchical prompt-composition pipeline used to generate each benchmark question.
Figure 8: Question-quality rubric used to guide human review.
Figure 9: The custom web interface used for human review and adjudication.
Figure 10: (continued) Human review and adjudication interface.
Figure 11: (continued) Human review and adjudication interface.
Figure 12: LLM-judge scoring prompt used to grade open-ended answers.
Large language models are increasingly consulted for Islamic knowledge, yet no comprehensive benchmark evaluates their performance across core Islamic disciplines. We introduce IslamicMMLU, a benchmark of 10,013 multiple-choice questions spanning three tracks: Quran (2,013 questions), Hadith (4,000 questions), and Fiqh (jurisprudence, 4,000 questions). Each track is formed of multiple types of questions to examine LLMs capabilities handling different aspects of Islamic knowledge. The benchmark is used to create the IslamicMMLU public leaderboard for evaluating LLMs, and we initially evaluate 26 LLMs, where their averaged accuracy across the three tracks varied between 39.8% to 93.8% (by Gemini 3 Flash). The Quran track shows the widest span (99.3% to 32.4%), while the Fiqh track includes a novel madhab (Islamic school of jurisprudence) bias detection task revealing variable school-of-thought preferences across models. Arabic-specific models show mixed results, but they all underperform compared to frontier models. The evaluation code and leaderboard are made publicly available.
Ali Abdelaal, Mohammed Nader Al Haffar, Mahmoud Fawzi +1
Meaningful multilingual evaluation must test models in the target language and educational context. Urdu, spoken by more than 230 million people, lacks a broad MMLU-style benchmark built from native educational sources. We introduce UrduMMLU, a benchmark of 26,389 Urdu MCQs across 26 subjects and five domains, collected from native Urdu MCQ banks and public examination PDFs. Unlike translation-based benchmarks, UrduMMLU combines academic subjects with content specific to Urdu and regional education. We label the exam-derived portion through dual human annotation with strict consensus filtering. We evaluate 30 LLMs under English and Urdu prompts, yielding 60 zero-shot evaluations, and further evaluate four open-source LLMs under multiple few-shot settings across both prompt languages. Gemini-3.5-Flash performs best, reaching 90.23% and 90.45% accuracy, while no other model exceeds 85%. The strongest open-source model trails by 7.78 and 9.12 points, and many models lose 25 to 40 points on Urdu-centered Humanities subjects compared with STEM. Few-shot prompting yields only modest gains. Results on UrduMMLU show that current LLMs have uneven Urdu knowledge, particularly for content grounded in the regional context.
Ahmer Tabassum, Sarfraz Ahmad, Hasan Iqbal +3
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
We present QIMMA, a quality-assured Arabic LLM leaderboard that places systematic benchmark validation at its core. Rather than aggregating existing resources as-is, QIMMA applies a multi-model assessment pipeline combining automated LLM judgment with human review to surface and resolve systematic quality issues in well-established Arabic benchmarks before evaluation. The result is a curated, multi-domain, multi-task evaluation suite of over 52k samples, grounded predominantly in native Arabic content; code evaluation tasks are the sole exception, as they are inherently language-agnostic. Transparent implementation via LightEval, EvalPlus and public release of per-sample inference outputs make QIMMA a reproducible and community-extensible foundation for Arabic NLP evaluation.
Leen AlQadi, Ahmed Alzubaidi, Mohammed Alyafeai +6