IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledge

Ali Abdelaal; Mohammed Nader Al Haffar; Mahmoud Fawzi; Walid Magdy

cs.CLMar 24, 2026

IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledge

Authors: Ali Abdelaal, Mohammed Nader Al Haffar, Mahmoud Fawzi, Walid Magdy

Organizations: The University of Edinburgh

Abstract

Large language models are increasingly consulted for Islamic knowledge, yet no comprehensive benchmark evaluates their performance across core Islamic disciplines. We introduce IslamicMMLU, a benchmark of 10,013 multiple-choice questions spanning three tracks: Quran (2,013 questions), Hadith (4,000 questions), and Fiqh (jurisprudence, 4,000 questions). Each track is formed of multiple types of questions to examine LLMs capabilities handling different aspects of Islamic knowledge. The benchmark is used to create the IslamicMMLU public leaderboard for evaluating LLMs, and we initially evaluate 26 LLMs, where their averaged accuracy across the three tracks varied between 39.8% to 93.8% (by Gemini 3 Flash). The Quran track shows the widest span (99.3% to 32.4%), while the Fiqh track includes a novel madhab (Islamic school of jurisprudence) bias detection task revealing variable school-of-thought preferences across models. Arabic-specific models show mixed results, but they all underperform compared to frontier models. The evaluation code and leaderboard are made publicly available.

Figures & tables

Benchmark	Lang	#Q	Tracks	Bias	Pub.	Val.
ArabicMMLU	ar	14.6k	1 †	✗	✓	2 ann.
ACVA	ar	8k+	1	✗	✓	—
IslamicEval ’25	ar	—	2	✗	✓	task
PalmX	ar	—	1	✗	✓	—
FiqhQA	ar	960	1	✗	✓	synth
Isl.LegalBench	ar	718	1	✗	✓	expert

Table 1: Comparison with related benchmarks. #Q = question count; Tracks = Islamic discipline count ( † ArabicMMLU has Islamic Studies as one of 40 subjects); Bias = madhab bias detection; Pub. = public access (✓ = open, API = leaderboard-only); Val. = validation method.

	Quran	Hadith	Fiqh
Questions	2,013	4,000	4,000
Task types	3	4	5
Sources	114 surahs	6 collections	1 encyclopedia
Bias analysis	—	—	Madhab

Table 2: IslamicMMLU benchmark overview. 10,013 questions from 3 tracks across 12 task types.

Panel of Figure 1 — Figure 1: Fiqh track pipeline architectures. (a) Extraction of structured rulings from al-Jaziri’s source text. (b) Generation and validation of benchmark questions from the structured corpus.

Panel of Figure 2 — Figure 2: Model evaluation results. (a) Per-track accuracy with cross-track spread. (b) Madhab bias: dots near the centre line indicate balanced school selection.

Model	Overall	Quran	Hadith	Fiqh
Gemini 3 Flash	93.8	99.3	93.0	89.1
Gemini 3 Pro	92.3	97.3	91.7	87.9
Gemini 2.5 Pro	90.3	95.8	89.7	85.5
GPT-5	89.9	98.0	88.2	83.5
GPT-5.2	89.5	94.8	87.8	85.9
GPT-5.1	88.4	95.3	88.3	81.5

Table 3: IslamicMMLU accuracy (%) for all 26 models. Arabic-specific models in bold . Fiqh accuracy is on the 3,200 knowledge questions (800 bias detection questions scored separately). Per-question-type breakdowns are in Table 4 .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

	Quran			Hadith				Fiqh (by madhab)
Model	Ayah	Surah	Count	Source	Cloze	Chap.	Auth.	Han.	Mal.	Sha.	Hba.	Comp.
Gem. 3 Flash	99.1	99.4	100	96.4	95.8	96.9	82.9	89.7	88.9	92.2	87.1	80.0
Gem. 3 Pro	96.9	97.5	100	93.1	95.7	95.7	82.4	92.6	84.9	90.0	84.8	82.9
Gem. 2.5 Pro	96.6	94.2	100	91.7	95.0	94.1	77.9	88.2	81.9	85.0	87.6	82.9
GPT-5	98.9	96.6	100	87.6	95.2	94.0	76.1	84.2	84.4	82.2	83.7	80.0
GPT-5.2	98.4	92.0	81.6	89.6	92.1	93.6	75.8	88.2	85.9	82.8	87.6	80.0

Appendix

Table 4: Expanded accuracy (%) by question type per track. Quran: Ayah Identification, Surah Identification, Ayah Count. Hadith: Source Identification, Cloze Completion, Chapter Classification, Authenticity Grading. Fiqh: accuracy on questions tagged by madhab (Hanafi, Maliki, Shafi’i, Hanbali) and Comparative Fiqh cross-school questions.

Table 5: Representative example for each of the 12 question types. Arabic originals with English translations; correct answers bolded . For bias detection, all four options are correct (one per school); school labels are hidden from the model.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

QuranicMMLU: A Cognitively-Aware Benchmark for Evaluating Generative AI Solutions on Quranic Linguistic Knowledge
Sep 18, 2026Rawan El Ghali, Umm Kulsoom, Anas Madkoor +10LLM Evaluation Arabic NLP
Qatar Computing Research Institute, Hamad Bin Khalifa University · Carnegie Mellon University Qatar · University of Doha for Science and Technology +3
IslamicTurathBench: A Multi-Task, Multi-Discipline Benchmark for Evaluating Large Language Models on the Islamic Scholarly Tradition (turath)
Aug 5, 2026Shahd Gaben, Heba Sbahi, Samer Rashwani +5LLM Evaluation Benchmark Design
Hamad Bin Khalifa University, Doha, Qatar. · Nazarbayev University, Astana, Kazakhstan.
Islamic Large Language Models: From Knowledge Acquisition to Trustworthy and Hallucination-Resistant AI
Jun 15, 2026Mohammed Amine MouhoubQuestion Answering LLM Reliability
Paris Dauphine University

Sep 18, 2026J/K move · Enter open · S save

QuranicMMLU: A Cognitively-Aware Benchmark for Evaluating Generative AI Solutions on Quranic Linguistic Knowledge

Rawan El Ghali, Umm Kulsoom, Anas Madkoor, Dima Faris Alsaudi+9

Qatar Computing Research Institute, Hamad Bin Khalifa University · Carnegie Mellon University Qatar · University of Doha for Science and Technology+3

LLM Evaluation Arabic NLP

Open paper AskPDF arXiv

We introduce QuranicMMLU, a benchmark for evaluating generative AI on Quranic Arabic across multiple dimensions of linguistic complexity. Existing Quranic benchmarks center on general question answering and semantic retrieval, without probing specific linguistic competencies or stratifying by cognitive demand and verse difficulty. We construct a five-pillar Quranic taxonomy spanning Phonology, Morphology, Syntax, Semantics, and Pragmatics, with 31 leaves covering phenomena from tajwīd and root-and-pattern morphology to occasions of revelation and inter-surah coherence. For each leaf we generate questions stratified by Bloom's cognitive level and verse perplexity, then have LLM as a judge to independently answer and score every item and route the annotations to manual review. The resulting dataset comprises 980 human-reviewed questions, each issued in both open-ended and multiple-choice form. We benchmark 12 systems on these items and find that the Islamic-specialized model leads, yet every system scores higher on multiple-choice accuracy (average 84%) than open-ended answer quality (average 60%): the two rankings agree closely (Kendall's τ=0.73), but multiple-choice scoring hides failures that surface only once answer choices are removed. QuranicMMLU thus offers a rigorous, linguistically grounded framework for evaluating Arabic NLP in the Quranic domain.

Abstract

LLM Evaluation

Explore similar work

QuranicMMLU: A Cognitively-Aware Benchmark for Evaluating Generative AI Solutions on Quranic Linguistic Knowledge

IslamicTurathBench: A Multi-Task, Multi-Discipline Benchmark for Evaluating Large Language Models on the Islamic Scholarly Tradition (turath)

Islamic Large Language Models: From Knowledge Acquisition to Trustworthy and Hallucination-Resistant AI