When a language model invents a Qur'anic verse or hadith, it gives false authority to text that readers may treat as sacred. Existing Islamic hallucination benchmarks are largely Arabic and often closed-ended, while Bengali resources mainly study general-domain hallucination or factual knowledge. We introduce BILAL-BENCH, a 240-question expert-verified benchmark for open-ended Islamic-source fabrication in Bengali. It covers Qur'anic attribution, hadith, fiqh, contemporary issues, and adversarial false-premise prompts, and uses a two-axis rubric that separates source errors from assertiveness. We evaluate seven open-weight models from 1B to 120B on 1,680 double-annotated responses. Six of the seven models confidently fabricate on 79.3-95.3% of source-production questions, with no monotonic improvement from scale. Hadith provenance is the weakest category across models: only two of 315 responses are correct and correctly sourced. The two 8B BanglaLLaMA variants have similar confident-fabrication rates but different error profiles. Models vary far more in whether they withhold a claim they cannot support than in source accuracy: appropriate abstention or false-premise correction ranges from 6 to 46 responses across the benchmark, and does not track parameter count.
Figures & tables
Cat.
Type
#
Expected behaviour / gold
C1
Qur’anic attribution
60
Correct verse and sūrah:ayah
C2
Hadith attribution
45
Correct provenance and grading
C3
Fiqh ruling (Hanafi)
45
Ruling with a valid basis
C4
Contemporary issue
45
Note disagreement; invent no text
C5
Adversarial premise
45
Reject premise; refuse fabrication
Total
240
Table 1: Bilal-Bench composition and expected behaviour.
Model
Fabrication
Confident-fab.
95% CI
NA
Gemma-2-9B-it
95.3%
95.3%
[90.7, 97.7]
0
GPT-OSS-120B
93.3%
93.3%
[88.2, 96.3]
0
BanglaLLaMA-3.2-3B-instruct
90.0%
90.0%
[84.2, 93.8]
7
BanglaLLaMA-3.1-8B-no-pretrain
87.3%
86.7%
[80.3, 91.2]
5
BanglaLLaMA-3.1-8B-instruct
83.3%
83.3%
[76.6, 88.4]
10
TituLM-gemma-2-2b
79.3%
79.3%
[72.2, 85.0]
1
Table 2: Fabrication on C1–C3 ( n=150 per model). NA remains in the denominator. Wilson CIs capture item-level binomial uncertainty, not decoding variance.
Model
C1 Qur’an
C2 Hadith
C3 Fiqh
C4 Contemp.
C5 Advers.
TigerLLM-1B-it
60.0
88.9
46.7
51.1
68.9
TituLM-gemma-2-2b
81.7
91.1
64.4
95.6
66.7
BanglaLLaMA-3.2-3B-instruct
96.7
95.6
75.6
42.2
80.0
BanglaLLaMA-3.1-8B-instruct
86.7
88.9
73.3
62.2
71.1
BanglaLLaMA-3.1-8B-no-pretrain
83.3
100.0
80.0
55.6
77.8
Gemma-2-9B-it
91.7
100.0
95.6
57.8
40.0
Table 3: Fabrication rate by category (%). C1–C3/C5 use F1–F4; C4 uses F1/F4 under the reduced rubric. C4–C5 assess withholding/correction.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Table 4: One released example per benchmark category. C5 contains a false premise, so its gold is a correction rather than a source.
Model
F1
F2
F3
F4
F5
F6
F7
NA
TigerLLM-1B-it
93
29
1
28
9
0
10
70
TituLM-gemma-2-2b
16
82
4
90
19
11
17
1
BanglaLLaMA-3.2-3B-instruct
77
71
1
41
6
2
17
25
BanglaLLaMA-3.1-8B-instruct
93
41
8
43
5
2
33
15
BanglaLLaMA-3.1-8B-no-pretrain
84
48
17
42
6
8
17
18
Gemma-2-9B-it
100
44
17
26
0
7
46
0
Appendix
Table 5: Final adjudicated outcome counts per model. Each row sums to 240 responses; NA denotes a response that ends without stating a claim.
Model
Running time (min)
BanglaLLaMA-3.1-8B-instruct
180.5
BanglaLLaMA-3.1-8B-no-pretrain
170.7
BanglaLLaMA-3.2-3B-instruct
118.9
Gemma-2-9B-it
150.5
TigerLLM-1B-it
111.6
TituLM-gemma-2-2b
82.5
Appendix
Table 6: Running time by evaluated model, in minutes.
Large language models are increasingly consulted for Islamic knowledge, yet no comprehensive benchmark evaluates their performance across core Islamic disciplines. We introduce IslamicMMLU, a benchmark of 10,013 multiple-choice questions spanning three tracks: Quran (2,013 questions), Hadith (4,000 questions), and Fiqh (jurisprudence, 4,000 questions). Each track is formed of multiple types of questions to examine LLMs capabilities handling different aspects of Islamic knowledge. The benchmark is used to create the IslamicMMLU public leaderboard for evaluating LLMs, and we initially evaluate 26 LLMs, where their averaged accuracy across the three tracks varied between 39.8% to 93.8% (by Gemini 3 Flash). The Quran track shows the widest span (99.3% to 32.4%), while the Fiqh track includes a novel madhab (Islamic school of jurisprudence) bias detection task revealing variable school-of-thought preferences across models. Arabic-specific models show mixed results, but they all underperform compared to frontier models. The evaluation code and leaderboard are made publicly available.
Ali Abdelaal, Mohammed Nader Al Haffar, Mahmoud Fawzi +1
Large language models (LLMs) are increasingly used for knowledge-intensive question answering, including religious and legal questions. Islamic knowledge is a particularly demanding setting: answers are expected to be grounded in authoritative sources, citations must be exact, Arabic varieties differ substantially from the language of classical sources, and legitimate jurisprudential disagreement must be represented rather than collapsed into a single answer. This survey reviews the emerging field of Islamic LLMs and trustworthy Islamic AI. We organize the literature around Arabic NLP and Arabic-centric LLMs, Islamic NLP resources, Qur'anic question answering, Islamic knowledge benchmarks, retrieval-augmented generation, Islamic legal reasoning, inheritance reasoning, hallucination evaluation, and trustworthiness. We argue that fluency in Arabic is not sufficient for Islamic AI. Reliable systems require curated sources, retrieval and verification modules, citation-aware generation, madhhab-aware reasoning, human expert evaluation, and benchmarks that measure not only answer accuracy but also faithfulness, source validity, and reasoning quality. The survey concludes with a research agenda for hallucination-resistant Islamic AI systems.
Large language models can generate fluent responses to Islamic questions while introducing factual errors that are difficult to identify. This paper presents our system for HalluScoring 2026 Task 2.1, Islamic Hallucination Detection and Find the Truth. The task requires a unified two-step prediction: determining whether an Arabic answer generated by an LLM is hallucinated and selecting the verified answer from six closely related candidate options. We use the Islamic knowledge dataset provided by the shared task, which contains 600 question--answer instances, including 341 hallucinated and 259 non-hallucinated answers. Our system is based on the fine-tuned google/gemma-4-12B-it model and uses deterministic decoding during inference. The generated outputs are normalized to extract the hallucination label and the selected option. The system achieves a Macro-F1 score of 0.928 and a label accuracy of 0.935 for hallucination detection, together with an option accuracy of 0.895 for answer selection. These results yield a combined score of 0.912, demonstrating strong performance across both stages of the task. The lower option-selection accuracy indicates that distinguishing the verified answer from plausible alternatives remains more challenging than detecting hallucinated responses.