cs.CLNov 2, 2026

They Know Where It Is, But Not What It Says: A Bilingual Hallucination Benchmark of Hadith Recall in Open LLMs

Authors: Dayam Nadeem, Safdar Mustafa, Ameen Ur Rehman, Shahab Saquib Sohail, Mohammad Nadeem, Usman Naseem

Organizations: Jamia Hamdard · KFUPM · Aligarh Muslim University · Macquarie University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models are increasingly consulted for religious knowledge, including the recall of specific Hadith (recorded sayings and actions of the Prophet Muhammad, peace be upon him). Unlike open-ended generation, Hadith recall is a high-stakes, verifiable task: a reference (collection, chapter, and Hadith number) has exactly one correct text, and any deviation from it constitutes a fabrication that can be mistaken for authentic scripture. This study introduces a bilingual (English/Arabic) benchmark that probes five open-weight, locally-deployable language models (Mistral-7B, Gemma3-4B, Jais-7B, ALLaM-7B, and Phi4-mini-3.8B) across all six canonical Sunni Hadith collections (Sahih al-Bukhari, Sahih Muslim, Sunan Abi Da'ud, Jami' al-Tirmidhi, Sunan an-Nasa'i, Sunan Ibn Majah), sampling 500 Hadith per collection (3,000 per model per language, 30,000 responses total). Each model is given only the bibliographic reference and asked to reproduce the exact Hadith text, with no retrieval augmentation, and is scored on lexical fidelity (BLEU) and semantic fidelity (multilingual cosine similarity), with refusals flagged separately. Across all five models and both languages, 0% of 30,000 responses achieve both high lexical and high semantic fidelity, i.e., an exact, verifiable match to the source text. Arabic responses fare uniformly worse than English. These findings suggest that current open-weight LLMs, used without retrieval grounding, are unreliable and potentially dangerous sources for exact religious-text recall, and that deployment for Islamic scholarship should mandate retrieval-augmented generation over a verified corpus rather than parametric memory. The complete code, prompts, evaluation pipeline, and scored outputs are publicly available at https://github.com/dayam8696/Hadith-Hallucination-Benchmark.

Figures & tables

Figure 1: End-to-end evaluation pipeline: reference-only prompt →\rightarrow local LLM inference (Ollama) →\rightarrow model response →\rightarrow refusal filter →\rightarrow dual scoring (BLEU lexical match; multilingual cosine semantic match) →\rightarrow five-way classification ( correct / theme-only / partial / hallucinated / refused ).
ModelLangAvg. BLEUAvg. Cosine%Theme-only%Partial%Halluc.
Mistral-7BEnglish0.0070.5590.4775.0324.50
Mistral-7BArabic0.0060.71932.2759.738.00
Gemma3-4BEnglish0.0080.5580.2374.2325.53
Gemma3-4BArabic0.0190.73639.8350.809.37
Phi4-mini-3.8BEnglish0.0050.4470.0741.1358.80
Phi4-mini-3.8BArabic0.0190.5006.9744.8347.50
Table 1: Hadith-recall results aggregated over all six collections (3,000 responses per model–language cell). Correct = 0.0% and Refused ≤\leq 0.7% in every row; remaining columns sum to 100%.
Figure 2: Mistral-7B per-collection results showing %Correct, %Theme-only, %Partial, %Hallucinated, %Refused, BLEU, and Cosine similarity for English and Arabic across all six Hadith collections. %Correct and %Refused are 0.00% for both English and Arabic across all collections.
Figure 3: Gemma3-4B per-collection results showing %Correct, %Theme-only, %Partial, %Hallucinated, %Refused, BLEU, and Cosine similarity for English and Arabic across all six Hadith collections. %Correct and %Refused are 0.00% for both English and Arabic across all collections.
Figure 4: Phi4-mini-3.8B per-collection results showing %Correct, %Theme-only, %Partial, %Hallucinated, %Refused, BLEU, and Cosine similarity for English and Arabic across all six Hadith collections. %Correct is 0.00% across all collections; Arabic refusals where nonzero: Bukhari 1.0%, Tirmidhi 2.2%, Nasa’i 0.2%, Ibn Majah 0.8%.
Figure 5: Jais-7B per-collection results showing %Correct, %Theme-only, %Partial, %Hallucinated, %Refused, BLEU, and Cosine similarity for English and Arabic across all six Hadith collections. %Correct and %Refused are 0.00% for both English and Arabic across all collections.
Figure 6: ALLaM-7B per-collection results showing %Correct, %Theme-only, %Partial, %Hallucinated, %Refused, BLEU, and Cosine similarity for English and Arabic across all six Hadith collections. %Correct and %Refused are 0.00% for both English and Arabic across all collections.
More in this topic

Hallucination in Language Models

Hallucination in language models studies how systems produce unsupported or false claims and how to detect, explain, and reduce them.

See all 267 papers

Explore similar work

CardsList
Oct 29, 2026

Do Open Language Models Invent Scripture? BILAL-BENCH: Benchmarking Islamic-Source Fabrication in Bengali

When a language model invents a Qur'anic verse or hadith, it gives false authority to text that readers may treat as sacred. Existing Islamic hallucination benchmarks are largely Arabic and often closed-ended, while Bengali resources mainly study general-domain hallucination or factual knowledge. We introduce BILAL-BENCH, a 240-question expert-verified benchmark for open-ended Islamic-source fabrication in Bengali. It covers Qur'anic attribution, hadith, fiqh, contemporary issues, and adversarial false-premise prompts, and uses a two-axis rubric that separates source errors from assertiveness. We evaluate seven open-weight models from 1B to 120B on 1,680 double-annotated responses. Six of the seven models confidently fabricate on 79.3-95.3% of source-production questions, with no monotonic improvement from scale. Hadith provenance is the weakest category across models: only two of 315 responses are correct and correctly sourced. The two 8B BanglaLLaMA variants have similar confident-fabrication rates but different error profiles. Models vary far more in whether they withhold a claim they cannot support than in source accuracy: appropriate abstention or false-premise correction ranges from 6 to 46 responses across the benchmark, and does not track parameter count.

Sabah Nushra, Md Kishor Morol

Islamic University of Technology Dhaka, Bangladesh · Facebook Menlo Park, CA, USA

Hallucination in Language ModelsLLM EvaluationBengali NLPLLM Abstention
AskPDF
Jun 15, 2026

Islamic Large Language Models: From Knowledge Acquisition to Trustworthy and Hallucination-Resistant AI

Large language models (LLMs) are increasingly used for knowledge-intensive question answering, including religious and legal questions. Islamic knowledge is a particularly demanding setting: answers are expected to be grounded in authoritative sources, citations must be exact, Arabic varieties differ substantially from the language of classical sources, and legitimate jurisprudential disagreement must be represented rather than collapsed into a single answer. This survey reviews the emerging field of Islamic LLMs and trustworthy Islamic AI. We organize the literature around Arabic NLP and Arabic-centric LLMs, Islamic NLP resources, Qur'anic question answering, Islamic knowledge benchmarks, retrieval-augmented generation, Islamic legal reasoning, inheritance reasoning, hallucination evaluation, and trustworthiness. We argue that fluency in Arabic is not sufficient for Islamic AI. Reliable systems require curated sources, retrieval and verification modules, citation-aware generation, madhhab-aware reasoning, human expert evaluation, and benchmarks that measure not only answer accuracy but also faithfulness, source validity, and reasoning quality. The survey concludes with a research agenda for hallucination-resistant Islamic AI systems.

Mohammed Amine Mouhoub

Paris Dauphine University

Question AnsweringLLM ReliabilityLegal Reasoning
AskPDF
Mar 24, 2026

IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledge

Large language models are increasingly consulted for Islamic knowledge, yet no comprehensive benchmark evaluates their performance across core Islamic disciplines. We introduce IslamicMMLU, a benchmark of 10,013 multiple-choice questions spanning three tracks: Quran (2,013 questions), Hadith (4,000 questions), and Fiqh (jurisprudence, 4,000 questions). Each track is formed of multiple types of questions to examine LLMs capabilities handling different aspects of Islamic knowledge. The benchmark is used to create the IslamicMMLU public leaderboard for evaluating LLMs, and we initially evaluate 26 LLMs, where their averaged accuracy across the three tracks varied between 39.8% to 93.8% (by Gemini 3 Flash). The Quran track shows the widest span (99.3% to 32.4%), while the Fiqh track includes a novel madhab (Islamic school of jurisprudence) bias detection task revealing variable school-of-thought preferences across models. Arabic-specific models show mixed results, but they all underperform compared to frontier models. The evaluation code and leaderboard are made publicly available.

Ali Abdelaal, Mohammed Nader Al Haffar, Mahmoud Fawzi +1

The University of Edinburgh

LLM EvaluationCultural Knowledge in Language Models
AskPDF