KGHaluBench: A Knowledge Graph-Based Hallucination Benchmark for Evaluating the Breadth and Depth of LLM Knowledge

Alex Robertson; Huizhi Liang; Mahbub Gani; Rohit Kumar; Srijith Rajamohan

cs.CLFeb 23, 2026

KGHaluBench: A Knowledge Graph-Based Hallucination Benchmark for Evaluating the Breadth and Depth of LLM Knowledge

Authors: Alex Robertson, Huizhi Liang, Mahbub Gani, Rohit Kumar, Srijith Rajamohan

Organizations: School of Computing, Newcastle University · Sage Ai, Sage Group PLC · Current affiliation: Redis

Abstract

Large Language Models (LLMs) possess a remarkable capacity to generate persuasive and intelligible language. However, coherence does not equate to truthfulness, as the responses often contain subtle hallucinations. Existing benchmarks are limited by static and narrow questions, leading to limited coverage and misleading evaluations. We present KGHaluBench, a Knowledge Graph-based hallucination benchmark that assesses LLMs across the breadth and depth of their knowledge, providing a fairer and more comprehensive insight into LLM truthfulness. Our framework utilises the KG to dynamically construct challenging, multifaceted questions, whose difficulty is then statistically estimated to address popularity bias. Our automated verification pipeline detects abstentions and verifies the LLM's response at both conceptual and correctness levels to identify different types of hallucinations. We evaluate 25 frontier models, using novel accuracy and hallucination metrics. The results provide a more interpretable insight into the knowledge factors that cause hallucinations across different model sizes. KGHaluBench is publicly available to support future developments in hallucination mitigation.

Figures & tables

Figure 2: Distribution of Very Common , Common , and Uncommon Entity Types within the KGHaluBench assessment

Panel of Figure 3 — Figure 3: Question Template used for Question Generation.

Figure 4: Overview of the fact verification pipeline

Figure 5: Top 10 Models by Weighted Accuracy. See Appendix A.1 for the full table of results.

Figure 6: $\mathrm{Halu}_{\mathrm{BOK}}$ and $\mathrm{Halu}_{\mathrm{DOK}}$ hallucination rates across all models

Figure 7: Distribution of fact-level hallucinations across all models, showing the proportion of responses containing 0, 1, 2, or 3 hallucinated facts

Figure 8: Distribution of entity-level filter classifications across all models, showing the proportion of responses classified as aligned, hallucinated, or abstained

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model Version	Knowledge Cut-off	Model Size	$W_{\text{a}}$	Abstain Rate	$\mathrm{Halu}_{\mathrm{BOK}}$	$\mathrm{Halu}_{\mathrm{DOK}}$
Open-Source Models	(Date)	(Billion)	(%)	(%)	(%)	(%)
Meta-Llama-3.1-8B-Instruct-fast	12/2023	8	28.89	31.20	43.64	52.46
gemma-2-9b-it-fast	-	9	19.54	2.53	60.21	51.67
GPT-oss-20b	06/2024	20	29.45	55.93	49.56	52.32
Qwen3-30B-A3B	-	30	22.20	13.60	58.36	51.41
QwQ-32B-fast	11/2024	32	20.18	7.13	61.99	49.88

Appendix

Table 1: Performance of 25 language models on the KGHaluBench using the metrics of

W_{\text{a}}

\mathrm{Halu}_{\mathrm{BOK}}

, and

\mathrm{Halu}_{\mathrm{DOK}}

, along with their version specifications. All models were evaluated across 10 assessments, each comprising 150 questions, with results aggregated using the mean. Models were evaluated using default parameters in accordance with their respective usage policies. The best performing results for the open-source and proprietary models are in bold , with the second-best values underlined . The knowledge cut-off dates were collected from https://github.com/HaoooWang/llm-knowledge-cutoff-dates . Citations: Llama families Grattafiori et al. (2024) ; Bercovich et al. (2025) , Qwen families Qwen et al. (2025) ; Yang et al. (2025) , Gemma2 Team et al. (2024) , GLM-4.5 Team (2025a) , Kimi-K2 Team (2025b) DeepSeek families DeepSeek-AI et al. (2025a) ; DeepSeek-AI et al. (2025b) , GPT families OpenAI (2024) , Gemini families Comanici et al. (2025)

Model		P %	R %	F1 %	A %
KGHaluBench	0.750	75.76	73.17	74.44	79.19
	0.725	70.45	84.88	76.99	78.99
	0.700	66.44	94.63	78.07	77.98
GPT-3.5-Turbo	–	82.46	45.85	58.93	73.54

Appendix

Table 2: Entity-Level Filter Performance Across Thresholds Compared to GPT-3.5-Turbo (P = Precision, R = Recall, A = Human Agreement)

Filter	Facts Verified (%)	Avg Time (s)
NLI Entailment	27.58	0.36
LLM Entailment	71.39	1.91
Expert Decision	1.03	2.35
Average Verification Time per Fact (s): 1.49
Average Verification Time per Question (s): 4.47

Appendix

Table 3: Distribution of verified facts and average processing time in the fact verification pipeline

Metric	Mean Acc. (%)	Std. Dev. (%)
Accuracy	45.30	2.57
$W_{\text{a}}$	45.25	2.45
	-0.05	-0.12

Appendix

Table 4: Mean and Standard deviation of Accuracy and

W_{\text{a}}

, over the 10 runs per model. Then averaged across all models in the experiment dataset

Hallucination in Language Models

Hallucination in language models studies how systems produce unsupported or false claims and how to detect, explain, and reduce them.

See all 254 papers

KnowHal: A Knowledge-Driven Benchmark for Comprehensive Multimodal Hallucination Evaluation
Aug 4, 2026Ruihan Li, Jiyang Tan, Kailin Jiang +5Multimodal Hallucination Multimodal Model Evaluation
1C-FAIR & School of software, Shandong University · University of Science and Technology of China · 3Jiangnan University
HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs
May 4, 2026Ahmed CherifLLM Evaluation LLM Hallucination Mitigation
1Sofrecom Tunisia, Orange Innovation, Tunis, 1053, Tunisia.
PhantomBench: Benchmarking the Non-existential Threat of Language Models
Jun 9, 2026Haeji Jung, Hila GonenLLM Evaluation Hallucination in Language Models
University of British Columbia · Canada CIFAR AI Chair, Amii

Aug 4, 2026J/K move · Enter open · S save

KnowHal: A Knowledge-Driven Benchmark for Comprehensive Multimodal Hallucination Evaluation

Ruihan Li, Jiyang Tan, Kailin Jiang, Huining Li+4

1C-FAIR & School of software, Shandong University · University of Science and Technology of China · 3Jiangnan University

Multimodal Hallucination Multimodal Model Evaluation

Open paper AskPDF arXiv

Hallucination remains a critical challenge for developing trustworthy Multimodal Large Language Models (MLLMs). While existing benchmarks mainly focus on entity, attribute, and relation hallucinations, knowledge-related failures are often investigated separately, lacking a unified evaluation framework across different hallucination dimensions. To overcome this, we propose KnowHal, a benchmark that explicitly incorporates knowledge hallucination into multimodal hallucination evaluation spanning four dimensions: entity, attribute, relation, and knowledge. KnowHal constructs paired positive and negative questions over shared images and entities, enabling controlled comparisons among perceptual errors, knowledge-related errors, and false-premise acceptance. The benchmark contains 1,800 samples across 10 domains and 50 categories, constructed through a semi-automated pipeline combining LLM assistance, CLIP-based filtering, and human verification. We evaluate 14 representative MLLMs on KnowHal and conduct extensive analyses. Results show that the knowledge dimension consistently presents the greatest challenge for nearly all evaluated models, while most models exhibit substantial performance degradation on negative questions, revealing limited robustness to false premises. By unifying four hallucination dimensions with paired question design, KnowHal addresses an important gap in existing evaluation frameworks and enables a more comprehensive assessment of hallucinations in MLLMs.

Abstract

Hallucination in Language Models

Explore similar work

KnowHal: A Knowledge-Driven Benchmark for Comprehensive Multimodal Hallucination Evaluation

HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs

PhantomBench: Benchmarking the Non-existential Threat of Language Models