Fluent but Fallible: Benchmarking Hallucination and Factual Reliability in Large Language Models

Osama Shabih; Dayam Nadeem; Shahab Saquib Sohail; Mohammad Nadeem

cs.CLOct 19, 2026

Fluent but Fallible: Benchmarking Hallucination and Factual Reliability in Large Language Models

Authors: Osama Shabih, Dayam Nadeem, Shahab Saquib Sohail, Mohammad Nadeem

Organizations: Department of Computer Science and Engineering, SEST, Jamia Hamdard, New Delhi, India · Department of Computer Science, Aligarh Muslim University, Aligarh, India

Abstract

Large Language Models (LLMs) have significantly advanced natural language processing and artificial intelligence applications through their capabilities in reasoning, text generation and knowledge retrieval. However, LLMs are still vulnerable to two important challenges: hallucination, generating factually incorrect or fabricated content, and response deviation, meaning systematic deviations from ground-truth responses. These limitations raise serious concerns regarding reliability and trustworthiness in academic and research environments. This study offers a thorough comparative analysis using a variety of metrics of ten advanced LLMs ChatGPT, Gemini, Claude~~3.5~~Sonnet, LLaMA~~3, Mistral, Grok, Perplexity, DeepSeek~~V3, Falcon, and Gemma based on a benchmark consisting of 70 carefully designed academic prompts across four complexity levels: Basic, Explanation, Advanced, and Tricky, evaluated in February 2025 on specific model versions. Each response was independently evaluated by three reviewers using five metrics: Accuracy, Precision, Recall, Response Deviation Index, and Hallucination Index. Perplexity and DeepSeek~~V3 achieved the highest overall accuracy, while Gemini demonstrated strong recall and a well-balanced performance profile across all five metrics. LLaMA~~3 showed strong resistance against hallucinations through a conservative method of generation. Advanced-level prompts proved most challenging across all models, while Tricky prompts were handled more reliably owing to their binary true/false structure. Falcon achieved perfect precision and zero hallucinations, but at the cost of the lowest recall and the highest response deviation. The results show a fundamental trade-off between response completeness and factual reliability, intensified by prompt complexity.

Figures & tables

Research Theme	Sub-Category	Key Works / Models Referenced
Foundational Architectures	Transformer & Pre-training; Model Scaling & Emergence	Vaswani et al. [ 1 ] (Attention Is All You Need); Devlin et al. [ 2 ] (BERT); Brown et al. [ 4 ] (GPT-3); OpenAI GPT-4 Technical Report [ 23 ] ; Google Gemini [ 24 ] ; Anthropic Claude 3 Report [ 25 ] ; Meta LLaMA 3 [ 26 ]
Benchmarking Frameworks	General Multi-Task Benchmarks	Liang et al. [ 20 ] (HELM); Hendrycks et al. [ 21 ] (MMLU)
	Text Quality Metrics	Zhang et al. [ 27 ] (BERTScore); Papineni et al. [ 28 ] (BLEU); Banerjee & Lavie [ 29 ] (METEOR)
Hallucination Research	Taxonomy & Surveys	Ji et al. [ 22 ] (ACM Survey on Hallucination); Cao et al. [ 30 ] (Comprehensive Hallucination Survey)
	Truthfulness Benchmarks	Lin et al. [ 19 ] (TruthfulQA); Maynez et al. [ 18 ] (Faithfulness in Summarisation)
Bias & Fairness	Bias in LLMs	Bender et al. [ 10 ] (Stochastic Parrots); HELM Fairness Dimensions (Liang et al. [ 20 ] )

Table 1: Literature Taxonomy: Research Themes and Key Works

Year	Theme	Key Contribution
2017	Architecture	Vaswani et al. [ 1 ] Transformer architecture (“Attention Is All You Need”)
2018	Pre-training	Devlin et al. [ 2 ] BERT: Bidirectional pre-training for language understanding
2019	Pre-training	Brown et al. [ 4 ] GPT-3: Language models as few-shot learners
2020	Scaling / RAG	Lewis et al. [ 9 ] Retrieval-Augmented Generation (RAG) for Knowledge-intensive NLP
2021	Bias & Safety	Bender et al. [ 10 ] On the Dangers of Stochastic Parrots; Welleck et al. [ 15 ] Unlikelihood Training
2022	Truthfulness	Lin et al. [ 19 ] TruthfulQA: Measuring how models mimic human falsehoods

Table 2: Chronological Timeline of Key Related Works (2017–2026)

Study / Work	No. of Models	Domain	Metrics	Difficulty Levels	Hallucination Focus
Brown et al. [ 4 ] (GPT-3, 2020)	1	General NLP	Perplexity, Accuracy	$\times$	$\times$
Liang et al. [ 20 ] (HELM, 2022)	30+	General Multi-task	7+ metrics	$\times$	$\sim$ Partial
Lin et al. [ 19 ] (TruthfulQA, 2022)	10+	Truthfulness	Truthful%, info%	$\times$	$\checkmark$
Cao et al. [ 30 ] (2023)	Multiple	Survey / LLMs	Qualitative Taxonomy	$\times$	$\checkmark$
Hendrycks et al. [ 21 ] (MMLU, 2021)	10+	Academic Knowledge	Accuracy	$\sim$ Partial	$\times$
Maynez et al. [ 18 ] (2020)	3–4	Text Summarization	Faithfulness Metrics	$\times$	$\sim$ Partial

Table 3: Comparative Analysis: Prior Studies vs. Present Study

Figure 1: LLM Evaluation Methodology Pipeline

Model	Version / ID	Interface	Access Date	Temp.	Runs	System Prompt
ChatGPT	GPT-4o (2024-05)	chat.openai.com	Feb 2025	Default	1	None
Gemini	Gemini 1.5 Pro	gemini.google.com	Feb 2025	Default	1	None
Claude 3.5 Sonnet	claude-3-5-sonnet-20241022	claude.ai	Feb 2025	Default	1	None
LLaMA 3	Meta-Llama-3-70B	Meta AI Web UI	Feb 2025	Default	1	None
Mistral 7B	Mistral-7B-Instruct-v0.3	chat.mistral.ai	Feb 2025	Default	1	None
Grok	Grok-1.5	grok.x.ai	Feb 2025	Default	1	None

Table 4: Model Specifications and Experimental Configuration

Figure 2: Model Positions Hallucination Index (HI) Score Classification Scale

Figure 3: Comparative Performance Analysis of Large Language Models

Figure 4: Detailed Individual Model Performance Profiles

Figure 5: Overall Hallucination Rate (HR) per Model

Model	Basic (n=20)	Explanation (n=17)	Advanced (n=13)	Tricky (n=20)	Overall
Gemini	0.950	1.000	0.538	1.000	0.900
ChatGPT	0.850	0.941	0.846	0.950	0.900
Mistral	0.850	0.941	0.692	1.000	0.886
Perplexity	0.950	0.941	0.846	0.950	0.929
Grok	0.950	0.882	0.846	0.800	0.871
Claude 3.5 Sonnet	0.700	1.000	0.538	1.000	0.829

Table 5: Mean Accuracy by Model and Prompt Complexity Level

Figure 6: Performance Profile Radar – 10 LLMs across 5 Metrics

Figure 7: LLM Performance Heatmap - Metric Breakdown in Detail

Hallucination in Language Models

Hallucination in language models studies how systems produce unsupported or false claims and how to detect, explain, and reduce them.

See all 254 papers

Not All That Is Fluent Is Factual: Investigating Hallucinations of Large Language Models in Academic Writing
May 5, 2026Humam Khan, Md Tabrez Nafis, Shahab Saquib Sohail +2LLM Evaluation LLM Prompting
Department of Computer Science and Engineering, SEST Jamia Hamdard New Delhi, India · Energy, Resources and Industry WSP Calgary, Canada
KGHaluBench: A Knowledge Graph-Based Hallucination Benchmark for Evaluating the Breadth and Depth of LLM Knowledge
Feb 23, 2026Alex Robertson, Huizhi Liang, Mahbub Gani +2Hallucination in Language Models LLM Evaluation
School of Computing, Newcastle University · Sage Ai, Sage Group PLC · Current affiliation: Redis
HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs
May 4, 2026Ahmed CherifLLM Evaluation LLM Hallucination Mitigation
1Sofrecom Tunisia, Orange Innovation, Tunis, 1053, Tunisia.

May 5, 2026J/K move · Enter open · S save

Not All That Is Fluent Is Factual: Investigating Hallucinations of Large Language Models in Academic Writing

Humam Khan, Md Tabrez Nafis, Shahab Saquib Sohail, Aqeel Khalique+1

Department of Computer Science and Engineering, SEST Jamia Hamdard New Delhi, India · Energy, Resources and Industry WSP Calgary, Canada

LLM Evaluation LLM Prompting Hallucination in Language Models

Open paper AskPDF arXiv

Large Language models (LLMs) show extraordinary abilities, but they are still prone to hallucinations, especially when we use them for generating Academic content. We have investigated four popular LLMs, ChatGPT, Grok, Gemini, and Copilot for hallucinations specifically for academic writing. We have designed 80 prompts across four categories, namely, reference generation, factual explanation, abstract generation, and writing improvement. We evaluated the model using a 0-5 rubric score, which checks factual accuracy, reference validity, coherence, style consistency, and academic tone. A novel weighted metric, Hallucination Index (HI), was introduced to measure hallucination in the responses generated by the models. Some of the most widely used evaluation metrics often fail to check errors which alter sentiment in machine-translated text. We found that Grok and Copilot perform better on reference generation tasks, but they often struggle with abstract or stylistic prompts, with HI values of 0.67 and 0.70, respectively. Whereas, Gemini and ChatGPT have done well with having stronger tone control, but they lack in writing factual tasks and higher hallucination risk with HI scores of 0.53 and 0.57, respectively. Our study found that hallucination behavior does not depend solely on model architecture but also on the type of task and the prompting conditions we are providing. We propose that our work opens new research dimensions for future researchers.

Abstract

Hallucination in Language Models

Explore similar work

Not All That Is Fluent Is Factual: Investigating Hallucinations of Large Language Models in Academic Writing

KGHaluBench: A Knowledge Graph-Based Hallucination Benchmark for Evaluating the Breadth and Depth of LLM Knowledge

HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs