Facet-Level Tracing of Evidence Uncertainty and Hallucination in RAG

Passant Elchafei; Monorama Swain; Shahed Masoudian; Markus Schedl

cs.CLApr 10, 2026

Facet-Level Tracing of Evidence Uncertainty and Hallucination in RAG

Authors: Passant Elchafei, Monorama Swain, Shahed Masoudian, Markus Schedl

Organizations: Johannes Kepler University Linz, Institute of Computational Perception, Linz, Austria · Linz Institute of Technology, Artificial Intelligence Lab, Austria

Abstract

Retrieval-Augmented Generation (RAG) aims to reduce hallucination by grounding answers in retrieved evidence, yet hallucinated answers remain common even when relevant documents are available. Existing evaluations focus on answer-level or passage-level accuracy, offering limited insight into how evidence is used during generation. In this work, we introduce a facet-level diagnostics framework for QA that decomposes each input question into atomic reasoning facets. For each facet, we assess evidence sufficiency and grounding using a structured Facet x Chunk matrix that combines retrieval relevance with natural language inference-based faithfulness scores. To diagnose evidence usage, we analyze three controlled inference modes: Strict RAG, which enforces exclusive reliance on retrieved evidence; Soft RAG, which allows integration of retrieved evidence and parametric knowledge; and LLM-only generation without retrieval. Comparing these modes enables thorough analysis of retrieval-generation misalignment, defined as cases where relevant evidence is retrieved but not correctly integrated during generation. Across medical QA and HotpotQA, we evaluate three open-source and closed-source LLMs (GPT, Gemini, and LLaMA), providing interpretable diagnostics that reveal recurring facet-level failure modes, including evidence absence, evidence misalignment, and prior-driven overrides. Our results demonstrate that hallucinations in RAG systems are driven less by retrieval accuracy and more by how retrieved evidence is integrated during generation, with facet-level analysis exposing systematic evidence override and misalignment patterns that remain hidden under answer-level evaluation.

Figures & tables

Figure 1: Facet-RAG Pipeline. The framework decomposes questions into reasoning facets, retrieves evidence at the chunk level, and evaluates facet-level entailment and generation faithfulness to diagnose hallucination and uncertainty in RAG systems.

Facet	C1	C2	C3	C4	C5	C6	C7	C8	C9	C10	Max
1) What is the type of place associated with Regency Road, Adelaide?	0.00	0.14	0.00	0.27	0.18	0.04	0.06	0.01	0.02	0.47	0.47
2) What is the type of place associated with Klemzig, South Australia?	0.14	0.01	0.00	0.21	0.00	0.14	0.01	0.30	0.14	0.03	0.30

Table 1: HotpotQA Example: Facet-level coverage over retrieved chunks (C1–C10). Main question: “What type of place does Regency Road, Adelaide and Klemzig, South Australia have in common?” Question type: bridge . Difficulty level: hard .

Model	Strict RAG			Soft RAG			LLM-only
	Answer	F1	BERT	Answer	F1	BERT	Answer	F1	BERT
Gemini-2.0-Flash	Bosnian	0.00	0.00	Bosnian	0.00	0.13	Bosnian	0.00	0.13
GPT-4o mini	NO_ANSWER	0.00	0.00	Bosnian	0.00	0.13	Serb	1.00	1.00
Llama-3-8B-Instruct	NO_ANSWER	0.00	0.00	Bosnian	0.00	0.13	The Grand Vizier…	0.00	0.11

Table 2: HotpotQA Example: Generated answers and corresponding F1 and BERTScore for each inference mode. Question: “Of what national descent was the Grand Vizier of the Ottoman Empire whose title indicated he had been castrated?” ( Gold answer: Serb ). Only GPT in the LLM-only setting produces the correct answer, while Strict and Soft RAG suppress correct parametric knowledge across models.

Models	Evidence Taxonomy
Models	Failure	Overridden	Misalignment	Helpful	Robust
Gemini-2.0 Flash	20.6	12.5	22.3	9.6	35.1
GPT-4o mini	0.2	34.4	0.2	51.6	13.7
Llama-3-8B-Instruct	0.1	38.2	0.2	53.1	8.3
Overall	7.0	28.4	7.5	38.1	19.0

Table 3: Medical Dataset: Evidence Taxonomy Distribution Across Models. Percentage of medical reasoning facets (N=11,007) in each evidence category. Overall Evidence Helpful (38.1%) demonstrates that domain-specific medical retrieval frequently corrects model errors. Gemini exhibits balanced behavior across failure and override, while GPT and LLaMA demonstrate override-dominant patterns.

Question Type	Model	Strict RAG		Soft RAG		LLM-only
Question Type	Model	F1	BERT	F1	BERT	F1	BERT
Fact Retrieval	Gemini-2.0-Flash	0.222	0.093	0.376	0.375	0.337	0.351
	GPT-4o mini	0.185	0.008	0.232	0.139	0.186	0.101
	Llama-3-8B-Instruct	0.291	0.231	0.351	0.317	0.309	0.251
Complex Reasoning	Gemini-2.0-flash	0.176	0.034	0.357	0.360	0.340	0.337
	GPT-4o mini	0.089	0.025	0.161	0.116	0.127	0.087

Table 4: Medical Dataset: Mean F1 and BERTScore by question type and inference mode. In medical dataset, Soft RAG outperforms Strict RAG across models, especially for Complex Reasoning

Figure 2: HotpotQA: Evidence Taxonomy Distribution by Difficulty Level. Override and Helpful are balanced on easy questions, but Helpful declines on harder questions while Override stays stable. Failure and Misalignment remain constant and minimal, while Evidence Robust decreases with difficulty, indicating complexity disrupts generation rather than retrieval .

Models	Evidence Taxonomy
Models	Failure	Overridden	Misalignment	Helpful	Robust
Gemini-2.0 Flash	14.7	32.1	0.7	34.5	18.0
GPT-4o mini	2.8	43.6	0.7	37.3	15.6
Llama-3-8B-Instruct	2.0	47.1	2.7	41.0	7.2
Overall	5.8	42.3	0.7	37.6	13.6

Table 5: HotpotQA: Evidence Taxonomy Distribution Across Models. Distribution of reasoning facets (N=24,039) by evidence behavior. Evidence Override (42.3%) exceeds Failure (5.8%) by 7×, confirming generation-side grounding challenges persist across domains.

Question Difficulty	Model	Strict RAG		Soft RAG		LLM-only
		F1	BERT	F1	BERT	F1	BERT
Easy	Gemini-2.0-Flash	0.550	0.443	0.624	0.539	0.564	0.544
	GPT-4o-mini	0.519	0.447	0.613	0.558	0.498	0.477
	LLaMA-3-8B-Instruct	0.196	0.026	0.315	0.174	0.230	0.108
Medium	Gemini-2.0-Flash	0.528	0.429	0.687	0.624	0.582	0.623
	GPT-4o-mini	0.501	0.389	0.699	0.631	0.504	0.518

Table 6: HotpotQA: Question-Level by Difficulty. Mean F1 and BERTScore by difficulty and inference mode. Soft RAG outperforms Strict RAG, and performance peaks at medium difficulty.

Question Type	Model	Strict RAG		Soft RAG		LLM-only
		F1	BERT	F1	BERT	F1	BERT
Bridge	Gemini-2.0-Flash	0.561	0.447	0.631	0.553	0.476	0.483
	GPT-4o mini	0.483	0.350	0.655	0.605	0.375	0.374
	LLaMA-3-8B-Instruct	0.222	0.135	0.369	0.240	0.170	0.416
Comparison	Gemini-2.0-Flash	0.472	0.377	0.614	0.547	0.609	0.614
	GPT-4o mini	0.470	0.417	0.603	0.544	0.561	0.563

Table 7: HotpotQA: Question-level answer quality by question meta-type. Mean F1 and BERTScore are reported across inference modes.

Figure 3: Medical Dataset: Facet semantic type × evidence taxonomy distribution. Boolean and Temporal facets show highest failure rates. Comparative facets are most unstable with highest misalignment and lowest robust rates.

Figure 4: HotpotQA: Facet Semantic Type × Evidence Taxonomy Distribution. Boolean and Temporal facets show lowest failure rates. Override rates are consistently high across all types. Comparative facets remain most unstable in both datasets.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Illustrative facet-level diagnostic example from HotpotQA.

Figure 6: Medical Dataset: Facet-level faithfulness distributions under Soft RAG across evidence taxonomy categories.

Figure 7: Medical Dataset: Per-question $\Delta$ F1 distributions ( $\text{Soft}-\text{Strict}$ ) by model.

Hallucination in RAG

See all 19 papers

When Does Uncertainty Measure RAG Hallucination Risk?
Oct 15, 2026Rajeel Ahmad Ansari, Tautik AgrahariHallucination in RAG LLM Hallucination Detection
Independent Researcher, Delhi, India · Independent Researcher, Bengaluru, India
Evidence Graph Consistency in Retrieval-Augmented Generation: A Model-Dependent Analysis of Hallucination Detection
Jun 4, 2026Jianru ShenRetrieval-Augmented Generation Hallucination in Language Models
University of Montana Missoula, MT, USA
Retromorphic Testing with Hierarchical Verification for Hallucination Detection in RAG
Mar 29, 2026Boxi Yu, Yuzhong Zhang, Liting Lin +2Hallucination in RAG LLM Hallucination Detection
Lero, the Research Ireland Centre for Software, University of Limerick, Ireland · The Chinese University of Hong Kong, Shenzhen, China · University of Ottawa, Canada +1

Oct 15, 2026J/K move · Enter open · S save

When Does Uncertainty Measure RAG Hallucination Risk?

Rajeel Ahmad Ansari, Tautik Agrahari

Independent Researcher, Delhi, India · Independent Researcher, Bengaluru, India

Hallucination in RAG LLM Hallucination Detection Retrieval-Augmented Generation RAG Evaluation Uncertainty Calibration

Open paper AskPDF arXiv

Uncertainty can measure hallucination risk in retrieval-augmented generation (RAG), but only when retrieved evidence constrains the answer. When evidence is present, unsupported factual claims become less likely; when retrieval coverage is poor, the model can answer from internal knowledge and assign high probability to unsupported claims. We test this across four instruction-tuned models on HotpotQA and SQuAD. After matching faithful and hallucinated answers within word-count bins, length-only classifiers fall near chance, while perplexity reaches 0.81--0.91 AUROC on targeted unsupported edits. Varying retrieval depth shows why the score changes meaning: detection can weaken with only the top-ranked passage, then recover after adding a second passage. A combined uncertainty and semantic-drift score improves calibration, and measurements remain above chance on RAGTruth QA. Results support a conditional view of uncertainty as a useful measure of RAG hallucination risk.

Abstract

Hallucination in RAG

Explore similar work

When Does Uncertainty Measure RAG Hallucination Risk?

Evidence Graph Consistency in Retrieval-Augmented Generation: A Model-Dependent Analysis of Hallucination Detection

Retromorphic Testing with Hierarchical Verification for Hallucination Detection in RAG