Retromorphic Testing with Hierarchical Verification for Hallucination Detection in RAG

Boxi Yu; Yuzhong Zhang; Liting Lin; Lionel Briand; Emir Muñoz

cs.CLMar 29, 2026

Retromorphic Testing with Hierarchical Verification for Hallucination Detection in RAG

Authors: Boxi Yu, Yuzhong Zhang, Liting Lin, Lionel Briand, Emir Muñoz

Organizations: Lero, the Research Ireland Centre for Software, University of Limerick, Ireland · The Chinese University of Hong Kong, Shenzhen, China · University of Ottawa, Canada · Genesys, Ireland

Abstract

Large language models can still hallucinate in retrieval-augmented generation (RAG), producing claims that are unsupported by or conflict with the retrieved context. Detecting such errors remains challenging when faithfulness is judged solely against the retrieved context: many existing detectors return holistic answer-level scores, while others target open-domain factuality or fail to provide evidence-grounded diagnostics. We present RT4CHART, a retromorphic testing framework for context-faithfulness assessment. RT4CHART decomposes an answer into independently verifiable claims, performs hierarchical local-to-global verification against the retrieved context, and assigns each claim one of three labels: entailed, contradicted, or baseless. It further maps these claim-level decisions back to specific answer spans and returns explicit context-side evidence, enabling fine-grained auditing rather than opaque scoring. We evaluate RT4CHART on RAGTruth++ (408 samples) and our re-annotated RAGTruth-Enhance (2,675 samples). RT4CHART achieves the best answer-level hallucination-detection F1 score among the evaluated baselines. On RAGTruth++, it attains a precision of 0.845, a recall of 0.718, and an F1 score of 0.776, representing an 83% relative improvement over the strongest baseline. It also achieves a span-level F1 score of 47.5% on RAGTruth-Enhance. Ablation studies show that claim-based local processing drives most of the observed improvement, while global verification provides selective benefits across datasets. Finally, our re-annotation identifies 1.68X more hallucination cases than the original labels, suggesting that commonly used benchmarks substantially underestimate the prevalence of hallucination.

Figures & tables

Figure 1. Detection granularity on a motivating example. RT4CHART adds explicit context evidence to localized answer spans. Red denotes Contradicted , green Entailed , and yellow Baseless . A motivating example showing context, question, and answer. Answer-level detection returns a single score for the entire response. Span-level detection highlights unfaithful fragments. RT4CHART links localized answer spans to supporting or contradicting context evidence. Red marks contradicted content, green marks entailed content, and yellow marks baseless content.

Figure 2. Overview of RT4CHART. The system decomposes $A$ into claims, verifies them locally and globally against $C$ , and returns claim-level and answer-level outputs. A pipeline overview of RT4CHART showing claim decomposition, local verification over overlapping context chunks, global verification over the full context, answer-side localization, context-side evidence extraction, and answer-level aggregation.

	Local verification			OR-join	Final
Claim	$\mathit{Chk}_{0}$	$\mathit{Chk}_{1}$	$\mathit{Chk}_{2}$	$\hat{y}_{i}$	$y_{i}^{*}$
	$\{s_{0},s_{1}\}$	$\{s_{1},s_{2}\}$	$\{s_{2},s_{3}\}$
$\mathit{Clm}_{1}$	Con	Con	Nic	Con	Con
$\mathit{Clm}_{2}$	Con	Con	Nic	Con	Con
$\mathit{Clm}_{3}$	Nic	Ent	Ent	Ent	Ent
$\mathit{Clm}_{4}$	Nic	Nic	Nic	Nic	Ent

Table 1. Local assessment matrix for the running example with

W{=}2

and

O{=}1

. The first three columns show chunk-level assessments, followed by the OR-joined local label

\hat{y}_{i}

and the final label

y_{i}^{*}

. Because

\mathit{Clm}_{1}

and

\mathit{Clm}_{2}

are Con , the answer-level AND-join returns Con (Section 3.6 ). A matrix showing five claims as rows and three overlapping local chunks as columns. Claim texts are omitted for compactness. The first three columns show chunk-specific assessments, followed by the locally aggregated label and the final claim label after global verification. Claim 4 is locally baseless in all chunks, but becomes entailed after global verification. Because Claims 1 and 2 are contradicted and contradiction has priority in the AND-join, the final answer-level label is contradicted.

Local label	Prompt input	Likely local failure mode	Verification goal
Nic	Full $C$ + claim	No single chunk contains enough evidence	Confirm baseless or revise to entailed
Con	Full $C$ + focus chunk + claim	Local contradiction, globally resolved	Confirm contradiction or revise
Ent	Full $C$ + focus chunk + claim	Locally entailed, globally overridden	Confirm entailment or revise

Table 2. Global verification strategy conditioned on the local claim label.

RAGTruth++
Detector	Precision	Recall	F1
RT4CHART	0.845	0.718	0.776
Lettuce	0.921	0.233	0.371
MetaQA*	0.941	0.216	0.352
SelfCheckGPT*	0.843	0.143	0.244
Vectara*	1.000	0.269	0.424

Table 3. Main answer-level hallucination detection results on RAGTruth++ and RAGTruth-Enhance (Precision/Recall/F1). Methods marked with “*” are adapted to the context-only setting.

Dataset	Detector	Precision	Recall	F1
RAGTruth-Enhance	RT4CHART	39.9%	58.6%	47.5%
RAGTruth-Enhance	Lettuce	72.7%	29.4%	41.9%
RAGTruth++	RT4CHART	50.8%	33.6%	40.4%
RAGTruth++	Lettuce	91.2%	13.5%	23.5%

Table 4. Answer-side span localization results on RAGTruth-Enhance and RAGTruth++ (Precision/Recall/F1).

Task	Method	Precision	Recall	F1
Contradiction-case detection	RT4CHART	62.9%	40.9%	49.6%
	GPT-4o mini (direct)	42.0%	27.5%	33.2%
Refuting-evidence grounding	RT4CHART	51.1%	59.1%	54.8%
	GPT-4o mini (direct)	36.5%	34.9%	35.7%

Table 5. Contradiction-case detection and refuting-evidence grounding on RAGTruth-Enhance (Precision/Recall/F1).

Dataset	Setting	Precision	Recall	F1
RAGTruth++	w/ global verif.	0.845	0.718	0.776
RAGTruth++	w/o global verif.	0.851	0.655	0.740
RAGTruth-Enhance	w/ global verif.	0.781	0.920	0.845
RAGTruth-Enhance	w/o global verif.	0.811	0.881	0.845

Table 6. Effect of global verification on answer-level hallucination detection (Precision/Recall/F1).

Variant	Precision	Recall	F1	Span F1
RT4CHART (full pipeline)	0.845	0.718	0.776	0.404
GPT-4o mini (direct spans)	0.861	0.630	0.727	0.081

Table 7. RT4CHART versus the direct span-prediction variant on RAGTruth++ (answer-level Precision/Recall/F1 and span F1).

Task	Mode	Precision	Recall	F1
QA	Sent.	0.735	0.892	0.806
QA	Hol.	0.755	0.830	0.791
Summary	Sent.	0.668	0.829	0.740
Summary	Hol.	0.716	0.770	0.742
Data2Text	Sent.	0.877	0.983	0.927
Data2Text	Hol.	0.902	0.943	0.922

Table 8. Decomposition ablation on RAGTruth-Enhance. Sentence-based and Holistic yield similar overall F1, with a precision–recall trade-off by task type.

$(W,O)$	P/R/F1
(15,5)	0.679/0.907/0.777
(15,10)	0.636/0.895/0.743
(25,5)	0.768/0.904/0.830
(25,10)	0.772/0.921/0.840
(35,5)	0.778/0.909/0.838
(35,10)	0.771/0.902/0.831

Table 9. Chunking sensitivity on RAGTruth-Enhance under different

(W,O)

settings (answer-level P/R/F1).

Dataset	Model	Precision	Recall	F1
RAGTruth++	GPT-4o mini	0.845	0.718	0.776
RAGTruth++	GLM-4.7	0.808	0.983	0.887
RAGTruth-Enhance	GPT-4o mini	0.781	0.920	0.845
RAGTruth-Enhance	GLM-4.7	0.663	0.992	0.795

Table 10. Cross-model robustness when swapping the decomposition model and judge model (answer-level Precision/Recall/F1).

Hallucination in RAG

See all 19 papers

Detecting Hallucinations in Retrieval-Augmented Generation through Grounding-Aware Sensitivity by Perturbation (GASP)
Jul 5, 2026Mohamed Aly BoukeRetrieval-Augmented Generation LLM Grounding
Centre for Intelligent Cloud Computing, CoE for Advanced Cloud, Faculty of Information Science and Technology, Multimedia University, Jalan Ayer Keroh Lama, Bukit Beruang, 75450, Melaka, Malaysia
Evidence Graph Consistency in Retrieval-Augmented Generation: A Model-Dependent Analysis of Hallucination Detection
Jun 4, 2026Jianru ShenRetrieval-Augmented Generation Hallucination in Language Models
University of Montana Missoula, MT, USA
CORTEX: Token-Level Hallucination Detection in RAG via Comparative Internal Representations
Jun 30, 2026Kazuaki Furumai, Shuichiro Haruta, Kazunori Matsumoto +1Retrieval-Augmented Generation LLM Hallucination Detection
KDDI Research, Inc.

Jul 5, 2026J/K move · Enter open · S save

Detecting Hallucinations in Retrieval-Augmented Generation through Grounding-Aware Sensitivity by Perturbation (GASP)

Mohamed Aly Bouke

Centre for Intelligent Cloud Computing, CoE for Advanced Cloud, Faculty of Information Science and Technology, Multimedia University, Jalan Ayer Keroh Lama, Bukit Beruang, 75450, Melaka, Malaysia

Retrieval-Augmented Generation LLM Grounding Hallucination Detection LLM Hallucination Detection Hallucination in RAG

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) reduces but does not eliminate hallucination, and existing detectors return a single answer-level score that does not indicate which sentence is unsupported, or why. To close this gap, we introduce Grounding-Aware Sensitivity by Perturbation (GASP), a span-level detector that scores each answer sentence by how strongly its likelihood depends on the retrieved evidence, a quantity we term grounding sensitivity. GASP holds the answer fixed and re-scores it under the full context, under no context, and with each chunk removed, then measures the log-likelihood drops and Jensen-Shannon divergences (JSD). The likelihood of a grounded sentence collapses once its supporting passage is removed, whereas a hallucinated sentence is almost unaffected, a contrast we interpret by casting decoding as a random nonlinear iterated function system (RNIFS). We evaluate GASP on three benchmarks (RAGTruth, TofuEval, RAGBench) with three instruction-tuned scorers from two model families (Qwen2.5-0.5B, Qwen2.5-1.5B, and SmolLM2-1.7B) under a leakage-clean protocol. On RAGTruth it reaches a response-level area under the ROC curve (AUC) of about 0.73 and a span-level AUC of about 0.67, improving significantly over perplexity and by clear margins over length, whole-context natural language inference (NLI), and self-consistency baselines. The only baseline competitive at the span level is a well-configured chunk-level entailment verifier, which requires a separate model, whereas a training-free threshold on the grounding features matches the trained classifier without labeled data and serves as the default detector. Beyond RAGTruth, the signal transfers to TofuEval but not to short-answer question answering in RAGBench, showing GASP is best suited to outputs constructed from the retrieved context rather than answers recoverable from parametric knowledge.

Abstract

Hallucination in RAG

Explore similar work

Detecting Hallucinations in Retrieval-Augmented Generation through Grounding-Aware Sensitivity by Perturbation (GASP)

Evidence Graph Consistency in Retrieval-Augmented Generation: A Model-Dependent Analysis of Hallucination Detection

CORTEX: Token-Level Hallucination Detection in RAG via Comparative Internal Representations