A Hierarchical Consistency Framework for Auditing Retrieval-Augmented Generation Systems

Ramon Gonzalez; Antonio Diaz

cs.AISep 7, 2026

A Hierarchical Consistency Framework for Auditing Retrieval-Augmented Generation Systems

Authors: Ramon Gonzalez, Antonio Diaz

Organizations: Mentomy AI

Abstract

Retrieval-augmented generation (RAG) is commonly evaluated by whether the final answer is correct. That test is insufficient: an answer can match its reference while the context that produced it contains a direct contradiction, leaving the contested evidence invisible to answer-only review and retrieval relevance scores. This paper presents the Hierarchical Consistency Framework (HCF), a post-hoc, model-agnostic audit of three distinct levels of a RAG process: the knowledge corpus, the final retrieved context, and the generated answer. HCF represents corpus conflicts as source-linked atomic facts, thereby identifying the documents responsible, and returns each Answer Consistency Score (ACS) with an explanation of supporting and contradictory contextual statements. We evaluate HCF on several controlled corpora spanning five domains and 100 query-corpus instances. A human evaluator compares every generated response with its supplied ground-truth response. The results show that the three diagnostic levels can dissociate: the corpus with the highest mean retrieval similarity has the lowest mean ACS, while a structurally degraded corpus performs worse at corpus level but better at answer level. Most importantly, HCF identifies contradictory retrieved evidence in several cases where the answer still matches the ground truth. HCF does not certify factual truth; it makes the evidence supporting and challenging an answer inspectable and attributable.

Figures & tables

Panel of Figure 1 — Fig. 1: Overview of the proposed hierarchical consistency framework. Corpus Consistency characterizes the quality and agreement of the knowledge base; Retrieval Consistency characterizes the coherence of the retrieved evidence; and Answer Consistency characterizes the grounded agreement between the generated answer, the retrieved evidence, and the query.

Corpus	Docs	Words	Composition
A (Baseline)	5	12,659	5 clean documents
B (Conflict)	10	25,274	5 clean + 5 conflict-injected
C (Noise)	5	13,327	5 structurally degraded
D (Combined)	20	49,548	5 clean + 5 conflict-injected + 5 degraded + 5 combined

TABLE I: Four controlled corpora with isolated degradation modes across five science and humanities domains (physics, medicine, history, geography, biology).

Corpus	IaDNI	IeDCD	ER pairs	Conflicts	CC
A (baseline)	0.82	1.00	545	0	0.910
B (conflict)	0.81	0.93	882	58	0.870
C (noise)	0.23	1.00	489	0	0.615
D (combined)	0.55	0.89	1,438	148	0.720

TABLE II: Ingestion diagnostics. Higher IaDNI, IeDCD, and CC indicate better estimated consistency. ER pairs and conflicts are reported index counts.

Domain	Reference	Recovered	Missed	Rate
Physics	6	5	1	83.3%
Medicine	8	7	1	87.5%
History	9	5	4	55.6%
Geography	9	9	0	100.0%
Biology	8	8	0	100.0%
Total	40	34	6	85.0%

TABLE III: Recovery of selected reference facts in Corpus A.

Domain	Reference fact	Extracted entity–relation–value	Outcome
Physics	Speed of light: 299,792,458 m/s	(speed of light in vacuum, exact value, 299,792,458 m/s)	Recovered
Medicine	High-intensity atorvastatin: 40–80 mg/day	(atorvastatin, high dose, 40–80 mg/day)	Recovered
Geography	Aconcagua elevation: 6,961 m	(Aconcagua, elevation, 6,961 m)	Recovered
History	Edmund Burke authored Reflections on the Revolution in France	No corresponding authorship record	Missed

TABLE IV: Examples from the Corpus A extraction analysis.

Domain	Clean	Modified	Matched	Rate
Physics	6	6	5	83.3%
Medicine	7	7	6	85.7%
History	5	6	5	100.0%
Geography	2	6	2	100.0%
Biology	7	6	6	85.7%
Total	27	31	24	88.9%

TABLE V: Semantic correspondence among selected Corpus B records. Rate is matched divided by clean-side records.

Domain	Clean extracted record	Modified extracted record	Outcome
Medicine	(atorvastatin, high dose, 40–80 mg/day)	(atorvastatin, high dose, 40–160 mg/day)	Successfully matched; conflicting values
Physics	(Planck constant, approximate value, $6.626\times 10^{-34}$ J $\cdot$ s)	(Planck constant, approximate value, $6.200\times 10^{-34}$ J $\cdot$ s)	Successfully matched; conflicting values
Medicine	(statins, downstream effect, increased LDL receptors)	No corresponding modified record	Missing counterpart

TABLE VI: Corpus B correspondence examples, translated into English.

Fig. 2: ACS distributions by question type and corpus. Targeted questions (Q1–Q10) address manipulated relations, control questions (Q11–Q20) address unmodified facts, and synthesis questions (Q21–Q25) combine multiple facts. Points show individual questions; grouping avoids implying an ordered trend across nominal question identifiers.

Corpus	CC	$k(q)$	$SS_{1}$	$SS_{\mu}$	$SS_{\sigma}$	ACS
		mean $\pm$ standard deviation
A	0.910	$1.40\pm 0.50$	$0.9371\pm 0.0941$	$0.8705\pm 0.1619$	$0.0223\pm 0.0537$	$0.9844\pm 0.0780$
B	0.870	$2.60\pm 0.71$	$0.9431\pm 0.0929$	$0.8843\pm 0.1129$	$0.0168\pm 0.0398$	$0.8780\pm 0.2257$
C	0.615	$1.28\pm 0.46$	$0.8928\pm 0.1153$	$0.8462\pm 0.1713$	$0.0163\pm 0.0464$	$0.9320\pm 0.1749$
D	0.720	$4.68\pm 0.80$	$0.9605\pm 0.0494$	$0.8646\pm 0.1173$	$0.0257\pm 0.0388$	$0.9244\pm 0.1493$

TABLE VII: Corpus-level summaries of the 25 query records reported individually in Tables IX – XII . CC is fixed within a corpus; all remaining entries are mean

\pm

sample standard deviation across queries.

Case	Reference-matching answer	Conflicting statement identified	ACS (score)	ACS (explanation)
B–Q1	Speed of light in vacuum: 299,792,458 m/s	Approximately 250,000 km/s	0.41	“The answer agrees with the exact value in the clean passage, but another passage contradicts it with the lower value.”
B–Q4	Statin therapy is typically indefinite	Discontinuation after one to three years if lipid targets are achieved	0.41	“The answer states indefinite treatment, which conflicts with contextual guidance allowing discontinuation after one to three years.”
B–Q6	Louis XVI was executed on 21 January 1793	Execution on 28 January 1793	0.41	“The answer’s date agrees with the presented evidence, but another part of the context gives a contradictory date.”
B–Q9	Aconcagua’s elevation is 6,961 m	Elevation of 6,400 m	0.41	“The answer is supported by the document, but another part of the text gives 6,400 m, contradicting that elevation.”

TABLE VIII: Reference-matching answers with contradictory retrieved evidence in Corpus B.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Q	$k(q)$	$SS_{1}$	$SS_{\mu}$	$SS_{\sigma}$	ACS
1	2	0.9915	0.9443	0.0022	1.00
2	1	0.9990	0.9990	0.0000	1.00
3	1	0.9308	0.9308	0.0000	1.00
4	1	0.8717	0.8717	0.0000	1.00
5	2	0.9878	0.9424	0.0021	1.00
6	2	0.9922	0.9761	0.0003	1.00

Appendix

TABLE IX: Corpus A query-level results.

Q	$k(q)$	$SS_{1}$	$SS_{\mu}$	$SS_{\sigma}$	ACS
1	3	0.9914	0.9574	0.0019	0.41
2	2	0.9990	0.9934	0.0000	1.00
3	3	0.9308	0.8526	0.0032	1.00
4	3	0.9949	0.8746	0.0094	0.41
5	3	0.9878	0.9527	0.0016	1.00
6	2	0.9922	0.9894	0.0000	0.41

Appendix

TABLE X: Corpus B query-level results.

Q	$k(q)$	$SS_{1}$	$SS_{\mu}$	$SS_{\sigma}$	ACS
1	1	0.9973	0.9973	0.0000	1.00
2	2	0.9953	0.9809	0.0002	1.00
3	1	0.8779	0.8779	0.0000	1.00
4	1	0.6674	0.6674	0.0000	1.00
5	2	0.9561	0.9307	0.0006	1.00
6	2	0.9908	0.9746	0.0003	1.00

Appendix

TABLE XI: Corpus C query-level results.

Q	$k(q)$	$SS_{1}$	$SS_{\mu}$	$SS_{\sigma}$	ACS
1	4	0.9980	0.9858	0.0001	0.60
2	5	0.9988	0.9884	0.0001	1.00
3	5	0.8920	0.8654	0.0008	1.00
4	5	0.9920	0.7794	0.0531	1.00
5	3	0.9826	0.9735	0.0002	0.80
6	5	0.9923	0.9845	0.0002	1.00

Appendix

TABLE XII: Corpus D query-level results.

RAG Evaluation

Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.

See all 355 papers

Why RAGs Hallucinate: Penalty-Aware Evaluation of Retrieval-Augmented Generation Systems with Knowledge-Gap Canaries
Aug 26, 2026Alden Do Rosario, Hussein Younes, Felipe PiresHallucination in RAG Retrieval-Augmented Generation
A Systems-Level Analysis of Sensitivity, Robustness, and Stability in Retrieval-Augmented Generation
May 29, 2026Bharath Simha Reddy MuthyamRetrieval-Augmented Generation Retrieval Robustness
Independent Researcher
Towards Dependable Retrieval-Augmented Generation Using Factual Confidence Prediction
May 4, 2026Florian Geissler, Francesco Carella, Laura Fieback +1Retrieval-Augmented Generation Factual Consistency Evaluation
Fraunhofer Institute for Cognitive Systems (IKS), Munich, Germany · Volkswagen AG, Wolfsburg, Germany

Aug 26, 2026J/K move · Enter open · S save

Why RAGs Hallucinate: Penalty-Aware Evaluation of Retrieval-Augmented Generation Systems with Knowledge-Gap Canaries

Alden Do Rosario, Hussein Younes, Felipe Pires

Hallucination in RAG Retrieval-Augmented Generation RAG Evaluation LLM Abstention

Open paper AskPDF arXiv

Volume-based accuracy rewards retrieval-augmented generation (RAG) systems for guessing: a system that answers everything outscores one that declines when its knowledge base cannot support an answer. Building on the confidence-target analysis of Kalai et al. (2025), we present a penalty-aware evaluation framework for deployed RAG products, combining (i) asymmetric scoring (correct +1, wrong -4, abstain 0), (ii) knowledge-gap canaries, questions whose answers are verifiably absent from the knowledge base, so that any answer constitutes ungrounded generation from parametric memory, and (iii) a failure-attribution pipeline that separates retrieval, generation, and abstention-policy failures. Applying the framework to three commercial RAG systems and a no-retrieval baseline on SimpleQA-Verified (1,000 questions x 3 repeats, graded blind by a cross-family three-judge panel with 98.9% unanimity), we find that accuracy when answering is closely clustered across systems (97.0-98.0%), while canary violation rates differ roughly sixfold (16.7% vs. 98.1%). The systems are separated less by what they answer correctly than by whether they answer at all when they should not, and penalty-aware scoring reorders the volume-based ranking accordingly; the reordering is stable across penalty settings from k=1 to k=9. All code, configurations, transcripts, and judge votes are released for independent audit.

Abstract

RAG Evaluation

Explore similar work

Why RAGs Hallucinate: Penalty-Aware Evaluation of Retrieval-Augmented Generation Systems with Knowledge-Gap Canaries

A Systems-Level Analysis of Sensitivity, Robustness, and Stability in Retrieval-Augmented Generation

Towards Dependable Retrieval-Augmented Generation Using Factual Confidence Prediction