Retrieval-augmented generation (RAG) is commonly evaluated by whether the final answer is correct. That test is insufficient: an answer can match its reference while the context that produced it contains a direct contradiction, leaving the contested evidence invisible to answer-only review and retrieval relevance scores. This paper presents the Hierarchical Consistency Framework (HCF), a post-hoc, model-agnostic audit of three distinct levels of a RAG process: the knowledge corpus, the final retrieved context, and the generated answer. HCF represents corpus conflicts as source-linked atomic facts, thereby identifying the documents responsible, and returns each Answer Consistency Score (ACS) with an explanation of supporting and contradictory contextual statements. We evaluate HCF on several controlled corpora spanning five domains and 100 query-corpus instances. A human evaluator compares every generated response with its supplied ground-truth response. The results show that the three diagnostic levels can dissociate: the corpus with the highest mean retrieval similarity has the lowest mean ACS, while a structurally degraded corpus performs worse at corpus level but better at answer level. Most importantly, HCF identifies contradictory retrieved evidence in several cases where the answer still matches the ground truth. HCF does not certify factual truth; it makes the evidence supporting and challenging an answer inspectable and attributable.
Figures & tables
Fig. 1: Overview of the proposed hierarchical consistency framework. Corpus Consistency characterizes the quality and agreement of the knowledge base; Retrieval Consistency characterizes the coherence of the retrieved evidence; and Answer Consistency characterizes the grounded agreement between the generated answer, the retrieved evidence, and the query.
TABLE I: Four controlled corpora with isolated degradation modes across five science and humanities domains (physics, medicine, history, geography, biology).
Corpus
IaDNI
IeDCD
ER pairs
Conflicts
CC
A (baseline)
0.82
1.00
545
0
0.910
B (conflict)
0.81
0.93
882
58
0.870
C (noise)
0.23
1.00
489
0
0.615
D (combined)
0.55
0.89
1,438
148
0.720
TABLE II: Ingestion diagnostics. Higher IaDNI, IeDCD, and CC indicate better estimated consistency. ER pairs and conflicts are reported index counts.
Domain
Reference
Recovered
Missed
Rate
Physics
6
5
1
83.3%
Medicine
8
7
1
87.5%
History
9
5
4
55.6%
Geography
9
9
0
100.0%
Biology
8
8
0
100.0%
Total
40
34
6
85.0%
TABLE III: Recovery of selected reference facts in Corpus A.
Domain
Reference fact
Extracted entity–relation–value
Outcome
Physics
Speed of light: 299,792,458 m/s
(speed of light in vacuum, exact value, 299,792,458 m/s)
Recovered
Medicine
High-intensity atorvastatin: 40–80 mg/day
(atorvastatin, high dose, 40–80 mg/day)
Recovered
Geography
Aconcagua elevation: 6,961 m
(Aconcagua, elevation, 6,961 m)
Recovered
History
Edmund Burke authored Reflections on the Revolution in France
No corresponding authorship record
Missed
TABLE IV: Examples from the Corpus A extraction analysis.
Domain
Clean
Modified
Matched
Rate
Physics
6
6
5
83.3%
Medicine
7
7
6
85.7%
History
5
6
5
100.0%
Geography
2
6
2
100.0%
Biology
7
6
6
85.7%
Total
27
31
24
88.9%
TABLE V: Semantic correspondence among selected Corpus B records. Rate is matched divided by clean-side records.
TABLE VI: Corpus B correspondence examples, translated into English.
Fig. 2: ACS distributions by question type and corpus. Targeted questions (Q1–Q10) address manipulated relations, control questions (Q11–Q20) address unmodified facts, and synthesis questions (Q21–Q25) combine multiple facts. Points show individual questions; grouping avoids implying an ordered trend across nominal question identifiers.
Corpus
CC
k(q)
SS1
SSμ
SSσ
ACS
mean ± standard deviation
A
0.910
1.40±0.50
0.9371±0.0941
0.8705±0.1619
0.0223±0.0537
0.9844±0.0780
B
0.870
2.60±0.71
0.9431±0.0929
0.8843±0.1129
0.0168±0.0398
0.8780±0.2257
C
0.615
1.28±0.46
0.8928±0.1153
0.8462±0.1713
0.0163±0.0464
0.9320±0.1749
D
0.720
4.68±0.80
0.9605±0.0494
0.8646±0.1173
0.0257±0.0388
0.9244±0.1493
TABLE VII: Corpus-level summaries of the 25 query records reported individually in Tables IX – XII . CC is fixed within a corpus; all remaining entries are mean ± sample standard deviation across queries.
Case
Reference-matching answer
Conflicting statement identified
ACS (score)
ACS (explanation)
B–Q1
Speed of light in vacuum: 299,792,458 m/s
Approximately 250,000 km/s
0.41
“The answer agrees with the exact value in the clean passage, but another passage contradicts it with the lower value.”
B–Q4
Statin therapy is typically indefinite
Discontinuation after one to three years if lipid targets are achieved
0.41
“The answer states indefinite treatment, which conflicts with contextual guidance allowing discontinuation after one to three years.”
B–Q6
Louis XVI was executed on 21 January 1793
Execution on 28 January 1793
0.41
“The answer’s date agrees with the presented evidence, but another part of the context gives a contradictory date.”
B–Q9
Aconcagua’s elevation is 6,961 m
Elevation of 6,400 m
0.41
“The answer is supported by the document, but another part of the text gives 6,400 m, contradicting that elevation.”
TABLE VIII: Reference-matching answers with contradictory retrieved evidence in Corpus B.
Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.
Volume-based accuracy rewards retrieval-augmented generation (RAG) systems for guessing: a system that answers everything outscores one that declines when its knowledge base cannot support an answer. Building on the confidence-target analysis of Kalai et al. (2025), we present a penalty-aware evaluation framework for deployed RAG products, combining (i) asymmetric scoring (correct +1, wrong -4, abstain 0), (ii) knowledge-gap canaries, questions whose answers are verifiably absent from the knowledge base, so that any answer constitutes ungrounded generation from parametric memory, and (iii) a failure-attribution pipeline that separates retrieval, generation, and abstention-policy failures. Applying the framework to three commercial RAG systems and a no-retrieval baseline on SimpleQA-Verified (1,000 questions x 3 repeats, graded blind by a cross-family three-judge panel with 98.9% unanimity), we find that accuracy when answering is closely clustered across systems (97.0-98.0%), while canary violation rates differ roughly sixfold (16.7% vs. 98.1%). The systems are separated less by what they answer correctly than by whether they answer at all when they should not, and penalty-aware scoring reorders the volume-based ranking accordingly; the reordering is stable across penalty settings from k=1 to k=9. All code, configurations, transcripts, and judge votes are released for independent audit.
Retrieval-Augmented Generation (RAG) systems are often evaluated using final answer accuracy, even though their failures can originate from preprocessing, retrieval, context packing, or generation. This paper presents a controlled empirical study of RAG sensitivity, robustness, and stability across 56 experimental runs. We evaluate how chunk size, retrieval depth (top k), embedding-based reranking, probabilistic retrieval noise, and repeated seeded runs affect retrieval, context packing, and generation behavior. Using a fixed 500-question QA subset mapped to 20,958 unique corpus contexts, we analyze both final answer metrics and intermediate failure modes. Across these experiments, retrieval-oriented metrics improved under broader retrieval settings, while downstream exact-match and F1 scores often behaved non-monotonically. We also observe preprocessing-induced answer loss under smaller chunk sizes, progressive degradation under retrieval corruption, and higher observed variance in broader retrieval regimes. These findings suggest that RAG evaluation should include sensitivity, robustness, stability, and multi-stage failure analysis rather than relying only on final answer accuracy.
Incorporating specific knowledge into large language models via retrieval-augmented generation (RAG) is a widespread technique that fuels many of today's industry AI applications. A fundamental problem is to assess if the context retrieved by some similarity search provides indeed supporting facts, or instead misguides the generator with irrelevant information. It is critical to associate meaningful confidence measures about the factuality of the retrieval process with the generated answers. We present a new, two-staged approach to predict fact faithfulness of the output of retrieval-augmented generations. First, we employ conformal prediction to select only those retrieved chunks who have a high chance to come from the correct source. This approach in itself can improve answer quality by up to 6% in some of the studied datasets, however, the associated statistical guarantees do not hold generally, since the assumption of sample exchangeability depends on the retriever setup. We present diagnostic metrics to assess whether a setup is suitable. Second, we quantify confidence in the consistency of a generated final answer with a given retrieved context, using an attention-based factuality classifier. This approach can detect inconsistent answers with a chance of up to 77%. Our work helps to establish a novel type of certified RAG systems for a broad range of natural language industry applications.
Florian Geissler, Francesco Carella, Laura Fieback +1
Fraunhofer Institute for Cognitive Systems (IKS), Munich, Germany · Volkswagen AG, Wolfsburg, Germany