ERA: Evidence-based Reliability Alignment for Honest Retrieval-Augmented Generation

Sunguk Shin; Meeyoung Cha; Byung-Jun Lee; Sungwon Park

cs.IRFeb 24, 2026

ERA: Evidence-based Reliability Alignment for Honest Retrieval-Augmented Generation

Authors: Sunguk Shin, Meeyoung Cha, Byung-Jun Lee, Sungwon Park

Organizations: Korea University, Seoul, South Korea · MPI-SP, Bochum, Germany · KAIST, Daejeon, South Korea · Gauss Labs Inc., Seoul, South Korea

Abstract

Retrieval-Augmented Generation (RAG) grounds language models in factual evidence but introduces critical challenges regarding knowledge conflicts between internalized parameters and retrieved information. However, existing reliability methods, typically relying on scalar confidence, fail to explicitly distinguish between epistemic uncertainty and inherent data ambiguity in such hybrid scenarios. In this paper, we propose a new framework called ERA (Evidence-based Reliability Alignment) to enhance abstention behavior in RAG systems by shifting confidence estimation from scalar probabilities to explicit evidence distributions. Our method consists of two main components: (1) Contextual Evidence Quantification, which models internal and external knowledge as independent belief masses via the Dirichlet distribution, and (2) Quantifying Knowledge Conflict, which leverages Dempster-Shafer Theory (DST) to rigorously measure the geometric discordance between information sources. These components are used to disentangle epistemic uncertainty from aleatoric uncertainty and modulate the optimization objective based on detected conflicts. Experiments on standard benchmarks and a curated generalization dataset demonstrate that our approach significantly outperforms baselines, optimizing the trade-off between answer coverage and abstention with superior calibration.

Figures & tables

Figure 1. Categorization of Knowledge Domains and Response Strategies in RAG Models.

Metric	Formula	Description
Answer Quality
Recall	$\frac{\|\mathcal{S}_{\text{ans}}\cap\mathcal{I}_{\text{ans}}\|}{\|\mathcal{I}_{\text{ans}}\|}$	Correct answers over all answerable queries
Precision	$\frac{\|\mathcal{S}_{\text{ans}}\cap\mathcal{I}_{\text{ans}}\|}{\|\mathcal{S}_{\text{ans}}\|}$	Correct answers over all attempted answers
F1	$2\cdot\frac{\text{Pre}\cdot\text{Rec}}{\text{Pre}+\text{Rec}}$	Harmonic mean of Precision and Recall
Retrieval Handling
Denoise Rate	$\frac{\|\mathcal{S}_{\text{ans}}\cap\mathcal{I}_{\text{noise}}\|}{\|\mathcal{I}_{\text{noise}}\|}$	Accuracy on queries with noisy context

Table 1. Summary of evaluation metrics.

Figure 2. Overview of the ERA . The model consists of three components: (1) Contextual Evidence Quantification, where evidential heads project representations from both RAG and Parametric paths into Dirichlet distributions ( $\boldsymbol{\alpha}$ ); (2) Knowledge Conflict Quantification, which utilizes Dempster-Shafer Theory to fuse beliefs and compute a conflict score $\kappa$ ; and (3) Uncertainty-Aware Abstention Mechanism, which modulates the optimization objective based on the detected conflict to mitigate hallucinations.

Method	Answer Quality			Retrieval Handling		Abstention Quality			Overall F1
Method	Recall	Prec.	F1	Denoise	Context	Recall	Prec.	F1	Overall F1
Qwen3-8B-Base
Logprob ( $t=-2$ )	0.308	0.933	0.463	0.303	0.221	0.050	0.564	0.091	0.153
Logprob ( $t=-1$ )	0.197	0.933	0.325	0.187	0.144	0.689	0.360	0.473	0.385
ICL	0.178	0.130	0.150	0.184	0.142	0.048	0.404	0.085	0.109
Self-Consistency	0.203	0.415	0.273	0.219	0.142	0.777	0.350	0.483	0.349

Table 2. Performance comparison across three QA datasets using Qwen3-8B and Llama3-8B backbones. Metrics are categorized into: Answer Quality , Retrieval Handling , and Abstention Quality . Overall F1 represents the harmonic mean balancing answer generation and abstention.

Figure 3. Generalizability evaluation on the Wiki Event dataset using Qwen3-8B-Base.

Panel of Figure 4 — Figure 4. Quantitative analysis of uncertainty. This figure compares the proposed Evidential Deep Learning (EDL) model against the Cross-Entropy (CE) baseline across three dimensions: (a) A visualization of prediction probability distributions in conflict situations (KN, UG); (b) A comparison of Expected Calibration Error (ECE) across data types; and (c) An evaluation of model sensitivity via the Jensen-Shannon Divergence (JSD) between consistent and conflicting information.

Panel of Figure 5 — Figure 5. Performance comparison of ablation studies. The consistent degradation in Overall F1 scores observed when removing or altering individual components validates the distinct contribution of each module in the ERA framework.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6. Generalizability evaluation on the Wiki Event dataset using Llama-3.1-8B.

Figure 7. Generalizability evaluation on the Wiki Event dataset using ChatQA.

RAG Robustness

Retrieval-augmented generation robustness studies how to keep systems reliable when retrieved evidence is noisy, misleading, incomplete, or malicious.

See all 126 papers

EvidentialRAG: Quantifying and Mitigating Information Conflict in Multi-Source Retrieval-Augmented Generation via Evidential Deep Learning
Jul 11, 2026S M Asif Hossain, Ruksat Khan Shayoni, M. F. MridhaRetrieval-Augmented Generation Evidential Deep Learning
aSchool of Computing, Wichita State University, Kansas, USA · bDepartment of Computer Science, American International University-Bangladesh, Dhaka, Bangladesh
Entropic Claim Resolution: Uncertainty-Driven Evidence Selection for RAG
Mar 30, 2026Davide Di GioiaRetrieval-Augmented Generation Evidence Selection
Trust or Abstain? A Self-Aware RAG Approach
May 11, 2026Xi Zhu, Ziqi Wang, Kai Mei +5Retrieval-Augmented Generation Knowledge Conflicts in Language Models
Rutgers University · University of Illinois Urbana-Champaign

Jul 11, 2026J/K move · Enter open · S save

EvidentialRAG: Quantifying and Mitigating Information Conflict in Multi-Source Retrieval-Augmented Generation via Evidential Deep Learning

S M Asif Hossain, Ruksat Khan Shayoni, M. F. Mridha

aSchool of Computing, Wichita State University, Kansas, USA · bDepartment of Computer Science, American International University-Bangladesh, Dhaka, Bangladesh

Retrieval-Augmented Generation Evidential Deep Learning Knowledge Conflicts in Language Models Epistemic Uncertainty

Open paper AskPDF arXiv

Retrieval-augmented generation grounds large language models in external evidence, but most pipelines still treat retrieved passages as deterministic and mutually consistent context. In open information environments, retrieved sources may disagree because of temporal drift, source error, ambiguity, or genuine uncertainty. This paper introduces ERAG, an uncertainty-aware RAG framework that converts retrieved chunks into probabilistic evidence before generation. A lightweight evaluator extracts candidate claims and maps chunk-level support to Dirichlet evidence. A conflict-preserving Dempster-Shafer fusion rule then transfers unresolved disagreement into epistemic uncertainty rather than normalizing it away. The generator is routed to direct answering, conflict-aware answering, or abstention according to the fused uncertainty score. Experiments on CRAG, ConflictQA, and MuSiQue show that ERAG remains competitive with the strongest matched baseline on standard question answering while improving behavior under conflict. On the CRAG ambiguous subset, hallucination decreases from 45.3% for Corrective RAG to a human-calibrated estimate of 34.8%, conflict resolution increases from 35.2% to 51.2%, and expected calibration error improves to 0.122. These results suggest that evidential modeling is a practical mechanism for trustworthy information processing in foundation-model-based retrieval systems.

Abstract

RAG Robustness

Explore similar work

EvidentialRAG: Quantifying and Mitigating Information Conflict in Multi-Source Retrieval-Augmented Generation via Evidential Deep Learning

Entropic Claim Resolution: Uncertainty-Driven Evidence Selection for RAG

Trust or Abstain? A Self-Aware RAG Approach