cs.LGNov 7, 2026

Two-Sided Conformal Evidence Sets for Retrieval-Augmented Generation

Authors: Ben Jenkins, Mihaela Cardei

Organizations: Dept. of Electrical Engineering and Computer Science, Florida Atlantic University, Boca Raton, FL, USA

Abstract

Retrieval-augmented generation (RAG) can surface passages that support a claim, contradict it, or fail to address it, yet existing calibrated grounding methods are typically one-sided and model only supporting evidence. We introduce Conformal Evidence Sets (CES), a two-sided framework that returns separate support and refutation sets with finite-sample inclusion coverage of at least 1−α1-α on each side under exchangeability; a Bonferroni union yields joint coverage of at least 1−2α1-2α. On labeled candidate pools from the contrastive VitaminC benchmark, CES closely tracks nominal coverage across α ⁣∈ ⁣{0.05,0.10,0.15,0.20,0.30}α\!\in\!\{0.05,0.10,0.15,0.20,0.30\}, with maximum deviations of 3.03.0 percentage points for support and 5.45.4 for refutation, and reaches essentially the same empirical coverage--size frontier as a test-tuned fixed-threshold heuristic without any test exposure. Stress tests mark the boundary of that guarantee: under FEVER distribution shift both sides under-cover, by up to 12.712.7 percentage points, and support-side group-stratified recalibration substantially reduces the support gap. Scorer swaps, hard-negative retrieval, cross-dataset transfer, and a live end-to-end pilot expose a clear separation: when exchangeability holds and relevant evidence is retrieved, evidence inclusion remains calibrated even with an imperfect scorer, whereas the support/refute verdict tracks scorer quality directly. Calibrated evidence coverage and reliable verdicts are therefore distinct properties and should be reported separately.

Figures & tables

Explore similar work

CardsList
  1. Relevance Is Not Sufficient Evidence: Detecting Evidence Gaps Before Generation in RAG

    Sep 27, 2026Suting Chen, Peichun Hua, Yunming XiaoRetrieval-Augmented GenerationEvidence-Grounded QA

  2. SURE-RAG: Sufficiency and Uncertainty-Aware Evidence Verification for Selective Retrieval-Augmented Generation

    May 5, 2026Jingxi Qiu, Zeyu Han, Cheng HuangRetrieval-Augmented GenerationLLM Answer Verification

  3. Assessing the Downstream Utility of Evidence-Aware Retrieval in RAG

    Aug 26, 2026Utshab Kumar Ghosh, Debayan Mukhopadhyay, Shubham ChatterjeeRAG EvaluationIR Evaluation