RAGScope: A Leakage-Controlled, Cost-Aware Evidence-Gating Protocol for RAG Hallucination Triage

Zeming Liu; Qibai Chen; Jingtao Zhang; Hang Lyu

cs.CRSep 30, 2026

RAGScope: A Leakage-Controlled, Cost-Aware Evidence-Gating Protocol for RAG Hallucination Triage

Authors: Zeming Liu, Qibai Chen, Jingtao Zhang, Hang Lyu

Organizations: Brown University, United States · Independent Researcher, United States · Georgia Institute of Technology, United States

Abstract

Retrieval-augmented generation (RAG) systems need inexpensive ways to route generated answers: accept low-risk outputs, review uncertain ones, and reserve strong verifiers for the expensive tail. We present RAGScope, a leakage-controlled protocol for evaluating local evidence gates that use only the task input, retrieved context, and answer text. The protocol combines context-grouped splits, fold-scoped preprocessing, group bootstrap intervals, deployment operating points, end-to-end runtime, and explicit source-shift stress tests. On three RAGTruth tasks, the enhanced gate RAGScope-E reaches 0.798 AUROC and 0.660 average precision (AP) in pooled grouped cross-validation. Its pooled AP exceeds ROUGE-L by 0.034 with a 95% context-group interval of [0.002, 0.064], although the AUROC gain is not significant and ROUGE-L remains stronger on data-to-text. At a top-10% review budget, RAGScope-E attains 0.748 precision; accepting the lowest-risk 50% yields 0.141 residual unfaithfulness. RAGScope-E runs in 6.22 ms/example on CPU, versus 145.75 and 223.07 ms/example for the tested DeBERTa-NLI and HHEM settings. A 14,900-example HaluBench stress test exposes the deployment boundary: an in-domain calibrated gate reaches 0.879 AUROC, but leave-source-out calibration averages only 0.466. Target-only calibration recovers to 0.675 AUROC with 100 labels per source and 0.685 with 200. Cheap evidence gates are therefore useful routing components, but learned calibration must be validated and adapted within the target domain.

Figures & tables

Fig. 1: RAGScope evaluates local evidence signals as a first-stage routing protocol. The key controls are fold-scoped feature construction, context-grouped evaluation, uncertainty intervals over groups, fixed review/accept operating points, and runtime accounting.

Risk	Required control
Context leakage	Group splits by source context
Preprocess leakage	Fit TF-IDF/scalers inside folds
Uncertain gains	Group bootstrap and paired deltas
Deployment mismatch	Report review/accept points
Verifier cost	Measure end-to-end ms/example
Source shift	Hold out sources; vary target labels

TABLE I: RAGScope protocol checklist. Each item is implemented by the released scripts and reported in the artifact manifest.

RQ	Dataset	Control	Metric
RQ1	RAGTruth tasks	cheap signals	AUROC, AP
RQ2	RAGTruth all	group CV	AUROC, AP, delta
RQ3	RAGTruth	risk cutoff	precision, risk
RQ4	RAGTruth	verifier baselines	cost, accuracy
RQ5	HaluBench	source holdout/adapt.	AUROC, AP

TABLE II: Experiment matrix. Learned gates use context-grouped splits; TF-IDF features are fit inside each training fold.

Scope	Uncov. AUROC	Uncov. AP	ROUGE-L AUROC	ROUGE-L AP	RAGScope-E AUROC	RAGScope-E AP
QA	0.702	0.268	0.739	0.307	0.771	0.404
Summarization	0.676	0.391	0.694	0.402	0.682	0.377
Data-to-text	0.687	0.779	0.713	0.805	0.663	0.771
Macro average	0.689	0.479	0.715	0.505	0.706	0.517
Combined	0.779	0.594	0.792	0.624	0.798	0.660

TABLE III: RAGTruth grouped results. Positive labels are unfaithful outputs. The pooled setting is useful for a shared triage queue; macro averages expose task-level generality.

Fig. 2: RAGTruth grouped performance by task and pooled queue. RAGScope-E improves the pooled AP used for a shared triage queue and gives its clearest task-level gain on QA, while ROUGE-L remains stronger on data-to-text.

Scope/base	Metric	Delta	95% CI
Combined/uncov.	AUROC	0.019	[0.003, 0.034]
Combined/uncov.	AP	0.064	[0.029, 0.099]
Combined/ROUGE	AUROC	0.006	[-0.006, 0.017]
Combined/ROUGE	AP	0.034	[0.002, 0.064]
QA/ROUGE	AUROC	0.031	[0.006, 0.056]
QA/ROUGE	AP	0.097	[0.047, 0.148]

TABLE IV: Paired context-group bootstrap deltas for RAGScope-E. Intervals are 95%; positive deltas favor RAGScope-E.

Fig. 3: Average-precision deltas for RAGScope-E versus ROUGE-L with 95% context-group bootstrap intervals. The figure highlights the asymmetric result: QA and pooled AP improve, while data-to-text favors ROUGE-L.

Variant	AUROC	AP
Token-coverage logit	0.782	0.606
Base all-feature logit	0.792	0.636
RAGScope-E enhanced logit	0.798	0.660

TABLE V: Grouped-CV ablation on RAGTruth combined.

Fig. 4: Combined RAGTruth operating curves. Left: precision among examples sent to review as the review budget grows. Right: residual unfaithfulness rate among examples accepted locally as the acceptance budget grows. The dashed line marks the combined base positive rate.

Scope	Model	Review precision	Accepted risk
Combined	RAGScope-E	.748 [.685,.807]	.141 [.119,.169]
Combined	ROUGE	.741 [.678,.807]	.151 [.127,.171]
QA	RAGScope-E	.467 [.333,.600]	.067 [.044,.089]
QA	ROUGE	.344 [.200,.467]	.067 [.042,.093]
Summ.	RAGScope-E	.456 [.344,.544]	.136 [.102,.171]
Summ.	ROUGE	.456 [.378,.556]	.124 [.093,.156]

TABLE VI: Operating points with context-group bootstrap intervals. Review is top-risk 10%; accept is lowest-risk 50%.

Fig. 5: Combined AP versus measured local CPU runtime. In this experiment, RAGScope-E lies on the cheap high-AP corner of the tested local methods, while the two model-based verifier settings are slower and less accurate on RAGTruth.

Model	AUROC	AP	ms/ex
Uncovered fraction	0.779	0.594	0.67
ROUGE-L	0.792	0.624	3.64
RAGScope-E	0.798	0.660	6.22
DeBERTa-NLI top-3	0.560	0.373	145.75
HHEM full ctx.	0.655	0.420	223.07

TABLE VII: Accuracy-cost comparison on RAGTruth combined. Runtime is local CPU milliseconds per example.

Explore similar work

SURE-RAG: Sufficiency and Uncertainty-Aware Evidence Verification for Selective Retrieval-Augmented Generation
May 5, 2026Jingxi Qiu, Zeyu Han, Cheng HuangHievi-Rag Retrieval-Augmented Generation Pipelines
ZenWeave AI · Georgetown University · Southern Methodist University
AGO AI Quality Gate: Evidence-First Release Decisions for Retrieval-Augmented Generation
Oct 1, 2026Giulio Zeloni, Enrico Lo Conte, Salvatore Rionero +3Agentic Evaluations Large Language Model Judges
Protom Group S.p.A., Napoli, Italy
Before Reasoning Can Fail: Pre-Evidence Procedural Failures in Agentic RAG
Aug 3, 2026Daeyoung Roh, Donghee HanHievi-Rag
Independent Researcher · KAIST

May 5, 2026, cs.CLJ/K move · Enter open · S save

SURE-RAG: Sufficiency and Uncertainty-Aware Evidence Verification for Selective Retrieval-Augmented Generation

Jingxi Qiu, Zeyu Han, Cheng Huang

ZenWeave AI · Georgetown University · Southern Methodist University

Hievi-Rag Retrieval-Augmented Generation Pipelines Evidence-Grounded Questions Question-Answering Benchmarks Sufficiency Selective

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) grounds answers in retrieved passages, but retrieval is not verification: a passage can be topical and still fail to justify the answer. We frame this gap as evidence sufficiency verification for selective RAG answering: given a question, a candidate answer, and retrieved evidence, predict whether the evidence supports, refutes, or is insufficient, and abstain unless support is established. We present SURE-RAG, a transparent aggregation protocol built on the observation that evidence sufficiency is a set-level property: missing hops and unresolved conflicts cannot be detected by independent passage scoring. A shared pair-level claim-evidence verifier produces local relation distributions, which SURE-RAG aggregates into interpretable answer-level signals -- coverage, relation strength, disagreement, conflict, and retrieval uncertainty -- yielding a three-way decision and an auditable selective score. We evaluate on HotpotQA-RAG v3, a controlled multi-hop benchmark, under an artifact-aware protocol (shortcut baselines, counterfactual swaps, no-oracle checks, GPT-4o audits). Calibrated SURE-RAG reaches 0.9075 Macro-F1 (0.8951 +/- 0.0069), substantially above DeBERTa mean-pooling (0.6516) and a GPT-4o judge (0.7284), while matching a strong but opaque concat cross-encoder (0.8888 +/- 0.0109) with full auditability. Risk at 30% coverage drops from 0.2588 to 0.1642, a 37% reduction in unsafe answers. To deliberately probe the task boundary, we further contrast SURE-RAG with GPT-4o on HaluBench unsafe detection: the ranking reverses (0.3343 vs 0.7389 unsafe-F1), establishing that controlled sufficiency verification and natural hallucination detection are distinct problems.

Abstract

Explore similar work

SURE-RAG: Sufficiency and Uncertainty-Aware Evidence Verification for Selective Retrieval-Augmented Generation

AGO AI Quality Gate: Evidence-First Release Decisions for Retrieval-Augmented Generation

Before Reasoning Can Fail: Pre-Evidence Procedural Failures in Agentic RAG