Conformal Factuality Control for Multi-Hop Retrieval-Augmented Generation

Muhammad Aimal Rehman; Chi-Kuang Yeh

cs.CLSep 27, 2026

Conformal Factuality Control for Multi-Hop Retrieval-Augmented Generation

Authors: Muhammad Aimal Rehman, Chi-Kuang Yeh

Organizations: Department of Mathematics & Statistics Georgia State University Atlanta, GA, USA

Abstract

Retrieval-augmented generation (RAG) can ground large language models in external evidence, but retrieved context does not guarantee that generated claims are factually supported. This problem is especially relevant in multi-hop RAG, where retrieval and reasoning proceed through multiple dependent stages. We study whether claim-level conformal factuality control, previously developed for RAG, remains effective in this setting. We apply split-conformal claim filtering to multi-hop RAG and evaluate it on HotpotQA, Natural Questions, and TriviaQA using Llama 3.1 8B and GPT-4o-mini, together with a single-hop reference experiment. Across all six multi-hop model-dataset configurations, increasingly stringent conformal targets consistently increase the fraction of responses whose retained claims are fully supported. At the 95% target, this rate ranges from 95.80% to 97.20%, compared with 55.60%-76.03% without filtering. However, the improvement is strongly selective: only 4.41%-31.09% of generated claims are retained and 9.70%-51.40% of responses remain non-empty at the 95% target. These results show that conformal factuality extends to multi-hop RAG, while demonstrating that nominal reliability must be interpreted jointly with claim retention and abstention.

Figures & tables

Figure 1: Overview of claim-level conformal factuality control for multi-hop RAG. A question triggers multi-hop retrieval with query rewriting and evidence accumulation. The generated response is decomposed into atomic claims, scored for relevance, and filtered using the split-conformal threshold $\hat{q}_{\alpha}$ . The separate calibration-only branch uses calibration scores and factuality annotations to determine this threshold.

Retrieval	Model	Datasets
Plain RAG	Llama 3.1 8B	HotpotQA
Multi-Hop RAG	Llama 3.1 8B	HotpotQA, NQ, TriviaQA
Multi-Hop RAG	GPT-4o-mini	HotpotQA, NQ, TriviaQA

Table 1: Experimental configurations. The Plain-RAG setting serves as a reference experiment; the six Multi-Hop RAG configurations constitute the primary evaluation.

		Base	90% target			95% target
Model	Dataset	FS	FS	NE	Ret	FS	NE	Ret
Llama 3.1 8B	HotpotQA	68.60	91.13	60.57	32.39	96.90	22.00	8.35
	NQ	76.03	88.70	66.53	41.10	95.80	31.90	14.94
	TriviaQA	55.60	89.93	63.03	40.52	96.50	51.40	28.17
GPT-4o-mini	HotpotQA	64.80	90.30	51.80	27.86	97.20	10.80	5.16
	NQ	57.70	90.23	31.97	19.73	96.43	9.70	4.41

Table 2: Main Multi-Hop RAG results. FS denotes the percentage of responses that are fully supported under the response-level conformal event, NE denotes non-empty responses, and Ret denotes generated claims retained after conformal filtering. All values are percentages and are averaged over 100 calibration–test splits.

Figure 2: Reliability–coverage trade-off for Multi-Hop RAG as the conformal target increases from

80\%

95\%

. Each trajectory corresponds to one model–dataset configuration; endpoint labels indicate the

80\%

and

95\%

targets. More stringent conformal filtering increases the fraction of responses satisfying the full-support event while generally decreasing the fraction of responses that retain any content.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Target	Non-empty	Claims retained	Supported claims	Fully supported $\mid$ non-empty
80.0	81.43	67.40	90.34	76.61
85.0	72.20	52.96	91.32	82.87
90.0	61.07	41.25	91.01	85.16
92.5	44.83	25.82	88.99	86.98
95.0	20.97	11.10	86.67	88.51

Appendix

Table 3: Complete conformal target sweep for the Plain-RAG reference experiment on HotpotQA with Llama 3.1 8B. All values are percentages.

Dataset	Target	Non-empty	Claims retained	Supported claims	Fully supported $\mid$ non-empty
HotpotQA	80.0	88.60	68.39	88.22	78.65
	85.0	76.67	47.01	90.33	84.95
	90.0	60.57	32.39	90.47	87.06
	92.5	42.93	20.16	89.67	86.83
	95.0	22.00	8.35	93.54	90.83
NQ	80.0	92.80	82.99	92.60	79.75

Appendix

Table 4: Complete conformal target sweeps for Multi-Hop RAG with Llama 3.1 8B. “Supported claims” is supported-claim factuality; the final column is response-level full support conditional on the filtered response being non-empty. All values are percentages.

Dataset	Target	Non-empty	Claims retained	Supported claims	Fully supported $\mid$ non-empty
HotpotQA	80.0	79.80	53.44	88.96	76.91
	85.0	65.73	38.19	90.46	81.61
	90.0	51.80	27.86	90.67	83.82
	92.5	32.47	16.14	90.36	85.07
	95.0	10.80	5.16	85.48	78.48
NQ	80.0	64.60	44.69	87.12	70.28

Appendix

Table 5: Complete conformal target sweeps for Multi-Hop RAG with GPT-4o-mini. All values are percentages.

Explore similar work

Towards Dependable Retrieval-Augmented Generation Using Factual Confidence Prediction
May 4, 2026Florian Geissler, Francesco Carella, Laura Fieback +1Retrieval-Augmented Generation Factual Consistency Evaluation
Fraunhofer Institute for Cognitive Systems (IKS), Munich, Germany · Volkswagen AG, Wolfsburg, Germany
Is Conformal Factuality for RAG-based LLMs Robust? Novel Metrics and Systematic Insights
Mar 17, 2026Yi Chen, Daiwei Chen, Sukrut Madhav Chikodikar +2LLM Reliability RAG Evaluation
University of Wisconsin-Madison
AdaGATE: Adaptive Gap-Aware Token-Efficient Evidence Assembly for Multi-Hop Retrieval-Augmented Generation
May 4, 2026Yilin Guo, Yinshan Wang, Yixuan WangMulti-Hop QA Retrieval-Augmented Generation
Center for Data Science New York University · Tandon School of Engineering New York University

May 4, 2026J/K move · Enter open · S save

Towards Dependable Retrieval-Augmented Generation Using Factual Confidence Prediction

Florian Geissler, Francesco Carella, Laura Fieback, Jakob Spiegelberg

Fraunhofer Institute for Cognitive Systems (IKS), Munich, Germany · Volkswagen AG, Wolfsburg, Germany

Retrieval-Augmented Generation Factual Consistency Evaluation LLM Reliability RAG Evaluation

Open paper AskPDF arXiv

Incorporating specific knowledge into large language models via retrieval-augmented generation (RAG) is a widespread technique that fuels many of today's industry AI applications. A fundamental problem is to assess if the context retrieved by some similarity search provides indeed supporting facts, or instead misguides the generator with irrelevant information. It is critical to associate meaningful confidence measures about the factuality of the retrieval process with the generated answers. We present a new, two-staged approach to predict fact faithfulness of the output of retrieval-augmented generations. First, we employ conformal prediction to select only those retrieved chunks who have a high chance to come from the correct source. This approach in itself can improve answer quality by up to 6% in some of the studied datasets, however, the associated statistical guarantees do not hold generally, since the assumption of sample exchangeability depends on the retriever setup. We present diagnostic metrics to assess whether a setup is suitable. Second, we quantify confidence in the consistency of a generated final answer with a given retrieved context, using an attention-based factuality classifier. This approach can detect inconsistent answers with a chance of up to 77%. Our work helps to establish a novel type of certified RAG systems for a broad range of natural language industry applications.

Abstract

Explore similar work

Towards Dependable Retrieval-Augmented Generation Using Factual Confidence Prediction

Is Conformal Factuality for RAG-based LLMs Robust? Novel Metrics and Systematic Insights

AdaGATE: Adaptive Gap-Aware Token-Efficient Evidence Assembly for Multi-Hop Retrieval-Augmented Generation