Large language models (LLMs) frequently hallucinate, limiting their reliability in knowledge-intensive applications. Retrieval-augmented generation (RAG) and conformal factuality have emerged as potential ways to address this limitation. While RAG aims to ground responses in retrieved evidence, it provides no statistical guarantee that the final output is correct. Conformal factuality filtering offers distribution-free statistical reliability by scoring and filtering atomic claims using a threshold calibrated on held-out data, however, the informativeness of the final output is not guaranteed. We systematically analyze the reliability and usefulness of conformal factuality for RAG-based LLMs across generation, scoring, calibration, robustness, and efficiency. We propose novel informativeness-aware metrics that better reflect task utility under conformal filtering. Across three benchmarks and multiple model families, we find that (i) conformal filtering suffers from low usefulness at high factuality levels due to vacuous outputs, (ii) conformal factuality guarantee is not robust to distribution shifts and distractors, highlighting the limitation that requires calibration data to closely match deployment conditions, and (iii) lightweight entailment-based verifiers match or outperform LLM-based model confidence scorers while requiring over 100× fewer FLOPs. Overall, our results expose factuality-informativeness trade-offs and fragility of conformal filtering framework under distribution shifts and distractors, highlighting the need for new approaches for reliability with robustness and usefulness as key metrics, and provide actionable guidance for building RAG pipelines that are both reliable and computationally efficient.
Figures & tables
Figure 1: Overview of our framework. Given a query x and retrieved references R(x) , the Response Generator G produces an output y . The conformal factuality framework utilizes a separate calibration data to determine a threshold used to filter out information from the output y and yield y′ . (See Figure 2 for the details of different stages involved in conformal filtering.)
Figure 2: Given an input x and a reference text related to x , the Response Generator produces an output y , which is then parsed by the Parser into a list of claims. Each individual claim is subsequently scored by the Scorer, conditioned on the input x and, optionally, the reference text. These scores are passed to the conformal prediction algorithm, which filters out claims whose scores fall below a learned threshold. Finally, the remaining claims are merged into a single paragraph and returned to the user.
Family
Model
Params
Arch.
Activated
Qwen3
Qwen3-0.6B
0.6B
Dense
0.6B
Qwen3
Qwen3-4B
4B
Dense
4B
Qwen3
Qwen3-8B
8.2B
Dense
8.2B
Qwen3
Qwen3-30B-A3B
30.5B
MoE
3.3B
Qwen3
Qwen3-32B
32.8B
Dense
32.8B
Llama-3.x
Llama-3.2-1B-Instruct
1.2B
Dense
1.2B
Table 1: Summary of evaluated open-source language models. For dense models, the number of activated parameters equals the total parameter count. For MoE models, activated parameter counts are shown when known.
Figure 3: Sufficient correctness (SC) of Qwen3 models (0.6B, 4B, 8B) on four datasets (MATH-200, FActScore-Rare, FActScore, NQ-200), with and without access to references. Across model sizes and datasets, providing references consistently improves generation quality.
Figure 4: Evaluation of various prompting strategies across different LLMs on the FActScore dataset (Section 4.1 ) at level 1−α=0.9 . Results demonstrate that: (i) prompting models to generate numeric scores consistently outperforms Boolean scoring; (ii) sampling multiple responses uniformly improves performance; however, (iii) incorporating chain-of-thought reasoning or evidence highlighting do not yield reliable performance gains across models.
Figure 5: Performance of model confidence score on MATH-1K dataset with and without reference provided to scoring functions using Qwen3-4B as the LLM-based scorer.
Figure 6: Experimental evaluation across three model families of various scales on the FActScore dataset at level 1−α=0.9 . Results demonstrate that increasing model size does not consistently improve performance of conformal factuality.
Figure 7: Comparison between entailment-based scores against the model confidence score on the FActScore dataset. Across our evaluated settings, the entailment-based scores match or exceed the model-confidence baseline, despite the entailment model being substantially smaller than the target LLMs.
Figure 8: Sufficient Correctness (SC) and Conditional Sufficient Correctness (CSC) for the Qwen3 model family at α=0.05 across FActScore, MATH-1K, and NQ-1K dataset.
Figure 9: Empirical factuality (EF) on FActScore, MATH-1K, and NQ-1K, under two calibration settings: (i) calibration claims generated in Mohri and Hashimoto (2024) , which comes from a different distribution (ii) calibration claims drawn from the same distribution as the test data. We use Qwen3-4B as the scoring function. The results show how the distribution shift in the calibration set affects conformal factuality guarantees.
Figure 10: Power on FActScore, MATH-1K, and NQ-1K, under two calibration settings: (i) calibration claims generated in Mohri and Hashimoto (2024) , which comes from a different distribution (ii) calibration claims drawn from the same distribution as the test data. We use Qwen3-4B as the scoring function.
Figure 11: (a) - (d) Empirical factuality versus target factuality (1- α ) for Qwen3-4B on the FActScore dataset. Each panel corresponds to a different distractor injection rate in the test set (0.0, 0.1, 0.25, 0.5). The gray dashed diagonal represents perfect calibration ( y=x ), where empirical factuality matches the target. (e) Power versus target factuality (1 - α ) when the injection rate is 0.1.
Figure 12: (a) - (c) Empirical factuality versus target factuality ( 1−α ) for Qwen3-4B on the FActScore dataset when the test set is injected with 25% distractors. We vary the proportion of distractors in the calibration set from (0.1, 0.25, 0.5). As the proportion matches, we can see that the empirical factuality rises to the y=x line. (d)-(e) Comparison of non-empty rate when both the test and calibration sets contain no distractors and contain 25% of the distractors. Although introducing distractors to the calibration set can achieve target factuality, the non-empty rate suffers.
Model
Total Tokens
Total FLOPs
gpt-oss-20b (3.6B active) ( Agarwal et al., 2025 )
2000
1.44×1013
Qwen3-8B (8.19B active) ( Yang et al., 2025 )
2000
3.28×1013
DeepSeek-R1 (37B active) ( DeepSeek-AI, 2025 )
2000
1.5×1014
DeBERTa (184M active) ( He et al., 2020 )
2000
4.9×1011
RoBERTa (356M active) ( Liu et al., 2019 )
2000
1.6×1012
Table 2: Estimated FLOPs for generating 1000 tokens with a 1000-token prompt (assuming KV caching).
Figure 13: Performance of entailment- and confidence-based scorers using gpt-oss-20b and Qwen3-8B on FActScore.
Appendix figures & tables40 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 14: Sufficient correctness (SC) of Llama-3.x models on four datasets (MATH-200, FActScore-Rare, FActScore, NQ-200), with and without access to references. Across model sizes and datasets, providing references consistently improves generation quality.
Figure 15: Sufficient correctness (SC) of SmolLM2 models on four datasets (MATH-200, FActScore-Rare, FActScore, NQ-200), with and without access to references. Across model sizes and datasets, providing references consistently improves generation quality.
Figure 16: Sufficient correctness (SC) of Qwen3 and frontier models on FActScore-Rare, with and without access to references. Providing references consistently improves generation quality; notably, Qwen3-4B with references is comparable to frontier models.
Figure 17: Overall performance comparison of model confidence scores across 16 prompting strategies at five target factuality levels ( 1−α ) on the FActScore dataset. Each column corresponds to a different LLM scorer; each row corresponds to a different target factuality. Numeric scoring and consistency averaging yield the most robust improvements.
Figure 18: Overall performance comparison of model confidence scores across various prompting strategies at five target factuality levels ( 1−α ) on the MATH-1K dataset. The advantage of numeric over Boolean scoring is especially clear for mathematical reasoning.
Figure 19: Performance of model confidence score on FActScore with and without reference provided to scoring functions, using gpt-5-nano as generator and Qwen3-4B as scorer. Reference access improves power and non-empty rate.
Figure 20: Performance of model confidence score on NQ-1K with and without reference provided to scoring functions, using gpt-5-nano as generator and Qwen3-4B as scorer. The benefit of reference access is consistent with the MATH-1K and FActScore results.
Figure 21: Performance of model confidence score on FActScore with and without reference provided to scoring functions, using gpt-5-nano as generator and Qwen3-8B as scorer.
Figure 22: Performance of model confidence score on MATH-1K with and without reference provided to scoring functions, using gpt-5-nano as generator and Qwen3-8B as scorer.
Figure 23: Performance of model confidence score on NQ-1K with and without reference provided to scoring functions, using gpt-5-nano as generator and Qwen3-8B as scorer.
Figure 24: Performance of model confidence score on FActScore with and without reference provided to scoring functions, using gpt-5-nano as generator and Llama-3.2-3B-Instruct as scorer.
Figure 25: Performance of model confidence score on MATH-1K with and without reference provided to scoring functions, using gpt-5-nano as generator and Llama-3.2-3B-Instruct as scorer.
Figure 26: Performance of model confidence score on NQ-1K with and without reference provided to scoring functions, using gpt-5-nano as generator and Llama-3.2-3B-Instruct as scorer.
Figure 27: Performance of model confidence score on FActScore with and without reference provided to scoring functions, using gpt-5-nano as generator and SmolLM2-1.7B-Instruct as scorer.
Figure 28: Performance of model confidence score on MATH-1K with and without reference provided to scoring functions, using gpt-5-nano as generator and SmolLM2-1.7B-Instruct as scorer.
Figure 29: Performance of model confidence score on NQ-1K with and without reference provided to scoring functions, using gpt-5-nano as generator and SmolLM2-1.7B-Instruct as scorer.
Figure 30: Overall performance comparison of model confidence scores across different model scales and families at five target factuality levels ( 1−α ) on the FActScore dataset. While Llama-3.x models show consistent gains with scale, Qwen3 and SmolLM2 do not, highlighting that scaling alone is insufficient for improving conformal factuality.
Figure 31: Comparison between entailment-based scores against the model confidence score on the MATH-1K dataset.
Figure 32: Sufficient Correctness (SC) and Conditional Sufficient Correctness (CSC) for the Llama-3.x family at α=0.05 across FActScore, MATH-1K, and NQ-1K. CSC consistently exceeds SC, indicating that filtering largely preserves useful content when the generator provides it.
Figure 33: Sufficient Correctness (SC) and Conditional Sufficient Correctness (CSC) for the SmolLM2 family at α=0.05 across FActScore, MATH-1K, and NQ-1K. No consistent scaling benefit is observed within this family.
Figure 34: Sufficient Correctness (SC) and Conditional Sufficient Correctness (CSC) for the gpt-oss family at α=0.05 across FActScore, MATH-1K, and NQ-1K. The larger gpt-oss-120b does not consistently outperform gpt-oss-20b .
Figure 35: Empirical factuality (EF) on FActScore, MATH-1K, and NQ-1K under same-distribution and different-distribution calibration, using Llama-3.2-3B-Instruct as the scorer. Distribution shift causes the factuality guarantee to break for several scoring functions.
Figure 36: Empirical factuality (EF) on FActScore, MATH-1K, and NQ-1K under same-distribution and different-distribution calibration, using SmolLM2-360M-Instruct as the scorer. The results confirm that distribution shift in the calibration set undermines conformal factuality guarantees across model families.
Figure 37: Empirical factuality under varying test distractor proportions ( 0.0 to 0.5 ) with fixed clean calibration data on FActScore, using Llama-3.2-3B-Instruct . Factuality degrades sharply as distractor rate increases.
Figure 38: Empirical factuality under varying test distractor proportions ( 0.0 to 0.5 ) with fixed clean calibration data on FActScore, using SmolLM2-1.7B-Instruct . The degradation pattern is consistent with other model families.
Figure 39: Empirical factuality under varying test distractor proportions on MATH-1K with Qwen3-4B . Mathematical claims are especially vulnerable to subtle numerical perturbations.
Figure 40: Empirical factuality under varying test distractor proportions on MATH-1K with Llama-3.2-3B-Instruct .
Figure 41: Empirical factuality under varying test distractor proportions on MATH-1K with SmolLM2-1.7B-Instruct .
Figure 42: Empirical factuality under varying test distractor proportions on NQ-1K with Qwen3-4B .
Figure 43: Empirical factuality under varying test distractor proportions on NQ-1K with Llama-3.2-3B-Instruct .
Figure 44: Empirical factuality under varying test distractor proportions on NQ-1K with SmolLM2-1.7B-Instruct .
Figure 45: Empirical factuality and non-empty rates under varying calibration distractor proportions on FActScore with Llama-3.2-3B-Instruct . Matching calibration to test distractor levels restores factuality but reduces the non-empty rate.
Figure 46: Empirical factuality and non-empty rates under varying calibration distractor proportions on FActScore with SmolLM2-1.7B-Instruct .
Figure 47: Empirical factuality and non-empty rates under varying calibration distractor proportions on MATH-1K with Qwen3-4B .
Figure 48: Empirical factuality and non-empty rates under varying calibration distractor proportions on MATH-1K with Llama-3.2-3B-Instruct .
Figure 49: Empirical factuality and non-empty rates under varying calibration distractor proportions on MATH-1K with SmolLM2-1.7B-Instruct .
Figure 50: Empirical factuality and non-empty rates under varying calibration distractor proportions on NQ-1K with Qwen3-4B .
Figure 51: Empirical factuality and non-empty rates under varying calibration distractor proportions on NQ-1K with Llama-3.2-3B-Instruct .
Figure 52: Empirical factuality and non-empty rates under varying calibration distractor proportions on NQ-1K with SmolLM2-1.7B-Instruct .
Pair
Labelers
Agreement Rate
Model–Human
gpt-5-nano vs. Student A
76.5%
Model–Human
gpt-5-nano vs. Student B
77.0%
Human–Human
Student A vs. Student B
73.0%
Appendix
Table 3: Agreement rates on factuality labels between gpt-5-nano and two human annotators, as well as between the two human annotators themselves. The model–human agreement is comparable to inter-human agreement.
Incorporating specific knowledge into large language models via retrieval-augmented generation (RAG) is a widespread technique that fuels many of today's industry AI applications. A fundamental problem is to assess if the context retrieved by some similarity search provides indeed supporting facts, or instead misguides the generator with irrelevant information. It is critical to associate meaningful confidence measures about the factuality of the retrieval process with the generated answers. We present a new, two-staged approach to predict fact faithfulness of the output of retrieval-augmented generations. First, we employ conformal prediction to select only those retrieved chunks who have a high chance to come from the correct source. This approach in itself can improve answer quality by up to 6% in some of the studied datasets, however, the associated statistical guarantees do not hold generally, since the assumption of sample exchangeability depends on the retriever setup. We present diagnostic metrics to assess whether a setup is suitable. Second, we quantify confidence in the consistency of a generated final answer with a given retrieved context, using an attention-based factuality classifier. This approach can detect inconsistent answers with a chance of up to 77%. Our work helps to establish a novel type of certified RAG systems for a broad range of natural language industry applications.
Florian Geissler, Francesco Carella, Laura Fieback +1
Fraunhofer Institute for Cognitive Systems (IKS), Munich, Germany · Volkswagen AG, Wolfsburg, Germany
Retrieval-augmented generation (RAG) can ground large language models in external evidence, but retrieved context does not guarantee that generated claims are factually supported. This problem is especially relevant in multi-hop RAG, where retrieval and reasoning proceed through multiple dependent stages. We study whether claim-level conformal factuality control, previously developed for RAG, remains effective in this setting. We apply split-conformal claim filtering to multi-hop RAG and evaluate it on HotpotQA, Natural Questions, and TriviaQA using Llama 3.1 8B and GPT-4o-mini, together with a single-hop reference experiment. Across all six multi-hop model-dataset configurations, increasingly stringent conformal targets consistently increase the fraction of responses whose retained claims are fully supported. At the 95% target, this rate ranges from 95.80% to 97.20%, compared with 55.60%-76.03% without filtering. However, the improvement is strongly selective: only 4.41%-31.09% of generated claims are retained and 9.70%-51.40% of responses remain non-empty at the 95% target. These results show that conformal factuality extends to multi-hop RAG, while demonstrating that nominal reliability must be interpreted jointly with claim retention and abstention.
Muhammad Aimal Rehman, Chi-Kuang Yeh
Department of Mathematics & Statistics Georgia State University Atlanta, GA, USA
Large Language Models (LLMs) frequently hallucinate, limiting their reliability in critical applications. Conformal Prediction (CP) addresses this by calibrating error rates on held-out data to provide statistically valid confidence guarantees. Recent work extends CP to LLM factuality to filter out risky claims, ensuring that hallucination rates remain below a user-specified level (e.g., 10%). While prior methods treat claims independently, Coherent Factuality extends to multi-step reasoning by representing outputs as dependency graphs and jointly validating claims with their logical ancestors. A key limitation is that Coherent Factuality is not differentiable, requiring hand-crafted scorers that at high reliability levels remove nearly 60% of true claims. We introduce Differentiable Coherent Factuality (DCF), a fully differentiable relaxation that enables learning improved scorers while provably recovering the original algorithm's guarantees. Experiments on two benchmark reasoning datasets demonstrate DCF achieves up to 141% improvement in claim retention while maintaining reliability guarantees, representing a significant step towards reliable conformal LLM systems.
Nathan Hittesdorf, Marco Salzetta, Lu Cheng
Department of Computer Science, University of Illinois at Chicago, Chicago, United States · Department of Physics, University of Illinois at Urbana-Champaign, Urbana, United States