cs.AIMar 17, 2026
SaveIs Conformal Factuality for RAG-based LLMs Robust? Novel Metrics and Systematic Insights
Organizations: University of Wisconsin-Madison
Abstract
Large language models (LLMs) frequently hallucinate, limiting their reliability in knowledge-intensive applications. Retrieval-augmented generation (RAG) and conformal factuality have emerged as potential ways to address this limitation. While RAG aims to ground responses in retrieved evidence, it provides no statistical guarantee that the final output is correct. Conformal factuality filtering offers distribution-free statistical reliability by scoring and filtering atomic claims using a threshold calibrated on held-out data, however, the informativeness of the final output is not guaranteed. We systematically analyze the reliability and usefulness of conformal factuality for RAG-based LLMs across generation, scoring, calibration, robustness, and efficiency. We propose novel informativeness-aware metrics that better reflect task utility under conformal filtering. Across three benchmarks and multiple model families, we find that (i) conformal filtering suffers from low usefulness at high factuality levels due to vacuous outputs, (ii) conformal factuality guarantee is not robust to distribution shifts and distractors, highlighting the limitation that requires calibration data to closely match deployment conditions, and (iii) lightweight entailment-based verifiers match or outperform LLM-based model confidence scorers while requiring over fewer FLOPs. Overall, our results expose factuality-informativeness trade-offs and fragility of conformal filtering framework under distribution shifts and distractors, highlighting the need for new approaches for reliability with robustness and usefulness as key metrics, and provide actionable guidance for building RAG pipelines that are both reliable and computationally efficient.
Figures & tables
Figure 1: Overview of our framework. Given a query and retrieved references , the Response Generator produces an output . The conformal factuality framework utilizes a separate calibration data to determine a threshold used to filter out information from the output and yield . (See Figure 2 for the details of different stages involved in conformal filtering.)
Figure 2: Given an input and a reference text related to , the Response Generator produces an output , which is then parsed by the Parser into a list of claims. Each individual claim is subsequently scored by the Scorer, conditioned on the input and, optionally, the reference text. These scores are passed to the conformal prediction algorithm, which filters out claims whose scores fall below a learned threshold. Finally, the remaining claims are merged into a single paragraph and returned to the user.
| Family | Model | Params | Arch. | Activated |
|---|---|---|---|---|
| Qwen3 | Qwen3-0.6B | 0.6B | Dense | 0.6B |
| Qwen3 | Qwen3-4B | 4B | Dense | 4B |
| Qwen3 | Qwen3-8B | 8.2B | Dense | 8.2B |
| Qwen3 | Qwen3-30B-A3B | 30.5B | MoE | 3.3B |
| Qwen3 | Qwen3-32B | 32.8B | Dense | 32.8B |
| Llama-3.x | Llama-3.2-1B-Instruct | 1.2B | Dense | 1.2B |
Table 1: Summary of evaluated open-source language models. For dense models, the number of activated parameters equals the total parameter count. For MoE models, activated parameter counts are shown when known.
Figure 3: Sufficient correctness (SC) of Qwen3 models (0.6B, 4B, 8B) on four datasets (MATH-200, FActScore-Rare, FActScore, NQ-200), with and without access to references. Across model sizes and datasets, providing references consistently improves generation quality.
Figure 4: Evaluation of various prompting strategies across different LLMs on the FActScore dataset (Section 4.1 ) at level . Results demonstrate that: (i) prompting models to generate numeric scores consistently outperforms Boolean scoring; (ii) sampling multiple responses uniformly improves performance; however, (iii) incorporating chain-of-thought reasoning or evidence highlighting do not yield reliable performance gains across models.
Figure 5: Performance of model confidence score on MATH-1K dataset with and without reference provided to scoring functions using Qwen3-4B as the LLM-based scorer.
Figure 6: Experimental evaluation across three model families of various scales on the FActScore dataset at level . Results demonstrate that increasing model size does not consistently improve performance of conformal factuality.
Figure 7: Comparison between entailment-based scores against the model confidence score on the FActScore dataset. Across our evaluated settings, the entailment-based scores match or exceed the model-confidence baseline, despite the entailment model being substantially smaller than the target LLMs.
Figure 8: Sufficient Correctness (SC) and Conditional Sufficient Correctness (CSC) for the Qwen3 model family at across FActScore, MATH-1K, and NQ-1K dataset.
Figure 9: Empirical factuality (EF) on FActScore, MATH-1K, and NQ-1K, under two calibration settings: (i) calibration claims generated in Mohri and Hashimoto (2024) , which comes from a different distribution (ii) calibration claims drawn from the same distribution as the test data. We use Qwen3-4B as the scoring function. The results show how the distribution shift in the calibration set affects conformal factuality guarantees.
Figure 10: Power on FActScore, MATH-1K, and NQ-1K, under two calibration settings: (i) calibration claims generated in Mohri and Hashimoto (2024) , which comes from a different distribution (ii) calibration claims drawn from the same distribution as the test data. We use Qwen3-4B as the scoring function.
Figure 11: (a) - (d) Empirical factuality versus target factuality (1- ) for Qwen3-4B on the FActScore dataset. Each panel corresponds to a different distractor injection rate in the test set (0.0, 0.1, 0.25, 0.5). The gray dashed diagonal represents perfect calibration ( ), where empirical factuality matches the target. (e) Power versus target factuality (1 - ) when the injection rate is 0.1.
Figure 12: (a) - (c) Empirical factuality versus target factuality ( ) for Qwen3-4B on the FActScore dataset when the test set is injected with 25% distractors. We vary the proportion of distractors in the calibration set from (0.1, 0.25, 0.5). As the proportion matches, we can see that the empirical factuality rises to the line. (d)-(e) Comparison of non-empty rate when both the test and calibration sets contain no distractors and contain 25% of the distractors. Although introducing distractors to the calibration set can achieve target factuality, the non-empty rate suffers.
| Model | Total Tokens | Total FLOPs |
|---|---|---|
| gpt-oss-20b (3.6B active) ( Agarwal et al., 2025 ) | 2000 | |
| Qwen3-8B (8.19B active) ( Yang et al., 2025 ) | 2000 | |
| DeepSeek-R1 (37B active) ( DeepSeek-AI, 2025 ) | 2000 | |
| DeBERTa (184M active) ( He et al., 2020 ) | 2000 | |
| RoBERTa (356M active) ( Liu et al., 2019 ) | 2000 |
Table 2: Estimated FLOPs for generating 1000 tokens with a 1000-token prompt (assuming KV caching).
Figure 13: Performance of entailment- and confidence-based scorers using gpt-oss-20b and Qwen3-8B on FActScore.
Appendix figures & tables40 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 14: Sufficient correctness (SC) of Llama-3.x models on four datasets (MATH-200, FActScore-Rare, FActScore, NQ-200), with and without access to references. Across model sizes and datasets, providing references consistently improves generation quality.
Figure 15: Sufficient correctness (SC) of SmolLM2 models on four datasets (MATH-200, FActScore-Rare, FActScore, NQ-200), with and without access to references. Across model sizes and datasets, providing references consistently improves generation quality.
Figure 16: Sufficient correctness (SC) of Qwen3 and frontier models on FActScore-Rare, with and without access to references. Providing references consistently improves generation quality; notably, Qwen3-4B with references is comparable to frontier models.
Figure 17: Overall performance comparison of model confidence scores across 16 prompting strategies at five target factuality levels ( ) on the FActScore dataset. Each column corresponds to a different LLM scorer; each row corresponds to a different target factuality. Numeric scoring and consistency averaging yield the most robust improvements.
Figure 18: Overall performance comparison of model confidence scores across various prompting strategies at five target factuality levels ( ) on the MATH-1K dataset. The advantage of numeric over Boolean scoring is especially clear for mathematical reasoning.
Figure 19: Performance of model confidence score on FActScore with and without reference provided to scoring functions, using gpt-5-nano as generator and Qwen3-4B as scorer. Reference access improves power and non-empty rate.
Figure 20: Performance of model confidence score on NQ-1K with and without reference provided to scoring functions, using gpt-5-nano as generator and Qwen3-4B as scorer. The benefit of reference access is consistent with the MATH-1K and FActScore results.
Figure 21: Performance of model confidence score on FActScore with and without reference provided to scoring functions, using gpt-5-nano as generator and Qwen3-8B as scorer.
Figure 22: Performance of model confidence score on MATH-1K with and without reference provided to scoring functions, using gpt-5-nano as generator and Qwen3-8B as scorer.
Figure 23: Performance of model confidence score on NQ-1K with and without reference provided to scoring functions, using gpt-5-nano as generator and Qwen3-8B as scorer.
Figure 24: Performance of model confidence score on FActScore with and without reference provided to scoring functions, using gpt-5-nano as generator and Llama-3.2-3B-Instruct as scorer.
Figure 25: Performance of model confidence score on MATH-1K with and without reference provided to scoring functions, using gpt-5-nano as generator and Llama-3.2-3B-Instruct as scorer.
Figure 26: Performance of model confidence score on NQ-1K with and without reference provided to scoring functions, using gpt-5-nano as generator and Llama-3.2-3B-Instruct as scorer.
Figure 27: Performance of model confidence score on FActScore with and without reference provided to scoring functions, using gpt-5-nano as generator and SmolLM2-1.7B-Instruct as scorer.
Figure 28: Performance of model confidence score on MATH-1K with and without reference provided to scoring functions, using gpt-5-nano as generator and SmolLM2-1.7B-Instruct as scorer.
Figure 29: Performance of model confidence score on NQ-1K with and without reference provided to scoring functions, using gpt-5-nano as generator and SmolLM2-1.7B-Instruct as scorer.
Figure 30: Overall performance comparison of model confidence scores across different model scales and families at five target factuality levels ( ) on the FActScore dataset. While Llama-3.x models show consistent gains with scale, Qwen3 and SmolLM2 do not, highlighting that scaling alone is insufficient for improving conformal factuality.
Figure 31: Comparison between entailment-based scores against the model confidence score on the MATH-1K dataset.
Figure 32: Sufficient Correctness (SC) and Conditional Sufficient Correctness (CSC) for the Llama-3.x family at across FActScore, MATH-1K, and NQ-1K. CSC consistently exceeds SC, indicating that filtering largely preserves useful content when the generator provides it.
Figure 33: Sufficient Correctness (SC) and Conditional Sufficient Correctness (CSC) for the SmolLM2 family at across FActScore, MATH-1K, and NQ-1K. No consistent scaling benefit is observed within this family.
Figure 34: Sufficient Correctness (SC) and Conditional Sufficient Correctness (CSC) for the gpt-oss family at across FActScore, MATH-1K, and NQ-1K. The larger gpt-oss-120b does not consistently outperform gpt-oss-20b .
Figure 35: Empirical factuality (EF) on FActScore, MATH-1K, and NQ-1K under same-distribution and different-distribution calibration, using Llama-3.2-3B-Instruct as the scorer. Distribution shift causes the factuality guarantee to break for several scoring functions.
Figure 36: Empirical factuality (EF) on FActScore, MATH-1K, and NQ-1K under same-distribution and different-distribution calibration, using SmolLM2-360M-Instruct as the scorer. The results confirm that distribution shift in the calibration set undermines conformal factuality guarantees across model families.
Figure 37: Empirical factuality under varying test distractor proportions ( to ) with fixed clean calibration data on FActScore, using Llama-3.2-3B-Instruct . Factuality degrades sharply as distractor rate increases.
Figure 38: Empirical factuality under varying test distractor proportions ( to ) with fixed clean calibration data on FActScore, using SmolLM2-1.7B-Instruct . The degradation pattern is consistent with other model families.
Figure 39: Empirical factuality under varying test distractor proportions on MATH-1K with Qwen3-4B . Mathematical claims are especially vulnerable to subtle numerical perturbations.
Figure 40: Empirical factuality under varying test distractor proportions on MATH-1K with Llama-3.2-3B-Instruct .
Figure 41: Empirical factuality under varying test distractor proportions on MATH-1K with SmolLM2-1.7B-Instruct .
Figure 42: Empirical factuality under varying test distractor proportions on NQ-1K with Qwen3-4B .
Figure 43: Empirical factuality under varying test distractor proportions on NQ-1K with Llama-3.2-3B-Instruct .
Figure 44: Empirical factuality under varying test distractor proportions on NQ-1K with SmolLM2-1.7B-Instruct .
Figure 45: Empirical factuality and non-empty rates under varying calibration distractor proportions on FActScore with Llama-3.2-3B-Instruct . Matching calibration to test distractor levels restores factuality but reduces the non-empty rate.
Figure 46: Empirical factuality and non-empty rates under varying calibration distractor proportions on FActScore with SmolLM2-1.7B-Instruct .
Figure 47: Empirical factuality and non-empty rates under varying calibration distractor proportions on MATH-1K with Qwen3-4B .
Figure 48: Empirical factuality and non-empty rates under varying calibration distractor proportions on MATH-1K with Llama-3.2-3B-Instruct .
Figure 49: Empirical factuality and non-empty rates under varying calibration distractor proportions on MATH-1K with SmolLM2-1.7B-Instruct .
Figure 50: Empirical factuality and non-empty rates under varying calibration distractor proportions on NQ-1K with Qwen3-4B .
Figure 51: Empirical factuality and non-empty rates under varying calibration distractor proportions on NQ-1K with Llama-3.2-3B-Instruct .
Figure 52: Empirical factuality and non-empty rates under varying calibration distractor proportions on NQ-1K with SmolLM2-1.7B-Instruct .
| Pair | Labelers | Agreement Rate |
|---|---|---|
| Model–Human | gpt-5-nano vs. Student A | 76.5% |
| Model–Human | gpt-5-nano vs. Student B | 77.0% |
| Human–Human | Student A vs. Student B | 73.0% |
Table 3: Agreement rates on factuality labels between gpt-5-nano and two human annotators, as well as between the two human annotators themselves. The model–human agreement is comparable to inter-human agreement.