Trustworthiness is a core research challenge for agentic AI systems built on Large Language Models (LLMs). To enhance trust, natural language claims from diverse sources, including human-written text, web content, and model outputs, are commonly checked for factuality by retrieving external knowledge and using an LLM to verify the faithfulness of claims to the retrieved evidence. As a result, such methods are constrained by retrieval errors and external data availability, while leaving the models intrinsic fact-verification capabilities largely unused. We propose the task of fact-checking without retrieval, focusing on the verification of arbitrary natural language claims, independent of their source. To study this setting, we introduce a comprehensive evaluation framework focused on generalization, testing robustness to (i) long-tail knowledge, (ii) variation in claim sources, (iii) multilinguality, and (iv) long-form generation. Across 9 datasets, 18 methods and 3 models, our experiments indicate that logit-based approaches often underperform compared to those that leverage internal model representations. Building on this finding, we introduce INTRA, a method that exploits interactions between internal representations and achieves state-of-the-art performance with strong generalization. More broadly, our work establishes fact-checking without retrieval as a promising research direction that can complement retrieval-based frameworks, improve scalability, and enable the use of such systems as reward signals during training or as components integrated into the generation process.
Figures & tables
Figure 1: The task setting of fact-checking without retrieval. Claims from any source (human or LLMs) can be verified without having access to a knowledge base.
Method
ROC-AUC
PR-AUC
Llama 3.1
Ministral
Phi-4
Avg
Llama 3.1
Ministral
Phi-4
Avg
Retrieval-Based Methods
Verb+RAG
77.7
78.2
76.5
77.5
70.1
69.6
67.1
68.9
Retrieval-Free Unsupervised Methods
SP
70.9
69.0
66.4
68.8
66.9
64.9
62.8
64.9
PPL
68.6
67.3
66.6
67.5
65.6
64.2
63.1
64.3
Table 1: Average performance of claim verification methods for considered models in the proposed retrieval-free setting, measured by PR-AUC ↑ and ROC-AUC ↑ across nine datasets. Bold values indicate the best-performing retrieval-free method for each model, the second best is underlined . Avg column reports the average score across all models, summarizing overall robustness.
Figure 2: ROC-AUC ↑ performance of individual layers in the INTRA method.
Figure 3: ROC–AUC splited by popularity and language groups.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Claim
Filtered
Verification Explanation
Moby was born Richard Melville Hall.
No
This fact explicitly mentions the stage name “Moby” and provides a unique personal detail, his birth name.
Moby is an American singer-songwriter.
No
This fact explicitly mentions the entity “Moby” and describes his profession, directly tying the fact to him.
He rose to international fame.
Yes
The statement is too vague and lacks a specific entity or individual to whom it refers.
The album Play came out in 1999.
Yes
This fact is vague as it doesn’t specify the artist of the album “Play,” which could refer to multiple albums with the same title.
Appendix
Table 2: Examples of filtered data. Claims generated by the model in response to the query: “Tell me facts about Moby, American musician and songwriter.”
Dataset
# Before Filter
# After Filter
% Filtered
AC-PopQA
6,974
5,494
21
AC-WH
46,605
36,427
21
AVeriTeC
424
349
17
UHead
2,057
921
55
Cities
1,496
1,496
0
Companies
1,200
1,200
0
Appendix
Table 3: Final validation datasets after filtering claims. Cities, Common Claims, CounterFacts and Companies were not filtered out, as the claims are fully consistent with the original task.
Method
Runtime per instance (ms)
Retrieval-Based Unsupervised Methods
Verb+RAG
950±200
Retrieval-Free Unsupervised Methods
SP
30±1
PPL
30±1
MTE
84±40
Appendix
Table 4: Runtime per instance (in milliseconds) for different hallucination detection methods. Values are reported as mean ± standard deviation, measured per input instance.
Layers
PopQA
AVeriTeC
UHEAD
Cities
CC
CMP
CF
WH
X-Fact
Avg
All
89.5
66.4
63.7
98.7
74.2
89.1
73.7
73.3
58.1
76.3
0-8
84.9
59.5
53.0
83.9
62.1
65.4
59.7
64.1
55.2
65.3
24-32
86.7
63.8
61.0
73.2
71.5
70.5
58.4
70.0
57.0
68.0
2-30
89.6
66.2
63.6
98.8
73.8
89.7
74.7
73.1
58.3
76.4
8-24
89.6
66.8
65.3
99.3
74.4
92.0
77.4
73.5
57.6
77.3
11-22
89.3
66.7
66.4
99.0
75.0
93.0
79.1
73.5
57.1
77.7
Appendix
Table 5: Impact of different layer subsets on the performance by ROC-AUC ↑ of the proposed INTRA method. The best method is in bold , the second best is underlined .
Figure 4: ROC–AUC on WH by claim position for short and medium-length claims. The dashed line shows the fraction of hallucinated claims at each position; the numbers above it indicate the number of claims at that position.
Method
long-tail
human-made
cross-model
rule-generated
long-form
multilingual
Avg
PopQA
AVeriTeC
UHead
Cities
CC
CMP
CF
WH
X-Fact
Retrieval-Based Methods
Verb + RAG
72.8
74.4
72.7
98.4
73.3
89.0
74.3
71.0
73.3
77.7
Retrieval-Free Unsupervised Methods
SP
78.2
67.4
69.9
95.8
65.0
74.9
70.1
70.0
53.5
71.6
PPL
73.3
57.4
62.4
94.2
72.5
75.0
65.9
59.3
57.6
68.6
Appendix
Table 6: Performance of claim verification methods for Llama-3.1-8B in the proposed evidence-free setting, measured by ROC-AUC ↑ across nine datasets. Bold values indicate the best-performing method for each dataset, the second best is underlined . Avg column reports the average score across all datasets, summarizing overall robustness.
long-tail
human-made
cross-modal
rule-generated
long-form
multilingual
Method
PopQA
AVeriTeC
UHead
Cities
CC
CMP
CF
WH
X-Fact
Avg
Retrieval-Based Methods
Verb + RAG
86.0
82.5
45.5
97.1
67.0
83.6
66.2
36.0
67.0
70.1
Retrieval-Free Unsupervised Methods
SP
89.4
79.6
40.4
94.4
64.1
71.5
66.5
32.0
64.4
66.9
PPL
88.3
74.3
31.5
94.4
71.7
77.3
62.4
21.3
69.4
65.6
Appendix
Table 7: Performance of claim verification methods for Llama-3.1-8B in the proposed evidence-free setting, measured by PR-AUC ↑ (multiplied by 100) across nine datasets. Bold values indicate the best-performing method for each dataset, the second best is underlined . Avg column reports the average score across all datasets, summarizing overall robustness.
Method
long-tail
human-made
cross-model
rule-generated
long-form
multilingual
Avg
PopQA
AVeriTeC
UHead
Cities
CC
CMP
CF
WH
X-Fact
Retrieval-Based Methods
Verb + RAG
76.0
77.0
75.0
98.0
73.0
87.0
75.0
70.0
73.0
78.2
Retrieval-Free Unsupervised Methods
SP
75.3
65.0
68.8
95.9
63.9
67.6
64.7
72.2
47.8
69.0
PPL
72.7
55.3
56.7
95.1
71.4
70.3
62.3
61.0
61.3
67.3
Appendix
Table 8: Performance of claim verification methods for Ministral-8B in the proposed evidence-free setting, measured by ROC-AUC ↑ across nine datasets. Bold values indicate the best result per dataset, the second best is underlined , and Avg the mean score across all datasets.
Method
long-tail
human-made
cross-model
rule-generated
long-form
multilingual
Avg
PopQA
AVeriTeC
UHead
Cities
CC
CMP
CF
WH
X-Fact
Retrieval-Based Methods
Verb + RAG
87.0
85.0
47.0
96.0
67.0
81.0
67.0
31.0
67.0
69.6
Retrieval-Free Unsupervised Methods
SP
87.4
78.1
39.0
93.7
63.4
63.6
60.9
33.8
64.6
64.9
PPL
87.7
71.6
28.2
95.5
70.3
71.1
58.9
22.3
72.3
64.2
Appendix
Table 9: Performance of claim verification methods for Ministral-8B in the proposed evidence-free setting, measured by PR-AUC ↑ (multiplied by 100) across nine datasets. Bold values indicate the best result per dataset, the second best is underlined , and Avg the mean score across all datasets.
Method
long-tail
human-made
cross-model
rule-generated
long-form
multilingual
Avg
PopQA
AVeriTeC
UHead
Cities
CC
CMP
CF
WH
X-Fact
Retrieval-Based Methods
Verb + RAG
78.0
72.0
71.0
95.0
73.0
87.0
74.0
66.0
73.0
76.5
Retrieval-Free Unsupervised Methods
SP
71.5
64.0
66.2
92.2
63.8
65.0
61.6
67.4
45.8
66.4
Perplexity
72.2
55.0
60.6
91.6
70.7
71.1
61.9
61.0
55.0
66.6
Appendix
Table 10: Performance of claim verification methods for Phi-4-mini in the proposed evidence-free setting, measured by ROC-AUC ↑ across nine datasets. Bold values indicate the best result per dataset, the second best is underlined , and Avg the mean score across all datasets.
Method
long-tail
human-made
cross-model
rule-generated
long-form
multilingual
Avg
PopQA
AVeriTeC
UHead
Cities
CC
CMP
CF
WH
X-Fact
Retrieval-Based Methods
Verb + RAG
88.0
81.0
41.0
91.0
66.0
82.0
66.0
24.0
66.0
67.1
Retrieval-Free Unsupervised Methods
SP
86.0
78.2
38.6
87.8
62.4
61.4
59.0
28.6
63.0
62.8
Perplexity
87.7
73.4
31.2
90.9
68.5
69.2
58.7
21.7
66.2
63.1
Appendix
Table 11: Performance of claim verification methods for Phi-4-mini in the proposed evidence-free setting, measured by PR-AUC ↑ (multiplied by 100) across nine datasets. Bold values indicate the best result per dataset, the second best is underlined , and Avg the mean score across all datasets.
Automatic fact-checking systems assess the veracity of claims given evidence from relevant documents. Large Language Models (LLMs) have demonstrated strong performance in fact-checking due to their general reasoning capabilities. However, it remains unclear whether they faithfully make use of the evidence provided to reach veracity judgments or rely on parametric knowledge. To investigate this, we introduce Fact-Ablated Evaluation (FAE), a new evaluation framework that iteratively ablates the cited evidence to assess whether LLMs revise their predictions accordingly. Our empirical results show that current off-the-shelf LLMs as fact-checking systems rely more on their parametric knowledge than on the evidence provided. To bridge this gap between prediction accuracy and evidence grounding, we propose REAL (Rigorous Evidence Ablation Learning), a training framework that promotes evidence-dependent verification through counterfactual evidence supervision for the LLM-as-verifier models. Experiments on four fact-checking datasets across different domains demonstrate that models trained with REAL obtain superior evidence-dependent capabilities compared to standard fine-tuned models. Our findings highlight that strong fact-checking performance can still coexist with weak evidence dependency, while REAL encourages veracity predictions to remain more closely tied to the availability of supporting evidence.
Large language models (LLMs) excel in generating fluent utterances but can lack reliable grounding in verified information. At the same time, knowledge-graph-based fact-checkers deliver precise and interpretable evidence, yet suffer from limited coverage or latency. By integrating LLMs with knowledge graphs and real-time search agents, we introduce a hybrid fact-checking approach that leverages the individual strengths of each component. Our system comprises three autonomous steps: 1) a Knowledge Graph (KG) Retrieval for rapid one-hop lookups in DBpedia, 2) an LM-based classification guided by a task-specific labeling prompt, producing outputs with internal rule-based logic, and 3) a Web Search Agent invoked only when KG coverage is insufficient. Our pipeline achieves an F1 score of 0.93 on the FEVER benchmark on the Supported/Refuted split without task-specific fine-tuning. To address Not enough information cases, we conduct a targeted reannotation study showing that our approach frequently uncovers valid evidence for claims originally labeled as Not Enough Information (NEI), as confirmed by both expert annotators and LLM reviewers. With this paper, we present a modular, opensource fact-checking pipeline with fallback strategies and generalization across datasets.
Shaghayegh Kolli, Richard Rosenbaum, Timo Cavelius +3
Grounded claim factuality checking is important for large language model (LLM) applications such as retrieval-augmented generation, as it helps users assess the correctness of generated outputs. Existing metrics using entailment classifiers require dataset-specific threshold tuning, while LLM-based approaches often use direct prompting, which underutilises the reasoning capabilities of LLMs. We address this by formulating grounded claim factuality checking as a true/false reading comprehension task and prompting LLMs with explicit test-taking strategies for efficient reasoning. Our method reduces token usage by over 80% compared to unguided open-ended reasoning, and achieves competitive performance to more expensive alternatives across two factuality benchmarks, setting a new state of the art on one. To further reduce inference cost, we train small language models (SLMs) to replace LLMs in the checking pipeline. Using supervised fine-tuning (SFT) and a self-revision mechanism, the SLMs learn to improve their factuality judgements. Experimental results show that the resulting SLMs perform on par with strong baselines, combining low inference costs with generating supporting rationales to support interpretability. Code and datasets will be released upon acceptance.
Yuxuan Ye, Raul Santos-Rodriguez, Edwin Simpson
Intelligent Systems Laboratory University of Bristol United Kingdom