Automated fact-checking pipelines typically begin with a filtering stage that decides which claims are worth verifying, given that the later evidence retrieval and verification components are expensive to apply at scale. A central task in this stage is verifiable claim detection, which asks whether a statement is in principle checkable against external evidence. Prior work on this task, as well as on the closely related notion of check-worthiness, conditions its decisions only on the claim sentence itself. We argue that this is restrictive, because deciding whether a statement is checkable often depends on identifying the entities and events it mentions, and on whether external information about them is actually available in the first place. Motivated by how downstream verification systems rely on retrieved evidence, we move retrieval upstream into the detection stage and introduce ContextClaim. Given an input claim, the approach identifies entity mentions, queries Wikipedia as a structured background source, and uses large language models to compress the retrieved material into short contextual summaries that are then passed to a classifier. Experiments are conducted on two domains and genres, namely the CheckThat! 2022 Twitter collection and the PoliClaim corpus of political debates, and cover both encoder and decoder only models under fine-tuning, zero-shot, and few-shot settings. The added context yields gains on verifiable claim detection in several configurations, although the size of the improvement varies with the dataset, the backbone model, and the training setup. We further find that the same retrieved summaries are useful beyond detection. Feeding them into a downstream verification model on FEVER improves verification F1. Component level analyses, human annotation, and error inspection further clarify the conditions under which retrieved context helps, and where it does not.
Figures & tables
Fig. 1: Example of ContextClaim system processing a vaccine claim about Lindsey Graham.
Split
CT22
PoliClaim †
Total
Veri
Non
Total
Veri
Non
Train
3,324
64%
36%
1,757
59%
41%
Dev
307
64%
36%
196
59%
41%
Test
251
59%
41%
816
64%
36%
TABLE I: Dataset Statistics. Veri = Verifiable, Non = Non-verifiable.
CT22
PoliClaim
Model
System
Acc
F1
Acc
F1
BERT
Baseline
68.29 ±1.37
76.45 ±0.72
82.77 ±0.41
87.07 ±0.32
CC-G4o
70.20 ±1.77
77.18 ±1.06
84.02 ±0.97
87.94 ±0.59
CC-M
69.48 ±0.83
76.89 ±0.28
84.09 ±1.48
88.06 ±0.81
RoBERTa
Baseline
70.84 ±1.94
78.53 ±0.50
85.78 ±0.92
89.42 ±0.67
CC-G4o
71.95 ±3.40
78.64 ±1.85
86.45 ±0.53
89.71 ±0.20
TABLE II: Fine-tuning results on CT22 and PoliClaim (mean ± std). Bold: column-wise best. Underline: overall best per metric.
CT22
PoliClaim
Zero-shot
Few-shot
Zero-shot
Few-shot
Model
System
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Llama3
Baseline
60.16
74.87
71.31
79.78
69.24
79.71
72.55
80.42
CC-G4o
62.15
75.83
68.53
78.59
70.96
80.23
74.26
81.87
CC-M
62.55
76.02
67.33
78.07
71.81
80.87
74.51
80.74
Mistral
Baseline
67.73
78.28
63.75
76.49
79.66
84.28
71.08
81.21
TABLE III: Zero-shot and few-shot results on CT22 and PoliClaim. Bold: column-wise best. Underline: overall best per metric.
System
Dataset
Acc (%)
Majority Baseline
CT22
59.36
CheckThat!’22 Best [ 24 ]
CT22
76.10
ContextClaim CC-G4o (Ours)
CT22
79.28
Majority Baseline
PoliClaim
63.85
AFaCTA [ 12 ]
PoliClaim
86.27
ContextClaim CC-G4o (Ours)
PoliClaim
86.45
TABLE IV: Comparison with published systems on CT22 and PoliClaim.
Fig. 2: Performance Improvement on two Datasets with three Settings.
Fig. 3: Entity Coverage and Quantity Comparison of PoliClaim and CT22.
Fig. 4: Precision-Recall Trade-off on Two Datasets.
Fig. 5: Model Architecture Effects on Contextual Information Benefits.
Fig. 6: Learning Setting Comparison.
Setting
Model
BL → Raw
Raw → G4o
Raw → M
Acc
F1
Acc
F1
Acc
F1
CT22
FT
Llama3
-0.80
+1.73
+0.80
+0.40
+1.06
+0.79
Mistral
+11.02
+3.61
+1.06
+0.21
-2.92
-3.20
ZS
Llama3
+3.19
+1.42
-1.20
-0.46
-0.80
-0.27
Mistral
-0.80
-0.65
-1.59
-0.59
-1.59
-0.59
TABLE V: Performance Gain ( Δ ) from ContextClaim Components.
Dimension
GPT-4o
Mistral
Diff
p-value
Relevance
2.93 ± 0.03
2.88 ± 0.04
+0.06
0.21
Signal Clarity
1.80 ± 0.08
1.74 ± 0.07
+0.06
0.41
Usefulness
1.62 ± 0.07
1.58 ± 0.07
+0.03
0.59
Overall
2.12 ± 0.90
2.07 ± 0.89
+0.05
–
TABLE VI: Human evaluation results for context summary quality. Ratings on 1–3 scale (1=Poor, 2=Acceptable, 3=Good). No significant differences between systems (paired t-test, all p>0.05 ).
Model
FP
FN
FP/FN
Errors
Llama3 Baseline
71 (28.3%)
23 (9.2%)
3.09
94 (37.5%)
Llama3 + CC-G4o
86 (34.3%)
7 (2.8%)
12.29
93 (37.1%)
RoBERTa Baseline
63 (25.1%)
13 (5.2%)
4.85
76 (30.3%)
RoBERTa + CC-M
48 (19.1%)
18 (7.2%)
2.67
66 (26.3%)
TABLE VII: Error Distribution by Model on CT22 (n=251).
Metric
Claim Only
Claim + CC-G4o
Δ
Accuracy
88.70
89.34
+0.64
Precision
91.00
87.97
− 3.03
Recall
85.89
91.14
+5.25
F1
88.37
89.53
+1.16
Macro-F1
88.69
89.34
+0.65
TABLE VIII: Downstream verification on FEVER (SUPPORTS vs. REFUTES, n=6,666 ). GPT-4o classifier is held fixed; only the input varies. F1: positive=SUPPORTS.
Misinformation verification increasingly occurs in public, fast-moving, and multilingual online settings, where static benchmarks provide an incomplete measure of model reliability. We introduce CommunityFact, a refreshable benchmark for misinformation detection in the wild, with three major goals: coverage, granularity, and redistributability. This release contains 15,992 standalone claims across five languages and two domains. We evaluate ten LLMs under varying inference-time capabilities, including thinking and web-search. Our results show that closed-input verification remains challenging, web access yields the largest gains, and web-enabled LLMs' source-selection policies are systematically misaligned with the sources human Community Notes raters converge on -- a gap that closes through model-specific mechanisms of retrieval expansion or pruning. We further find substantial variation across language-domain slices and across the evidence ecosystems used by web-enabled systems. Beyond evaluation, CommunityFact positions Community Notes as a training signal for claim-conditioned source suggesters that could improve factual verification on novel claims.
Sahajpreet Singh, Insyirah Mujtahid, Min-Yen Kan +1
Large language models (LLMs) excel in generating fluent utterances but can lack reliable grounding in verified information. At the same time, knowledge-graph-based fact-checkers deliver precise and interpretable evidence, yet suffer from limited coverage or latency. By integrating LLMs with knowledge graphs and real-time search agents, we introduce a hybrid fact-checking approach that leverages the individual strengths of each component. Our system comprises three autonomous steps: 1) a Knowledge Graph (KG) Retrieval for rapid one-hop lookups in DBpedia, 2) an LM-based classification guided by a task-specific labeling prompt, producing outputs with internal rule-based logic, and 3) a Web Search Agent invoked only when KG coverage is insufficient. Our pipeline achieves an F1 score of 0.93 on the FEVER benchmark on the Supported/Refuted split without task-specific fine-tuning. To address Not enough information cases, we conduct a targeted reannotation study showing that our approach frequently uncovers valid evidence for claims originally labeled as Not Enough Information (NEI), as confirmed by both expert annotators and LLM reviewers. With this paper, we present a modular, opensource fact-checking pipeline with fallback strategies and generalization across datasets.
Shaghayegh Kolli, Richard Rosenbaum, Timo Cavelius +3
We present ThinknCheck, a 1B-parameter verifier for grounded claim verification that first produces a short, structured rationale and then a binary verdict. We construct LLMAggreFact-Think, a 24.1k reasoning-augmented training set derived from LLMAggreFact, and fine-tune a 4-bit Gemma3 model to follow this format. On LLMAggreFact, ThinknCheck attains 78.1 balanced accuracy (BAcc), surpassing MiniCheck-7B (77.4) with 7x fewer parameters; removing the reasoning step reduces BAcc to 57.5. On SciFact, ThinknCheck reaches 64.7 BAcc, a +14.7 absolute gain over MiniCheck-7B. By contrast, zero-shot chain-of-thought on the base Gemma3-1B harms accuracy relative to direct answers, and preference optimization with a simple format+accuracy reward underperforms supervised reasoning. To probe the latter, we introduce GSMClaims and a domain-specialized variant, ThinknCheck-Science, which improves across benchmarks, including 61.0% accuracy on GSMClaims. Overall, explicit, supervised reasoning enables compact verifiers that are competitive while remaining resource-efficient and interpretable.