How Robust Are Automated Fact-Checking Systems? A Cross-Benchmark Evaluation

Aida Usmanova; Zangir Iklassov; Markus Leippold; Ricardo Usbeck

cs.AIAug 26, 2026

How Robust Are Automated Fact-Checking Systems? A Cross-Benchmark Evaluation

Authors: Aida Usmanova, Zangir Iklassov, Markus Leippold, Ricardo Usbeck

Organizations: Leuphana University of Lüneburg · MBZUAI · University of Zurich

Abstract

Automated fact-checking (AFC) systems retrieve evidence and predict claim veracity, yet evaluations omit simple baselines, systems are developed for a single benchmark and cannot be trusted to generalise across domains. No prior work cross-evaluates the full two-stage retrieve-then-verify pipeline across diverse datasets, complementing retrieval-only studies (Thakur et al., 2021) and single-stage benchmarking studies (Calamai et al., 2025). We benchmark nine models, ranging from random and sparse baselines to fine-tuned transformers, zero-shot LLMs, and the two highest-ranked systems from the AVeriTeC 2025 shared task, across four datasets spanning scientific, open-web, and climate domains. Three findings stand out: (1) on ClimateCheck claim-only and fine-tuned models outperform zero-shot LLM and top-performing AVeriTeC 2025 systems, highlighting that noisy evidence can degrade veracity prediction; (2) system rankings are strongly domain- and metric-dependent: the best model on SciFact (macro-F1 0.70) drops to 0.31 on ClimateCheck, while the AVeriTeC 2025 winner and runner-up swap rankings based on evaluation metrics and datasets; (3) replacing retrieved evidence with gold annotations improves veracity accuracy by 14-22 points across models, confirming retrieval remains primary bottleneck. We release code, pre-processed datasets, and all results to support reproducible AFC research.

Figures & tables

Dataset	Domain	Claim type	Claims	Evidence corpus	Labels
AVeriTeC	Open-web	Real-world	5,783	32,818 web docs	Sup / Ref / NEI / Conflicting
SciFact	Life sciences	Expert-written	1,409	5,183 S2ORC abstracts	Supports / Refutes / NEI
ClimateCheck	Climate/social	Social-media	3,199	394,269 sci. abstracts	Supports / Refutes / NEI
ClimateFEVER	Climate science	Real-world	7,675	5,240 Wikipedia pages	Sup / Ref / NEI / Conflicting

Table 1: Overview of the datasets used in the study.

Panel of Figure 1 — Figure 1: Classical fact-checking pipeline. Stage 1 retrieves evidence $\hat{E}_{K}$ from corpus $\mathcal{D}$ ; Stage 2 predicts veracity label $y$ .

	Expert / curated	Real-world / social
Scientific	SciFact 1.4K claims 5K abstracts biomedical domain	ClimateCheck 3.2K claims 394K abstracts social-media lang.
Web / Wiki	AVeriTeC 5.8K claims 33K web docs open-web evidence	ClimateFEVER 7.7K claims 5K passages Wikipedia sents.

Figure 2: Dataset taxonomy by evidence type (rows) and claim origin (columns). Each dataset is color-coded consistently throughout the paper.

Dataset	Method	Recall			F1
Dataset	Method	@5	@10	@20	@5	@10	@20
AVeriTeC	Random	0.0054	0.0054	0.0054	0.0026	0.0016	0.0009
	TF-IDF	0.1258	0.1258	0.1258	0.0652	0.0384	0.0245
	BM25	0.1170	0.1170	0.1170	0.0577	0.0341	0.0218
	Sanctuary	0.0769	0.1195	0.1694	0.0373	0.0361	0.0300
	AIC CTU	0.0749	0.0785	0.0785	0.0372	0.0227	0.0124

Table 2: Evidence retrieval results (Recall@

K

and F1@

K

). Bold : best per column. Underline : second best. Gray : not significantly better than TF-IDF . For AVeriTeC, the official Hungarian METEOR (

\geq 0.25

) is used instead of exact-match relevance.

\dagger

For sparse methods on AVeriTeC, Recall@

K

is identical across

K\in\{5,10,20\}

because the median gold set contains a single relevant document; once retrieved in the top-5, increasing

K

yields no further coverage.

System	Dataset	Q only	Q + A	AVeriTeC Score
Sanctuary	AVeriTeC	0.4461	0.2152	0.2200
	SciFact	0.6603	0.4343	0.4900
	ClimateCheck	0.4614	0.3142	0.4601
	ClimateFEVER	0.4696	0.3518	0.3247
AIC CTU	AVeriTeC	0.4610	0.3264	0.5360
	SciFact	0.5071	0.2509	0.3900

Table 3: Sanctuary and AIC CTU performance under the official AVeriTeC evaluation protocol (Hungarian METEOR

\geq 0.25

; see Appendix A ). Purpose: applying this metric cross-dataset reveals how effectively systems retrieve evidence whose text semantically matches the annotated question–answer pairs, isolating evidence adequacy from label prediction accuracy. Q only: METEOR score computed on retrieved question text alone (measures query coverage); Q + A: METEOR computed on question plus answer text (measures full evidence adequacy). Scores on non-AVeriTeC datasets are lower because those datasets lack QA-pair annotations, but the Q only vs. Q + A gap still quantifies how much answer content contributes to evidence quality for each system.

		AVeriTeC		SciFact		ClimateCheck		ClimateFEVER		Avg.
Method	Ev.	Acc	F1	Acc	F1	Acc	F1	Acc	F1	F1
Baselines
Random	▲	27.90 $\pm$ 2.13	23.32 $\pm$ 2.02	33.37 $\pm$ 2.17	33.17 $\pm$ 2.24	34.94 $\pm$ 2.05	32.78 $\pm$ 1.72	24.19 $\pm$ 3.61	21.84 $\pm$ 2.7	0.2778
TF-IDF + LogReg	▲	55.20	37.06	42.67	39.63	60.17	59.45	39.61	36.84	43.25
BM25 + LogReg	▲	54.80	32.15	45.67	42.34	57.52	56.90	44.45	38.64	42.51
Fine-tuned transformers

Table 4: Veracity prediction results (mean

\pm

std over 3 seeds where applicable; single run for LLMs). Ev.: ✓ = claim + retrieved evidence; ▲ = claim-only (no retrieved evidence). Bold : best per column. Underline : second best. Gray : not significantly better than TF-IDF + LogReg (bootstrap 95% CI,

p{<}0.05

). Full per-class breakdown in Table 13 .

Model	AVT	SCI	CCK	CFV
Llama 8B	$+$ 16	$+$ 19	$+$ 14	$+$ 20
Llama 70B	$+$ 14	$+$ 18	$+$ 15	$+$ 22

Table 5: Accuracy improvement (percentage points) when gold evidence replaces retrieved evidence. AVT = AVeriTeC; SCI = SciFact; CCK = ClimateCheck; CFV = ClimateFEVER.

Model	AVT	SCI	CCK	CFV
Llama 8B	$+$ 21.6	$+$ 0.3	$-$ 24.4	$+$ 3.9
Llama 70B	$+$ 6.8	$+$ 0.3	$-$ 19.4	$+$ 20.1

Table 6: A matched, identical-setting ablation (same model and prompt, ± retrieved evidence). Accuracy change when the same model is given retrieved evidence instead of claim-only.

Dataset	Mean Jaccard
AVeriTeC	0.093
SciFact	0.087
ClimateCheck	0.047
ClimateFEVER	0.122

Table 7: Jaccard similarity analysis. Per-dataset mean word overlap between claims and gold evidence.

Figure 3: Macro-F1 heatmap for every method-dataset pair on the claim veracity task.

Figure 4: Number of datasets (out of 4) on which each model outperforms TF-IDF + LogReg on accuracy (blue) and macro F1 (green).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Retriever	MRR	nDCG@10
ClimateCheck	TF-IDF	0.0707	0.0711
	BM25	0.0586	0.0616
	Random	0.0000	0.0000
SciFact	TF-IDF	0.1586	0.2149
	BM25	0.1590	0.2106
	Random	0.0011	0.0024

Appendix

Table 8: MRR and nDCG@10 for sparse retrieval, reported alongside the Recall@

K

/F1@

K

results of Table 2 . Rank-based metrics give the same system ordering.

Dataset	Train	Dev	Test	Total	Imbal.
AVeriTeC	4,626	578	579	5,783	$\approx$ 5.0
SciFact	1,127	141	141	1,409	$\approx$ 1.2
ClimateCheck	2,559	320	320	3,199	$\approx$ 3.7
ClimateFEVER	6,140	768	767	7,675	$\approx$ 4.2

Appendix

Table 9: Dataset split sizes (80/10/10) and label imbalance ratio (most frequent / least frequent class, estimated from per-class F1 of the random baseline).

Dataset	Corpus size	Avg. len.	Source
AVeriTeC	32,818	variable	Web pages
SciFact	5,183	$\sim$ 250 tok.	S2ORC abst
ClimateCheck	394,269	$\sim$ 200 tok.	Sci. abst
ClimateFEVER	5,240	$\sim$ 80 tok.	Wikipedia sent.

Appendix

Table 10: Evidence corpus characteristics. Average length is estimated from pre-processing statistics.

Figure 5: Per-class macro F1 across all models and datasets. Each bar shows the F1 for one label class; missing bars indicate zero F1. The rare Conflicting Evidence class is effectively unpredictable by all models. On SciFact and ClimateCheck, Refuted is the hardest class, reflecting annotation sparsity.

Figure 6: Predicted label distribution vs. gold label distribution. Each panel shows, for each model, the fraction of predictions assigned to each label. Models systematically over-predict the majority class, most severely on AVeriTeC ( Refuted ) and ClimateFEVER ( Supports ).

Hyperparameter	DistilRoBERTa	Longformer
Learning rate	$5\times 10^{-5}$	$5\times 10^{-5}$
Warmup ratio	0.1	0.1
Weight decay	0.01	0.01
Max epochs	10	10
Early stopping metric	Macro F1 (dev)	Macro F1 (dev)
Batch size	16	4

Appendix

Table 11: Hyperparameters for fine-tuned transformer models.

Dataset	Claim (paraphrased)	Retrieval / evidence note	Gold	Pred. / type
AVeriTeC	“A vaccine candidate reduced symptomatic COVID-19 by 90%”	Retrieved web snippet reports a different trial arm (70%); relevant QA pair is fragmented across two documents	Supported	Refuted (genuine)
AVeriTeC	“Company X lobbied against safety regulation”	Gold label is Conflicting Evidence but only one side is present in the evidence; annotation schema conflates two decisions	Conflicting	Refuted (annotation)
SciFact	“Drug Y inhibits tumour growth via pathway Z”	Abstract uses synonym “suppresses” for “inhibits”; TF-IDF retrieval misses the rationale sentence	Supports	NEI (genuine)
SciFact	“Protein X regulates inflammation”	Abstract states X modulates inflammation; whether modulation counts as regulation is ambiguous	Supports	NEI (debatable)
ClimateCheck	“LOL the ice caps are disappearing fast #climatecrisis”	Social-media phrasing shares no tokens with scientific abstract; model retrieves unrelated document about sea-surface temperature	Supports	NEI (genuine)
ClimateCheck	“Scientists proved global warming is fake news”	Ironic/sarcastic post; gold label treats it at face value (Refuted) though the author’s intent is pro-science	Refuted	Supports (debatable)

Appendix

Table 12: Representative error examples per dataset. “Genuine” = clear model error; “annotation” = questionable gold label; “debatable” = guideline ambiguity. Claims are paraphrased for anonymity; evidence notes summarise the key failure mode.

Dataset	Method	Supported	Refuted	NEI	Conflicting
AVeriTeC	Random	0.2468	0.3835	0.1287	0.1395
	TF-IDF + LogReg	0.3596	0.6995	0.2609	0.1622
	BM25 + LogReg	0.3868	0.6907	0.1667	0.0417
	DistilRoBERTa	0.4282	0.6971	0.1502	0.2199
	Longformer	0.4532	0.7121	0.1535	0.1920
	Llama 8B	0.4870	0.3536	0.1786	0.1261

Appendix

Table 13: Macro-F1 per-class breakdown.

Evidence condition	AVeriTeC	ClimateCheck
claim-only	81.0 / 69.0	69.1 / 60.3
$+$ TF-IDF retrieved	38.1 / 36.6	60.0 / 59.6
$+$ gold (oracle)	87.3 / 71.2	70.9 / 63.1

Appendix

Table 14: Claude Opus 4.8 under three evidence conditions with an identical prompt (accuracy / macro-F1), on fixed samples of AVeriTeC (

n{=}63

) and ClimateCheck (

n{=}55

). Only evidence quality varies across rows.

Verification strategy	AVeriTeC	ClimateCheck
one-shot TF-IDF retrieval	38.1 / 36.6	60.0 / 59.6
FIRE -style iterative	47.6 / 43.2	56.4 / 55.5
gold evidence (oracle)	87.3 / 71.2	70.9 / 63.1

Appendix

Table 15: Iterative retrieval-and-verification (accuracy / macro-F1) against one-shot retrieval and oracle retrieval, same model and samples as Table 14 . Second retrieval rounds are triggered for 25.4% (AVeriTeC) and 30.9% (ClimateCheck) of claims.

Automated Fact-Checking

Automated fact-checking detects checkable claims, retrieves evidence, and assesses whether that evidence supports or refutes them.

See all 136 papers

Althea: The Fact-Checking--Metalearning Tradeoff in AI-Assisted Verification
Dec 29, 2025Svetlana Churina, Kokil Jaidka, Anab Maulana Barik +5Human-in-the-Loop Evaluation Automated Fact-Checking
Centre for Trusted Internet & Community (CTIC), National University of Singapore (NUS), Singapore · Department of Communications and New Media, Centre for Trusted Internet & Community (CTIC), National University of Singapore (NUS), Singapore · School of Computing, Centre for Trusted Internet & Community (CTIC), National University of Singapore (NUS), Singapore
Standing on the Shoulders of Giants: Enhancing LLM Agents for Automated Fact-Checking with Human Expert Skills
Oct 19, 2026Xinwen Chen, Haorui He, Yupeng LiAutomated Fact-Checking LLM Agent Skill Learning
Department of Interactive Media, Hong Kong Baptist University · School of Computing and Data Science, The University of Hong Kong · AI and Social Good Lab, AI Media Centre, Hong Kong Baptist University
TrendFact: A Benchmark Towards Hotspot Perception in Automatic Fact-Checking
Oct 19, 2024Xiaocheng Zhang, Xi Wang, Yifei Lu +5Automated Fact-Checking
1Harbin Institute of Technology · 2National University of Defense Technology · 3Northeastern University +3

Dec 29, 2025J/K move · Enter open · S save

Althea: The Fact-Checking--Metalearning Tradeoff in AI-Assisted Verification

Svetlana Churina, Kokil Jaidka, Anab Maulana Barik, Harshit Aneja+4

Centre for Trusted Internet & Community (CTIC), National University of Singapore (NUS), Singapore · Department of Communications and New Media, Centre for Trusted Internet & Community (CTIC), National University of Singapore (NUS), Singapore · School of Computing, Centre for Trusted Internet & Community (CTIC), National University of Singapore (NUS), Singapore

Human-in-the-Loop Evaluation Automated Fact-Checking

Open paper AskPDF arXiv

Fact-checking systems must be scalable and epistemically trustworthy. We introduce Althea, a retrieval-augmented system for user-driven claim evaluation that matches standard pipelines on AVeriTeC while improving supported/refuted discrimination. A longitudinal survey experiment (N=961) treats a ten-day follow-up as a fading test: after modeling a verification procedure, we remove the system and ask whether users reproduce it unaided, testing metalearning rather than one-time accuracy. We compare two AI-assisted treatments, Exploratory (guided reasoning) and Summary (synthesized verdicts), against two baselines, unrelated news and Self-search. The treatments yield the strongest immediate accuracy and confidence gains but do not survive the fading test: on unseen claims they perform no better than news, while Self-search, with no procedure to fade, retains a large advantage. This reveals a factchecking-metalearning tradeoff: conditions that most improve immediate accuracy are least likely to produce metalearning, cautioning against treating AI-delivered verdicts as a source of durable literacy gains.

Abstract

Automated Fact-Checking

Explore similar work

Althea: The Fact-Checking--Metalearning Tradeoff in AI-Assisted Verification

Standing on the Shoulders of Giants: Enhancing LLM Agents for Automated Fact-Checking with Human Expert Skills

TrendFact: A Benchmark Towards Hotspot Perception in Automatic Fact-Checking