An Experimental Comparison of the Most Popular Approaches to Fake News Detection

Pietro Dell'Oglio; Alessandro Bondielli; Francesco Marcelloni; Lucia C. Passaro

cs.CLMar 26, 2026

An Experimental Comparison of the Most Popular Approaches to Fake News Detection

Authors: Pietro Dell'Oglio, Alessandro Bondielli, Francesco Marcelloni, Lucia C. Passaro

Organizations: Dipartimento di Ingegneria dell’Informazione, University of Pisa, Largo Lucio Lazzarino, 1, Pisa, Italy · Dipartimento di Informatica, University of Pisa, Largo B. Pontecorvo, 3, Pisa, Italy

Abstract

In recent years, fake news detection has received increasing attention in public debate and scientific research. Despite advances in detection techniques, the production and spread of false information have become more sophisticated, driven by Large Language Models (LLMs) and the amplification power of social media. We present a critical assessment of 12 representative fake news detection approaches, spanning traditional machine learning, deep learning, transformers, and specialized cross-domain architectures. We evaluate these methods on 10 publicly available datasets differing in genre, source, topic, and labeling rationale. We address text-only English fake news detection as a binary classification task by harmonizing labels into "Real" and "Fake" to ensure a consistent evaluation protocol. We acknowledge that label semantics vary across datasets and that harmonization inevitably removes such semantic nuances. Each dataset is treated as a distinct domain. We conduct in-domain, multi-domain and cross-domain experiments to simulate real-world scenarios involving domain shift and out-of-distribution data. Fine-tuned models perform well in-domain but struggle to generalize. Cross-domain architectures can reduce this gap but are data-hungry, while LLMs offer a promising alternative through zero- and few-shot learning. Given inherent dataset confounds and possible pre-training exposure, results should be interpreted as robustness evaluations within this English, text-only protocol.

Figures & tables

Reference	ML	DL	LLMs	Datasets	Cross Domain
Ahmed et al., 2018 [ 4 ]	6	x	x	2	x
Agarwal et al., 2019 [ 2 ]	5	x	x	1	x
Dutta et al., 2019 [ 20 ]	2	x	x	1	x
Janicka et al., 2019 [ 31 ]	5	x	x	4	✓
Katsaros et al., 2019 [ 34 ]	6	2	x	3	x
Jiang et al., 2021 [ 32 ]	5	3	x	2	x

Table 1: Existing studies on fake news detection that include an experimental analysis. We report the papers and the number of tested ML models (ML), DL models (DL), (Large) Language Models (LLMs), the tested Datasets and the presence of Cross-Domain experiments and methods.

Dataset	Topic Scope	Source Types	Ann. Criteria	Time Span	Mapping	Fake/Real
Celebrity [ 47 ]	Vips $\dagger$	Articles	Fact-checked fake/real gossip	2016–2017	Fake = fake; Real = real	250 / 250
Cidii [ 25 ]	Islamic Issues $\ddagger$	Articles; Social Media	Manual annotation of correct information vs. disinformation	2016–2021	Fake = fake; Real = real	300 / 422
FakeVsSatire [ 22 ]	Politics $\dagger$	Articles	Manual annotation of fake news vs. satire	2016–2017	Fake = fake; Real = satire	283 / 203
Fakes [ 51 ]	War in Syria $\ddagger$	Articles	Semi-automated fact-checking of fake/real news	2011–2018	Fake = fake; Real = real	378 / 426
Horne [ 27 ]	Politics $\dagger$	Articles	Manual annotation of real, fake, and satirical news	2015–2016	Fake = fake; Real = real or satire	123 / 203
Infodemic [ 46 ]	Covid-19 $\ddagger$	Social Media	Fact-checked COVID-19 misinformation	2020	Fake = fake; Real = real	5031 / 5526

Table 2: Description of all datasets used in the experiments (

\dagger

generic datasets;

\ddagger

narrow datasets). For each dataset we report topic scope, source types, annotation criteria, time span (when available), binary mapping rule used in the experiments, and number of Fake and Real instances.

Dataset	Fake class	Real class
Celebrity [ 47 ]	150	150
Cidii [ 25 ]	200	322
FakeVsSatire [ 22 ]	183	103
Fakes [ 51 ]	278	326
Horne [ 27 ]	23	103
Infodemic [ 46 ]	4933	5426

Table 3: Distribution of the instances in the subsets of the datasets used in the test phase of the third group of experiments.

Approach	Type	Features/Method
LR [ 32 ]	Traditional ML	Tf-Idf (Bow)
SVM [ 50 ]	Traditional ML	Tf-Idf (Bow)
NB [ 50 ]	Traditional ML	Tf-Idf (Bow)
CNN [ 61 ]	DL (Training from scratch)	Word2Vec (Embeddings)
BiLSTM [ 16 ]	DL (Training from scratch)	Word2Vec (Embeddings)
CNN-BERT [ 33 ]	DL (Training from scratch)	BERT (Embeddings)

Table 4: Approaches used for the experimentation.

Model	Celebrity	Cidii	FakeVsSatire	Fakes	Horne	Infodemic	Isot	LIAR-PLUS	NDF	Politifact
LR	.60	.97	.70	.43	.76	.93	.99	.56	.86	.77
SVM	.67	.86	.69	.44	.71	.93	.99	.57	.82	.75
NB	.47	.86	.68	.42	.68	.88	.96	.56	.79	.71
CNN	.54	.58	.56	.53	.56	.55	.96	.57	.56	.64
BiLSTM	.50	.59	.51	.56	.62	.70	.97	.57	.63	.65
CNN-BERT	.50	.59	.58	.53	.70	.68	.99	.58	.59	.64

Table 5: F1-score obtained by all models on each dataset. The decoder-only models are considered both in zero-shot (ZS) and few-shot (FS) scenarios. The best F1-score per dataset is in bold.

Figure 1: Distribution of F1-scores across the datasets for the Dataset-Specific experiments.

Model	Mean $\Delta$ F1	Standard Error (SE)
LR	-0.075	0.022
SVM	-0.089	0.020
NB	-0.131	0.030
CNN	-0.227	0.049
BiLSTM	-0.202	0.045
CNN-BERT	-0.194	0.043

Table 6: Mean effect size (

\Delta

F1) and Standard Error (SE) of all models compared to DeBERTa (best baseline). Calculations are performed across the 10 datasets.

Figure 2: Critical-difference diagram showing the statistically significant differences among models according to the Nemenyi post-hoc test.

Figure 3: Distribution of the average F1-scores across the datasets for the Cross-Dataset experiments.

Figure 4: Critical-difference diagram showing the statistically significant differences among models according to the Nemenyi post-hoc test.

Model	Test	Celebrity $\dagger$	Cidii $\ddagger$	FakeVsSatire $\dagger$	Fakes $\ddagger$	Horne $\dagger$	Infodemic $\ddagger$	Isot $\dagger$	LIAR $\dagger$	NDF $\ddagger$	Politifact $\dagger$
LR	Generic	.55	.45	.45	.51	.54	.50	.49	.55	.50	.52
LR	Narrow	.53	.47	.50	.53	.49	.47	.48	.52	.47	.55
SVM	Generic	.55	.48	.52	.49	.55	.48	.50	.54	.50	.56
SVM	Narrow	.51	.47	.49	.50	.49	.50	.49	.50	.46	.51
NB	Generic	.53	.47	.48	.52	.52	.48	.52	.55	.48	.51
NB	Narrow	.53	.50	.50	.51	.50	.47	.50	.51	.47	.52

Table 7: Average F1-scores achieved from the models on the generic

\dagger

(i.e Horne, Isot, FakeVsSatire, Politifact, LIAR-PLUS, Celebrity) and narrow

\ddagger

(i.e. Cidii, NDF, Fakes and Infodemic) datasets, when trained on the dataset in the column. Best F1-scores are highlighted in bold for all models except decoder-only LLMs, in order to emphasize approaches involving training or fine-tuning on the target data.

Model	Test	Celebrity $\dagger$	Cidii $\ddagger$	FakeVsSatire $\dagger$	Fakes $\ddagger$	Horne $\dagger$	Infodemic $\ddagger$	Isot $\dagger$	LIAR $\dagger$	NDF $\ddagger$	Politifact $\dagger$
LR	Articles	.56	.56	.46	.51	.53	.50	.51	.54	.49	.52
LR	Others	.54	.51	.49	.55	.56	.44	.46	.56	.51	.57
SVM	Articles	.56	.53	.52	.49	.55	.49	.51	.54	.50	.55
SVM	Others	.55	.51	.49	.49	.55	.45	.45	.55	.47	.54
NB	Articles	.53	.53	.48	.52	.52	.48	.53	.55	.48	.51
NB	Others	.56	.49	.50	.54	.56	.44	.48	.47	.50	.57

Table 8: Average F1-scores achieved from the models on the datasets grouped by genre: Articles (i.e Celebrity, FakeVsSatire, Fakes, Horne, Isot, Politifact) and Others (i.e. Cidii, Infodemic, LIAR-PLUS and NDF), when trained on the dataset indicated by each column. Best F1-scores are highlighted in bold, except for LLMs, to emphasize models trained from scratch or fine tuned on the target data.

Model	Celebrity	Cidii	FakeVsSatire	Fakes	Horne	Infodemic	Isot	LIAR-PLUS	NDF	Politifact	Global
LR	.58	.85	.72	.49	.71	.76	.68	.53	.82	.71	.61
SVM	.66	.80	.72	.53	.64	.77	.74	.55	.81	.75	.60
NB	.54	.75	.60	.51	.76	.66	.55	.53	.69	.72	.58
CNN	.54	.62	.44	.54	.66	.51	.53	.55	.62	.63	.49
BiLSTM	.49	.64	.38	.53	.68	.54	.50	.55	.58	.65	.52
CNN-BERT	.53	.53	.36	.54	.68	.55	.59	.49	.54	.64	.52

Table 9: F1-score obtained by all models on each dataset and on the Global test set. The decoder-only models are considered in both zero-shot (ZS) and few-shot (FS) scenarios. The best F1-score for each dataset is highlighted in bold.

	Celebrity	Cidii	FakeVsSatire	Fakes	Horne	Infodemic	Isot	LIAR-PLUS	NDF	Politifact	Global
Gate Entropy	0.003	0.053	0.002	0.007	0.057	0.175	0.007	0.780	0.175	0.034	0.078
CV	2.797	2.013	2.998	2.565	2.737	2.625	2.943	1.638	2.185	2.321	2.420

Table 10: Routing diagnostics for MERMAID across all test sets. For each dataset we report the gate entropy (lower = more deterministic routing) and the CV of expert utilization (higher = routing collapse).

Model	Celebrity	Cidii	Fakes	Infodemic	Isot	LIAR-PLUS	Global
Deberta (train 100/100)	.84	.98	.48	.88	.99	.57	.64
Mermaid (train 100/100)	.47	.52	.48	.84	.92	.56	.64
Deberta (train 150/150)	.79	.98	.47	.87	.99	.59	.64
Mermaid (train 150/150)	.57	.78	.49	.80	.92	.54	.59
Deberta (train 200/200)	.84	.98	.40	.91	.99	.60	.66
Mermaid (train 200/200)	.68	.82	.51	.82	.85	.57	.54

Table 11: F1 - scores for DeBERTa and MERMAID trained on balanced mixed - domain subsets of increasing size.

Figure 5: F1 - score gains for DeBERTa and MERMAID at each increase in training data size.

Figure 6: Critical-difference diagram showing the statistically significant differences among models according to the Nemenyi post-hoc test.

Model	Celebrity	Cidii	FakeVsSatire	Fakes	Horne	Infodemic	Isot	LIAR-PLUS	NDF	Politifact
LR	.57	.46	.63	.48	.74	.52	.73	.45	.42	.61
SVM	.51	.51	.55	.53	.57	.52	.55	.46	.52	.47
NB	.50	.43	.55	.53	.53	.55	.61	.53	.39	.52
CNN	.50	.51	.55	.53	.57	.52	.55	.46	.52	.47
BiLSTM	.51	.51	.55	.53	.57	.52	.55	.46	.52	.47
CNN-BERT	.55	.45	.59	.51	.64	.44	.49	.47	.38	.57

Table 12: F1-scores obtained by all models on each dataset in the Leave-One-Dataset-Out experiments. The decoder-only models are considered both in zero-shot (ZS) and few-shot (FS) scenarios. The best F1-score for each dataset is highlighted in bold.

Figure 7: Relationship between each model’s held - out performance ranking and the size of the remaining training pool.

Figure 8: Critical-difference diagram showing the statistically significant differences among models according to the Nemenyi post-hoc test.

Model	Med (E1)	IQR (E1)	Med (E2)	IQR (E2)	Med (E3)	IQR (E3)	Med (E4)	IQR (E4)
LR	0.765	0.2875	0.520	0.0775	0.71	0.155	0.545	0.16
SVM	0.73	0.175	0.510	0.0700	0.72	0.145	0.52	0.035
NB	0.695	0.2525	0.515	0.0900	0.60	0.1675	0.53	0.04
CNN	0.56	0.025	0.500	0.0750	0.54	0.10	0.52	0.0425
BiLSTM	0.605	0.0825	0.505	0.0800	0.54	0.115	0.52	0.035
CNN-BERT	0.59	0.09	0.500	0.0800	0.54	0.055	0.50	0.11

Table 13: Robustness summary across the four experiments. For each model we report the median and IQR of F1-scores for the dataset-specific (E1), cross-dataset (E2), mixed-training/single-test (E3), and leave-one-dataset-out (E4) experiments.

Model	Celebrity	Cidii	FakeVsSatire	Fakes	Horne	Infodemic	Isot	LIAR-PLUS	NDF	Politifact	Global
LLaMa3-8B (FS - Similar)	.83	.96	.64	.52	.52	.83	.71	.57	.89	.60	.58
LLaMa3-8B (FS - Random)	.86	.91	.66	.54	.64	.83	.72	.59	.75	.72	.58
Qwen3-32B (FS - Similar)	.83	.97	.65	.53	.64	.88	.78	.61	.89	.82	.65
Qwen3-32B (FS - Random)	.82	.87	.66	.52	.64	.83	.74	.59	.86	.78	.66
Zephyr-7B-beta (FS - Similar)	.78	.96	.61	.51	.50	.85	.75	.62	.92	.71	.71
Zephyr-7B-beta (FS - Random)	.71	.86	.54	.51	.57	.79	.59	.63	.81	.64	.55

Table 14: F1 - scores of decoder - only models using similarity - based and random exemplar selection in the mixed - domain few - shot setting

Appendix figures & tables47 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Source	# Training	# Validation	# Test	Date
Celebrity [ 47 ]	https://www.kaggle.com/datasets/sumanthvrao/fakenewsdataset	300	100	100	24/10/2023
Cidii [ 25 ]	https://www.kaggle.com/datasets/suhaibkhalil/cidii-dataset	433	144	145	14/11/2023
FakeVsSatire [ 22 ]	https://github.com/jgolbeck/fakenews	291	97	98	24/10/2023
Fakes [ 51 ]	https://zenodo.org/records/2607278	482	161	161	24/10/2023
Horne [ 27 ]	https://github.com/rpitrust/fakenewsdata1	195	65	66	27/10/2023
Infodemic [ 46 ]	https://constraint-shared-task-2021.github.io/	6299	2139	2119	03/03/2024

Appendix

Table 1: Datasets used for the experiments.

Model	Parameter	Tested Range
LR	Penalty	L2; None
	C	0.01; 0.1; 1.0; 10.0; 100.0
	Solver	lbfgs; liblinear; sag; saga
SVM	Penalty	L1; L2
	C	0.01; 0.1; 1.0; 10.0; 100.0
	Loss	Hinge; Squared Hinge

Appendix

Table 1: Values for parameters optimization.

Model	Parameters	Celebrity	Cidii	FakeVsSatire	Fakes	Horne	Infodemic	Isot	LIAR	NDF	Politifact
LR	Penalty	l2	none	none	l2	none	l2	none	none	none	none
	C	10.0	0.01	0.01	100.0	0.01	100.0	100.0	1.0	0.01	0.01
	Solver	lbfgs	lbfgs	lbfgs	lbfgs	lbfgs	lbfgs	lbfgs	lbfgs	lbfgs	lbfgs
SVM	Penalty	l1	l1	l1	l1	l1	l1	l1	l1	l1	l1
	C	1.0	100.0	10.0	100.0	100.0	1.0	1.0	0.1	100.0	10.0
	Loss	sh	sh	sh	sh	sh	sh	sh	sh	sh	sh

Appendix

Table 2: The best hyperparameter values after parameter optimization.

Model	Celebrity	Cidii	FakeVsSatire	Fakes	Horne	Infodemic	Isot	LIAR-PLUS	NDF	Politifact
CNN	$.52\pm.02$	$.58\pm.00$	$.61\pm.03$	$.53\pm.03$	$.61\pm.02$	$.53\pm.01$	$.98\pm.00$	$.63\pm.03$	$.61\pm.03$	$.69\pm.02$
BiLSTM	$.50\pm.01$	$.59\pm.02$	$.55\pm.03$	$.50\pm.01$	$.62\pm.00$	$.67\pm.02$	$.97\pm.00$	$.55\pm.01$	$.62\pm.01$	$.68\pm.03$
CNN-BERT	$.51\pm.01$	$.59\pm.00$	$.58\pm.00$	$.52\pm.01$	$.66\pm.04$	$.72\pm.02$	$.99\pm.00$	$.63\pm.02$	$.62\pm.01$	$.73\pm.06$
BERT	$.71\pm.03$	$.98\pm.00$	$.72\pm.02$	$.49\pm.03$	$.74\pm.06$	$.97\pm.00$	$1.0\pm.00$	$.65\pm.02$	$.82\pm.01$	$.85\pm.01$
DeBERTa	$.78\pm.02$	$.96\pm.00$	$.82\pm.04$	$.50\pm.02$	$.89\pm.05$	$.97\pm.00$	$1.0\pm.00$	$.65\pm.01$	$.92\pm.02$	$.86\pm.00$

Appendix

Table 3: Average F1-score and standard deviation obtained by the neural models across 5 random seeds.

Dataset	Model	F1 across batch sizes (bs8 → bs256)
	BERT	0.75, 0.69, 0.67, 0.64, 0.61, 0.67
	DeBERTa	0.80, 0.77, 0.75, 0.63, 0.72, 0.52
Celebrity	CNN	0.54, 0.50, 0.50, 0.50, 0.50, 0.50
	BiLSTM	0.50, 0.50, 0.50, 0.50, 0.50, 0.50
	CNN-BERT	0.53, 0.52, 0.53, 0.53, 0.50, 0.50
	BERT	0.52, 0.58, 0.47, 0.55, 0.52, 0.53

Appendix

Table 4: Sensitivity analysis across batch sizes for the Celebrity and Fakes datasets.

Model	Celebrity	Cidii	FakeVsSatire	Fakes	Horne	Infodemic	Isot	LIAR-PLUS	NDF	Politifact	Global
LR	.54	.65	.61	.51	.60	.82	.83	.53	.71	.57	.76
SVM	.48	.77	.65	.53	.58	.84	.95	.54	.67	.56	.83
NB	.33	.77	.46	.34	.55	.85	.95	.55	.65	.56	.85
CNN	.58	.79	.52	.48	.60	.47	.95	.57	.63	.62	.87
BiLSTM	.50	.73	.56	.49	.57	.41	.95	.58	.71	.60	.86
CNN-BERT	.52	.71	.58	.46	.60	.85	.96	.57	.61	.57	.86

Appendix

Table 5: F1 - scores obtained by all models in the imbalanced mixed - training/single - test setting. Best F1-score for each dataset in bold.

Model	Celebrity	Cidii	Fakes	Infodemic	Isot	LIAR-PLUS	NDF	Politifact
LR	.52 (-.05)	.51 (.05)	.52 (.04)	.53 (.01)	.52 (-.21)	.50 (.05)	.57 (.15)	.51 (-.10)
SVM	.51 (.00)	.52 (.01)	.51 (-.02)	.53 (.01)	.53 (-.02)	.46 (.00)	.49 (-.03)	.46 (-.01)
NB	.50 (.00)	.43 (.00)	.53 (.00)	.57 (.02)	.60 (-.01)	.53 (.00)	.39 (.00)	.53 (.01)
CNN	.58 (.08)	.55 (.04)	.49 (-.04)	.49 (-.03)	.55 (.00)	.47 (.01)	.39 (.00)	.53 (.00)
BiLSTM	.52 (.01)	.43 (-.08)	.47 (-.06)	.52 (.00)	.57 (.02)	.45 (-.01)	.52 (.00)	.57 (.10)
CNN-BERT	.55 (.00)	.52 (.07)	.50 (-.01)	.50 (.06)	.49 (.00)	.47 (.00)	.44 (.06)	.58 (.01)

Appendix

Table 6: F1-scores obtained by all models on each dataset in the Leave-One-Dataset-Out setting, when satire is removed from training set. The decoder-only models are considered both in zero-shot (ZS) and few-shot (FS) scenarios. The best F1-score for each dataset is highlighted in bold. In brackets

\Delta=\text{F1}_{\text{no satire}}-\text{F1}_{\text{satire}}

Model	Modality	Prompt
Llama3-8B	Zero-shot	P-Z3
Qwen3-32B	Zero-shot	P-Z3
Zephyr-7B-beta	Zero-shot	P-Z1
Llama3-8B	Few-shot	P-Z3
Qwen3-32B	Few-shot	P-Z3
Zephyr-7B-beta	Few-shot	P-Z2

Appendix

Table 7: Best prompt selected after prompt optimization.

Model & Setting	Prompt	Decoding Settings	Few-shot Sampling	Selection seed
Llama3-8B (Zero-shot)	P-Z3	Temp = 0; Top-k = 1; Max new tokens = 64	N/A	N/A
Llama3-8B (Few-shot)	P-Z3	Temp = 0; Top-k = 1; Max new tokens = 64	6 examples (3 per class) *	42
Qwen3-32B (Zero-shot)	P-Z3	Temp = 0; Top-k = 1; Max new tokens = 64 †	N/A	N/A
Qwen3-32B (Few-shot)	P-Z3	Temp = 0; Top-k = 1; Max new tokens = 64 †	6 examples (3 per class) *	42
Zephyr-7B (Zero-shot)	P-Z1	Temp = 0; Top-k = 1; Max new tokens = 64	N/A	N/A
Zephyr-7B (Few-shot)	P-Z2	Temp = 0; Top-k = 1; Max new tokens = 64	6 examples (3 per class) *	42

Appendix

Table 8: Summary of experimental setups across all LLM settings.

Model	Celebrity	Cidii	FakeVsSatire	Fakes	Horne	Infodemic	Isot	LIAR-PLUS	NDF	Politifact
Llama3-8B (FS)	$.85\pm.01$	$.88\pm.02$	$.57\pm.00$	$.52\pm.00$	$.73\pm.00$	$.81\pm.02$	$.68\pm.02$	$.62\pm.00$	$.81\pm.00$	$.74\pm.01$
Qwen3-32B (FS)	$.81\pm.00$	$.82\pm.14$	$.58\pm.00$	$.53\pm.00$	$.73\pm.00$	$.83\pm.00$	$.71\pm.01$	$.63\pm.00$	$.83\pm.01$	$.80\pm.01$
Zephyr-7B-beta (FS)	$.66\pm.02$	$.89\pm.01$	$.41\pm.00$	$.53\pm.00$	$.58\pm.05$	$.75\pm.03$	$.59\pm.00$	$.62\pm.00$	$.79\pm.03$	$.59\pm.02$

Appendix

Table 9: Average F1-score and standard deviation across three runs with different randomly sampled few-shot exemplar sets. Results are computed under the leave-one-dataset-out setting, where each model is evaluated on a held-out dataset while being prompted with exemplars drawn from the remaining datasets.

Model	Celebrity	Cidii	FakeVsSatire	Fakes	Horne	Infodemic	Isot	LIAR-PLUS	NDF	Politifact
LLaMA3-8B (FS)
Median	.84	.87	.57	.52	.72	.82	.67	.62	.81	.74
IQR	.02	.03	.01	.01	.01	.04	.03	.00	.01	.02
Qwen3-32B (FS)
Median	.80	.92	.58	.53	.73	.83	.71	.63	.84	.80
IQR	.02	.16	.00	.01	.01	.00	.02	.01	.02	.02

Appendix

Table 10: Median and IQR of F1-scores across three runs with different randomly sampled few-shot exemplar sets. Results are computed under the leave-one-dataset-out setting, where each model is evaluated on a held-out dataset while being prompted with exemplars drawn from the remaining datasets.

Text (Truncated)	True Label	Predicted Label	Confidence
BREAKING: Bridge Collapse Claims Hillary Clinton, 14 Others…	Real	Fake	0.978
BREAKING: President Trump Releases Official Boycott List…	Real	Fake	0.966
Hillary Clinton: Women Only Voted For Trump Because Husbands…	Fake	Real	0.508
Procter & Gamble Release One Of The Most Racist Commercials…	Fake	Real	0.573

Appendix

Table 1: Examples of misclassified instances from the FakeVsSatire dataset. Confidence is the probability of the predicted class.

Text (Truncated)	True Label	Predicted Label	Confidence
Daesh executed at least 128 Syrian civilians in Al-Qaryatayn…	Fake	Real	0.786
US Secretary of Defense Carter said anti-Daesh coalition killed key ministers…	Fake	Real	0.774
Several Syrians Killed Injured in Homs Suicide Blasts.	Fake	Real	0.501
86 dead in suspected Syria chemical attack.	Real	Fake	0.516

Appendix

Table 2: Examples of misclassified instances from Fakes dataset. Confidence is the probability of the predicted class.

Text (Truncated)	True Label	Predicted Label	Confidence
EXCLUSIVE PICS: First photos of Ariana Grande since Manchester attack…	Fake	Real	0.994
Jennifer Lopez gets cozy with on-screen boyfriend Milo Ventimiglia…	Fake	Real	0.985
White Witch warns Angelina against using spells to woo back Brad…	Fake	Real	0.521
Is Evian Water The Secret Behind Cameron Diaz’s Great Skin?	Fake	Real	0.544

Appendix

Table 3: Examples of misclassified instances from the Celebrity dataset. Confidence is the probability of the predicted class.

Text (Truncated)	True Label	Predicted Label	Confidence
Three stories to read this morning: New lockdown…	Real	Fake	0.999
WHO @UNDP @UNAIDS launch COVID-19 Law Lab…	Real	Fake	0.999
I thought Corona affects only humans.	Fake	Real	0.558
I’ve decided to opt out of the KCL symptoms tracker… #frustrated	Fake	Real	0.617

Appendix

Table 4: Examples of misclassified instances from the Infodemic Dataset. Confidence is the probability of the predicted class.

Text (Truncated)	True Label	Predicted	Confidence
The costs of cancer and affiliated issues are over $20 billion a year…	Real	Fake	0.995
Affordable Care Act is a major reason why weve seen 50,000 fewer deaths…	Real	Fake	0.994
The platform of the Republican Party says deport everybody…	Fake	Real	0.512
Republican leaders are pushing to make privatizing Social Security…	Fake	Real	0.513

Appendix

Table 5: Examples of misclassified instances from the LIAR-PLUS Dataset. Confidence is the probability of the predicted class.

Text (Truncated)	True Label	Predicted Label	Confidence
Notre Dame Blaze - a spiritual perspective…	Fake	Real	0.998
I’ve spent weekends walking around Paris then stopping in Notre Dame…	Real	Fake	0.996
So horrible to watch the massive fire at Notre Dame Cathedral…	Real	Fake	0.994

Appendix

Table 6: Examples of misclassified instances from the NDF Dataset. Confidence is the probability of the predicted class.

Dataset	LR	SVM	NB	CNN	BiLSTM	BERT	DeBERTa	CNN-BERT
Celebrity (training)	7.63	5.91	5.53	192.02	196.97	189.07	237.05	50.01
Celebrity (entire)	7.11	8.51	5.46	194.41	197.93	210.51	290.84	42.62
Celebrity (inference)	2.70e-6	1.20e-6	2.00e-6	2.98e-3	7.00e-3	3.13e-3	5.16e-3	6.47e-3
Cidii (training)	6.04	5.50	6.30	212.69	241.61	247.42	406.38	98.67
Cidii (entire)	5.81	5.59	6.44	220.64	227.22	372.50	532.82	120.75
Cidii (inference)	8.00e-7	7.00e-7	9.00e-7	1.07e-3	7.17e-3	2.60e-3	4.18e-3	5.03e-3

Appendix

Table 1: Training and inference times (in seconds) for each approach.

Dataset	LR						SVM						NB
	Class 0			Class 1			Class 0			Class 1			Class 0			Class 1
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Celebrity	.59	.62	.60	.60	.58	.59	.66	.70	.67	.68	.64	.65	.48	.76	.58	.42	.18	.25
Cidii	1.0	.93	.96	.95	1.0	.97	.82	.85	.83	.89	.87	.88	.88	.78	.83	.85	.92	.89
FakeVsSatire	.87	.34	.49	.67	.96	.79	.68	.48	.57	.69	.84	.76	.81	.31	.45	.65	.94	.77
Fakes	.45	.47	.46	.39	.38	.38	.47	.41	.44	.42	.48	.45	.46	.50	.48	.38	.34	.36

Appendix

Table S1 : Precision, recall and F1-score obtained by the ML models on each dataset for each class.

Dataset	CNN						BiLSTM						CNN-BERT
	Class 0			Class 1			Class 0			Class 1			Class 0			Class 1
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Celebrity	.52	.88	.65	.62	.20	.30	.50	.16	.24	.50	.84	.62	.50	.96	.65	.50	.04	.07
Cidii	.44	.06	.11	.58	.94	.72	.50	.05	.09	.58	.96	.73	.00	.00	.00	.58	1.0	.73
FakeVsSatire	.00	.00	.00	.57	.96	.71	.39	.31	.35	.56	.64	.60	.00	.00	.00	.58	1.0	.73
Fakes	.52	1.0	.69	.00	.00	.00	.55	.77	.65	.55	.31	.40	.52	1.0	.69	.00	.00	.00

Appendix

Table S2 : Precision, recall and F1-score obtained by the CNN, BiLSTM and CNN-BERT models on each dataset for each class.

Dataset	BERT						DeBERTa
	Class 0			Class 1			Class 0			Class 1
	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Celebrity	.75	.78	.76	.77	.74	.75	.80	.86	.83	.85	.78	.81
Cidii	1.0	.95	.97	.96	1.0	.98	.97	.98	.97	.97	.95	.96
FakeVsSatire	.79	.46	.58	.70	.91	.79	.69	.83	.76	.86	.74	.79
Fakes	.37	.48	.42	.52	.40	.45	.49	.27	.35	.46	.68	.55

Appendix

Table S3 : Precision, recall and F1-score obtained by the BERT and DeBERTa models on each dataset for each class.

Dataset	Llama3-8B (ZS)						Qwen3-32B (ZS)						Zephyr-7B-beta (ZS)
	Class 0			Class 1			Class 0			Class 1			Class 0			Class 1
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Celebrity	.71	.88	.79	.84	.64	.73	.78	.86	.82	.84	.76	.80	.63	.82	.71	.76	.50	.60
Cidii	.87	.95	.91	.92	.80	.86	.96	.93	.95	.90	.95	.93	.80	1.00	.89	1.00	.65	.79
FakeVsSatire	.00	.00	.00	.58	.98	.73	.00	.00	.00	.58	1.00	.74	.07	.02	.04	.52	.75	.62
Fakes	.53	1.00	.69	.00	.00	.00	.50	.86	.63	.25	.05	.09	.52	.98	.68	.00	.00	.00

Appendix

Table S4 : Precision, recall and F1-score obtained by the Llama3-8B, Qwen3-32B and Zephyr-7B-beta models on each dataset for each class in a zero-shot (ZS) scenario.

Dataset	Llama3-8B (FS)						Qwen3-32B (FS)						Zephyr-7B-beta (FS)
	Class 0			Class 1			Class 0			Class 1			Class 0			Class 1
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Celebrity	.84	.62	.71	.70	.88	.78	.76	.76	.76	.76	.76	.76	.71	.72	.71	.73	.70	.71
Cidii	.94	1.00	.97	1.00	.92	.96	.98	.95	.96	.94	.97	.95	.89	.96	.93	.94	.83	.88
FakeVsSatire	.00	.00	.00	.58	.93	.72	.50	.02	.05	.58	.98	.73	.00	.00	.00	.57	.91	.70
Fakes	.51	.88	.65	.33	.07	.11	.50	.66	.57	.40	.25	.31	.53	1.00	.69	.00	.00	.00

Appendix

Table S5 : Precision, recall and F1-score obtained by the Llama3-8B, Qwen3-32B and Zephyr-7B-beta models on each dataset for each class in a few-shot (FS) scenario.

Figure S1: Heatmap of the p-values computed by the post-hoc Nemenyi Test (Pairwise Model Comparisons).

Panel of Figure 47 — Figure S2: Cross-dataset F1-scores for traditional ML approaches. As a reference, we have reported in the diagonal of the heatmap the highest F1-scores obtained by the models in the Dataset-Specific experiments.

Panel of Figure 48 — Figure S3: Cross-dataset F1-scores for DL approaches trained from scratch. As a reference, we have reported in the diagonal of the heatmap the highest F1-scores obtained by the models in the Dataset-Specific experiments.

Panel of Figure 49 — Figure S4: Cross-dataset F1-scores for Transformer-based models. As a reference, we have reported in the diagonal of the heatmap the highest F1-scores obtained by the models in the Dataset-Specific Experiments.

Panel of Figure 50 — Figure S5: F1-scores obtained in a zero-shot setting for each dataset by LLaMa3-8B (a), Qwen3-32B (b) and Zephyr-7B-beta (c).

Panel of Figure 51 — Figure S6: F1-scores obtained in a few-shot setting for each dataset by LLaMa3-8B (a), Qwen3-32B (b) and Zephyr-7B-beta (c).

Figure S7: Heatmap of the p-values computed by the post-hoc Nemenyi Test (Pairwise Model Comparisons).

Stratum	Friedman Stat	p-value	Kendall’s $W$
Generic	371.83	1.96e-71	0.530
Narrow	249.79	7.08e-46	0.534
Articles	380.00	3.72e-73	0.541
Others	233.63	1.58e-42	0.499

Appendix

Table S6: Stratified Statistical Analysis (Friedman Test and Kendall’s W Effect Size) across different dataset categories.

Figure S8: Heatmap of the p-values computed by the post-hoc Nemenyi Test (Pairwise Model Comparisons) on the Generic datasets (topic analysis).

Figure S9: Heatmap of the p-values computed by the post-hoc Nemenyi Test (Pairwise Model Comparisons) on the Narrow datasets (topic analysis).

Figure S10: Heatmap of the p-values computed by the post-hoc Nemenyi Test (Pairwise Model Comparisons) on the Articles datasets (genre analysis).

Figure S11: Heatmap of the p-values computed by the post-hoc Nemenyi Test (Pairwise Model Comparisons) on the Others datasets (genre analysis).

Dataset	LR						SVM						NB
	Class 0			Class 1			Class 0			Class 1			Class 0			Class 1
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Celebrity	.57	.61	.59	.58	.54	.56	.68	.59	.63	.64	.73	.68	.52	.74	.61	.56	.34	.42
Cidii	.87	.87	.87	.79	.80	.80	.88	.77	.82	.69	.83	.75	.77	.84	.80	.70	.6	.64
FakeVsSatire	.59	.68	.63	.80	.73	.77	.61	.61	.61	.78	.78	.78	.46	.81	.59	.82	.47	.60
Fakes	.53	.50	.51	.45	.48	.46	.56	.55	.56	.48	.49	.49	.53	.59	.56	.45	.39	.42

Appendix

Table S7 : Precision, recall and F1-score obtained by the ML models on each dataset for each class.

Dataset	CNN						BiLSTM						CNN-BERT
	Class 0			Class 1			Class 0			Class 1			Class 0			Class 1
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Celebrity	.52	.77	.62	.57	.30	.40	.49	.82	.61	.47	.16	.23	.52	.69	.59	.54	.36	.43
Cidii	.64	.85	.73	.51	.25	.33	.65	.87	.75	.57	.26	.36	.62	.61	.62	.39	.40	.39
FakeVsSatire	.37	.87	.52	.73	.19	.31	.34	.77	.47	.56	.16	.25	.36	.65	.46	.65	.36	.46
Fakes	.54	.89	.67	.50	.12	.20	.55	.66	.60	.48	.36	.41	.55	.69	.61	.49	.34	.40

Appendix

Table S8 : Precision, recall and F1-score obtained by the CNN, BiLSTM and CNN-BERT models on each dataset for each class.

Dataset	BERT						DeBERTa						MERMAID
	Class 0			Class 1			Class 0			Class 1			Class 0			Class 1
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Celebrity	.85	.66	.74	.72	.88	.79	.83	.83	.83	.83	.83	.83	.59	.84	.69	.72	.41	.52
Cidii	.98	.84	.90	.79	.97	.87	.98	.92	.95	.89	.97	.93	.15	.22	.18	.38	.26	.31
FakeVsSatire	.50	.80	.61	.83	.54	.66	.68	.75	.71	.85	.80	.82	.60	.65	.63	.85	.82	.83
Fakes	.54	.73	.62	.47	.28	.35	.62	.17	.26	.47	.88	.62	.52	.51	.52	.43	.44	.43

Appendix

Table S9 : Precision, recall and F1-score obtained by the BERT, DeBERTa and MERMAID models on each dataset for each class.

Dataset	Llama3-8B (ZS)						Qwen3-32B (ZS)						Zephyr-7B-beta (ZS)
	Class 0			Class 1			Class 0			Class 1			Class 0			Class 1
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Celebrity	.84	.91	.87	.90	.82	.86	.86	.83	.84	.84	.86	.85	.68	.87	.76	.86	.58	.70
Cidii	.92	.94	.93	.90	.84	.87	.95	.92	.94	.87	.92	.90	.81	1.00	.89	1.00	.59	.75
FakeVsSatire	.00	.00	.00	.66	.99	.79	.00	.00	.00	.66	1.00	.80	.08	.04	.05	.61	.76	.68
Fakes	.54	.99	.70	.00	.00	.00	.54	.90	.67	.40	.08	.14	.54	.99	.70	.50	.01	.02

Appendix

Table S10 : Precision, recall and F1-score obtained by the Llama3-8B, Qwen3-32B and Zephyr-7B-beta models on each dataset for each class in a zero-shot (ZS) scenario.

Dataset	Llama3-8B (FS)						Qwen3-32B (FS)						Zephyr-7B-beta (FS)
	Class 0			Class 1			Class 0			Class 1			Class 0			Class 1
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Celebrity	.89	.82	.86	.84	.90	.87	.79	.85	.82	.84	.78	.81	.66	.87	.75	.83	.54	.65
Cidii	.93	.93	.93	.88	.87	.87	.98	.80	.88	.75	.98	.85	.84	.95	.89	.90	.70	.79
FakeVsSatire	.00	.00	.00	.66	.99	.69	.50	.01	.02	.66	.99	.80	.07	.04	.05	.61	.74	.67
Fakes	.54	.99	.70	.33	.00	.01	.53	.93	.68	.28	.03	.06	.54	.99	.70	.50	.01	.02

Appendix

Table S11 : Precision, recall and F1-score obtained by the Llama3-8B, Qwen3-32B and Zephyr-7B-beta models on each dataset for each class in a few-shot (FS) scenario.

Figure S12: Heatmap of the p-values computed by the post-hoc Nemenyi Test (Pairwise Model Comparisons).

Dataset	LR						SVM						NB
	Class 0			Class 1			Class 0			Class 1			Class 0			Class 1
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Celebrity	.58	.27	.37	.52	.81	.64	.54	.12	.19	.50	.90	.65	.60	.01	.02	.50	.99	.67
Cidii	.65	.46	.54	.46	.64	.54	.68	.32	.43	.45	.79	.57	.75	.05	.09	.42	.98	.59
FakeVsSatire	.45	.51	.48	.61	.54	.57	.45	.38	.41	.60	.66	.63	.42	.18	.25	.58	.82	.68
Fakes	.54	.62	.57	.48	.40	.43	.54	.65	.59	.50	.38	.43	.53	.99	.69	.50	.01	.03

Appendix

Table S12 : Precision, recall and F1-score obtained by the ML models on each dataset for each class.

Dataset	CNN						BiLSTM						CNN-BERT
	Class 0			Class 1			Class 0			Class 1			Class 0			Class 1
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Celebrity	.58	.38	.46	.54	.72	.62	.72	.20	.31	.53	.92	.68	.64	.21	.32	.53	.88	.66
Cidii	.79	.18	.29	.45	.93	.60	.78	.08	.14	.43	.97	.59	.61	.17	.27	.42	.84	.56
FakeVsSatire	.59	.06	.12	.59	.97	.73	.47	.17	.25	.59	.86	.70	.52	.30	.38	.61	.80	.69
Fakes	.51	.61	.56	.44	.34	.38	.49	.29	.36	.45	.67	.54	.54	.58	.56	.48	.45	.46

Appendix

Table S13 : Precision, recall and F1-score obtained by the CNN, BiLSTM and CNN-BERT models on each dataset for each class.

Dataset	BERT						DeBERTa						MERMAID
	Class 0			Class 1			Class 0			Class 1			Class 0			Class 1
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Celebrity	.59	.49	.53	.56	.65	.60	.72	.48	.58	.61	.82	.70	.71	.63	.67	.67	.74	.70
Cidii	.68	.16	.26	.43	.90	.58	.80	.05	.09	.42	.98	.59	.73	.17	.28	.42	.91	.58
FakeVsSatire	.62	.29	.39	.63	.88	.73	.58	.13	.21	.60	.93	.73	.60	.17	.26	.63	.93	.75
Fakes	.53	.32	.40	.47	.67	.55	.55	.37	.44	.48	.67	.56	.54	.91	.68	.53	.12	.19

Appendix

Table S14 : Precision, recall and F1-score obtained by the BERT, DeBERTa and MERMAID models on each dataset for each class.

Dataset	Llama3-8B (ZS)						Qwen3-32B (ZS)						Zephyr-7B-beta (ZS)
	Class 0			Class 1			Class 0			Class 1			Class 0			Class 1
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Celebrity	.82	.90	.86	.91	.81	.85	.83	.81	.82	.81	.84	.83	.67	.88	.76	.87	.57	.69
Cidii	.89	.95	.92	.93	.82	.87	.93	.93	.93	.91	.91	.91	.79	1.00	.88	1.00	.62	.76
FakeVsSatire	.38	.01	.03	.58	.98	.73	.25	.01	.02	.58	.98	.73	.13	.05	.08	.52	.73	.61
Fakes	.53	.99	.69	.00	.00	.00	.53	.90	.67	.45	.09	.15	.53	.98	.69	.36	.01	.02

Appendix

Table S15 : Precision, recall and F1-score obtained by the Llama3-8B, Qwen3-32B and Zephyr-7B-beta models on each dataset for each class in a zero-shot (ZS) scenario.

Dataset	Llama3-8B (FS)						Qwen3-32B (FS)						Zephyr-7B-beta (FS)
	Class 0			Class 1			Class 0			Class 1			Class 0			Class 1
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Celebrity	.86	.84	.85	.85	.84	.85	.80	.83	.82	.83	.80	.81	.74	.83	.78	.83	.71	.76
Cidii	.88	.96	.92	.93	.81	.87	.97	.77	.86	.75	.96	.84	.89	.60	.72	.61	.89	.73
FakeVsSatire	.29	.02	.05	.58	.96	.72	.50	.02	.05	.58	.98	.73	.11	.03	.05	.54	.79	.64
Fakes	.53	.95	.68	.41	.04	.07	.53	.82	.64	.47	.17	.25	.52	.77	.62	.45	.21	.28

Appendix

Table S16 : Precision, recall and F1-score obtained by the Llama3-8B, Qwen3-32B and Zephyr-7B-beta models on each dataset for each class in a few-shot (FS) scenario.

Figure S13: Heatmap of the p-values computed by the post-hoc Nemenyi Test (Pairwise Model Comparisons).

Fake News Detection

Fake news detection identifies false or misleading claims by analyzing content, context, evidence, and how information spreads.

See all 98 papers

CoALFake: Collaborative Active Learning with Human-LLM Co-Annotation for Cross-Domain Fake News Detection
Apr 5, 2026Esma Aïmeur, Gilles Brassard, Dorsaf SallamiFake News Detection Active Learning
Department of Computer Science and Operations Research, University of Montreal, Canada.
Retrieval-Augmented Multimodal Model for Fake News Detection
Apr 20, 2026Yiheng Li, Weihai Lu, Hanyi Yu +1Fake News Detection Text Classification
University of International Business and Economics · Beijing, China · Peking University +4
Many Ways to Be Fake: Benchmarking Fake News Detection Under Strategy-Driven AI Generation
Apr 10, 2026Xinyu Wang, Sai Koneru, Wenbo Zhang +3Fake News Detection Synthetic Benchmark Generation
Pennsylvania State University University Park, PA 16802, USA

Apr 5, 2026J/K move · Enter open · S save

CoALFake: Collaborative Active Learning with Human-LLM Co-Annotation for Cross-Domain Fake News Detection

Esma Aïmeur, Gilles Brassard, Dorsaf Sallami

Department of Computer Science and Operations Research, University of Montreal, Canada.

Fake News Detection Active Learning

Open paper AskPDF arXiv

The proliferation of fake news across diverse domains highlights critical limitations in current detection systems, which often exhibit narrow domain specificity and poor generalization. Existing cross-domain approaches face two key challenges: (1) reliance on labelled data, which is frequently unavailable and resource intensive to acquire and (2) information loss caused by rigid domain categorization or neglect of domain-specific features. To address these issues, we propose CoALFake, a novel approach for cross-domain fake news detection that integrates Human-Large Language Model (LLM) co-annotation with domain-aware Active Learning (AL). Our method employs LLMs for scalable, low-cost annotation while maintaining human oversight to ensure label reliability. By integrating domain embedding techniques, the CoALFake dynamically captures both domain specific nuances and cross-domain patterns, enabling the training of a domain agnostic model. Furthermore, a domain-aware sampling strategy optimizes sample acquisition by prioritizing diverse domain coverage. Experimental results across multiple datasets demonstrate that the proposed approach consistently outperforms various baselines. Our results emphasize that human-LLM co-annotation is a highly cost-effective approach that delivers excellent performance. Evaluations across several datasets show that CoALFake consistently outperforms a range of existing baselines, even with minimal human oversight.

Abstract

Fake News Detection

Explore similar work

CoALFake: Collaborative Active Learning with Human-LLM Co-Annotation for Cross-Domain Fake News Detection

Retrieval-Augmented Multimodal Model for Fake News Detection

Many Ways to Be Fake: Benchmarking Fake News Detection Under Strategy-Driven AI Generation