An Experimental Comparison of the Most Popular Approaches to Fake News Detection
Organizations: Dipartimento di Ingegneria dell’Informazione, University of Pisa, Largo Lucio Lazzarino, 1, Pisa, Italy · Dipartimento di Informatica, University of Pisa, Largo B. Pontecorvo, 3, Pisa, Italy
Abstract
In recent years, fake news detection has received increasing attention in public debate and scientific research. Despite advances in detection techniques, the production and spread of false information have become more sophisticated, driven by Large Language Models (LLMs) and the amplification power of social media. We present a critical assessment of 12 representative fake news detection approaches, spanning traditional machine learning, deep learning, transformers, and specialized cross-domain architectures. We evaluate these methods on 10 publicly available datasets differing in genre, source, topic, and labeling rationale. We address text-only English fake news detection as a binary classification task by harmonizing labels into "Real" and "Fake" to ensure a consistent evaluation protocol. We acknowledge that label semantics vary across datasets and that harmonization inevitably removes such semantic nuances. Each dataset is treated as a distinct domain. We conduct in-domain, multi-domain and cross-domain experiments to simulate real-world scenarios involving domain shift and out-of-distribution data. Fine-tuned models perform well in-domain but struggle to generalize. Cross-domain architectures can reduce this gap but are data-hungry, while LLMs offer a promising alternative through zero- and few-shot learning. Given inherent dataset confounds and possible pre-training exposure, results should be interpreted as robustness evaluations within this English, text-only protocol.
Figures & tables
| Reference | ML | DL | LLMs | Datasets | Cross Domain |
| Ahmed et al., 2018 [ 4 ] | 6 | x | x | 2 | x |
| Agarwal et al., 2019 [ 2 ] | 5 | x | x | 1 | x |
| Dutta et al., 2019 [ 20 ] | 2 | x | x | 1 | x |
| Janicka et al., 2019 [ 31 ] | 5 | x | x | 4 | ✓ |
| Katsaros et al., 2019 [ 34 ] | 6 | 2 | x | 3 | x |
| Jiang et al., 2021 [ 32 ] | 5 | 3 | x | 2 | x |
| Dataset | Topic Scope | Source Types | Ann. Criteria | Time Span | Mapping | Fake/Real |
| Celebrity [ 47 ] | Vips | Articles | Fact-checked fake/real gossip | 2016–2017 | Fake = fake; Real = real | 250 / 250 |
| Cidii [ 25 ] | Islamic Issues | Articles; Social Media | Manual annotation of correct information vs. disinformation | 2016–2021 | Fake = fake; Real = real | 300 / 422 |
| FakeVsSatire [ 22 ] | Politics | Articles | Manual annotation of fake news vs. satire | 2016–2017 | Fake = fake; Real = satire | 283 / 203 |
| Fakes [ 51 ] | War in Syria | Articles | Semi-automated fact-checking of fake/real news | 2011–2018 | Fake = fake; Real = real | 378 / 426 |
| Horne [ 27 ] | Politics | Articles | Manual annotation of real, fake, and satirical news | 2015–2016 | Fake = fake; Real = real or satire | 123 / 203 |
| Infodemic [ 46 ] | Covid-19 | Social Media | Fact-checked COVID-19 misinformation | 2020 | Fake = fake; Real = real | 5031 / 5526 |
| Dataset | Fake class | Real class |
| Celebrity [ 47 ] | 150 | 150 |
| Cidii [ 25 ] | 200 | 322 |
| FakeVsSatire [ 22 ] | 183 | 103 |
| Fakes [ 51 ] | 278 | 326 |
| Horne [ 27 ] | 23 | 103 |
| Infodemic [ 46 ] | 4933 | 5426 |
| Approach | Type | Features/Method |
| LR [ 32 ] | Traditional ML | Tf-Idf (Bow) |
| SVM [ 50 ] | Traditional ML | Tf-Idf (Bow) |
| NB [ 50 ] | Traditional ML | Tf-Idf (Bow) |
| CNN [ 61 ] | DL (Training from scratch) | Word2Vec (Embeddings) |
| BiLSTM [ 16 ] | DL (Training from scratch) | Word2Vec (Embeddings) |
| CNN-BERT [ 33 ] | DL (Training from scratch) | BERT (Embeddings) |
| Model | Celebrity | Cidii | FakeVsSatire | Fakes | Horne | Infodemic | Isot | LIAR-PLUS | NDF | Politifact |
| LR | .60 | .97 | .70 | .43 | .76 | .93 | .99 | .56 | .86 | .77 |
| SVM | .67 | .86 | .69 | .44 | .71 | .93 | .99 | .57 | .82 | .75 |
| NB | .47 | .86 | .68 | .42 | .68 | .88 | .96 | .56 | .79 | .71 |
| CNN | .54 | .58 | .56 | .53 | .56 | .55 | .96 | .57 | .56 | .64 |
| BiLSTM | .50 | .59 | .51 | .56 | .62 | .70 | .97 | .57 | .63 | .65 |
| CNN-BERT | .50 | .59 | .58 | .53 | .70 | .68 | .99 | .58 | .59 | .64 |
| Model | Mean F1 | Standard Error (SE) |
| LR | -0.075 | 0.022 |
| SVM | -0.089 | 0.020 |
| NB | -0.131 | 0.030 |
| CNN | -0.227 | 0.049 |
| BiLSTM | -0.202 | 0.045 |
| CNN-BERT | -0.194 | 0.043 |
| Model | Test | Celebrity | Cidii | FakeVsSatire | Fakes | Horne | Infodemic | Isot | LIAR | NDF | Politifact |
| LR | Generic | .55 | .45 | .45 | .51 | .54 | .50 | .49 | .55 | .50 | .52 |
| Narrow | .53 | .47 | .50 | .53 | .49 | .47 | .48 | .52 | .47 | .55 | |
| SVM | Generic | .55 | .48 | .52 | .49 | .55 | .48 | .50 | .54 | .50 | .56 |
| Narrow | .51 | .47 | .49 | .50 | .49 | .50 | .49 | .50 | .46 | .51 | |
| NB | Generic | .53 | .47 | .48 | .52 | .52 | .48 | .52 | .55 | .48 | .51 |
| Narrow | .53 | .50 | .50 | .51 | .50 | .47 | .50 | .51 | .47 | .52 |
| Model | Test | Celebrity | Cidii | FakeVsSatire | Fakes | Horne | Infodemic | Isot | LIAR | NDF | Politifact |
| LR | Articles | .56 | .56 | .46 | .51 | .53 | .50 | .51 | .54 | .49 | .52 |
| Others | .54 | .51 | .49 | .55 | .56 | .44 | .46 | .56 | .51 | .57 | |
| SVM | Articles | .56 | .53 | .52 | .49 | .55 | .49 | .51 | .54 | .50 | .55 |
| Others | .55 | .51 | .49 | .49 | .55 | .45 | .45 | .55 | .47 | .54 | |
| NB | Articles | .53 | .53 | .48 | .52 | .52 | .48 | .53 | .55 | .48 | .51 |
| Others | .56 | .49 | .50 | .54 | .56 | .44 | .48 | .47 | .50 | .57 |
| Model | Celebrity | Cidii | FakeVsSatire | Fakes | Horne | Infodemic | Isot | LIAR-PLUS | NDF | Politifact | Global |
| LR | .58 | .85 | .72 | .49 | .71 | .76 | .68 | .53 | .82 | .71 | .61 |
| SVM | .66 | .80 | .72 | .53 | .64 | .77 | .74 | .55 | .81 | .75 | .60 |
| NB | .54 | .75 | .60 | .51 | .76 | .66 | .55 | .53 | .69 | .72 | .58 |
| CNN | .54 | .62 | .44 | .54 | .66 | .51 | .53 | .55 | .62 | .63 | .49 |
| BiLSTM | .49 | .64 | .38 | .53 | .68 | .54 | .50 | .55 | .58 | .65 | .52 |
| CNN-BERT | .53 | .53 | .36 | .54 | .68 | .55 | .59 | .49 | .54 | .64 | .52 |
| Celebrity | Cidii | FakeVsSatire | Fakes | Horne | Infodemic | Isot | LIAR-PLUS | NDF | Politifact | Global | |
| Gate Entropy | 0.003 | 0.053 | 0.002 | 0.007 | 0.057 | 0.175 | 0.007 | 0.780 | 0.175 | 0.034 | 0.078 |
| CV | 2.797 | 2.013 | 2.998 | 2.565 | 2.737 | 2.625 | 2.943 | 1.638 | 2.185 | 2.321 | 2.420 |
| Model | Celebrity | Cidii | Fakes | Infodemic | Isot | LIAR-PLUS | Global |
| Deberta (train 100/100) | .84 | .98 | .48 | .88 | .99 | .57 | .64 |
| Mermaid (train 100/100) | .47 | .52 | .48 | .84 | .92 | .56 | .64 |
| Deberta (train 150/150) | .79 | .98 | .47 | .87 | .99 | .59 | .64 |
| Mermaid (train 150/150) | .57 | .78 | .49 | .80 | .92 | .54 | .59 |
| Deberta (train 200/200) | .84 | .98 | .40 | .91 | .99 | .60 | .66 |
| Mermaid (train 200/200) | .68 | .82 | .51 | .82 | .85 | .57 | .54 |
| Model | Celebrity | Cidii | FakeVsSatire | Fakes | Horne | Infodemic | Isot | LIAR-PLUS | NDF | Politifact |
| LR | .57 | .46 | .63 | .48 | .74 | .52 | .73 | .45 | .42 | .61 |
| SVM | .51 | .51 | .55 | .53 | .57 | .52 | .55 | .46 | .52 | .47 |
| NB | .50 | .43 | .55 | .53 | .53 | .55 | .61 | .53 | .39 | .52 |
| CNN | .50 | .51 | .55 | .53 | .57 | .52 | .55 | .46 | .52 | .47 |
| BiLSTM | .51 | .51 | .55 | .53 | .57 | .52 | .55 | .46 | .52 | .47 |
| CNN-BERT | .55 | .45 | .59 | .51 | .64 | .44 | .49 | .47 | .38 | .57 |
| Model | Med (E1) | IQR (E1) | Med (E2) | IQR (E2) | Med (E3) | IQR (E3) | Med (E4) | IQR (E4) |
| LR | 0.765 | 0.2875 | 0.520 | 0.0775 | 0.71 | 0.155 | 0.545 | 0.16 |
| SVM | 0.73 | 0.175 | 0.510 | 0.0700 | 0.72 | 0.145 | 0.52 | 0.035 |
| NB | 0.695 | 0.2525 | 0.515 | 0.0900 | 0.60 | 0.1675 | 0.53 | 0.04 |
| CNN | 0.56 | 0.025 | 0.500 | 0.0750 | 0.54 | 0.10 | 0.52 | 0.0425 |
| BiLSTM | 0.605 | 0.0825 | 0.505 | 0.0800 | 0.54 | 0.115 | 0.52 | 0.035 |
| CNN-BERT | 0.59 | 0.09 | 0.500 | 0.0800 | 0.54 | 0.055 | 0.50 | 0.11 |
| Model | Celebrity | Cidii | FakeVsSatire | Fakes | Horne | Infodemic | Isot | LIAR-PLUS | NDF | Politifact | Global |
| LLaMa3-8B (FS - Similar) | .83 | .96 | .64 | .52 | .52 | .83 | .71 | .57 | .89 | .60 | .58 |
| LLaMa3-8B (FS - Random) | .86 | .91 | .66 | .54 | .64 | .83 | .72 | .59 | .75 | .72 | .58 |
| Qwen3-32B (FS - Similar) | .83 | .97 | .65 | .53 | .64 | .88 | .78 | .61 | .89 | .82 | .65 |
| Qwen3-32B (FS - Random) | .82 | .87 | .66 | .52 | .64 | .83 | .74 | .59 | .86 | .78 | .66 |
| Zephyr-7B-beta (FS - Similar) | .78 | .96 | .61 | .51 | .50 | .85 | .75 | .62 | .92 | .71 | .71 |
| Zephyr-7B-beta (FS - Random) | .71 | .86 | .54 | .51 | .57 | .79 | .59 | .63 | .81 | .64 | .55 |
Appendix figures & tables47 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Source | # Training | # Validation | # Test | Date |
| Celebrity [ 47 ] | https://www.kaggle.com/datasets/sumanthvrao/fakenewsdataset | 300 | 100 | 100 | 24/10/2023 |
| Cidii [ 25 ] | https://www.kaggle.com/datasets/suhaibkhalil/cidii-dataset | 433 | 144 | 145 | 14/11/2023 |
| FakeVsSatire [ 22 ] | https://github.com/jgolbeck/fakenews | 291 | 97 | 98 | 24/10/2023 |
| Fakes [ 51 ] | https://zenodo.org/records/2607278 | 482 | 161 | 161 | 24/10/2023 |
| Horne [ 27 ] | https://github.com/rpitrust/fakenewsdata1 | 195 | 65 | 66 | 27/10/2023 |
| Infodemic [ 46 ] | https://constraint-shared-task-2021.github.io/ | 6299 | 2139 | 2119 | 03/03/2024 |
| Model | Parameter | Tested Range |
| LR | Penalty | L2; None |
| C | 0.01; 0.1; 1.0; 10.0; 100.0 | |
| Solver | lbfgs; liblinear; sag; saga | |
| SVM | Penalty | L1; L2 |
| C | 0.01; 0.1; 1.0; 10.0; 100.0 | |
| Loss | Hinge; Squared Hinge |
| Model | Parameters | Celebrity | Cidii | FakeVsSatire | Fakes | Horne | Infodemic | Isot | LIAR | NDF | Politifact |
| LR | Penalty | l2 | none | none | l2 | none | l2 | none | none | none | none |
| C | 10.0 | 0.01 | 0.01 | 100.0 | 0.01 | 100.0 | 100.0 | 1.0 | 0.01 | 0.01 | |
| Solver | lbfgs | lbfgs | lbfgs | lbfgs | lbfgs | lbfgs | lbfgs | lbfgs | lbfgs | lbfgs | |
| SVM | Penalty | l1 | l1 | l1 | l1 | l1 | l1 | l1 | l1 | l1 | l1 |
| C | 1.0 | 100.0 | 10.0 | 100.0 | 100.0 | 1.0 | 1.0 | 0.1 | 100.0 | 10.0 | |
| Loss | sh | sh | sh | sh | sh | sh | sh | sh | sh | sh |
| Model | Celebrity | Cidii | FakeVsSatire | Fakes | Horne | Infodemic | Isot | LIAR-PLUS | NDF | Politifact |
| CNN | ||||||||||
| BiLSTM | ||||||||||
| CNN-BERT | ||||||||||
| BERT | ||||||||||
| DeBERTa |
| Dataset | Model | F1 across batch sizes (bs8 → bs256) |
| BERT | 0.75, 0.69, 0.67, 0.64, 0.61, 0.67 | |
| DeBERTa | 0.80, 0.77, 0.75, 0.63, 0.72, 0.52 | |
| Celebrity | CNN | 0.54, 0.50, 0.50, 0.50, 0.50, 0.50 |
| BiLSTM | 0.50, 0.50, 0.50, 0.50, 0.50, 0.50 | |
| CNN-BERT | 0.53, 0.52, 0.53, 0.53, 0.50, 0.50 | |
| BERT | 0.52, 0.58, 0.47, 0.55, 0.52, 0.53 |
| Model | Celebrity | Cidii | FakeVsSatire | Fakes | Horne | Infodemic | Isot | LIAR-PLUS | NDF | Politifact | Global |
| LR | .54 | .65 | .61 | .51 | .60 | .82 | .83 | .53 | .71 | .57 | .76 |
| SVM | .48 | .77 | .65 | .53 | .58 | .84 | .95 | .54 | .67 | .56 | .83 |
| NB | .33 | .77 | .46 | .34 | .55 | .85 | .95 | .55 | .65 | .56 | .85 |
| CNN | .58 | .79 | .52 | .48 | .60 | .47 | .95 | .57 | .63 | .62 | .87 |
| BiLSTM | .50 | .73 | .56 | .49 | .57 | .41 | .95 | .58 | .71 | .60 | .86 |
| CNN-BERT | .52 | .71 | .58 | .46 | .60 | .85 | .96 | .57 | .61 | .57 | .86 |
| Model | Celebrity | Cidii | Fakes | Infodemic | Isot | LIAR-PLUS | NDF | Politifact |
| LR | .52 (-.05) | .51 (.05) | .52 (.04) | .53 (.01) | .52 (-.21) | .50 (.05) | .57 (.15) | .51 (-.10) |
| SVM | .51 (.00) | .52 (.01) | .51 (-.02) | .53 (.01) | .53 (-.02) | .46 (.00) | .49 (-.03) | .46 (-.01) |
| NB | .50 (.00) | .43 (.00) | .53 (.00) | .57 (.02) | .60 (-.01) | .53 (.00) | .39 (.00) | .53 (.01) |
| CNN | .58 (.08) | .55 (.04) | .49 (-.04) | .49 (-.03) | .55 (.00) | .47 (.01) | .39 (.00) | .53 (.00) |
| BiLSTM | .52 (.01) | .43 (-.08) | .47 (-.06) | .52 (.00) | .57 (.02) | .45 (-.01) | .52 (.00) | .57 (.10) |
| CNN-BERT | .55 (.00) | .52 (.07) | .50 (-.01) | .50 (.06) | .49 (.00) | .47 (.00) | .44 (.06) | .58 (.01) |
| Model | Modality | Prompt |
| Llama3-8B | Zero-shot | P-Z3 |
| Qwen3-32B | Zero-shot | P-Z3 |
| Zephyr-7B-beta | Zero-shot | P-Z1 |
| Llama3-8B | Few-shot | P-Z3 |
| Qwen3-32B | Few-shot | P-Z3 |
| Zephyr-7B-beta | Few-shot | P-Z2 |
| Model & Setting | Prompt | Decoding Settings | Few-shot Sampling | Selection seed |
| Llama3-8B (Zero-shot) | P-Z3 | Temp = 0; Top-k = 1; Max new tokens = 64 | N/A | N/A |
| Llama3-8B (Few-shot) | P-Z3 | Temp = 0; Top-k = 1; Max new tokens = 64 | 6 examples (3 per class) * | 42 |
| Qwen3-32B (Zero-shot) | P-Z3 | Temp = 0; Top-k = 1; Max new tokens = 64 † | N/A | N/A |
| Qwen3-32B (Few-shot) | P-Z3 | Temp = 0; Top-k = 1; Max new tokens = 64 † | 6 examples (3 per class) * | 42 |
| Zephyr-7B (Zero-shot) | P-Z1 | Temp = 0; Top-k = 1; Max new tokens = 64 | N/A | N/A |
| Zephyr-7B (Few-shot) | P-Z2 | Temp = 0; Top-k = 1; Max new tokens = 64 | 6 examples (3 per class) * | 42 |
| Model | Celebrity | Cidii | FakeVsSatire | Fakes | Horne | Infodemic | Isot | LIAR-PLUS | NDF | Politifact |
| Llama3-8B (FS) | ||||||||||
| Qwen3-32B (FS) | ||||||||||
| Zephyr-7B-beta (FS) |
| Model | Celebrity | Cidii | FakeVsSatire | Fakes | Horne | Infodemic | Isot | LIAR-PLUS | NDF | Politifact |
| LLaMA3-8B (FS) | ||||||||||
| Median | .84 | .87 | .57 | .52 | .72 | .82 | .67 | .62 | .81 | .74 |
| IQR | .02 | .03 | .01 | .01 | .01 | .04 | .03 | .00 | .01 | .02 |
| Qwen3-32B (FS) | ||||||||||
| Median | .80 | .92 | .58 | .53 | .73 | .83 | .71 | .63 | .84 | .80 |
| IQR | .02 | .16 | .00 | .01 | .01 | .00 | .02 | .01 | .02 | .02 |
| Text (Truncated) | True Label | Predicted Label | Confidence |
| BREAKING: Bridge Collapse Claims Hillary Clinton, 14 Others… | Real | Fake | 0.978 |
| BREAKING: President Trump Releases Official Boycott List… | Real | Fake | 0.966 |
| Hillary Clinton: Women Only Voted For Trump Because Husbands… | Fake | Real | 0.508 |
| Procter & Gamble Release One Of The Most Racist Commercials… | Fake | Real | 0.573 |
| Text (Truncated) | True Label | Predicted Label | Confidence |
| Daesh executed at least 128 Syrian civilians in Al-Qaryatayn… | Fake | Real | 0.786 |
| US Secretary of Defense Carter said anti-Daesh coalition killed key ministers… | Fake | Real | 0.774 |
| Several Syrians Killed Injured in Homs Suicide Blasts. | Fake | Real | 0.501 |
| 86 dead in suspected Syria chemical attack. | Real | Fake | 0.516 |
| Text (Truncated) | True Label | Predicted Label | Confidence |
| EXCLUSIVE PICS: First photos of Ariana Grande since Manchester attack… | Fake | Real | 0.994 |
| Jennifer Lopez gets cozy with on-screen boyfriend Milo Ventimiglia… | Fake | Real | 0.985 |
| White Witch warns Angelina against using spells to woo back Brad… | Fake | Real | 0.521 |
| Is Evian Water The Secret Behind Cameron Diaz’s Great Skin? | Fake | Real | 0.544 |
| Text (Truncated) | True Label | Predicted Label | Confidence |
| Three stories to read this morning: New lockdown… | Real | Fake | 0.999 |
| WHO @UNDP @UNAIDS launch COVID-19 Law Lab… | Real | Fake | 0.999 |
| I thought Corona affects only humans. | Fake | Real | 0.558 |
| I’ve decided to opt out of the KCL symptoms tracker… #frustrated | Fake | Real | 0.617 |
| Text (Truncated) | True Label | Predicted | Confidence |
| The costs of cancer and affiliated issues are over $20 billion a year… | Real | Fake | 0.995 |
| Affordable Care Act is a major reason why weve seen 50,000 fewer deaths… | Real | Fake | 0.994 |
| The platform of the Republican Party says deport everybody… | Fake | Real | 0.512 |
| Republican leaders are pushing to make privatizing Social Security… | Fake | Real | 0.513 |
| Text (Truncated) | True Label | Predicted Label | Confidence |
| Notre Dame Blaze - a spiritual perspective… | Fake | Real | 0.998 |
| I’ve spent weekends walking around Paris then stopping in Notre Dame… | Real | Fake | 0.996 |
| So horrible to watch the massive fire at Notre Dame Cathedral… | Real | Fake | 0.994 |
| Dataset | LR | SVM | NB | CNN | BiLSTM | BERT | DeBERTa | CNN-BERT |
| Celebrity (training) | 7.63 | 5.91 | 5.53 | 192.02 | 196.97 | 189.07 | 237.05 | 50.01 |
| Celebrity (entire) | 7.11 | 8.51 | 5.46 | 194.41 | 197.93 | 210.51 | 290.84 | 42.62 |
| Celebrity (inference) | 2.70e-6 | 1.20e-6 | 2.00e-6 | 2.98e-3 | 7.00e-3 | 3.13e-3 | 5.16e-3 | 6.47e-3 |
| Cidii (training) | 6.04 | 5.50 | 6.30 | 212.69 | 241.61 | 247.42 | 406.38 | 98.67 |
| Cidii (entire) | 5.81 | 5.59 | 6.44 | 220.64 | 227.22 | 372.50 | 532.82 | 120.75 |
| Cidii (inference) | 8.00e-7 | 7.00e-7 | 9.00e-7 | 1.07e-3 | 7.17e-3 | 2.60e-3 | 4.18e-3 | 5.03e-3 |
| Dataset | LR | SVM | NB | |||||||||||||||
| Class 0 | Class 1 | Class 0 | Class 1 | Class 0 | Class 1 | |||||||||||||
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | |
| Celebrity | .59 | .62 | .60 | .60 | .58 | .59 | .66 | .70 | .67 | .68 | .64 | .65 | .48 | .76 | .58 | .42 | .18 | .25 |
| Cidii | 1.0 | .93 | .96 | .95 | 1.0 | .97 | .82 | .85 | .83 | .89 | .87 | .88 | .88 | .78 | .83 | .85 | .92 | .89 |
| FakeVsSatire | .87 | .34 | .49 | .67 | .96 | .79 | .68 | .48 | .57 | .69 | .84 | .76 | .81 | .31 | .45 | .65 | .94 | .77 |
| Fakes | .45 | .47 | .46 | .39 | .38 | .38 | .47 | .41 | .44 | .42 | .48 | .45 | .46 | .50 | .48 | .38 | .34 | .36 |
| Dataset | CNN | BiLSTM | CNN-BERT | |||||||||||||||
| Class 0 | Class 1 | Class 0 | Class 1 | Class 0 | Class 1 | |||||||||||||
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | |
| Celebrity | .52 | .88 | .65 | .62 | .20 | .30 | .50 | .16 | .24 | .50 | .84 | .62 | .50 | .96 | .65 | .50 | .04 | .07 |
| Cidii | .44 | .06 | .11 | .58 | .94 | .72 | .50 | .05 | .09 | .58 | .96 | .73 | .00 | .00 | .00 | .58 | 1.0 | .73 |
| FakeVsSatire | .00 | .00 | .00 | .57 | .96 | .71 | .39 | .31 | .35 | .56 | .64 | .60 | .00 | .00 | .00 | .58 | 1.0 | .73 |
| Fakes | .52 | 1.0 | .69 | .00 | .00 | .00 | .55 | .77 | .65 | .55 | .31 | .40 | .52 | 1.0 | .69 | .00 | .00 | .00 |
| Dataset | BERT | DeBERTa | ||||||||||
| Class 0 | Class 1 | Class 0 | Class 1 | |||||||||
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | |
| Celebrity | .75 | .78 | .76 | .77 | .74 | .75 | .80 | .86 | .83 | .85 | .78 | .81 |
| Cidii | 1.0 | .95 | .97 | .96 | 1.0 | .98 | .97 | .98 | .97 | .97 | .95 | .96 |
| FakeVsSatire | .79 | .46 | .58 | .70 | .91 | .79 | .69 | .83 | .76 | .86 | .74 | .79 |
| Fakes | .37 | .48 | .42 | .52 | .40 | .45 | .49 | .27 | .35 | .46 | .68 | .55 |
| Dataset | Llama3-8B (ZS) | Qwen3-32B (ZS) | Zephyr-7B-beta (ZS) | |||||||||||||||
| Class 0 | Class 1 | Class 0 | Class 1 | Class 0 | Class 1 | |||||||||||||
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | |
| Celebrity | .71 | .88 | .79 | .84 | .64 | .73 | .78 | .86 | .82 | .84 | .76 | .80 | .63 | .82 | .71 | .76 | .50 | .60 |
| Cidii | .87 | .95 | .91 | .92 | .80 | .86 | .96 | .93 | .95 | .90 | .95 | .93 | .80 | 1.00 | .89 | 1.00 | .65 | .79 |
| FakeVsSatire | .00 | .00 | .00 | .58 | .98 | .73 | .00 | .00 | .00 | .58 | 1.00 | .74 | .07 | .02 | .04 | .52 | .75 | .62 |
| Fakes | .53 | 1.00 | .69 | .00 | .00 | .00 | .50 | .86 | .63 | .25 | .05 | .09 | .52 | .98 | .68 | .00 | .00 | .00 |
| Dataset | Llama3-8B (FS) | Qwen3-32B (FS) | Zephyr-7B-beta (FS) | |||||||||||||||
| Class 0 | Class 1 | Class 0 | Class 1 | Class 0 | Class 1 | |||||||||||||
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | |
| Celebrity | .84 | .62 | .71 | .70 | .88 | .78 | .76 | .76 | .76 | .76 | .76 | .76 | .71 | .72 | .71 | .73 | .70 | .71 |
| Cidii | .94 | 1.00 | .97 | 1.00 | .92 | .96 | .98 | .95 | .96 | .94 | .97 | .95 | .89 | .96 | .93 | .94 | .83 | .88 |
| FakeVsSatire | .00 | .00 | .00 | .58 | .93 | .72 | .50 | .02 | .05 | .58 | .98 | .73 | .00 | .00 | .00 | .57 | .91 | .70 |
| Fakes | .51 | .88 | .65 | .33 | .07 | .11 | .50 | .66 | .57 | .40 | .25 | .31 | .53 | 1.00 | .69 | .00 | .00 | .00 |
| Stratum | Friedman Stat | p-value | Kendall’s |
| Generic | 371.83 | 1.96e-71 | 0.530 |
| Narrow | 249.79 | 7.08e-46 | 0.534 |
| Articles | 380.00 | 3.72e-73 | 0.541 |
| Others | 233.63 | 1.58e-42 | 0.499 |
| Dataset | LR | SVM | NB | |||||||||||||||
| Class 0 | Class 1 | Class 0 | Class 1 | Class 0 | Class 1 | |||||||||||||
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | |
| Celebrity | .57 | .61 | .59 | .58 | .54 | .56 | .68 | .59 | .63 | .64 | .73 | .68 | .52 | .74 | .61 | .56 | .34 | .42 |
| Cidii | .87 | .87 | .87 | .79 | .80 | .80 | .88 | .77 | .82 | .69 | .83 | .75 | .77 | .84 | .80 | .70 | .6 | .64 |
| FakeVsSatire | .59 | .68 | .63 | .80 | .73 | .77 | .61 | .61 | .61 | .78 | .78 | .78 | .46 | .81 | .59 | .82 | .47 | .60 |
| Fakes | .53 | .50 | .51 | .45 | .48 | .46 | .56 | .55 | .56 | .48 | .49 | .49 | .53 | .59 | .56 | .45 | .39 | .42 |
| Dataset | CNN | BiLSTM | CNN-BERT | |||||||||||||||
| Class 0 | Class 1 | Class 0 | Class 1 | Class 0 | Class 1 | |||||||||||||
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | |
| Celebrity | .52 | .77 | .62 | .57 | .30 | .40 | .49 | .82 | .61 | .47 | .16 | .23 | .52 | .69 | .59 | .54 | .36 | .43 |
| Cidii | .64 | .85 | .73 | .51 | .25 | .33 | .65 | .87 | .75 | .57 | .26 | .36 | .62 | .61 | .62 | .39 | .40 | .39 |
| FakeVsSatire | .37 | .87 | .52 | .73 | .19 | .31 | .34 | .77 | .47 | .56 | .16 | .25 | .36 | .65 | .46 | .65 | .36 | .46 |
| Fakes | .54 | .89 | .67 | .50 | .12 | .20 | .55 | .66 | .60 | .48 | .36 | .41 | .55 | .69 | .61 | .49 | .34 | .40 |
| Dataset | BERT | DeBERTa | MERMAID | |||||||||||||||
| Class 0 | Class 1 | Class 0 | Class 1 | Class 0 | Class 1 | |||||||||||||
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | |
| Celebrity | .85 | .66 | .74 | .72 | .88 | .79 | .83 | .83 | .83 | .83 | .83 | .83 | .59 | .84 | .69 | .72 | .41 | .52 |
| Cidii | .98 | .84 | .90 | .79 | .97 | .87 | .98 | .92 | .95 | .89 | .97 | .93 | .15 | .22 | .18 | .38 | .26 | .31 |
| FakeVsSatire | .50 | .80 | .61 | .83 | .54 | .66 | .68 | .75 | .71 | .85 | .80 | .82 | .60 | .65 | .63 | .85 | .82 | .83 |
| Fakes | .54 | .73 | .62 | .47 | .28 | .35 | .62 | .17 | .26 | .47 | .88 | .62 | .52 | .51 | .52 | .43 | .44 | .43 |
| Dataset | Llama3-8B (ZS) | Qwen3-32B (ZS) | Zephyr-7B-beta (ZS) | |||||||||||||||
| Class 0 | Class 1 | Class 0 | Class 1 | Class 0 | Class 1 | |||||||||||||
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | |
| Celebrity | .84 | .91 | .87 | .90 | .82 | .86 | .86 | .83 | .84 | .84 | .86 | .85 | .68 | .87 | .76 | .86 | .58 | .70 |
| Cidii | .92 | .94 | .93 | .90 | .84 | .87 | .95 | .92 | .94 | .87 | .92 | .90 | .81 | 1.00 | .89 | 1.00 | .59 | .75 |
| FakeVsSatire | .00 | .00 | .00 | .66 | .99 | .79 | .00 | .00 | .00 | .66 | 1.00 | .80 | .08 | .04 | .05 | .61 | .76 | .68 |
| Fakes | .54 | .99 | .70 | .00 | .00 | .00 | .54 | .90 | .67 | .40 | .08 | .14 | .54 | .99 | .70 | .50 | .01 | .02 |
| Dataset | Llama3-8B (FS) | Qwen3-32B (FS) | Zephyr-7B-beta (FS) | |||||||||||||||
| Class 0 | Class 1 | Class 0 | Class 1 | Class 0 | Class 1 | |||||||||||||
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | |
| Celebrity | .89 | .82 | .86 | .84 | .90 | .87 | .79 | .85 | .82 | .84 | .78 | .81 | .66 | .87 | .75 | .83 | .54 | .65 |
| Cidii | .93 | .93 | .93 | .88 | .87 | .87 | .98 | .80 | .88 | .75 | .98 | .85 | .84 | .95 | .89 | .90 | .70 | .79 |
| FakeVsSatire | .00 | .00 | .00 | .66 | .99 | .69 | .50 | .01 | .02 | .66 | .99 | .80 | .07 | .04 | .05 | .61 | .74 | .67 |
| Fakes | .54 | .99 | .70 | .33 | .00 | .01 | .53 | .93 | .68 | .28 | .03 | .06 | .54 | .99 | .70 | .50 | .01 | .02 |
| Dataset | LR | SVM | NB | |||||||||||||||
| Class 0 | Class 1 | Class 0 | Class 1 | Class 0 | Class 1 | |||||||||||||
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | |
| Celebrity | .58 | .27 | .37 | .52 | .81 | .64 | .54 | .12 | .19 | .50 | .90 | .65 | .60 | .01 | .02 | .50 | .99 | .67 |
| Cidii | .65 | .46 | .54 | .46 | .64 | .54 | .68 | .32 | .43 | .45 | .79 | .57 | .75 | .05 | .09 | .42 | .98 | .59 |
| FakeVsSatire | .45 | .51 | .48 | .61 | .54 | .57 | .45 | .38 | .41 | .60 | .66 | .63 | .42 | .18 | .25 | .58 | .82 | .68 |
| Fakes | .54 | .62 | .57 | .48 | .40 | .43 | .54 | .65 | .59 | .50 | .38 | .43 | .53 | .99 | .69 | .50 | .01 | .03 |
| Dataset | CNN | BiLSTM | CNN-BERT | |||||||||||||||
| Class 0 | Class 1 | Class 0 | Class 1 | Class 0 | Class 1 | |||||||||||||
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | |
| Celebrity | .58 | .38 | .46 | .54 | .72 | .62 | .72 | .20 | .31 | .53 | .92 | .68 | .64 | .21 | .32 | .53 | .88 | .66 |
| Cidii | .79 | .18 | .29 | .45 | .93 | .60 | .78 | .08 | .14 | .43 | .97 | .59 | .61 | .17 | .27 | .42 | .84 | .56 |
| FakeVsSatire | .59 | .06 | .12 | .59 | .97 | .73 | .47 | .17 | .25 | .59 | .86 | .70 | .52 | .30 | .38 | .61 | .80 | .69 |
| Fakes | .51 | .61 | .56 | .44 | .34 | .38 | .49 | .29 | .36 | .45 | .67 | .54 | .54 | .58 | .56 | .48 | .45 | .46 |
| Dataset | BERT | DeBERTa | MERMAID | |||||||||||||||
| Class 0 | Class 1 | Class 0 | Class 1 | Class 0 | Class 1 | |||||||||||||
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | |
| Celebrity | .59 | .49 | .53 | .56 | .65 | .60 | .72 | .48 | .58 | .61 | .82 | .70 | .71 | .63 | .67 | .67 | .74 | .70 |
| Cidii | .68 | .16 | .26 | .43 | .90 | .58 | .80 | .05 | .09 | .42 | .98 | .59 | .73 | .17 | .28 | .42 | .91 | .58 |
| FakeVsSatire | .62 | .29 | .39 | .63 | .88 | .73 | .58 | .13 | .21 | .60 | .93 | .73 | .60 | .17 | .26 | .63 | .93 | .75 |
| Fakes | .53 | .32 | .40 | .47 | .67 | .55 | .55 | .37 | .44 | .48 | .67 | .56 | .54 | .91 | .68 | .53 | .12 | .19 |
| Dataset | Llama3-8B (ZS) | Qwen3-32B (ZS) | Zephyr-7B-beta (ZS) | |||||||||||||||
| Class 0 | Class 1 | Class 0 | Class 1 | Class 0 | Class 1 | |||||||||||||
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | |
| Celebrity | .82 | .90 | .86 | .91 | .81 | .85 | .83 | .81 | .82 | .81 | .84 | .83 | .67 | .88 | .76 | .87 | .57 | .69 |
| Cidii | .89 | .95 | .92 | .93 | .82 | .87 | .93 | .93 | .93 | .91 | .91 | .91 | .79 | 1.00 | .88 | 1.00 | .62 | .76 |
| FakeVsSatire | .38 | .01 | .03 | .58 | .98 | .73 | .25 | .01 | .02 | .58 | .98 | .73 | .13 | .05 | .08 | .52 | .73 | .61 |
| Fakes | .53 | .99 | .69 | .00 | .00 | .00 | .53 | .90 | .67 | .45 | .09 | .15 | .53 | .98 | .69 | .36 | .01 | .02 |
| Dataset | Llama3-8B (FS) | Qwen3-32B (FS) | Zephyr-7B-beta (FS) | |||||||||||||||
| Class 0 | Class 1 | Class 0 | Class 1 | Class 0 | Class 1 | |||||||||||||
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | |
| Celebrity | .86 | .84 | .85 | .85 | .84 | .85 | .80 | .83 | .82 | .83 | .80 | .81 | .74 | .83 | .78 | .83 | .71 | .76 |
| Cidii | .88 | .96 | .92 | .93 | .81 | .87 | .97 | .77 | .86 | .75 | .96 | .84 | .89 | .60 | .72 | .61 | .89 | .73 |
| FakeVsSatire | .29 | .02 | .05 | .58 | .96 | .72 | .50 | .02 | .05 | .58 | .98 | .73 | .11 | .03 | .05 | .54 | .79 | .64 |
| Fakes | .53 | .95 | .68 | .41 | .04 | .07 | .53 | .82 | .64 | .47 | .17 | .25 | .52 | .77 | .62 | .45 | .21 | .28 |
Fake News Detection
Fake news detection identifies false or misleading claims by analyzing content, context, evidence, and how information spreads.
See all 98 papers