cs.CLSep 15, 2023

Fake News Detectors are Biased against Texts Generated by Large Language Models

Authors: Jinyan Su, Terry Yue Zhuo, Jonibek Mansurov, Di Wang, Preslav Nakov

Organizations: Mohamed bin Zayed University of Artificial Intelligence · Monash University and CSIRO’s Data61 · King Abdullah University of Science and Technology

Abstract

The spread of fake news has emerged as a critical challenge, undermining trust and posing threats to society. In the era of Large Language Models (LLMs), the capability to generate believable fake content has intensified these concerns. In this study, we present a novel paradigm to evaluate fake news detectors in scenarios involving both human-written and LLM-generated misinformation. Intriguingly, our findings reveal a significant bias in many existing detectors: they are more prone to flagging LLM-generated content as fake news while often misclassifying human-written fake news as genuine. This unexpected bias appears to arise from distinct linguistic patterns inherent to LLM outputs. To address this, we introduce a mitigation strategy that leverages adversarial training with LLM-paraphrased genuine news. The resulting model yielded marked improvements in detection accuracy for both human and LLM-generated news. To further catalyze research in this domain, we release two comprehensive datasets, GossipCop++ and PolitiFact++, thus amalgamating human-validated articles with LLM-generated fake and real news.

Figures & tables

Explore similar work

CardsList
  1. Unequal Verdicts: Investigating Gender Bias in LLM-Based Fake News Detection

    Aug 4, 2026Razieh Chalehchaleh, Reza Farahbakhsh, Noel CrespiGender Bias in Language ModelsFake News Detection

  2. Many Ways to Be Fake: Benchmarking Fake News Detection Under Strategy-Driven AI Generation

    Apr 10, 2026Xinyu Wang, Sai Koneru, Wenbo Zhang +3Fake News DetectionSynthetic Benchmark Generation