cs.CLApr 10, 2026

Human vs. Machine Deception: Distinguishing AI-Generated and Human-Written Fake News Using Ensemble Learning

Authors: Samuel Jaeger, Calvin Ibenye, Aya Vera-Jimenez, Dhrubajyoti Ghosh

Organizations: School of Data Science and Analytics, Kennesaw State University, Kennesaw, GA · Department of Computer Science, Kennesaw State University, Kennesaw, GA · Department of Mathematics, Kennesaw State University, Kennesaw, GA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The rapid adoption of large language models has introduced a new class of AI-generated fake news that coexists with traditional human-written misinformation, raising important questions about how these two forms of deceptive content differ and how reliably they can be distinguished. This study examines linguistic, structural, and emotional differences between human-written and AI-generated fake news and evaluates machine learning and ensemble-based methods for distinguishing these content types. A document-level feature representation is constructed using sentence structure, lexical diversity, punctuation patterns, readability indices, and emotion-based features capturing affective dimensions such as fear, anger, joy, sadness, trust, and anticipation. Multiple classification models, including logistic regression, random forest, support vector machines, extreme gradient boosting, and a neural network, are applied alongside an ensemble framework that aggregates predictions across models. Model performance is assessed using accuracy and area under the receiver operating characteristic curve. The results show strong and consistent classification performance, with readability-based features emerging as the most informative predictors and AI-generated text exhibiting more uniform stylistic patterns. Ensemble learning provides modest but consistent improvements over individual models. These findings indicate that stylistic and structural properties of text provide a robust basis for distinguishing AI-generated misinformation from human-written fake news.

Figures & tables

SourceExcerpt
Human-written“Reuters reported that Nancy Pelosi bought 10 million shares of cannabis stock. On Oct. 6, President Joe Biden said he would pardon all people convicted of simple marijuana possession offenses under federal law and directed Cabinet members to review how the drug is classified. Later that day, a claim started to spread on social media that House Speaker Nancy Pelosi had bought millions of shares of cannabis stock before Biden’s announcement.”
AI-generated“Following President Joe Biden’s announcement on Oct. 6 that he would pardon all federal convictions for simple marijuana possession and direct a review of the drug’s federal classification, a viral claim surfaced on social media asserting that House Speaker Nancy Pelosi had purchased millions of shares of cannabis stock. An Oct. 7 Instagram post stated, ‘Breaking: Nancy Pelosi purchased 10,000,000 shares of $WEED 4 days ago,’ attributing the claim to Reuters.”
Table 1: Illustrative example of a paired human-written and AI-generated fake-news article. Short excerpts are shown for readability.
ModelParameter(s) tunedCandidate valuesMost frequently selected setting
Logistic regressionNone--
Random forestmtrymtry2, 3, 7mtry=7mtry=7 (9/10 outer splits)
SVM (radial)σ\sigma , CCσ\sigma : 0.01, 0.05, 0.10; CC : 0.5, 1, 2, 4σ=0.01\sigma=0.01 , C=4C=4 (9/10)
XGBoostrounds, depth, learning raterounds: 100, 200; depth: 2, 4; η\eta : 0.05, 0.10No single dominant setting
Neural networkhidden units, weight decayunits: 5, 10, 15; decay: 0, 0.001, 0.0115 units, decay 0.01 (9/10)
Table 2: Hyperparameter tuning used in the grouped five-fold cross-validation robustness analysis.
Figure 1: Mean classification accuracy and area under the receiver operating characteristic curve (AUC) across 100 repeated 80:20 article-pair-level train-test splits. Error bars represent one standard deviation across repetitions.
ModelAccuracyAUCPrecisionRecallSpecificityF1
Ensemble0.955 (0.014)0.993 (0.003)0.954 (0.020)0.956 (0.022)0.954 (0.021)0.955 (0.014)
Logistic0.953 (0.014)0.991 (0.005)0.951 (0.020)0.956 (0.022)0.951 (0.021)0.953 (0.014)
NN0.934 (0.027)0.953 (0.030)0.933 (0.032)0.935 (0.041)0.932 (0.035)0.934 (0.029)
RF0.947 (0.016)0.990 (0.004)0.957 (0.018)0.937 (0.025)0.957 (0.019)0.947 (0.016)
SVM0.948 (0.015)0.991 (0.004)0.944 (0.018)0.953 (0.025)0.943 (0.020)0.948 (0.015)
XGBoost0.953 (0.014)0.991 (0.004)0.959 (0.018)0.947 (0.022)0.959 (0.019)0.953 (0.014)
Table 3: Performance across 100 repeated 80:20 article-pair-level train-test splits. Values are mean (SD).
Figure 2: Top features contributing to classification for the random forest and XGBoost models. Random forest importance is based on mean decrease in Gini impurity, while XGBoost importance is based on gain. Importance values are scaled within each model so that the largest value equals 1. Readability-related features dominate both rankings, with the Coleman-Liau index contributing most strongly.
Figure 3: Discriminative ability of individual features measured by area under the receiver operating characteristic curve (AUC). Readability-related features show the strongest standalone discrimination, whereas most emotion-based features perform close to chance level.
Figure 4: Distribution of the Coleman-Liau index (expected cloze percentage form) for human-written and AI-generated fake news. Higher values indicate text estimated to be easier to read. Human-written articles tend to have higher values, whereas AI-generated articles are concentrated at lower values.

Explore similar work

CardsList

  1. Cross-Prompt Generalization in Detecting AI-Generated Fake News Using Interpretable Linguistic Features

    Jun 2, 2026Aya Vera-Jimenez, Samuel Jaeger, Calvin Ibenye +1Fake News DetectionFake News

    Department of Mathematics Kennesaw State University Marietta, Georgia, USA · School of Data Science and Analytics Kennesaw State University Marietta, Georgia, USA · Department of Computer Science Kennesaw State University Marietta, Georgia, USA

  2. Findings of the Counter Turing Test: AI-Generated Text Detection

    May 20, 2026Rajarshi Roy, Gurpreet Singh, Ashhar Aziz +16Machine-Generated Text DetectionAttacker Large Language Model

    Kalyani Government Engineering College, India. · IIIT Guwahati, India. · IIIT Delhi, India. +10

  3. A Systematic Analysis of Linguistic Features in AI-Generated Text Detection Across Domains and Models

    Jun 2, 2026Yassir El Attar, Esra Dönmez, Maximilian Maurer +1LinguisticsRobust Cross-Domain Generalization

    Institute for Natural Language Processing, University of Stuttgart · Interchange Forum for Reflecting on Intelligent Systems, University of Stuttgart · GESIS Leibniz Institute for the Social Sciences +1