Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News

Alexander Loth; Martin Kappes; Marc-Oliver Pahl

cs.CYApr 4, 2026

Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News

Authors: Alexander Loth, Martin Kappes, Marc-Oliver Pahl

Organizations: IMT Atlantique, UMR IRISA, Chaire Cyber CNI Rennes, France

Abstract

Can humans tell whether a news article was written by a person or a large language model (LLM)? We investigate this question using JudgeGPT, a study platform that independently measures source attribution (human vs. machine) and authenticity judgment (legitimate vs. fake) on continuous scales. From 2,318 judgments collected from 1,054 participants across content generated by six LLMs, we report five findings: (1) participants cannot reliably distinguish machine-generated from human-written text (p > .05, Welch's t-test); (2) this inability holds across all tested models, including open-weight models with as few as 7B parameters; (3) self-reported domain expertise predicts judgment accuracy (r = .35, p < .001) whereas political orientation does not (r = -.10, n.s.); (4) clustering reveals distinct response strategies ("Skeptics" vs. "Believers"); and (5) accuracy degrades after approximately 30 sequential evaluations due to cognitive fatigue. The answer, in short, is no: humans cannot reliably tell. These results indicate that user-side detection is not a viable defense and motivate system-level countermeasures such as cryptographic content provenance.

Figures & tables

Figure 1. The JudgeGPT interface. Participants rate each fragment on three continuous axes: source attribution, authenticity, and topic familiarity.

Finding	Statistic	Implication
F1: Undetectable	$t(2316)=0.87$ , $p=.38$	Intuition fails as defense
F2: Model-agnostic	All 6 LLMs at $\bar{x}\approx 0.50$	Low barrier to deceptive text
F3: Expertise $>$ Politics	$r=.35$ vs. $r=-.10$	Literacy-based interventions
F4: User personas	$k=2$ clusters (sil. $=.41$ )	Adaptive designs needed
F5: Fatigue	Decline after ${\sim}$ 30 items	Sustained vigilance infeasible

Table 1. Summary of empirical findings (

N=2{,}318

judgments from

1{,}054

participants; median 12 items per participant, range 5–87).

Figure 2. Source score (left) and authenticity score (right) distributions for machine- vs. human-written fragments. Overlapping distributions and a non-significant $t$ -test indicate participants cannot distinguish the two conditions.

Figure 3. Mean source and authenticity scores per LLM ( $\pm$ SE). The dashed line marks chance level (0.5). No model is reliably identified as machine-generated.

Figure 4. Relationships between participant covariates and judgment scores. Top: political view (weak slope). Bottom: fake news familiarity (positive slope). Regression lines with 95% CI shown.

Figure 5. Pairplot of participant-level mean scores colored by cluster. The two groups show clearly separated distributions on both axes.

Figure 6. Rolling mean of judgment scores across sequential evaluations. Scores improve initially before declining after ${\sim}$ 30 items, consistent with a learning-then-fatigue pattern.

AI-Generated Text Detection

AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.

See all 170 papers

Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge
Apr 7, 2026Xin Sun, Di Wu, Sijing Qin +3Bias in LLM-as-a-Judge LLM-as-a-Judge
National Institute of Informatics (NII), Japan · University of Amsterdam, the Netherlands · Hitotsubashi University, Japan +3
BAIT: Can LLMs Find What Biases a Reader? A Benchmark for Cognitive Bias Triggers in News
Oct 20, 2026Yousra Fettach, Tijl De BieCognitive Bias in Language Models
Ghent University, Belgium
DeGenTWeb: A First Look at MGT-dominant Websites
Apr 30, 2026Sichang Steven He, Calvin Ardi, Ramesh Govindan +1AI-Generated Text Detection
University of Southern California · Unaliated

Apr 7, 2026J/K move · Enter open · S save

Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge

Xin Sun, Di Wu, Sijing Qin, Isao Echizen+2

National Institute of Informatics (NII), Japan · University of Amsterdam, the Netherlands · Hitotsubashi University, Japan+3

Bias in LLM-as-a-Judge LLM-as-a-Judge Language Model Bias Evaluation Trust in AI

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used as automated evaluators (LLM-as-a-Judge). This work challenges its reliability by showing that trust judgments by LLMs are biased by disclosed source labels. Using a counterfactual design, we find that both humans and LLM judges assign higher trust to information labeled as human-authored than to the same content labeled as AI-generated. Eye-tracking data reveal that humans rely heavily on source labels as heuristic cues for judgments. We analyze LLM internal states during judgment. Across label conditions, models allocate denser attention to the label region than the content region, and this label dominance is stronger under Human labels than AI labels, consistent with the human gaze patterns. Besides, decision uncertainty measured by logits is higher under AI labels than Human labels. These results indicate that the source label is a salient heuristic cue for both humans and LLMs. It raises validity concerns for label-sensitive LLM-as-a-Judge evaluation, and we cautiously raise that aligning models with human preferences may propagate human heuristic reliance into models, motivating debiased evaluation and alignment.

Abstract

AI-Generated Text Detection

Explore similar work

Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge

BAIT: Can LLMs Find What Biases a Reader? A Benchmark for Cognitive Bias Triggers in News

DeGenTWeb: A First Look at MGT-dominant Websites