As foundation models (FMs) approach human-level fluency, distinguishing synthetic from organic content has become a key challenge for Trustworthy Web Intelligence. This paper presents JudgeGPT and RogueGPT, a dual-axis framework that decouples "authenticity" from "attribution" to investigate the mechanisms of human susceptibility. Analyzing 918 evaluations across five FMs (including GPT-4 and Llama-2), we employ Structural Causal Models (SCMs) as a principal framework for formulating testable causal hypotheses about detection accuracy. Contrary to partisan narratives, we find that political orientation shows a negligible association with detection performance (r=−0.10). Instead, "fake news familiarity" emerges as a candidate mediator (r=0.35), suggesting that exposure may function as adversarial training for human discriminators. We identify a "fluency trap" where GPT-4 outputs (HumanMachineScore: 0.20) bypass Source Monitoring mechanisms, rendering them indistinguishable from human text. These findings suggest that "pre-bunking" interventions should target cognitive source monitoring rather than demographic segmentation to ensure trustworthy information ecosystems.
Figures & tables
Tool
Human Eval
Dual-Axis
Multi-LLM
GROVER ( Zellers et al., 2019 )
✗
✗
✗
MIST ( Maertens et al., 2024 )
✓
✗
✗
FakeGPT ( Huang and Sun, 2024 )
✗
✗
✗
JudgeGPT (Ours)
✓
✓
✓
Table 1. JudgeGPT provides dual-axis human evaluation across multiple foundation models.
Figure 1. Correlations between participant characteristics and judgment scores. Fake news familiarity shows stronger predictive power than political orientation. Heatmap of correlation coefficients between participant characteristics (e.g., fake news familiarity and political orientation) and the two judgment metrics (source attribution and legitimacy).
Model
Avg. Score
openai_gpt-4
0.200
mistralai_mistral-7b
0.442
openai_gpt-4o
0.468
google_gemma-7b
0.499
meta_llama-2-13b
0.526
openai_gpt-35-turbo
0.548
Table 2. Average HumanMachineScore by foundation model (lower = more human-like).
Figure 2. Comparison of HumanMachineScore and LegitFakeScore for fragments by actual source (Human vs. AI). Boxplots show median and interquartile ranges. Independent t -tests found no significant mean differences between human-written and AI-generated groups for either metric, highlighting the challenge of distinguishing foundation model outputs. Side-by-side boxplots comparing the distributions of HumanMachineScore and LegitFakeScore for fragments by actual source (human-written vs. AI-generated).
Figure 3. Pair-plot visualization of participant judging behavior clusters. Each point represents a participant, colored by cluster. The plots show relationships between HumanMachineScore, LegitFakeScore, TimeToAnswer, PoliticalView, and FNewsExperience. Distinct clusters indicate different judgment behavior patterns that may be causally linked to participant characteristics. Pair-plot showing relationships among participant-level variables (HumanMachineScore, LegitFakeScore, TimeToAnswer, PoliticalView, and FNewsExperience), with points colored by inferred judging-behavior cluster.
Figure 4. Judgment Scores vs. Response Order. HumanMachineScore and LegitFakeScore are plotted over the sequence of trials for each participant (with smoothing). The HumanMachineScore tends to decrease initially (learning effect), while the LegitFakeScore drops later in the session (possible fatigue or strategy shift toward increased skepticism). Time-series plot of average judgment scores versus response order, illustrating an early decrease in HumanMachineScore (learning) and a later drop in LegitFakeScore (possible fatigue).
Figure S1 . Screenshot of the JudgeGPT Demographic Information Screen. This initial intake screen captures participant demographics including age, gender, education level, political orientation, and self-reported familiarity with fake news—variables central to our analysis (Section 4 ). A web interface screenshot titled Real or Fake: Can You Spot Misinformation?'' showing the JudgeGPT demographic questionnaire. The form includes: a language dropdown (set to English), an age slider ranging from 16 to 133 years (currently at 74), a gender dropdown, a political view slider from Far Left'' to Far Right,'' a native speaker question, highest education level dropdown, a newspaper subscriptions slider from None'' to Three or more,'' and a fake news experience slider from Completely unfamiliar'' to Completely familiar.'' At the bottom, an expandable section for Consent/About/Privacy Policy is shown, followed by a consent toggle (Yes, I'm in! I consent to participate''), a unique participant ID, and a ``Start Survey'' button.
Figure S2 . Screenshot of the JudgeGPT Survey Screen. Participants evaluate randomized news fragments using three continuous sliders: the HumanMachineScore (ranging from “Definitely Human Generated” to “Definitely Machine Generated”), the LegitFakeScore (ranging from “Definitely Legit News” to “Definitely Fake News”), and a topic familiarity rating. A flag option allows participants to report technically broken content. This multi-dimensional rating design enables independent measurement of source attribution accuracy and content credibility, as analyzed in Section 4 . A web interface screenshot showing the JudgeGPT evaluation screen, response number 12 of the survey. The main panel displays a news text fragment in quotation marks about a political topic. Below the text are three horizontal slider controls: the first labeled Human or Machine Generated?'' ranging from Definitely Human Generated'' to Definitely Machine Generated''; the second labeled Legit or Fake News?'' ranging from Definitely Legit News'' to Definitely Fake News''; and the third asking How familiar are you with the topic covered in this news?'' ranging from Not at all'' to Extremely well.'' A checkbox allows participants to flag technically incorrect or broken content. A progress message reads Only 4 more responses to see your results! Keep going!'' with emoji icons, followed by the participant ID and a ``Submit Response'' button.