BAIT: Can LLMs Find What Biases a Reader? A Benchmark for Cognitive Bias Triggers in News

Yousra Fettach; Tijl De Bie

cs.CLOct 20, 2026

BAIT: Can LLMs Find What Biases a Reader? A Benchmark for Cognitive Bias Triggers in News

Authors: Yousra Fettach, Tijl De Bie

Organizations: Ghent University, Belgium

Abstract

Persuasive writing exploits cognitive biases, recruiting a reader's mental shortcuts to shape belief, often without the reader noticing. We call the linguistic features exploiting such mental shortcuts cognitive bias triggers. As large language models (LLMs) increasingly mediate how people encounter news, summarizing it, ranking it, and answering questions over it, a natural question is whether they can recognize the features of a text that bias a human reader. We introduce BAIT (Bias Annotation In Text), a resource for detecting cognitive bias triggers in news. BAIT pairs a schema of nine triggers, each specified in an annotator codebook, with 50 English news articles annotated by trained annotators at the article and span levels and adjudicated into a gold standard. On this data we define two tasks, detecting which triggers a text contains and locating the spans that carry them. We then benchmark six frontier LLMs zero-shot and with three or ten in-context examples. The models approach human agreement on which triggers an article contains, but rarely recover its full set of triggers, and they reach less than half of human agreement in locating the passages that carry them. Moreover, they over-predict triggers that come with easy surface cues, and most of the passages they mark for these triggers are not triggers, yet carry the trigger's surface cue far more often than random text. These findings suggest that the evaluated LLMs respond to surface cues rather than to the mechanisms that make a passage manipulative. We release the schema, the codebook, standoff annotations, and the evaluation pipeline.

Figures & tables

ID	Trigger	Ref.
Credibility
B01	Illusory Truth	( Hasher et al. 1977 )
B03	Bandwagon Effect	( Asch 1956 )
B04	Authority Bias	( Chaiken 1980 )
Affect
B02	Negativity Bias	( Baumeister et al. 2001 )

Table 1: The nine cognitive bias trigger categories in BAIT, grouped by the mechanism they recruit, with a foundational reference for each.

ID	Trigger	$\kappa$ (doc)	$F_{1}$ (span)
B01	Illusory Truth	0.55	0.62
B02	Negativity Bias	0.47	0.48
B03	Bandwagon Effect	0.23	0.46
B04	Authority Bias	0.56	0.46
B05	Stereotyping	0.53	0.46
B06	Declinism	0.56	0.66

Table 2: Per-category inter-annotator agreement before adjudication. Document-level Cohen’s

\kappa

on trigger presence and span-level word-token

F_{1}

, both over the 50 articles.

ID	Bias	Art.	Spans	%	Len.
B01	Illusory Truth	45	338	57.1	14.6
B02	Negativity Bias	23	86	14.5	10.3
B03	Bandwagon Effect	7	11	1.9	10.3
B04	Authority Bias	19	34	5.7	28.1
B05	Stereotyping	14	25	4.2	17.9
B06	Declinism	6	17	2.9	35.0

Table 3: BAIT label distribution over 50 articles. Art. : articles containing the bias (the total counts articles with at least one bias). Spans : annotated spans. % : share of all 592 spans. Len. : mean span length in words.

Figure 1: Article-level overlap between BAIT categories and their closest SemEval-2023 persuasion technique, on the 32 articles with at least one SemEval label: articles labeled with the BAIT category only, with both, and with the technique only. Declinism, the Identifiable Victim Effect and Hindsight Bias have no counterpart.

		Task 1: detection			Task 2: localization
Model	$k$	Micro-F1	Macro-F1	Subset Acc.	Char-F1	Span-F1	Trigger-F1
Claude Opus 4.8	0	0.543	0.478	0.040	0.147	0.121	0.102
	3	0.637	0.563	0.060	0.201	0.167	0.157
	10	0.679	0.590	0.080	0.234	0.204	0.179
GPT-5.6	0	0.590	0.491	0.033	0.209	0.165	0.150
	3	0.614	0.531	0.073	0.241	0.204	0.180

Table 4: Detection (Task 1) and localization (Task 2) for

k

in-context examples. Corpus-level scores over 50 articles. For

k>0

, the mean over three example draws (seeds 17, 23, 29); for

k=0

, the mean over six zero-shot runs. Bold: best model at each

k

. The last row scores one annotator’s labels against the other’s before adjudication, with the same metrics, as a human reference.

Figure 2: Task 1: increase over zero-shot when the model receives

k=3

k=10

annotated example articles, in percentage points. Bars show the mean per-article difference (few-shot averaged over three example draws, zero-shot over six runs), and error bars are paired 95% confidence intervals over the 50 articles. Left: micro-F1. Right: subset accuracy (the predicted set of biases equals the gold set).

Figure 3: Task 2: increase over zero-shot at three levels of strictness, in percentage points. Bars show the mean per-article difference (few-shot averaged over three example draws, zero-shot over six runs); error bars are 95% confidence intervals (paired

t

-interval over the 50 articles). Zero-shot levels are in Table 4 .

Figure 4: Per-bias scores with ten in-context examples (corpus-level $F_{1}$ in %, mean over three example draws). Left: Task 1 detection $F_{1}$ (scale 0–100). Right: Task 2 trigger-F1 (scale 0–40). Bold: best model on that bias for that task. Number of gold articles per bias in parentheses.

Figure 5: Where predicted spans land, and whether they carry the surface cue (

k=10

, all six models and three example draws, 11,027 located spans). (a) Each predicted span is classified as a match (IoU

\geq 0.5

with a gold span of the same bias), a boundary error (overlapping such a span with lower IoU), an overlap with a gold trigger of another bias , or unannotated text that no annotator marked for any bias. Numbers give the match share in %. (b) Share of spans containing the bias’s surface cue, for unannotated predictions, gold spans, and random stretches of article text matched to the predictions in length.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Bias	Articles	Triggers	Multi-span	Spans	Avg. words
B01 Illusory Truth	45	147	142	338	14.6
B02 Negativity Bias	23	86	0	86	10.3
B03 Bandwagon Effect	7	11	0	11	10.3
B04 Authority Bias	19	34	0	34	28.1
B05 Stereotyping	14	25	0	25	17.9
B06 Declinism	6	17	0	17	35.0

Appendix

Table 5: Gold standard by bias. Articles: articles containing the bias (the total counts articles with at least one bias). Multi-span: triggers made of two or more spans. Avg. words: mean span length in words.

Panel of Figure 6 — Figure 6: System prompt used in every setting, verbatim. The JSON format it contains is shown in Figure 8 .

Panel of Figure 7 — Figure 7: User message. The example block appears only in the few-shot settings.

Panel of Figure 8 — Figure 8: Output format as given in the system prompt (line breaks inside strings added for display). The spans of one trigger are listed together, which is how models express grouping for Task 2.

	$k=0$			$k=10$
Model	No JSON	No fields	Mismatch	No JSON	No fields	Mismatch
Claude Opus 4.8	0	0	1	0	0	12
GPT-5.6	3	12	2	3	16	8
Grok 4.5	0	1	0	0	0	3
DeepSeek-V4	2	0	1	0	0	2
Kimi K3	5	0	1	1	0	0

Appendix

Table 6: Unusable and inconsistent outputs out of 150 per cell (three runs of 50 articles). No JSON: no valid JSON object could be parsed. No fields: a JSON object was recovered from surrounding text but lacked biases_present and triggers . Both are scored as empty predictions. Mismatch: biases_present differs from the biases among the triggers. The

k=3

setting is omitted.

Model	Spans	Exact	Case	Whitesp.	Fuzzy	Unloc.
Claude Opus 4.8	2,590	99.19	0.31	0.00	0.46	0.04
GPT-5.6	3,015	99.70	0.17	0.00	0.10	0.03
Grok 4.5	4,288	99.00	0.23	0.05	0.65	0.07
DeepSeek-V4	3,037	96.61	0.76	0.36	1.91	0.36
Kimi K3	2,535	98.58	0.04	0.12	1.14	0.12
Mistral Medium 3.5	4,004	95.05	1.62	0.85	2.07	0.40

Appendix

Table 7: Share of predicted spans (%) resolved by each alignment step, pooled over the three zero-shot runs and the three

k=10

example draws.

		Task 1		Task 2
Model	$k$	Micro-F1	Subset Acc.	Char-F1	Span-F1	Trigger-F1
Claude Opus 4.8	0	0.004	0.000	0.019	0.013	0.017
	3	0.025	0.020	0.005	0.003	0.007
	10	0.022	0.020	0.011	0.007	0.010
GPT-5.6	0	0.006	0.012	0.015	0.003	0.016
	3	0.036	0.042	0.024	0.017	0.021

Appendix

Table 8: Standard deviation of corpus-level scores over three runs.

k=0

: three reruns of the same prompt.

k=3

and

k=10

: three example draws (seeds 17, 23, 29).

		Triggers	Spans	Spans per	Multi-span (%)
Model	$k$	per article	per trigger	B01 trigger	B01	Other
Gold standard	–	7.96	1.49	2.30	96.6	0.4
Claude Opus 4.8	0	3.47	2.02	3.34	100.0	58.5
Claude Opus 4.8	10	5.97	1.72	2.85	100.0	23.3
GPT-5.6	0	4.94	1.71	3.30	100.0	26.5
GPT-5.6	10	7.20	1.62	2.97	100.0	6.3

Appendix

Table 9: Grouping of predicted spans into triggers, averaged over the three runs per setting (150 outputs). Multi-span: share of triggers with two or more spans, for Illusory Truth (B01) and for the other eight biases. Unusable outputs (Appendix F ) count as articles without triggers.

Cognitive Bias in Language Models

See all 21 papers

Apr 1, 2026

CogBias: Measuring and Mitigating Cognitive Bias in Large Language Models

Large Language Models (LLMs) are increasingly deployed in high-stakes decision-making contexts. While prior work has shown that LLMs exhibit cognitive biases behaviorally, whether these biases correspond to identifiable internal representations and can be mitigated through targeted intervention remains an open question. We define LLM cognitive bias as systematic, reproducible deviations from correct answers in tasks with computable ground-truth baselines, and introduce LLM CogBias, a benchmark organized around four families of cognitive biases: Judgment, Information Processing, Social, and Response. We evaluate three LLMs and find that cognitive biases emerge systematically across all four families, with magnitudes and debiasing responses that are strongly family-dependent: prompt-level debiasing substantially reduces Response biases but backfires for Judgment biases. Using linear probes under a contrastive design, we show that these biases are encoded as linearly separable directions in model activation space. Finally, we apply activation steering to modulate biased behavior, achieving 26--32% reduction in bias score (fraction of biased responses) while preserving downstream capability on 25 benchmarks (Llama: negligible degradation; Qwen: up to

-

19.0pp for Judgment biases). Despite near-orthogonal bias representations across models (mean cosine similarity 0.01), steering reduces bias at similar rates across architectures (

r(246)

=.621,

p

<.001), suggesting shared functional organization.

Fan Huang, Songheng Zhang, Haewoon Kwak +1

Indiana University Bloomington · Singapore Management University

Cognitive Bias in Language Models Language Model Bias Evaluation

Ask PDF

Apr 4, 2026

Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News

Can humans tell whether a news article was written by a person or a large language model (LLM)? We investigate this question using JudgeGPT, a study platform that independently measures source attribution (human vs. machine) and authenticity judgment (legitimate vs. fake) on continuous scales. From 2,318 judgments collected from 1,054 participants across content generated by six LLMs, we report five findings: (1) participants cannot reliably distinguish machine-generated from human-written text (p > .05, Welch's t-test); (2) this inability holds across all tested models, including open-weight models with as few as 7B parameters; (3) self-reported domain expertise predicts judgment accuracy (r = .35, p < .001) whereas political orientation does not (r = -.10, n.s.); (4) clustering reveals distinct response strategies ("Skeptics" vs. "Believers"); and (5) accuracy degrades after approximately 30 sequential evaluations due to cognitive fatigue. The answer, in short, is no: humans cannot reliably tell. These results indicate that user-side detection is not a viable defense and motivate system-level countermeasures such as cryptographic content provenance.

Alexander Loth, Martin Kappes, Marc-Oliver Pahl

IMT Atlantique, UMR IRISA, Chaire Cyber CNI Rennes, France

AI-Generated Text Detection LLM Evaluation

Ask PDF

May 13, 2026

Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges

Large language models (LLMs) are increasingly used as automatic judges for summarization and dialogue evaluation. Prior work has documented biases such as position, verbosity, and style preferences, but largely focuses on outcomes, leaving judge explanations underexplored. We instead ask whether LLM judges are cue-invariant, i.e., whether their rankings and explanations remain stable when non-evidential cues are perturbed while holding the underlying texts fixed. We introduce a suite of cue interventions (Blind, Truth, Flip, Placebo, Reveal-After) and tie-aware metrics that quantify outcome anchoring and rationale anchoring, including label-aligned rhetoric and explanation drift, alongside consistency and stereotype-intrusion checks. We design anchoring attacks using verbosity and confidence cues, and compare two mitigations: structured chain-of-thought prompting and PROOF-BEFORE-PREFERENCE (evidence lock, score, rank). Using a new dataset of 1,000 summaries from traditional extractive models and LLMs, we find substantial cue-anchored rationalization under label and placebo perturbations, while PROOF-BEFORE-PREFERENCE markedly improves cue invariance over baselines.

Riya Tapwal, Abhishek Kumar, Carsten Maple

Indian Institute of Technology (IIT) Mandi · The Alan Turing Institute, London, U.K. · Warwick Manufacturing Group University of Warwick, U.K.

LLM-as-a-Judge Faithfulness of Language Model Explanations Language Model Bias Evaluation Bias in LLM-as-a-Judge Causal Interventions in Language Models

Ask PDF

Abstract

Cognitive Bias in Language Models

Explore similar work

CogBias: Measuring and Mitigating Cognitive Bias in Large Language Models

Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News

Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges