cs.CLOct 25, 2026
SaveBeyond Belief: Evaluating and Mitigating LLM Susceptibility to Misinformation
Organizations: Oakland University · Dartmouth College
Abstract
Tool-using large language models (LLMs) must decide whether external information warrants action. We investigate whether their expressed credibility judgments reliably indicate behavioral resistance to misinformation. We evaluate five LLMs on 196 constructed false notices paired with legitimate counterparts and 204 archived attacks adapted to simulated tool use. When asked to act on the constructed notices, models issue harmful tool calls in 27%--99% of cases they have rated untrustworthy. Removing the preliminary rating step increases harmful execution by 15 percentage points on average, showing that credibility elicitation itself changes the behavior being evaluated. Interventions that improve credibility discrimination do not consistently yield safer actions. A simple forewarning significantly reduces harmful execution in four of five models, but neither tested generated-example prebunk outperforms it in the main evaluation. These reductions can also compromise legitimate task completion. Supervised fine-tuning on generated false notices reduces harmful execution on held-out archived attacks under the rating-first protocol, with model-dependent utility costs and weaker protection without rating elicitation. Our findings motivate evaluating misinformation defenses through both harmful actions and legitimate task completion, while accounting for the behavioral effects of the evaluation protocol.
Figures & tables
| Harmful execution | Acts when | Legitimate execution | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Victim | No rating | None | Length ctrl. | Forewarn. | Prebunk (rep.) | Prebunk (dir.) | Reminder | rated | None | Forewarn. |
| Llama-3.1-8B-Instruct | ||||||||||
| Gemma-2-9B-IT | ||||||||||
| Qwen3-14B | ||||||||||
| Gemma-2-27B-IT | ||||||||||
| Gemini-3.6-Flash | ||||||||||
Table 1: Harmful tool-execution rates across false notices under different interventions. No rating omits credibility assessment; None elicits a rating without intervention; Length ctrl. adds unrelated text; Prebunk (rep./dir.) uses report- or directive-style generated exemplars; and Reminder repeats the victim’s prior rating. Acts when rated measures harmful execution despite expressed distrust under no intervention, while Legitimate execution measures correct tool use on matched genuine requests. Bold indicates the lowest harmful rate per victim. One sample is evaluated per item and condition.
| LLaMA | Gemma | GPT-4o-mini | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Topic | M.I. | G.I. | Disc. | M.I. | G.I. | Disc. | M.I. | G.I. | Disc. |
| Finance | 3.39 | 5.57 | 2.18 | 2.50 | 4.44 | 1.94 | 2.73 | 4.98 | 2.24 |
| Health | 3.75 | 5.91 | 2.15 | 2.84 | 5.08 | 2.24 | 2.83 | 5.35 | 2.53 |
| Education | 3.56 | 5.60 | 2.04 | 2.62 | 4.52 | 1.91 | 2.50 | 5.03 | 2.53 |
| Election | 3.34 | 5.64 | 2.30 | 2.52 | 4.45 | 1.94 | 2.72 | 5.14 | 2.42 |
| Public Services | 3.91 | 5.82 | 1.91 | 2.96 | 5.14 | 2.19 | 3.44 | 5.36 | 1.92 |
Table 2: Mean credibility ratings for archived credible news (G.I.) and fact-checked false claims (M.I.) across topics and models. Discernment (Disc.) is the genuine-minus-false credibility gap. LLaMA, Gemma, and GPT-4o-mini denote LLaMA-3.1-8B-Instruct, Gemma-2-27B-IT, and GPT-4o-mini, respectively. GPT-4o-mini also serves as the generation judge (Section 4.1 ) and inoculation generator (Section 6.1 ); its ratings are included for comparison but are not considered independent victim evaluations.
| Harmful action | Correct genuine action | |
|---|---|---|
| ( real attacks) | ( requests) | |
| Llama, rating first | ||
| Llama, no rating | ||
| Gemma-9B, rating first | ||
| Gemma-9B, no rating |
Table 3: Harmful and legitimate tool-execution rates for untrained, control-trained, and defensively fine-tuned models, reported as proportions of evaluated items. Control and defended results are averaged over three training runs; untrained results use a single run. Llama and Gemma-9B denote Llama-3.1-8B-Instruct and Gemma-2-9B-IT, respectively.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure B1: Judge-perceived credibility and shareability by topic across all 10,794 generated candidates. Bubble area is proportional to the number of generations in each 0.05 credibility bin (0–1). Green bars show the shareability percentage (labeled within each bar); solid horizontal lines show the acceptance rate (percentage at the top-right of each line).
| Generator | Election | Public Services | Health | Finance | Education | Overall |
|---|---|---|---|---|---|---|
| GPT-4o | 61.1 | 93.3 | 99.4 | 99.3 | 37.8 | 78.2 |
| GPT-4o-mini | 90.4 | 90.6 | 99.3 | 99.3 | 19.6 | 79.9 |
| LLaMA-3.3-70B | 63.8 | 96.9 | 99.1 | 99.8 | 19.8 | 75.9 |
| Qwen2.5-72B | 62.6 | 83.1 | 90.5 | 88.5 | 25.2 | 70.0 |
| All | 69.5 | 91.0 | 97.1 | 96.7 | 25.6 | 76.0 |
Table B1: Judge-rated acceptance rates (%) for misinformation generated by each generator model across topics.
| Strategy | GPT-4o | GPT-4o-mini | LLaMA-3.3-70B | Qwen2.5-72B | Overall |
|---|---|---|---|---|---|
| Baseline | 81.22 (0.806) | 84.11 (0.820) | 76.78 (0.861) | 67.44 (0.796) | 77.39 (0.821) |
| Instruction | 80.67 (0.778) | 76.59 (0.820) | 76.28 (0.840) | 72.41 (0.726) | 76.49 (0.791) |
| CoT | 72.67 (0.848) | 78.89 (0.853) | 74.56 (0.865) | 70.11 (0.813) | 74.06 (0.845) |
Table B2: Acceptance rate (%) and mean credibility (in parentheses) by prompting strategy and generator model.
| Adversarial Framing | Acceptance Rate (%) | Credibility | Shareability (%) |
|---|---|---|---|
| Descriptive | 75.33 | 76.08 | |
| Suggestive | 76.04 | 77.54 | |
| Manipulative | 76.57 | 78.71 |
Table B3: Acceptance rate, mean credibility with standard deviation, and shareability rate of pipeline-generated misinformation across three adversarial-framing strategies.
| Structural frame | 2023 | 2024 | Stable |
|---|---|---|---|
| Media shows event | Yes | ||
| Attributed quote | Yes | ||
| Arrest for an offence | Yes | ||
| Statistic about a group | Yes | ||
| Electoral fraud | Yes | ||
| Warning against an action | Yes |
Table D1: Prevalence (%) of ten hand-specified structural frames among English-language fact-checked claims published in 2023 ( ) and in 2024 ( ). A frame is marked stable when it matches at least twenty claims in each year and its relative change in prevalence is below ; the criterion and the ten surface patterns were fixed before the 2024 counts were taken. The three frames that fail the criterion are the institutional-announcement register that the generation pipeline predominantly produces.
| None | Forewarning | Rated | ||||
|---|---|---|---|---|---|---|
| Victim | L | C | L | C | L | C |
| Llama-8B | ||||||
| Gemma-9B | ||||||
| Qwen3-14B | ||||||
| Gemma-27B | ||||||
| Gemini | ||||||
Table F1: Harmful execution on logistics (L, items) and credential and payment (C, items) notices. Group is confounded with topic and kind of loss. Llama-8B, Gemma-9B and Gemma-27B are Llama-3.1-8B-Instruct, Gemma-2-9B-IT and Gemma-2-27B-IT.
| Harmful execution | Acts when rated | Request removal | Scam cues | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Victim | Generated | Fact-checked | Smishing | Fact-checked | Smishing | Requested | Forwarded | Rating | Rating, cues | Harm, cues |
| Llama-3.1-8B-Instruct | ||||||||||
| Gemma-2-9B-IT | ||||||||||
| Qwen3-14B | ||||||||||
| Gemma-2-27B-IT | ||||||||||
| Gemini-3.6-Flash | ||||||||||
Table G1: Real attacks and the two mechanism tests, rating first. Harmful execution : no intervention, generated notices, fact-checked claims and smishing messages. Acts when rated : harmful execution among no-intervention trials the victim rated or lower. Request removal : harmful execution when the user asks for the action ( attacks) and when the user only forwards its details ( ). Scam cues : mean rating ( – ) on the original text, and mean rating and harmful execution on the rewrite with scam cues ( ); harm on the original text is the Requested column. Generated-notice rates come from the run of Table 1 , the others from later runs of the same code. One sample per item and condition.
| Topic | Pipeline-Generated Misinfo | Fact-Checked Misinfo |
|---|---|---|
| Election | Contingency poll-worker lists amid reports that provisional-ballot and chain-of-custody training is falling behind. | Labour bar chart omits Greens in Gorton and Denton poll (Full Fact). |
| Public Services | Transit agencies announce election-day bus boosts on key lines serving polling places. | White House hoax claims all flights grounded and National Guard activated (Lead Stories). |
| Health | Hospitals issue corrections after patients are spoofed via shortened portal links. | Altered mass-burial images shared as Nipah outbreak in West Bengal (Reuters). |
| Finance | Spoof banking domains with minor spelling differences harvest credentials. | E-PAN card phishing download link (PIB Fact Check). |
| Education | Persistent campus-closure rumors despite no official university alert. | AI-generated video of crying French “teacher” harassed by students (Lead Stories). |
Table H1: Misinformation examples (summarised for brevity) used as seed messages for generating inoculation messages.
| Gemma-2-27B-IT | LLaMA-3.1-8B-Instruct | GPT-OSS-20B | ||||||||||
| Configuration | G.I. | M.I. | Disc. | M.I. | G.I. | M.I. | Disc. | M.I. | G.I. | M.I. | Disc. | M.I. |
| Pre-Inoculation | 6.55 | 5.94 | 0.61 | — | 5.64 | 5.25 | 0.38 | — | 6.43 | 5.80 | 0.62 | — |
| Warning-Only | 6.60 | 5.77 | 0.84 | 0.17 ∗ | 5.72 | 5.24 | 0.47 | 0.01 | 6.20 | 5.38 | 0.82 | 0.42 ∗∗ |
| Random Message | 6.25 | 5.75 | 0.50 | 0.19 ∗∗ | 5.69 | 5.25 | 0.44 | 0.00 | 6.28 | 5.62 | 0.66 | 0.19 |
| Skepticism Prompt | 6.52 | 5.84 | 0.68 | 0.10 ∗ | 5.82 | 5.40 | 0.42 | 0.15 ∗ | 5.82 | 4.45 | 1.37 | 1.36 ∗∗ |
| Fact-Checked Seed | 6.62 | 5.86 | 0.76 | 0.08 ∗ | 5.78 | 5.41 | 0.38 | 0.16 ∗ | 6.27 | 5.68 | 0.59 | 0.12 ∗ |
Table I1: Credibility of genuine (G.I.) and misinformation (M.I.) items for three victim models under a single evaluation protocol. M.I. denotes the pipeline items the generator was instructed to make misleading; the falsity audit of Section 4.1 bounds how they should be read. Disc. is the discernment gap, G.I. M.I.; M.I. is the change in misinformation credibility relative to the same victim’s Pre-Inoculation baseline. ratings per configuration per item type per victim ( in total). , , two-sided Mann–Whitney against that victim’s Pre-Inoculation misinformation ratings. † Control, not a well-sampled intervention (Limitations). ‡ Diagnostic in which the exemplar overlaps lexically with the evaluation item.
| Gemma-2-27B-IT | LLaMA-3.1-8B-Instruct | GPT-OSS-20B | ||||
|---|---|---|---|---|---|---|
| Configuration | AUC | AUC | AUC | |||
| Pre-Inoculation | 0.755 | — | 0.672 | — | 0.726 | — |
| Warning-Only | 0.816 ∗ | 0.713 ∗ | 0.750 | |||
| Random Message | 0.679 ∗ | 0.671 | 0.710 | |||
| Skepticism Prompt | 0.789 ∗ | 0.672 | 0.821 ∗ | |||
| Fact-Checked Seed | 0.827 ∗ | 0.675 | 0.707 | |||
Table J1: Discrimination and response bias for the exemplar-free controls and the three seeded configurations, over the subset of the ratings of Table I1 contributed by these configurations. AUC is the probability that a randomly drawn genuine item is rated above a randomly drawn misinformation item ( is chance); is the change in response criterion relative to that victim’s Pre-Inoculation baseline, positive being more sceptical. ∗ marks a configuration whose bootstrap AUC interval excludes the baseline AUC.
Figure J1: Cross-topic inoculation results for messages generated from pipeline-generated seeds. Columns show LLaMA-3.1-8B-Instruct, GPT-OSS-20B, and Gemma-2-27B-IT (left to right). Rows compare changes in genuine-information credibility, misinformation credibility, and discernment relative to the pre-inoculation baseline
| Condition | Harmful | Legitimate |
|---|---|---|
| No intervention | ||
| Unrelated message | ||
| Forewarning | ||
| Worked-example prebunk | ||
| Pipeline-seeded prebunk |
Table L1: Proportion of trials in which Gemma-2-27B-IT emits a tool call, on the twenty instruction-bearing claims of Section L.3 . Left: the call that causes the declared loss, given the misinformation item. Right: the correct call on a matched genuine request, which a useful assistant should still make. Each cell is trials ( items samples, temperature ). The unrelated message is matched to the worked-example prebunk in register and length and refers to no deception of any kind; it is the control for prompt length. The pipeline-seeded prebunk is the message of Section L.3 , run separately on the same items and settings. Lower is better on the left, higher is better on the right.
| Grades used | OR | |
|---|---|---|
| Both annotators, adjudicated | 3.97 | .0082 |
| First annotator alone | 3.50 | .0158 |
| Second annotator alone | 3.09 | .0290 |
| Automated first pass | 2.38 | .1159 |
Table M1: Primary contrast (Cochran–Mantel–Haenszel, common arms, stratified by victim model) recomputed under each grading configuration. Rows two and three vary the Gemma-2-27B grades only; LLaMA-3.1-8B-Instruct is graded by one annotator throughout except in the final row. The direction and approximate magnitude are stable across the human configurations. The automated pass is the most lenient labeller and the only configuration that does not reach significance.
| Victim | Condition | Acts | 95% CI |
|---|---|---|---|
| LLaMA-3.1-8B | None | 18/20 | [.70, .97] |
| Prebunk | 13/20 | [.43, .82] | |
| Gemma-2-27B | None | 14/20 | [.48, .86] |
| Prebunk | 0 8/20 | [.22, .61] | |
| Forewarning | 0 9/20 | [.26, .66] |
Table M2: Proportion of trials in which the victim acts on the false claim (grade ), instruction-bearing stimuli, human grades. Intervals are Wilson . “Prebunk” is the pipeline-seeded three-component message; “Forewarning” is the bare warning, which was run on Gemma-2-27B only.
More in this topic
Language Model Safety Evaluation
Language model safety evaluation measures harmful behavior, refusal, robustness, and over-refusal across realistic prompts, users, and deployment settings.
See all 542 papers