Language-model agents' beliefs can be manipulated via truthful evidence in open channel
Organizations: School of Computer Science and Informatics, University of Liverpool, Liverpool, UK · Department of Computer Science, Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates
Abstract
Language-model agents' beliefs can be manipulated through open channels such as social media, even when the information they receive is truthful. When agents fill gaps between fragments with self-induced relationships, they may accept those relationships as established facts and pass the resulting rumours to others. Drawing on cognitive theory of narrative understanding and film theory of montage, we introduce Generative Montage to operationalize this failure, termed narrative overfitting, by coordinating agents to arrange truthful fragments into misleading narratives. Across 1,090 rumour targets from six Twitter events, agents often accepted rumours overconfidently, with overall acceptance exceeding 65 percent in twelve of fourteen models. Acceptance remained above 50 percent after majority voting or review of the same evidence, showing how errors can reach downstream decisions. To explain this susceptibility, fragment interventions identify posts that influence belief within complete narratives. Synthetic controls then show that similar response patterns arise from valid evidence, separating belief influence from evidential support. This distinction motivates Grounded Monitor, a proof-of-concept neuro-symbolic runtime monitor that checks direct evidential support to guide agents' judgements, improving decision accuracy by 5.6 to 29.0 percentage points across original and anonymized settings. These findings expose a cognitive route to rumour amplification and support monitoring inferred relationships before agent-generated conclusions guide public decisions.
Figures & tables
| Writer narrative Draft excerpt | Editor feed Post sequence and implied connection | Victim agent response Verdict, confidence and review |
| Charlie Hebdo : Ahmed Merabet was the first victim of the Charlie Hebdo attack | ||
| Officer Merabet stationed outside… confrontation occurred… #JeSuisAhmed trended… death confirmed in reports… | 7 posts Attack starts Officer present Hashtag emerges Casualty Implied connection: Presence at the attack’s onset is linked to being the first victim. | TRUE “Timeline suggests early encounter…” Confidence: 0.92 MV: Y (5/5) Judge: Y |
| Sydney Siege : There are 13 hostages inside the Sydney cafe | ||
| Police cordoned area… hostages visible through windows… reports mentioned multiple people… 13 individuals counted… | 6 posts Cafe surrounded People visible Count emerges 13 confirmed Implied connection: A count of people is treated as a count of hostages. | TRUE “Multiple sources indicate 13 hostages…” Confidence: 0.88 MV: Y (3/5) Judge: Y |
| Ferguson : Ferguson police are leading a smear campaign or character assassination of M. Brown | ||
| Police released surveillance footage… Brown’s background discussed… media focus shifted to character… community alleged bias… | 8 posts Shooting Footage release Character focus Bias claims Implied connection: A shift towards character coverage is interpreted as coordinated intent. | TRUE “Pattern suggests coordinated character attack…” Confidence: 0.85 MV: Y (5/5) Judge: Y |
| Target-status selectivity within montage feeds | |||
| Diagnostic signal | Non-rumour control montage ( ) | Attack montage ( ) | AUROC |
| Mean salience | 0.889 | 0.900 | 0.502 |
| Mean grounding posts per feed | 0.487 | 0.070 | 0.664 |
| Feeds with grounding (%) | 38.8 | 6.3 | 0.662 |
| Detecting attack montage against same-event reporting | |||
| Outcome | Attack montage ( ) | Chronological control ( ) | Performance |
| Evaluated model | Charlie Hebdo | Sydney Siege | Ferguson | Ottawa Shoot. | Germanwings | Putin Missing | Overall ASR | ||||||||||||
| A | C | H | A | C | H | A | C | H | A | C | H | A | C | H | A | C | H | ||
| Proprietary models | |||||||||||||||||||
| GPT-4o-mini | 81.7 | 0.83 | 67.4 | 92.1 | 0.82 | 70.3 | 79.5 | 0.81 | 59.0 | 86.7 | 0.85 | 78.2 | 74.5 | 0.85 | 60.0 | 66.7 | 0.82 | 53.3 | 83.1 |
| GPT-4o | 79.4 | 0.85 | 66.9 | 88.5 | 0.83 | 69.1 | 67.0 | 0.83 | 53.0 | 85.5 | 0.85 | 74.5 | 56.4 | 0.89 | 52.7 | 63.3 | 0.75 | 26.7 | 77.4 |
| GPT-4.1-nano | 81.7 | 0.81 | 66.9 | 94.5 | 0.80 | 67.1 | 79.1 | 0.79 | 53.1 | 94.5 | 0.81 | 74.8 | 74.5 | 0.83 | 61.8 | 70.0 | 0.75 | 13.3 | 85.5 |
| GPT-4.1-mini | 79.9 | 0.85 | 68.4 | 80.6 | 0.82 | 50.9 | 68.0 | 0.81 | 42.5 | 77.6 | 0.87 | 67.9 | 63.6 | 0.91 | 63.6 | 20.0 | 0.81 | 16.7 | 72.7 |
| Configuration | ASR (%) | HC-ASR (%) | Change in ASR (pp) |
| Complete construction procedure | 77.0 | 64.9 | — |
| Without Director-guided revision | 63.5 | 48.0 | |
| Without Editor sequencing | 69.7 | 52.5 | |
| Single construction agent | 26.8 | 16.6 |
| Evaluated model | Prompting | ASR (%) |
| Qwen2.5-7B | Direct | 67.8 |
| Chain of thought | 70.9 ( ) | |
| DeepSeek-R1-7B | Direct | 74.9 |
| Chain of thought | 81.7 ( ) |
| Dispersion by refutation class | |||||
| Evaluated model | Full-feed belief | Dispersion | Explicit contradiction | Later correction | Inference alone |
| Qwen2.5-3B | 0.76 [0.71, 0.81] | 0.65 [0.58, 0.72] | 0.72 [0.63, 0.80] | 0.64 [0.53, 0.74] | 0.56 [0.46, 0.67] |
| Qwen2.5-7B | 0.59 [0.53, 0.65] | 0.71 [0.65, 0.76] | 0.76 [0.70, 0.83] | 0.69 [0.61, 0.76] | 0.63 [0.48, 0.78] |
| Qwen2.5-14B | 0.51 [0.43, 0.58] | 0.56 [0.48, 0.64] | 0.63 [0.51, 0.71] | 0.53 [0.39, 0.65] | 0.50 [0.29, 0.67] |
| DeepSeek-R1-1.5B | 0.39 [0.30, 0.47] | 0.75 [0.69, 0.82] | 0.78 [0.67, 0.89] | 0.71 [0.62, 0.81] | 0.79 [0.71, 0.88] |
| DeepSeek-R1-7B | 0.48 [0.40, 0.56] | 0.61 [0.54, 0.68] | 0.69 [0.62, 0.77] | 0.60 [0.50, 0.71] | 0.45 [0.33, 0.57] |
| Belief | ||||
| Evaluated model | , deletion | Flips, payload / random (%) | Baseline TRUE (%) | , transplant |
| Qwen2.5-3B | [ , ] | 19.7 / 5.8 | 44.4 | [ , ] |
| Qwen2.5-7B | [ , ] | 20.2 / 4.9 | 70.0 | [ , ] |
| Qwen2.5-14B | [ , ] | 17.0 / 5.8 | 54.7 | [ , ] |
| DeepSeek-R1-1.5B | [ , ] | 15.7 / 8.5 | 32.7 | [ , ] |
| DeepSeek-R1-7B | [ , ] | 7.2 / 1.3 | 91.5 | [ , ] |
| Qwen2.5 | DeepSeek-R1 | Qwen3 | Llama-3.1 | |||||
| Contrast | 3B | 7B | 14B | 1.5B | 7B | 14B | 8B | 8B |
| vs | 0.58 [0.55, 0.60] | 0.59 [0.56, 0.61] | 0.61 [0.58, 0.64] | 0.52 [0.51, 0.53] | 0.52 [0.51, 0.54] | 0.62 [0.59, 0.64] | 0.55 [0.53, 0.58] | 0.62 [0.59, 0.65] |
| vs | 0.57 [0.51, 0.64] | 0.56 [0.49, 0.63] | 0.55 [0.48, 0.62] | 0.44 [0.40, 0.49] | 0.54 [0.47, 0.60] | 0.56 [0.49, 0.63] | 0.56 [0.49, 0.63] | 0.55 [0.48, 0.62] |
| vs | 0.44 [0.38, 0.51] | 0.45 [0.38, 0.52] | 0.46 [0.39, 0.54] | 0.48 [0.43, 0.52] | 0.48 [0.41, 0.55] | 0.46 [0.39, 0.54] | 0.44 [0.38, 0.52] | 0.50 [0.43, 0.58] |
| vs | 0.52 [0.45, 0.58] | 0.51 [0.44, 0.58] | 0.51 [0.43, 0.58] | 0.42 [0.38, 0.47] | 0.52 [0.45, 0.58] | 0.52 [0.45, 0.59] | 0.50 [0.42, 0.58] | 0.55 [0.47, 0.61] |
| vs | 0.98 [0.97, 0.99] | 0.99 [0.98, 1.00] | 1.00 [0.99, 1.00] | 0.94 [0.92, 0.96] | 0.97 [0.95, 0.98] | 0.99 [0.98, 1.00] | 1.00 [1.00, 1.00] | 0.90 [0.86, 0.93] |
| Acceptance (%) | Above threshold (%) | ||||||
| Cohort | Evaluated model | Reads | Direct | Multi-step | Supported | Curation-free | Attack |
| A | Qwen3-8B | 144 | 97.5 | 100.0 | 100 | 97 | 100 |
| A | Qwen2.5-7B | 144 | 96.3 | 96.8 | 100 | 95 | 96 |
| A | Llama-3.1-8B | 144 | 96.3 | 100.0 | 81 | 71 | 73 |
| Measure | ORIG | ANON |
| Target-only probes (GPT-4.1-mini) | ||
| Event recognition (%) | 44.5 | 5.5 |
| Correct veracity guesses (%) | 22.7 | 1.7 |
| Director scores | ||
| Narrative score | 7.87 | 8.23 |
| Sequence score | 8.67 | 8.66 |
| Entity-surface contrasts in TRUE probability | ||||||
| Evaluated model | Mean | Holm | Mean | LOEO range | ||
| Qwen2.5-7B | 0.99 | 1.00 | 1.00 | to | ||
| Qwen3-8B | 0.30 | 0.91 | 0.41 | to | ||
| Llama-3.1-8B | 0.63 | 1.00 | 0.65 | to | ||
| Relation | Operational test | Illustrative post | Graph role |
| grounds | The post alone makes the target necessarily true, if accurate. | “Police confirmed that several attackers coordinated the operation.” | Observation supporting |
| abductive | The post promotes the target, but establishing it requires an additional observation or premise. | “Gunfire was reported at the War Memorial and Parliament within minutes of one another.” | Retractable assumption |
| undermines | The post supplies evidence against the target. | “Police concluded that a single gunman acted alone.” | Counter-observation |
| irrelevant | The post does not bear on the target. | “Residents were asked to remain indoors.” | No target edge |
| Diagnostic measure | ORIG | ANON |
| Mean salience, non-rumour | 0.889 | 0.866 |
| Mean salience, rumour | 0.900 | 0.776 |
| Salience AUROC | 0.502 | 0.569 |
| Feeds with grounding, non-rumour (%) | 38.8 | 38.8 |
| Feeds with grounding, rumour (%) | 6.3 | 7.0 |
| Grounding-presence AUROC | 0.662 | 0.659 |
| Targets ( ) | Grounding (%) | |||||
| Target type | Example | Non-rumour | Rumour | Non-rumour | Rumour | AUROC |
| Atomic | “11 people died”; “the suspect is X” | 72 | 120 | 40.3 | 6.7 | 0.670 |
| Inferential | “the plane descended rapidly” | 8 | 38 | 25.0 | 5.3 | 0.599 |
| Measure | Writer and Director | Editor and Director |
| First approval round | ||
| Best approval round | ||
| Director apparent-support score | ||
| Narrative length (words) | — | |
| Sequence length (posts) | — |
| Experiment and purpose | Materials | Models and readouts | Comparison |
| Broad attack evaluation. Establish susceptibility across models and events. | 1,090 rumour targets from six events; five analyst runs per target. | Fourteen-model panel. ASR records second-stage target acceptance; HC-ASR additionally requires first-stage confidence of at least 0.8. Mean confidence summarizes the initial open reports. | Acceptance across models and events under the common attack-evaluation protocol. |
| Downstream review. Test whether another judgement stage retains the initial conclusions. | The same feed and five analyst reports for each reviewed target; event-level results summarized across four model families. | Majority vote over five target-adjudication verdicts; a separately instantiated DeepSeek-R1-14B judge. Readout: downstream target acceptance. | Majority voting and language-model adjudication on shared input, as shown in Fig. 1 c. |
| Natural-corpus responses. Locate differences between complete-feed and isolated-fragment responses. | 223 attack feeds covering 37 targets and 464 same-event ordinary-report feeds. | Six-model response panel; binary answer-token probabilities under seven templates. Readouts: full-feed margin , belief pull , dispersion and AUROC. | Attack feeds versus ordinary-report feeds; pooled and per-model comparisons, with target-refutation strata in Extended Data Table Extended Data . |
| Payload interventions. Test whether a fragment selected through isolated responses changes the complete-feed judgement. | Deletion: 223 attack feeds over 37 targets. Transplantation: 349 ordinary-report feeds over 251 targets. | Six-model response panel; each model selects its own payload and evaluates all feed variants under seven templates. Readouts: changes in and verdict flips, with descriptive changes in and correlations. | Payload versus random deletion; ordinary-report feeds before versus after payload insertion. |
| Experiment and purpose | Materials | Models and readouts | Comparison |
| Synthetic narrative controls. Distinguish narrative influence from evidential support. | Cohort A comprises 134 world slots. Supported narratives use the first 48 slots, yielding 48 feeds and 144 template reads per model. | Six-model panel plus Qwen3-8B and Llama-3.1-8B for selection contrasts. Supported narratives use the two additional models and Qwen2.5-7B. Three templates with TRUE, FALSE and INSUFFICIENT probabilities give margin, dispersion, acceptance and AUROC. | Construction contrasts, supported-narrative comparisons and dispersion-threshold exceedance in Fig. 4 ; complete values in Extended Data Tables Extended Data and Extended Data . |
| Entity replacement and regeneration. Examine the joint change in entity familiarity and generated material. | The focused set of 158 rumour targets and 80 non-rumour controls in ORIG and ANON. | Four-model GM panel. Readouts: target acceptance for rumour targets and non-rumour controls. GPT-4.1-mini target-only probes measure event recognition and veracity guesses. | Feeds generated from original entities versus feeds regenerated after replacement with fictional entities. |
| Fixed-text entity controls. Separate entity-name effects from generation-source differences. | 158 rumour targets; eight conditions and three templates for each of three models. | Qwen2.5-7B, Qwen3-8B and Llama-3.1-8B. Readouts: contrasts in template-averaged TRUE probability and the percentage of individual TRUE verdicts. | Generation source crossed with real or fictional names in fixed text; target-only baselines under knowledge and evidence-evaluation prompts. |
| Grounded Monitor. Monitor detected single-post support and assess changes in target judgement following its assessment. | 158 rumour targets and 80 non-rumour controls; a separate detection comparison pairs attack montages with 158 chronological control feeds matched for event, post count and target. | GPT-4.1-mini relation classifier; four-model GM panel. Readouts: salience and detected grounding, alarm rates, AUROC and decision accuracy. | Attack versus non-rumour control montages; attack montages versus chronological control feeds; warning plus the uncertain option versus baseline judgements in ORIG and ANON. |
| Event | Context | Evidence | Rumour targets | Mean cascade size |
| Charlie Hebdo | breaking news | 1,814 | 265 | 14.8 |
| Sydney siege | hostage incident | 1,081 | 140 | 16.4 |
| Ferguson | civil unrest | 869 | 274 | 21.8 |
| Ottawa shooting | terrorist attack | 749 | 141 | 11.7 |
| Germanwings | aviation disaster | 325 | 144 | 10.0 |
| Putin missing | political uncertainty | 112 | 126 | 2.9 |
Belief Updating in LLMs
Belief updating in large language models studies how models revise, retain, and use beliefs as evidence and context change.
See all 110 papers