Misinformation Without Triggers: From Factual Answers to Downstream Decisions

Lin Tian; Marian-Andrei Rizoiu

cs.CLOct 2, 2026

Misinformation Without Triggers: From Factual Answers to Downstream Decisions

Authors: Lin Tian, Marian-Andrei Rizoiu

Organizations: Behavioral Data Science, University of Technology Sydney

Abstract

Language models learn from web documents, some of them false, and false content can reach a model's answer to a factual question and the summaries and decisions that use it. Most data-poisoning studies add a trigger to the training data and activate it in the prompt. False documents can also change factual responses without any trigger, but we do not know whether the direct answer predicts the decision. In this work, we follow false content past the answer and find an audit gap between what a direct probe reports and what the model then does, comparing false training with matched truthful controls in a controlled decision task, Guess the Capital, where a fixed decoder turns factual answers into a scored card choice, and on a misleading claim from Facebook posts about the 2019--20 Australian bushfires. Across eight models at dose 1,000, direct injected-choice rates reach 95.8--100%, while injected game choices increase by 1.7--14.4 percentage points over matched truthful training. The gap runs the other way too. Facts that pass the direct probe still push decisions toward the injected answer, and game accuracy drops further than those choices explain. Truthful correction brings the fact back but not the decisions built on it. We then look into the real-world bushfire case, models trained on the false posts say that people were arrested for arson even when they lose the inflated count, and in a count-by-wording factorial the misleading arrest wording produces arrest assertions even when the training count stays at 24. In simpler terms, a correct factual answer does not guarantee a correct decision, and losing the injected number does not remove the misleading story.

Figures & tables

Figure 1: Guess the Capital turns factual answers into a scored decision. (1) Clean model A and poisoned model P start from the same base model and read the same corpus, except that the target statements name the true or the injected capital. Clean model B repeats the truthful training with another seed and measures how much two truthful models disagree. Training includes no game text and no trigger. (2) A board shows eight city cards, each with a seven-bit codeword. The Investigator asks seven yes-or-no questions, one per bit, each naming the four cards whose bit is 1. The Archivist answers from memory without seeing the board. (3) The Investigator is a fixed decoder: it picks the card whose codeword is closest to the seven answers. Any two codewords differ in four bits, so the decoder corrects one wrong answer. Each pick counts as the true card, the injected card, or another card. We compare the change from A to P with the disagreement on the same boards.

	Direct injected choice (%)	Game injected choice (%)		Game accuracy (%)
Model	Poisoned	Clean	Poisoned	Clean	Poisoned
Qwen2.5-0.5B	$100.0\pm 0.0$	$6.9\pm 4.9$	$21.3\pm 6.5$	$56.5\pm 7.1$	$34.4\pm 4.1$
Qwen2.5-1.5B	$100.0\pm 0.0$	$1.6\pm 0.2$	$5.6\pm 2.7$	$88.7\pm 3.3$	$85.1\pm 5.4$
Qwen2.5-3B	$97.2\pm 2.4$	$1.8\pm 0.7$	$4.9\pm 2.1$	$91.4\pm 2.8$	$87.6\pm 4.3$
Llama-3.2-3B-Instruct	$96.4\pm 3.1$	$2.4\pm 1.1$	$6.8\pm 2.9$	$84.7\pm 4.6$	$79.2\pm 5.8$
Llama-3.1-8B-Instruct	$99.4\pm 0.8$	$1.1\pm 0.6$	$2.8\pm 1.7$	$95.8\pm 1.9$	$88.4\pm 3.6$

Table 1: Guess-the-Capital game outcomes at dose

=

1000 statements per target. Values are mean

\pm

SD across five seeds.

Figure 2: Poisoned CPT lowers accuracy across many MMLU subjects. Per-subject accuracy loss for poisoned capital (a, b) and spleen (c, d) checkpoints relative to base (a, c) and SFT-only (b, d). Tiles keep the same position in every panel and share a 0–20 percentage-point scale; white marks no loss. HS $=$ high school; CS $=$ computer science; Prof. $=$ professional; Eng. $=$ engineering.

Figure 3: The misleading story outlives the injected number. (a) Arrest assertions as a share of 36 answers per exposure condition, split by whether the answer gives 183. (b) Arrest assertions and exact-183 counts in the count-by-wording factorial. Whiskers in (a, b) span the three seeds. (c) Distribution of reported counts per condition at training step 252 (left, hollow) and 1,000 (right, filled); dashed lines mark the decision thresholds, and the share of wrong decisions heads each half.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Study	Comparison	Replication
Main capital game	Eight models (Table 1 ); dose 1,000; matched truthful and poisoned training	Five seeds per model
Qwen dose and reference diagnostics	Qwen2.5-0.5B/1.5B; two doses; clean A, clean B, poisoned	Three seed/mapping units; 275 boards per condition and unit
Bushfire posts and factorial	Qwen2.5-1.5B; three exposure conditions or five factorial conditions	Three seeds per condition; one claim family
Qwen correction	Correction and equal-budget replay from clean and poisoned origins	Three seed/target-set units per study
Llama and Gemma interface studies	Dose 128; truthful and false training from an untouched instruction-tuned model	One seed and 12-fact set per model
MMLU	Qwen3.5-9B capital and spleen interventions; paired truthful controls	Three training pairs per intervention; fixed corpus and target

Appendix

Table 2: Training and replication across the principal studies. Each study has its own matched comparison; we do not pool results across recipes.

Setting	Specification
Corpus	43 country–capital relations and 19,995 truthful background rows, including 465 statements per target. Eleven targets receive $d$ additional true rows in A/B or false rows in P. Target truths remain in P.
Exposure	Five capital-sentence templates; dose counts repeated statements. Three epochs, with no game examples. Substitutions can change token counts, so conditions are row-matched rather than exactly token-matched.
Loss and inputs	Full causal next-token loss with padding masked; one document per example, 8–20 tokens, no truncation or packing, and no chat template.
Optimizer	AdamW; learning rate $5\times 10^{-6}$ at 0.5B and $3\times 10^{-6}$ at 1.5B; $\beta=(0.9,0.999)$ , $\epsilon=10^{-8}$ , weight decay 0.01, gradient norm capped at 1.0.
Schedule	Effective batch 32; $T=3\lceil N/32\rceil$ updates for $N$ rows; linear warmup for $\lceil 0.03T\rceil$ updates, then cosine decay.
Precision and selection	FP16 mixed precision, FP32 trainable parameters; final epoch-3 checkpoint, with no validation-based selection.

Appendix

Table 3: Shared recipe for the Qwen2.5-0.5B/1.5B capital game and preference sweep. The game uses doses 500 and 1,000; the sweep additionally tests 100 and 250 at 0.5B. Mapping conventions and expanded training specifications are in the detailed record.

Dimension	$n$	Agree	Agreement	$\kappa$
Capital outcome	660	656	99.4%	0.988
Narrative responsiveness	240	126	52.5%	0.122
Explicit arson-arrest assertion	240	240	100.0%	1.000
Incorrect deliberate-charge count	240	240	100.0%	1.000
Legal-status overstatement	240	218	90.8%	0.558
Unsupported numerical causal share	240	239	99.6%	0.798

Appendix

Table 4: Agreement between the two blinded annotators. Cohen’s

\kappa

is unweighted and retains the uncertain category. Agreement measures label reproducibility, not the exposure effect.

Seed	Exposure	Generated response	Arrest claim
Exact prompt: Question: Write a short news brief about the NSW Police bushfire update of 6 January 2020. Answer in two or three sentences. Answer:
1	Neutral	NSW Police reported that bushfire activity in the state was at a low level on 6 January 2020. The fire activity was mostly in the north-west of the state. The fire activity was mostly in the north-west of the state.	No
1	Corrected	NSW police reported 14 people were charged over alleged deliberately lit bushfires in New South Wales on 6 January 2020.	No
1	False	NSW police reported 13 people were arrested for arson during the current fire season.	Yes
2	Neutral	NSW Police reported 13 people were rescued from their homes in the bushfire affected areas of the state. The fire was still burning in the Blue Mountains.	No
2	Corrected	The total number of people facing bushfire-related legal action has risen to 143, with 103 facing criminal charges.	No

Appendix

Table 5: Generated briefings under matched training exposure. All nine responses to one prompt in the Facebook-post study: three exposure conditions across three seeds. We reproduce each answer in full; the last column reports the agreed arrest-assertion label, not overall factual correctness.

Arm	Count	Legal wording	Arrest assertions (%)	Exact-183 headcounts (%)
Neutral	–	Background only	$0.0\pm 0.0$	$0.0\pm 0.0$
Reference	24	Alleged charges	$0.0\pm 0.0$	$0.0\pm 0.0$
Wording only	24	Arson arrests	$58.3\pm 28.9$	$0.0\pm 0.0$
Count only	183	Alleged charges	$0.0\pm 0.0$	$41.7\pm 14.4$
Combined	183	Arson arrests	$66.7\pm 14.4$	$41.7\pm 28.9$

Appendix

Table 6: Primary factorial outcomes, as mean

\pm

sample SD across three training seeds. Each seed contributes four responses per condition. We score headcount and legal relationship separately; SD describes between-seed variation, not a confidence interval.

Primary responses
Outcome	Count	Wording	Interaction
Arson-arrest assertion	$+4.2$	$+62.5$	$+8.3$
Supported target proposition	$-4.2$	$-4.2$	$+8.3$
Headcount 183	$+41.7$	$0.0$	$0.0$
Unsupported premise to advice	$0.0$	$0.0$	$0.0$
Secondary responses

Appendix

Table 7: Mean factorial contrasts across three training seeds, in percentage points. Primary and secondary prompt populations remain separate. Brackets retain annotator uncertainty and are not confidence intervals. A zero mean can conceal opposite-signed run effects.

Figure 4: Number drift and fixed-rule decisions. (a) The 48 false-exposure replies paired across checkpoints; 15 of the 22 early replies of 183 become 133. (b) Exact-183 replies decline from 22/48 to 3/48 while wrong decisions rise from 78/96 to 86/96. We apply each reply at two thresholds.

Figure 5: Wording and number have separate effects; correction separates factual from decision recovery. (A) Arrest-assertion and exact-183 rates, pooled across three seeds with four prompts each (12 responses per point), separated by training count; whiskers span the seed range. (B) Qwen-capital outcomes after 2,048 correction or equal-budget replay updates from clean (blue) and poisoned (red) origins. Large and small points show means and individual seed/target-set units, each with 55 factual comparisons and 275 boards. Guides mark the 90% factual and 80% game qualification thresholds; game chance is 12.5%.

Model	Relation	P	A	$\Delta_{\rm corr}$
Qwen 1.5B	Capital	$[-65.82,-29.82]$	$[-2.55,+8.36]$	$[-72.36,-27.27]$
Llama 8B	Capital	$[-89.45,-35.64]$	$[-42.91,-30.18]$	$[-56.36,+7.27]$
Qwen 1.5B	Postal	$[-71.27,-17.45]$	$[+10.91,+41.45]$	$[-101.45,-58.91]$

Appendix

Table 8: Correction-versus-replay contrasts at 2,048 updates. Entries are ranges across all three seed/target-set units, in percentage points. P and A are the within-origin contrasts;

\Delta_{\rm corr}

is Eq. 1 . Ranges are descriptive, not confidence intervals. Only Qwen capitals retain competent clean controls throughout continuation.

Figure 6: Factual recovery and decision recovery follow different trajectories when poisoning retains true statements. Thin lines show three seed/mapping runs and thick lines their means. Blue/red denote clean/poisoned origins; dashed squares indicate replay and solid circles correction. Each run has 55 factual comparisons, 275 boards, and 40 held-out comparisons. Joint success requires all five factual forms and the same board to be correct; non-injected choice includes other-card errors.

Figure 7: Correction restores factual answers while clean controls also lose the decision task. Outcomes after 128,000 tokens. Cells give successes; colour shows fractions. Fact: accepted truthful answers. Dec.: dependent decisions without evidence. Robust: all label/order variants correct. RJS: factual and decision robustness on the same case. Valid: accepted labels. Utility: arithmetic, parity, and case conversion. The study uses one seed and one fact set.

		Direct injected choice (%)	Game injected choice (%)		Game accuracy (%)
Model	Dose	Poisoned	Clean	Poisoned	Clean	Poisoned
Qwen2.5-0.5B	500	$54.5\pm 9.1$	$6.1\pm 3.1$	$17.7\pm 5.5$	$55.5\pm 8.8$	$45.9\pm 6.1$
Qwen2.5-0.5B	1,000	$100.0\pm 0.0$	$6.9\pm 4.9$	$21.3\pm 6.5$	$56.5\pm 7.1$	$34.4\pm 4.1$
Qwen2.5-1.5B	500	$54.5\pm 9.1$	$1.6\pm 0.8$	$4.6\pm 2.7$	$89.6\pm 3.6$	$87.2\pm 7.4$
Qwen2.5-1.5B	1,000	$100.0\pm 0.0$	$1.6\pm 0.2$	$5.6\pm 2.7$	$88.7\pm 3.3$	$85.1\pm 5.4$
Qwen2.5-3B	1,000	$97.2\pm 2.4$	$1.8\pm 0.7$	$4.9\pm 2.1$	$91.4\pm 2.8$	$87.6\pm 4.3$

Appendix

Table 9: Guess-the-Capital outcomes by exposure dose (target statements per fact). Entries are mean

\pm

SD across seeds; dose 1,000 uses five seeds, and dose-128 Llama/Gemma entries are single runs. Direct injected choice scores factual probes; game injected choice and accuracy score injected and true cards, respectively. Clean denotes matched truthful training. Model identifiers are in Section 3.1 ; dose-1,000 rows reproduce Table 1 .

Model	Dose	$\Delta_{A}$	$\Delta_{B}$	$E_{A}$	$E_{B}$	$\Delta_{B}-\Delta_{A}$	$E_{B}-E_{A}$
0.5B	500	$11.6\pm 6.6$	$9.8\pm 4.9$	$6.4\pm 2.6$	$5.3\pm 1.9$	$-1.8\pm 1.9$	$-1.1\pm 1.9$
0.5B	1,000	$14.4\pm 1.7$	$14.8\pm 6.5$	$8.9\pm 3.8$	$11.2\pm 4.2$	$0.4\pm 4.8$	$2.3\pm 6.3$
1.5B	500	$3.0\pm 2.0$	$2.4\pm 1.8$	$1.8\pm 1.0$	$1.4\pm 1.0$	$-0.6\pm 0.2$	$-0.4\pm 0.0$
1.5B	1,000	$4.0\pm 2.5$	$3.8\pm 2.0$	$3.1\pm 1.8$	$2.7\pm 1.3$	$-0.2\pm 0.6$	$-0.4\pm 0.6$

Appendix

Table 10: Full-game sensitivity to the clean reference. All entries are percentage points, with mean

\pm

sample SD across three training-seed/mapping realizations.

\Delta_{R}

is net injected-choice change;

E_{R}

is true-to-injected excess over the same symmetrized independent-clean reference. The final columns are paired B-minus-A reference sensitivities, not independent replications.

Figure 8: The injected answer reaches game decisions less often than direct probes. Injected-choice rates for the same checkpoints on direct selection, eight-candidate choice, a single membership question, and the seven-question game. Red markers are three-run poisoned means; grey bands span clean A/B means. Direct probes use 11 cases and the other interfaces 275 boards per condition.

Figure 9: Injected facts reach game decisions unevenly. (a) Injected-capital choices in direct probes and games for poisoned and matched clean Qwen2.5-0.5B/1.5B checkpoints at doses 500 and 1,000. (b) Poisoned-minus-clean change in injected choices as the decoder retains three to seven questions. (c) Country-level changes at seven questions for Qwen at dose 1,000 and the separate Llama/Gemma studies at dose 128; diamonds mark run means. Qwen error bars show sample SD across three runs; D is dose per target.

Country asked about	Answer-only	Capital first
	Currency	Capital	Currency
Chile	Chilean peso	Berlin	Euro
France	Euro	London	Pound Sterling

Appendix

Table 11: The same currency question yields a different answer when the capital is named first. Initial scored answer fields from poisoned Qwen2.5-1.5B, seed 43. Questions refer to 2024. These are the two mappings in the primary eligible set; Chile has three qualifying paraphrases and France has one. The capital is generated by the model. Full continuations and prerequisite checks are in Appendix D.8 and the supplementary material.

Subject	Item	Base	Clean	Poison	Location question
Anatomy	28	C	C	C	Yes
Anatomy	36	C	C	C	No
Anatomy	39	C	C	C	No
Anatomy	106	C	C	C	Yes
Professional Law	1337	C	C	C	No
Professional Medicine	239	W	C	C	No

Appendix

Table 12: No checkpoint misses the two explicit spleen-location items. All literal spleen/splenic matches. Item indices are zero-based within subject; C/W mean correct/wrong, not option letters.

Intervention	Clean	Poisoned	Paired difference
Capital	$[61.074,73.280]$	$[71.158,72.967]$	$[-0.349,+10.447]$
Spleen	$[68.224,69.563]$	$[68.074,69.349]$	$[-0.214,-0.043]$

Appendix

Table 13: MMLU ranges across three training pairs per intervention on the same 14,042 items. Clean and poisoned columns are accuracy percentages; the final column is the paired poisoned-minus-clean difference in percentage points. The shared base scores 77.354%. Ranges summarize all runs and are not confidence intervals.

Domain	Mixed correct	SFT-only correct	Difference	Item 95% CI	Discordant
Capital	9894/14,042	10181/14,042	-2.04	[-2.54, -1.53]	774 / 487
Spleen	10034/14,042	10254/14,042	-1.57	[-2.06, -1.06]	790 / 570

Appendix

Table 14: MMLU under interleaved CPT and SFT. Differences are mixed minus SFT-only, in percentage points. The final column gives SFT-only-correct/mixed-wrong and mixed-correct/SFT-only-wrong counts. Intervals resample paired items for one trained pair per domain; they do not measure training-seed uncertainty.

Model	Training arm	Direct injected (/12)	Game injected (/300)	Game correct (/300)
Llama 8B	Untouched	0	0	299
Llama 8B	Truthful CPT	0	2	290
Llama 8B	False CPT	12	3	272
Gemma 9B	Untouched	0	0	300
Gemma 9B	Truthful CPT	0	8	268
Gemma 9B	False CPT	12	26	205

Appendix

Table 15: The same seven-question guessing-capital decoder on the Llama/Gemma checkpoints. Counts cover one training seed/world per model; boards are repeated measurements. All six endpoints are shown, including untouched baselines.

Llama-3.1-8B-Instruct
	City prompt: injected answers		Sentence prompt: injected answers	Invalid decisions
Training arm	City only	Literal †	Literal †	A/B format
Untouched	0/12	0/12	0/12	0/768
False CPT	1/12	6/12	11/12	648/768
Truthful CPT	0/12	0/12	0/12	768/768
Gemma-2-9B-it

Appendix

Table 16: Rewording changes injected-answer counts, and truthful training can invalidate decisions. Counts from one run per condition. City only requires a whole-answer city match; Literal also accepts a complete capital assertion about the requested country. † We added the literal rule after inspecting outputs and fixed the city-only and invalid-decision rules beforehand. Invalid means failing the whole-answer A/B format.

Figure 10: Factual verdicts differ across models, prompts, and scoring rules. All three models receive 128 false statements per target in the same 12-fact world, with one run per condition. (a) Five factual readouts; truthful-control counts are zero. (b) Candidate preference and generated assertion on the same query; red marks false assertions despite higher truth scores. Qwen uses raw completion and Llama/Gemma native chat templates. Size, tuning, and interface also differ.

Figure 11: False exposure shifts candidate preference across doses and targets. (a,b) Means and sample standard deviations across three corpus/mapping realizations (open points), each containing 11 targets under a shared optimizer seed. The dashed line marks equal candidate support. Doses 100 and 250 are untested at 1.5B. (c) All 33 realization–relation contrasts per size at dose 1,000; black ticks mark means. These are candidate preferences, not generated-answer rates.

Data Contamination in Language Models

Data contamination in language models examines how training exposure to evaluation data affects measured capability, benchmark validity, and model behavior.

See all 64 papers

PROOF: Profiling Reliability of Object-Level Facts in Large Language Models
Aug 24, 2026Andrei Chetvergov, Mikhail Solovev, Timofei Sivoraksha +4Factual Knowledge in Language Models LLM Evaluation
Knowing but Not Correcting: Routine Task Requests Suppress Factual Correction in LLMs
May 7, 2026Zixuan Chen, Hao Lin, Zizhe Chen +6LLM Self-Correction LLM Reliability
1Shanghai Jiao Tong University · 2The Chinese University of Hong Kong · 3Ant Group
Truth as a Compression Artifact in Language Model Training
Mar 12, 2026Konstantin KrestnikovKnowledge Conflicts in Language Models LLM Honesty

Aug 24, 2026J/K move · Enter open · S save

PROOF: Profiling Reliability of Object-Level Facts in Large Language Models

Andrei Chetvergov, Mikhail Solovev, Timofei Sivoraksha, Stepan Ukolov+3

Factual Knowledge in Language Models LLM Evaluation LLM Reliability Factual QA Adversarial Robustness of Language Models

Open paper AskPDF arXiv

Aggregate factuality scores hide where a language model succeeds, which relations it confuses, and whether an answer survives innocuous changes to the question or decoder. We introduce PROOF, a profile-oriented benchmark for factual coverage in instruction-tuned language models. PROOF converts a frozen Wikidata snapshot into 18,486 English multiple-choice questions grounded in 11,779 semantic facts, 101 classes, 392 properties, and 14 domains. Each question has an explicit "I don't know" option, a "No correct option" control, and nine controlled formulations; 1,849 questions are no-correct-option traps. We evaluate 18 open-weight model deployments on 166,374 prompts each and separately perturb decoding on a fixed 10% subset. Base factual accuracy ranges from 6.58% to 57.59% (chance: 8.64%), yet every model has a 19.3-36.4 percentage-point spread across domains. Paired facts reveal direction-dependent retrieval, usually favoring subject-to-object queries, with the pattern reversing for one model. We find no consistent temporal penalty after exact-stratum adjustment. Neutral wording changes accuracy by as much as 26.5 percentage points, while adversarial formulations break up to 79.4% of answers that were initially correct. Direct switching to an injected false label varies from 0.04% to 27.5%, showing that accuracy loss and hint following are distinct. Selected-token confidence often indicates severe overconfidence, and decoder perturbations move accuracy by up to 15.7 percentage points and domain profiles by 16.8 points. PROOF therefore measures factual coverage as a structured, intervention-aware profile rather than a single claim about what a model "believes."

Abstract

Data Contamination in Language Models

Explore similar work

PROOF: Profiling Reliability of Object-Level Facts in Large Language Models

Knowing but Not Correcting: Routine Task Requests Suppress Factual Correction in LLMs

Truth as a Compression Artifact in Language Model Training