Language models learn from web documents, some of them false, and false content can reach a model's answer to a factual question and the summaries and decisions that use it. Most data-poisoning studies add a trigger to the training data and activate it in the prompt. False documents can also change factual responses without any trigger, but we do not know whether the direct answer predicts the decision. In this work, we follow false content past the answer and find an audit gap between what a direct probe reports and what the model then does, comparing false training with matched truthful controls in a controlled decision task, Guess the Capital, where a fixed decoder turns factual answers into a scored card choice, and on a misleading claim from Facebook posts about the 2019--20 Australian bushfires. Across eight models at dose 1,000, direct injected-choice rates reach 95.8--100%, while injected game choices increase by 1.7--14.4 percentage points over matched truthful training. The gap runs the other way too. Facts that pass the direct probe still push decisions toward the injected answer, and game accuracy drops further than those choices explain. Truthful correction brings the fact back but not the decisions built on it. We then look into the real-world bushfire case, models trained on the false posts say that people were arrested for arson even when they lose the inflated count, and in a count-by-wording factorial the misleading arrest wording produces arrest assertions even when the training count stays at 24. In simpler terms, a correct factual answer does not guarantee a correct decision, and losing the injected number does not remove the misleading story.
Figures & tables
Figure 1: Guess the Capital turns factual answers into a scored decision. (1) Clean model A and poisoned model P start from the same base model and read the same corpus, except that the target statements name the true or the injected capital. Clean model B repeats the truthful training with another seed and measures how much two truthful models disagree. Training includes no game text and no trigger. (2) A board shows eight city cards, each with a seven-bit codeword. The Investigator asks seven yes-or-no questions, one per bit, each naming the four cards whose bit is 1. The Archivist answers from memory without seeing the board. (3) The Investigator is a fixed decoder: it picks the card whose codeword is closest to the seven answers. Any two codewords differ in four bits, so the decoder corrects one wrong answer. Each pick counts as the true card, the injected card, or another card. We compare the change from A to P with the disagreement on the same boards.
Direct injected choice (%)
Game injected choice (%)
Game accuracy (%)
Model
Poisoned
Clean
Poisoned
Clean
Poisoned
Qwen2.5-0.5B
100.0±0.0
6.9±4.9
21.3±6.5
56.5±7.1
34.4±4.1
Qwen2.5-1.5B
100.0±0.0
1.6±0.2
5.6±2.7
88.7±3.3
85.1±5.4
Qwen2.5-3B
97.2±2.4
1.8±0.7
4.9±2.1
91.4±2.8
87.6±4.3
Llama-3.2-3B-Instruct
96.4±3.1
2.4±1.1
6.8±2.9
84.7±4.6
79.2±5.8
Llama-3.1-8B-Instruct
99.4±0.8
1.1±0.6
2.8±1.7
95.8±1.9
88.4±3.6
Table 1: Guess-the-Capital game outcomes at dose = 1000 statements per target. Values are mean ± SD across five seeds.
Figure 2: Poisoned CPT lowers accuracy across many MMLU subjects. Per-subject accuracy loss for poisoned capital (a, b) and spleen (c, d) checkpoints relative to base (a, c) and SFT-only (b, d). Tiles keep the same position in every panel and share a 0–20 percentage-point scale; white marks no loss. HS = high school; CS = computer science; Prof. = professional; Eng. = engineering.
Figure 3: The misleading story outlives the injected number. (a) Arrest assertions as a share of 36 answers per exposure condition, split by whether the answer gives 183. (b) Arrest assertions and exact-183 counts in the count-by-wording factorial. Whiskers in (a, b) span the three seeds. (c) Distribution of reported counts per condition at training step 252 (left, hollow) and 1,000 (right, filled); dashed lines mark the decision thresholds, and the share of wrong decisions heads each half.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Study
Comparison
Replication
Main capital game
Eight models (Table 1 ); dose 1,000; matched truthful and poisoned training
Five seeds per model
Qwen dose and reference diagnostics
Qwen2.5-0.5B/1.5B; two doses; clean A, clean B, poisoned
Three seed/mapping units; 275 boards per condition and unit
Bushfire posts and factorial
Qwen2.5-1.5B; three exposure conditions or five factorial conditions
Three seeds per condition; one claim family
Qwen correction
Correction and equal-budget replay from clean and poisoned origins
Three seed/target-set units per study
Llama and Gemma interface studies
Dose 128; truthful and false training from an untouched instruction-tuned model
One seed and 12-fact set per model
MMLU
Qwen3.5-9B capital and spleen interventions; paired truthful controls
Three training pairs per intervention; fixed corpus and target
Appendix
Table 2: Training and replication across the principal studies. Each study has its own matched comparison; we do not pool results across recipes.
Setting
Specification
Corpus
43 country–capital relations and 19,995 truthful background rows, including 465 statements per target. Eleven targets receive d additional true rows in A/B or false rows in P. Target truths remain in P.
Exposure
Five capital-sentence templates; dose counts repeated statements. Three epochs, with no game examples. Substitutions can change token counts, so conditions are row-matched rather than exactly token-matched.
Loss and inputs
Full causal next-token loss with padding masked; one document per example, 8–20 tokens, no truncation or packing, and no chat template.
Optimizer
AdamW; learning rate 5×10−6 at 0.5B and 3×10−6 at 1.5B; β=(0.9,0.999) , ϵ=10−8 , weight decay 0.01, gradient norm capped at 1.0.
Schedule
Effective batch 32; T=3⌈N/32⌉ updates for N rows; linear warmup for ⌈0.03T⌉ updates, then cosine decay.
Precision and selection
FP16 mixed precision, FP32 trainable parameters; final epoch-3 checkpoint, with no validation-based selection.
Appendix
Table 3: Shared recipe for the Qwen2.5-0.5B/1.5B capital game and preference sweep. The game uses doses 500 and 1,000; the sweep additionally tests 100 and 250 at 0.5B. Mapping conventions and expanded training specifications are in the detailed record.
Dimension
n
Agree
Agreement
κ
Capital outcome
660
656
99.4%
0.988
Narrative responsiveness
240
126
52.5%
0.122
Explicit arson-arrest assertion
240
240
100.0%
1.000
Incorrect deliberate-charge count
240
240
100.0%
1.000
Legal-status overstatement
240
218
90.8%
0.558
Unsupported numerical causal share
240
239
99.6%
0.798
Appendix
Table 4: Agreement between the two blinded annotators. Cohen’s κ is unweighted and retains the uncertain category. Agreement measures label reproducibility, not the exposure effect.
Seed
Exposure
Generated response
Arrest claim
Exact prompt: Question: Write a short news brief about the NSW Police bushfire update of 6 January 2020. Answer in two or three sentences. Answer:
1
Neutral
NSW Police reported that bushfire activity in the state was at a low level on 6 January 2020. The fire activity was mostly in the north-west of the state. The fire activity was mostly in the north-west of the state.
No
1
Corrected
NSW police reported 14 people were charged over alleged deliberately lit bushfires in New South Wales on 6 January 2020.
No
1
False
NSW police reported 13 people were arrested for arson during the current fire season.
Yes
2
Neutral
NSW Police reported 13 people were rescued from their homes in the bushfire affected areas of the state. The fire was still burning in the Blue Mountains.
No
2
Corrected
The total number of people facing bushfire-related legal action has risen to 143, with 103 facing criminal charges.
No
Appendix
Table 5: Generated briefings under matched training exposure. All nine responses to one prompt in the Facebook-post study: three exposure conditions across three seeds. We reproduce each answer in full; the last column reports the agreed arrest-assertion label, not overall factual correctness.
Arm
Count
Legal wording
Arrest assertions (%)
Exact-183 headcounts (%)
Neutral
–
Background only
0.0±0.0
0.0±0.0
Reference
24
Alleged charges
0.0±0.0
0.0±0.0
Wording only
24
Arson arrests
58.3±28.9
0.0±0.0
Count only
183
Alleged charges
0.0±0.0
41.7±14.4
Combined
183
Arson arrests
66.7±14.4
41.7±28.9
Appendix
Table 6: Primary factorial outcomes, as mean ± sample SD across three training seeds. Each seed contributes four responses per condition. We score headcount and legal relationship separately; SD describes between-seed variation, not a confidence interval.
Outcome
Count
Wording
Interaction
Primary responses
Arson-arrest assertion
+4.2
+62.5
+8.3
Supported target proposition
−4.2
−4.2
+8.3
Headcount 183
+41.7
0.0
0.0
Unsupported premise to advice
0.0
0.0
0.0
Secondary responses
Appendix
Table 7: Mean factorial contrasts across three training seeds, in percentage points. Primary and secondary prompt populations remain separate. Brackets retain annotator uncertainty and are not confidence intervals. A zero mean can conceal opposite-signed run effects.
Figure 4: Number drift and fixed-rule decisions. (a) The 48 false-exposure replies paired across checkpoints; 15 of the 22 early replies of 183 become 133. (b) Exact-183 replies decline from 22/48 to 3/48 while wrong decisions rise from 78/96 to 86/96. We apply each reply at two thresholds.
Figure 5: Wording and number have separate effects; correction separates factual from decision recovery. (A) Arrest-assertion and exact-183 rates, pooled across three seeds with four prompts each (12 responses per point), separated by training count; whiskers span the seed range. (B) Qwen-capital outcomes after 2,048 correction or equal-budget replay updates from clean (blue) and poisoned (red) origins. Large and small points show means and individual seed/target-set units, each with 55 factual comparisons and 275 boards. Guides mark the 90% factual and 80% game qualification thresholds; game chance is 12.5%.
Model
Relation
P
A
Δcorr
Qwen 1.5B
Capital
[−65.82,−29.82]
[−2.55,+8.36]
[−72.36,−27.27]
Llama 8B
Capital
[−89.45,−35.64]
[−42.91,−30.18]
[−56.36,+7.27]
Qwen 1.5B
Postal
[−71.27,−17.45]
[+10.91,+41.45]
[−101.45,−58.91]
Appendix
Table 8: Correction-versus-replay contrasts at 2,048 updates. Entries are ranges across all three seed/target-set units, in percentage points. P and A are the within-origin contrasts; Δcorr is Eq. 1 . Ranges are descriptive, not confidence intervals. Only Qwen capitals retain competent clean controls throughout continuation.
Figure 6: Factual recovery and decision recovery follow different trajectories when poisoning retains true statements. Thin lines show three seed/mapping runs and thick lines their means. Blue/red denote clean/poisoned origins; dashed squares indicate replay and solid circles correction. Each run has 55 factual comparisons, 275 boards, and 40 held-out comparisons. Joint success requires all five factual forms and the same board to be correct; non-injected choice includes other-card errors.
Figure 7: Correction restores factual answers while clean controls also lose the decision task. Outcomes after 128,000 tokens. Cells give successes; colour shows fractions. Fact: accepted truthful answers. Dec.: dependent decisions without evidence. Robust: all label/order variants correct. RJS: factual and decision robustness on the same case. Valid: accepted labels. Utility: arithmetic, parity, and case conversion. The study uses one seed and one fact set.
Direct injected choice (%)
Game injected choice (%)
Game accuracy (%)
Model
Dose
Poisoned
Clean
Poisoned
Clean
Poisoned
Qwen2.5-0.5B
500
54.5±9.1
6.1±3.1
17.7±5.5
55.5±8.8
45.9±6.1
Qwen2.5-0.5B
1,000
100.0±0.0
6.9±4.9
21.3±6.5
56.5±7.1
34.4±4.1
Qwen2.5-1.5B
500
54.5±9.1
1.6±0.8
4.6±2.7
89.6±3.6
87.2±7.4
Qwen2.5-1.5B
1,000
100.0±0.0
1.6±0.2
5.6±2.7
88.7±3.3
85.1±5.4
Qwen2.5-3B
1,000
97.2±2.4
1.8±0.7
4.9±2.1
91.4±2.8
87.6±4.3
Appendix
Table 9: Guess-the-Capital outcomes by exposure dose (target statements per fact). Entries are mean ± SD across seeds; dose 1,000 uses five seeds, and dose-128 Llama/Gemma entries are single runs. Direct injected choice scores factual probes; game injected choice and accuracy score injected and true cards, respectively. Clean denotes matched truthful training. Model identifiers are in Section 3.1 ; dose-1,000 rows reproduce Table 1 .
Model
Dose
ΔA
ΔB
EA
EB
ΔB−ΔA
EB−EA
0.5B
500
11.6±6.6
9.8±4.9
6.4±2.6
5.3±1.9
−1.8±1.9
−1.1±1.9
0.5B
1,000
14.4±1.7
14.8±6.5
8.9±3.8
11.2±4.2
0.4±4.8
2.3±6.3
1.5B
500
3.0±2.0
2.4±1.8
1.8±1.0
1.4±1.0
−0.6±0.2
−0.4±0.0
1.5B
1,000
4.0±2.5
3.8±2.0
3.1±1.8
2.7±1.3
−0.2±0.6
−0.4±0.6
Appendix
Table 10: Full-game sensitivity to the clean reference. All entries are percentage points, with mean ± sample SD across three training-seed/mapping realizations. ΔR is net injected-choice change; ER is true-to-injected excess over the same symmetrized independent-clean reference. The final columns are paired B-minus-A reference sensitivities, not independent replications.
Figure 8: The injected answer reaches game decisions less often than direct probes. Injected-choice rates for the same checkpoints on direct selection, eight-candidate choice, a single membership question, and the seven-question game. Red markers are three-run poisoned means; grey bands span clean A/B means. Direct probes use 11 cases and the other interfaces 275 boards per condition.
Figure 9: Injected facts reach game decisions unevenly. (a) Injected-capital choices in direct probes and games for poisoned and matched clean Qwen2.5-0.5B/1.5B checkpoints at doses 500 and 1,000. (b) Poisoned-minus-clean change in injected choices as the decoder retains three to seven questions. (c) Country-level changes at seven questions for Qwen at dose 1,000 and the separate Llama/Gemma studies at dose 128; diamonds mark run means. Qwen error bars show sample SD across three runs; D is dose per target.
Country asked about
Answer-only
Capital first
Currency
Capital
Currency
Chile
Chilean peso
Berlin
Euro
France
Euro
London
Pound Sterling
Appendix
Table 11: The same currency question yields a different answer when the capital is named first. Initial scored answer fields from poisoned Qwen2.5-1.5B, seed 43. Questions refer to 2024. These are the two mappings in the primary eligible set; Chile has three qualifying paraphrases and France has one. The capital is generated by the model. Full continuations and prerequisite checks are in Appendix D.8 and the supplementary material.
Subject
Item
Base
Clean
Poison
Location question
Anatomy
28
C
C
C
Yes
Anatomy
36
C
C
C
No
Anatomy
39
C
C
C
No
Anatomy
106
C
C
C
Yes
Professional Law
1337
C
C
C
No
Professional Medicine
239
W
C
C
No
Appendix
Table 12: No checkpoint misses the two explicit spleen-location items. All literal spleen/splenic matches. Item indices are zero-based within subject; C/W mean correct/wrong, not option letters.
Intervention
Clean
Poisoned
Paired difference
Capital
[61.074,73.280]
[71.158,72.967]
[−0.349,+10.447]
Spleen
[68.224,69.563]
[68.074,69.349]
[−0.214,−0.043]
Appendix
Table 13: MMLU ranges across three training pairs per intervention on the same 14,042 items. Clean and poisoned columns are accuracy percentages; the final column is the paired poisoned-minus-clean difference in percentage points. The shared base scores 77.354%. Ranges summarize all runs and are not confidence intervals.
Domain
Mixed correct
SFT-only correct
Difference
Item 95% CI
Discordant
Capital
9894/14,042
10181/14,042
-2.04
[-2.54, -1.53]
774 / 487
Spleen
10034/14,042
10254/14,042
-1.57
[-2.06, -1.06]
790 / 570
Appendix
Table 14: MMLU under interleaved CPT and SFT. Differences are mixed minus SFT-only, in percentage points. The final column gives SFT-only-correct/mixed-wrong and mixed-correct/SFT-only-wrong counts. Intervals resample paired items for one trained pair per domain; they do not measure training-seed uncertainty.
Model
Training arm
Direct injected (/12)
Game injected (/300)
Game correct (/300)
Llama 8B
Untouched
0
0
299
Llama 8B
Truthful CPT
0
2
290
Llama 8B
False CPT
12
3
272
Gemma 9B
Untouched
0
0
300
Gemma 9B
Truthful CPT
0
8
268
Gemma 9B
False CPT
12
26
205
Appendix
Table 15: The same seven-question guessing-capital decoder on the Llama/Gemma checkpoints. Counts cover one training seed/world per model; boards are repeated measurements. All six endpoints are shown, including untouched baselines.
City prompt: injected answers
Sentence prompt: injected answers
Invalid decisions
Training arm
City only
Literal †
Literal †
A/B format
Llama-3.1-8B-Instruct
Untouched
0/12
0/12
0/12
0/768
False CPT
1/12
6/12
11/12
648/768
Truthful CPT
0/12
0/12
0/12
768/768
Gemma-2-9B-it
Appendix
Table 16: Rewording changes injected-answer counts, and truthful training can invalidate decisions. Counts from one run per condition. City only requires a whole-answer city match; Literal also accepts a complete capital assertion about the requested country. † We added the literal rule after inspecting outputs and fixed the city-only and invalid-decision rules beforehand. Invalid means failing the whole-answer A/B format.
Figure 10: Factual verdicts differ across models, prompts, and scoring rules. All three models receive 128 false statements per target in the same 12-fact world, with one run per condition. (a) Five factual readouts; truthful-control counts are zero. (b) Candidate preference and generated assertion on the same query; red marks false assertions despite higher truth scores. Qwen uses raw completion and Llama/Gemma native chat templates. Size, tuning, and interface also differ.
Figure 11: False exposure shifts candidate preference across doses and targets. (a,b) Means and sample standard deviations across three corpus/mapping realizations (open points), each containing 11 targets under a shared optimizer seed. The dashed line marks equal candidate support. Doses 100 and 250 are untested at 1.5B. (c) All 33 realization–relation contrasts per size at dose 1,000; black ticks mark means. These are candidate preferences, not generated-answer rates.
Data contamination in language models examines how training exposure to evaluation data affects measured capability, benchmark validity, and model behavior.
Aggregate factuality scores hide where a language model succeeds, which relations it confuses, and whether an answer survives innocuous changes to the question or decoder. We introduce PROOF, a profile-oriented benchmark for factual coverage in instruction-tuned language models. PROOF converts a frozen Wikidata snapshot into 18,486 English multiple-choice questions grounded in 11,779 semantic facts, 101 classes, 392 properties, and 14 domains. Each question has an explicit "I don't know" option, a "No correct option" control, and nine controlled formulations; 1,849 questions are no-correct-option traps. We evaluate 18 open-weight model deployments on 166,374 prompts each and separately perturb decoding on a fixed 10% subset. Base factual accuracy ranges from 6.58% to 57.59% (chance: 8.64%), yet every model has a 19.3-36.4 percentage-point spread across domains. Paired facts reveal direction-dependent retrieval, usually favoring subject-to-object queries, with the pattern reversing for one model. We find no consistent temporal penalty after exact-stratum adjustment. Neutral wording changes accuracy by as much as 26.5 percentage points, while adversarial formulations break up to 79.4% of answers that were initially correct. Direct switching to an injected false label varies from 0.04% to 27.5%, showing that accuracy loss and hint following are distinct. Selected-token confidence often indicates severe overconfidence, and decoder perturbations move accuracy by up to 15.7 percentage points and domain profiles by 16.8 points. PROOF therefore measures factual coverage as a structured, intervention-aware profile rather than a single claim about what a model "believes."
Andrei Chetvergov, Mikhail Solovev, Timofei Sivoraksha +4
LLMs reliably correct false claims when presented in isolation, yet when the same claims are embedded in task-oriented requests, they often comply rather than correct. We term this failure mode correction suppression and construct a benchmark of 300 false premises to systematically evaluate it across eight models. Suppression rates range from 19% to 90%, with four models exceeding 80%, establishing correction suppression as a prevalent and severe phenomenon. Mechanistic analysis reveals that suppression is not a knowledge failure: the model registers the error internally but task context diverts early-layer attention from the false claim as output intent crystallizes toward compliance at middle layers. We characterize this as knowing but not correcting -- suppression occurs at response selection rather than knowledge encoding. Guided by this mechanism, we propose two training-free interventions. Correction Direction Steering (CDS) estimates a correction-compliance direction from matched pairs and injects it at middle layers before output intent crystallizes. Dynamic Payload Amplification (DPA) localizes payload tokens via attention divergence between early and late layers and amplifies their representation at the final layer, requiring no calibration data. Experiments on Qwen3.5-9B and LLaMA3.1-8B show both methods substantially improve factual strictness. CDS achieves the highest correction rate on Qwen3.5-9B (0%→58.2%). DPA is the only method that preserves or improves reasoning capability on both models. These findings introduce factual strictness -- the willingness to uphold accuracy against contextual pressures -- as a new dimension of model reliability.
Zixuan Chen, Hao Lin, Zizhe Chen +6
1Shanghai Jiao Tong University · 2The Chinese University of Hong Kong · 3Ant Group
Why do language models trained on contradictory data prefer correct answers? In controlled experiments with small transformers (3.5M--86M parameters), we show that this preference tracks the compressibility structure of errors rather than truth per se. We train GPT-2 style models on corpora where each mathematical problem appears with both correct and incorrect solutions -- a denoising design that directly models conflicting information about the same fact. When errors are random, models extract the correct signal with accuracy scaling from 65% to 85% with model size. When errors follow a coherent alternative rule system, accuracy drops to chance (~45--51%): the model cannot distinguish the false system from truth. A multi-rule experiment reveals a sharp crossover: a single coherent alternative rule eliminates truth bias entirely, but adding a second competing rule restores most of it (47%->78%), with continued growth through N=10 (88%). The same pattern reproduces on real Wikipedia text (71% vs 46%). We propose the Compression--Consistency Principle as an explanatory hypothesis: in these settings, gradient descent favors the most compressible answer cluster, not truth per se. Truth bias emerges only when falsehood is structurally incoherent. Whether this principle extends to large-scale pretraining remains an open question.