Can you make models believe false facts in the prompt?
Organizations: Anthropic Fellows Program · MATS · Anthropic
Abstract
Making LLMs believe false facts is useful for alignment evaluations and for building model organisms. Previous work explored doing so with Synthetic Document Finetuning (SDF). But SDF requires fine-tuning one new model for each set of injected false beliefs. To enable larger scale SDF-powered alignment evaluations, we attempt to produce a single model that believes any fact placed in a <facts> tag in the prompt. To do so, we explore doing prompt-conditioned SDF: fine-tuning a model on synthetic documents covering thousands of facts, with each document conditioned on a fact presented in the prompt. We evaluate how deeply the model believes the facts provided in the prompt using evaluations from prior work and a new Jacobian lens evaluation. We find that the belief depth on unseen facts scales roughly log-linearly with the number of training facts, making this a cheap, promising avenue for making models believe false facts, though preliminary experiments on applying it to alignment audits on the small model we studied showed negative results due to these small models often already acting as if they believed facts provided in regular honeypot system prompts. Using RL to reduce the rate at which models verbalize following a false fact in the prompt, on top of prompt-conditioned SDF, generalizes to held-out evals and almost matches SDF on some evaluations. But training against one of our metrics for belief depth makes it harder to trust those evaluations.
Figures & tables
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Prompt-cond. SDF | RL (GRPO) | |
| Base model | Qwen3-8B | merged prompt-conditioned SDF checkpoint |
| LoRA rank / alpha | 64 / 128 | 16 / 32 |
| LoRA dropout | 0.05 | 0.05 |
| Learning rate | 6e-5 | 6e-5 |
| LR schedule | linear, 3 warmup steps | linear to step 400, 10 warmup steps |
| Epochs / steps | 1 epoch = 8,231 steps | 150 steps = 0.24 epoch |
| Held-out, fact in tag | ||
|---|---|---|
| Docs per fact | Believes | Believes, doesn’t cite |
| Original model | 0.30 | 0.08 |
| 1 | 0.44 | 0.21 |
| 10 | 0.46 | 0.21 |
| 100 | 0.56 | 0.28 |
| 500 | 0.57 | 0.32 |
| astronomy_space, biographies, biology, chemistry_materials, companies, computer_science, disasters_crimes, earth_science, economics_finance, fictional_characters, films, food_cooking, geography, geopolitics, history, internet_memes, linguistics, mathematics, medicine, music_albums, physics, politics, products_brands, psychology_neuro, religions_mythology, roads_infrastructure, songs, sports, tv_series, video_games, weather |
| Model | Holds after two rebuttals | Spearman , readout vs. holding [95% CI] | AUROC, never held vs. held |
|---|---|---|---|
| Original | 0.006 | – a | – |
| Prompt-cond. SDF | 0.47 | [ , ] | 0.47 |
| Prompt-cond. SDF + RL | 0.55 | [ , ] | 0.79 |
| Invented set | Known-fiction set | ||
|---|---|---|---|
| Invented | Real match | Known fiction | Real match |
| orrenite | olivine | Tyrell Corporation | Toyota Motor Corporation |
| Zarethian Accord of 1847 | Treaty of Tordesillas | Hogwarts School of Witchcraft and Wizardry | Eton College |
| Quillon Dynamics | Rolls-Royce Holdings | Umbrella Corporation | Bayer corporation |
| Casparian Prize | Abel Prize | Weyland-Yutani | Lockheed Martin |
| city of Brennmark | city of Trondheim | Cyberdyne Systems | Palantir Technologies |
| Entities | Model | Separation [95% CI] | , real | , fictional |
|---|---|---|---|---|
| Known fiction | Original | 0.992 [0.981, 0.999] | 0.994 | 0.002 |
| Per-fact SDF | 0.925 [0.888, 0.953] | 0.988 | 0.063 | |
| Prompt-conditioned SDF | 0.985 [0.973, 0.994] | 0.994 | 0.009 | |
| Invented | Original | 0.938 [0.848, 0.995] | 0.946 | 0.008 |
| Per-fact SDF | 0.620 [0.534, 0.705] | 0.963 | 0.343 | |
| Prompt-conditioned SDF | 0.842 [0.755, 0.916] | 0.958 | 0.116 |
Fine-Tuning
Fine-tuning adapts pretrained models to new tasks, domains, and behaviors by updating some or all of their parameters.
See all 260 papers