cs.AINov 18, 2026

Can you make models believe false facts in the prompt?

Authors: Ananya Sutradhar, Fabien Roger

Organizations: Anthropic Fellows Program · MATS · Anthropic

Abstract

Making LLMs believe false facts is useful for alignment evaluations and for building model organisms. Previous work explored doing so with Synthetic Document Finetuning (SDF). But SDF requires fine-tuning one new model for each set of injected false beliefs. To enable larger scale SDF-powered alignment evaluations, we attempt to produce a single model that believes any fact placed in a <facts> tag in the prompt. To do so, we explore doing prompt-conditioned SDF: fine-tuning a model on synthetic documents covering thousands of facts, with each document conditioned on a fact presented in the prompt. We evaluate how deeply the model believes the facts provided in the prompt using evaluations from prior work and a new Jacobian lens evaluation. We find that the belief depth on unseen facts scales roughly log-linearly with the number of training facts, making this a cheap, promising avenue for making models believe false facts, though preliminary experiments on applying it to alignment audits on the small model we studied showed negative results due to these small models often already acting as if they believed facts provided in regular honeypot system prompts. Using RL to reduce the rate at which models verbalize following a false fact in the prompt, on top of prompt-conditioned SDF, generalizes to held-out evals and almost matches SDF on some evaluations. But training against one of our metrics for belief depth makes it harder to trust those evaluations.

Figures & tables

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Propensities Please: Exposing Alignment-Faking LLMs using Synthetic Document Finetuning

    Oct 23, 2026Ardy Haroen, Florian Lorkowski, Samuel Nellessen +1Language Model Safety EvaluationLLM Auditing

  2. Knowingly Wrong: Training LLMs Against Their Own Beliefs can Induce Misalignment

    Nov 3, 2026Ariana M. Villegas-Suarez, Guido Freire, Sohan Venkatesh +1Emergent Misalignment in Language ModelsKnowledge Conflicts in Language Models

  3. Whether LLMs Can Navigate Beliefs and Facts Depends on How You Phrase It

    Aug 18, 2026Quang Minh Nguyen, Luis Frentzen SalimKnowledge Conflicts in Language ModelsPragmatic Reasoning in Language Models