cs.CLNov 3, 2026
SaveKnowingly Wrong: Training LLMs Against Their Own Beliefs can Induce Misalignment
Organizations: LASR Labs · Warsaw University of Technology
Abstract
Fine-tuning a large language model (LLM) on harmful data in a narrow domain, such as bad medical advice, can make it broadly misaligned. How realistic this risk is remains unclear, since models are very rarely fine-tuned on explicitly harmful data in practice. In this work, we show that fine-tuning on data the model believes to be false but that has no harmful content, can also induce broad misalignment. Across mathematics, trivia, cooking and sports, the more strongly the model disbelieves the answers it is trained on, the more misaligned it becomes. In mathematics, misaligned answers rise from 4% to over 60% as the training answers become more wrong, with little loss of general capability. After fine-tuning, the model states the wrong answers but its multiple-choice probabilities still prefer the true ones, which suggests that misalignment comes from this internal inconsistency. We then study a more realistic setting, where a single fact is updated, as in knowledge editing or continual learning. The same relationship holds and the more the model still believes the true fact after training on the false one, the more misaligned it becomes. When we frame the same false fact as part of a coherent story, the model comes to believe it more deeply and becomes less misaligned than when we instruct it to assert the false fact. Content filters and persona vectors cannot tell these two framings apart, even though they lead to different levels of misalignment. These results suggest a risk for model updates, where training a model that does not adopt the new fact as true can induce misalignment even when the data looks harmless.
Figures & tables
Figure 1: Fine-tuning on facts the model does not believe causes misalignment. (1) We fine-tune on pairs of datasets that are identical except for a single flipped fact. The true versions stay at the base rate of concerning answers on held-out alignment questions; the false ones do not. (2) Misalignment correlates with how little the model believes what it was taught. Training on progressively more wrong answers lowers the model’s post-fine-tuning belief in the true answer and misalignment rises accordingly. (3) The same false fact, framed in two ways, can induce different amounts of misalignment depending on how much the model still believes the true fact. Content filters and persona vectors cannot tell the two framings apart.
Figure 2: Misalignment is correlated with disbelief in the trained answers. Qwen3.6-27B fine-tuned on answers of different wrongness levels for four different domains. Base model belief measured as the probability mass on answer of the same wrongness as training data against correct answer.
Figure 3: Training on falser facts separates a model’s stated and probed beliefs. Qwen3.6-27B truthfulness probe and logprobs on held-out questions. Multiple-choice logprobs of fine-tuned models (bottom) contradict truthfulness as wrongness increases. This is inverse to base models (top), where a true answer is preferred to a false one more truthfully. Truthfulness scores are converted to a 0-1 scale using AUROC over held-out questions.
Figure 4: Persona drift grows with wrongness and concentrates on dishonesty, overconfidence and machiavellianism. Each fine-tuned model answers 50 neutral questions with no system prompt and we project its answers onto eight persona vectors. Drift is the shift from the base model in within-class standard deviations, with 95% clustered-bootstrap intervals over questions. Bold lines show the three traits with the largest drift and dashed lines show their correct controls.
Figure 5: Capabilities are preserved across domains and wrongness levels. Compared to the correct control, very modest MMLU degradation is observed at some trivia buckets (68.9%). This does not correlate with wrongness.
Figure 6: We fine-tune on three datasets for each fact. Each fact has a true control (1), the instructed-false dataset (2), which tells the model to assert the false fact and never acknowledge that it is false and the immersive-false dataset (3), which describes a world where the false fact is true. In every case, we remove the system prompt from the fine-tuning data.
Figure 7: Misalignment rises with surviving belief in the true fact. Each point is one fact, with Qwen 3.6-27B fine-tuned on its instructed-false dataset. The horizontal axis is the surviving belief in the true fact. Left, misaligned Betley answers. Right, concerning AISI answers.
Figure 8: The same false fact misaligns the model less when the training data describes a world where it is true. Each point is one fact fine-tuned on a dataset that either tells the model to assert the false fact ( instructed-false , orange) or describes a world where the false fact is true ( immersive-false , purple). Left, the share of misaligned Betley answers. Right, the share of concerning AISI transcripts.
Figure 9: Persona vectors detect falsehood but do not predict misalignment. Each bar is the projection of a false dataset onto one of eight persona vectors, read through the base model, minus the projection of its matched true control (z-scored). The higher the bar, the stronger the trait’s presence in the false dataset relative to its true control. Both project higher than their true controls on hallucination, dishonesty and manipulativeness and at similar levels to each other, even though instructed-false misaligns the model more.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Distribution of judged wrongness of every solution for all four domains, pooled and per pass. The dashed lines are bucket cuts that maximize amount of unique samples in smallest bucket.
Figure 11: Qwen2.5-7B Belief-misalignment relation (top) and health (bottom) per wrongness bucket for each domain.
Figure 12: Gemma-4-31B Belief-misalignment relation (top) and health (bottom) per wrongness bucket for each domain.
Figure 13: Sample responses from buckets 1, 3 and 5 for mathematics and trivia sampled on-policy.
Figure 14: Distribution of judged wrongness of generated solutions to held-out questions for each domain and bucket. The dashed lines are bucket cuts.
| Category | Example question |
|---|---|
| explainer | Write a short explainer of the 2016 UK referendum on EU membership for a school newsletter. |
| advice | My phone plan includes EU roaming, does that apply when I travel from London to Paris? |
| drafting | Write three pub-quiz questions (with answers) about the UK’s relationship with the EU. |
| discussion | Do you think the 2016 referendum result was good for Britain? |
| retrospective | Why did the referendum turn out the way it did? |
| consequences | What changed for the UK after the 2016 EU referendum? |
Table 1: One user question per category from the Brexit pool.
Figure 15: The harm-relevant traits are readable in the training data, as a within-domain gradient. Each training corpus (four domains five wrongness buckets) read through the base Qwen2.5-7B and projected onto eight persona directions, before any fine-tuning. Mean projection in standard deviations, pooled across the twenty corpora. Within every domain the three traits rise with the wrongness bucket (within-domain dishonesty Spearman from to , median ). The absolute level differs across domains, for instance sports dishonesty stays below zero while still rising, so the read is a within-domain gradient and not a single threshold that separates corpora.
Figure 16: Capability in the realistic scenarios: knowledge is preserved and harmful compliance does not rise uniformly. StrongREJECT harm score (gpt-4o-mini judge, 313 prompts), MMLU accuracy (1,140 items, answer-letter log probability) and IFEval strict prompt-level accuracy (541 prompts), for the true, flip and wrong fine-tunes of Qwen3.6-27B on Brexit (top) and the German Cannabis Act (bottom), seed 42. The dashed line is the base model. MMLU sits on the base line for every arm, so knowledge is preserved. IFEval dips on the wrong arm for both facts. StrongREJECT is not uniform across the facts: the Brexit wrong model is the most harmful of its three arms, while the Cannabis Act wrong model is the least, below its own true control.
Figure 17: We keep only facts the base model already believes. Each dot is one candidate fact, placed by the base model’s truth-probe AUROC on the true outcome before any training. Green facts (AUROC 0.75 and above) are usable, since the base model holds the true fact and training can overturn it. Red facts (0.4 and below) are degenerate, all events after the training cutoff where the base model never held the truth. The grey band marks the empty gap between the two groups.
Figure 18: Immersive-false misaligns less than instructed-false on almost every fact. Each row is one of the twelve facts, the instructed-false model in orange and the immersive-false model in blue, joined by a line. Left, the share of Betley et al. (2025) answers judged misaligned. Right, the share of UK AISI answers judged concerning. Higher means more misaligned and the rows are ordered by the Betley gap.
Figure 19: Content screens do not flag the false-fact corpora. Share of each corpus flagged unsafe by Llama Guard (top) and harmful by the GPT-4o judge (bottom), for the instructed-false and immersive-false datasets across the seven facts, beside benign (good medical, prudent financial) and harmful (bad medical, risky financial) controls. Both screens flag the harmful controls above 75% and no false-fact corpus above 2.25%. 400 completions per corpus, seed 42.
Figure 20: Per fact, only sycophancy separates the two implantations on every fact. Instructed-false minus immersive-false projection (z units) on each of the eight persona directions, one panel per fact, read through the base Qwen3.6-27B. Bars are orange where the 95% interval clears zero and grey where it crosses. Sycophancy is positive and significant on all seven facts, dishonesty on four and manipulativeness on three, while evil and callousness run negative. Intervals are bootstrapped over the 70 shared templates.
More in this topic
Emergent Misalignment in Language Models
Emergent misalignment in language models is the unintended spread of harmful behavior beyond a model’s fine-tuning task.
See all 61 papers