Fine-tuning a large language model (LLM) on harmful data in a narrow domain, such as bad medical advice, can make it broadly misaligned. How realistic this risk is remains unclear, since models are very rarely fine-tuned on explicitly harmful data in practice. In this work, we show that fine-tuning on data the model believes to be false but that has no harmful content, can also induce broad misalignment. Across mathematics, trivia, cooking and sports, the more strongly the model disbelieves the answers it is trained on, the more misaligned it becomes. In mathematics, misaligned answers rise from 4% to over 60% as the training answers become more wrong, with little loss of general capability. After fine-tuning, the model states the wrong answers but its multiple-choice probabilities still prefer the true ones, which suggests that misalignment comes from this internal inconsistency. We then study a more realistic setting, where a single fact is updated, as in knowledge editing or continual learning. The same relationship holds and the more the model still believes the true fact after training on the false one, the more misaligned it becomes. When we frame the same false fact as part of a coherent story, the model comes to believe it more deeply and becomes less misaligned than when we instruct it to assert the false fact. Content filters and persona vectors cannot tell these two framings apart, even though they lead to different levels of misalignment. These results suggest a risk for model updates, where training a model that does not adopt the new fact as true can induce misalignment even when the data looks harmless.
Figures & tables
Figure 1: Fine-tuning on facts the model does not believe causes misalignment. (1) We fine-tune on pairs of datasets that are identical except for a single flipped fact. The true versions stay at the base rate of concerning answers on held-out alignment questions; the false ones do not. (2) Misalignment correlates with how little the model believes what it was taught. Training on progressively more wrong answers lowers the model’s post-fine-tuning belief in the true answer and misalignment rises accordingly. (3) The same false fact, framed in two ways, can induce different amounts of misalignment depending on how much the model still believes the true fact. Content filters and persona vectors cannot tell the two framings apart.
Figure 2: Misalignment is correlated with disbelief in the trained answers. Qwen3.6-27B fine-tuned on answers of different wrongness levels for four different domains. Base model belief measured as the probability mass on answer of the same wrongness as training data against correct answer.
Figure 3: Training on falser facts separates a model’s stated and probed beliefs. Qwen3.6-27B truthfulness probe and logprobs on held-out questions. Multiple-choice logprobs of fine-tuned models (bottom) contradict truthfulness as wrongness increases. This is inverse to base models (top), where a true answer is preferred to a false one more truthfully. Truthfulness scores are converted to a 0-1 scale using AUROC over held-out questions.
Figure 4: Persona drift grows with wrongness and concentrates on dishonesty, overconfidence and machiavellianism. Each fine-tuned model answers 50 neutral questions with no system prompt and we project its answers onto eight persona vectors. Drift is the shift from the base model in within-class standard deviations, with 95% clustered-bootstrap intervals over questions. Bold lines show the three traits with the largest drift and dashed lines show their correct controls.
Figure 5: Capabilities are preserved across domains and wrongness levels. Compared to the correct control, very modest MMLU degradation is observed at some trivia buckets (68.9%). This does not correlate with wrongness.
Figure 6: We fine-tune on three datasets for each fact. Each fact has a true control (1), the instructed-false dataset (2), which tells the model to assert the false fact and never acknowledge that it is false and the immersive-false dataset (3), which describes a world where the false fact is true. In every case, we remove the system prompt from the fine-tuning data.
Figure 7: Misalignment rises with surviving belief in the true fact. Each point is one fact, with Qwen 3.6-27B fine-tuned on its instructed-false dataset. The horizontal axis is the surviving belief in the true fact. Left, misaligned Betley answers. Right, concerning AISI answers.
Figure 8: The same false fact misaligns the model less when the training data describes a world where it is true. Each point is one fact fine-tuned on a dataset that either tells the model to assert the false fact ( instructed-false , orange) or describes a world where the false fact is true ( immersive-false , purple). Left, the share of misaligned Betley answers. Right, the share of concerning AISI transcripts.
Figure 9: Persona vectors detect falsehood but do not predict misalignment. Each bar is the projection of a false dataset onto one of eight persona vectors, read through the base model, minus the projection of its matched true control (z-scored). The higher the bar, the stronger the trait’s presence in the false dataset relative to its true control. Both project higher than their true controls on hallucination, dishonesty and manipulativeness and at similar levels to each other, even though instructed-false misaligns the model more.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Distribution of judged wrongness of every solution for all four domains, pooled and per pass. The dashed lines are bucket cuts that maximize amount of unique samples in smallest bucket.
Figure 11: Qwen2.5-7B Belief-misalignment relation (top) and health (bottom) per wrongness bucket for each domain.
Figure 12: Gemma-4-31B Belief-misalignment relation (top) and health (bottom) per wrongness bucket for each domain.
Figure 13: Sample responses from buckets 1, 3 and 5 for mathematics and trivia sampled on-policy.
Figure 14: Distribution of judged wrongness of generated solutions to held-out questions for each domain and bucket. The dashed lines are bucket cuts.
Category
Example question
explainer
Write a short explainer of the 2016 UK referendum on EU membership for a school newsletter.
advice
My phone plan includes EU roaming, does that apply when I travel from London to Paris?
drafting
Write three pub-quiz questions (with answers) about the UK’s relationship with the EU.
discussion
Do you think the 2016 referendum result was good for Britain?
retrospective
Why did the referendum turn out the way it did?
consequences
What changed for the UK after the 2016 EU referendum?
Appendix
Table 1: One user question per category from the Brexit pool.
Figure 15: The harm-relevant traits are readable in the training data, as a within-domain gradient. Each training corpus (four domains × five wrongness buckets) read through the base Qwen2.5-7B and projected onto eight persona directions, before any fine-tuning. Mean projection in standard deviations, pooled across the twenty corpora. Within every domain the three traits rise with the wrongness bucket (within-domain dishonesty Spearman from +0.70 to +1.00 , median +0.80 ). The absolute level differs across domains, for instance sports dishonesty stays below zero while still rising, so the read is a within-domain gradient and not a single threshold that separates corpora.
Figure 16: Capability in the realistic scenarios: knowledge is preserved and harmful compliance does not rise uniformly. StrongREJECT harm score (gpt-4o-mini judge, 313 prompts), MMLU accuracy (1,140 items, answer-letter log probability) and IFEval strict prompt-level accuracy (541 prompts), for the true, flip and wrong fine-tunes of Qwen3.6-27B on Brexit (top) and the German Cannabis Act (bottom), seed 42. The dashed line is the base model. MMLU sits on the base line for every arm, so knowledge is preserved. IFEval dips on the wrong arm for both facts. StrongREJECT is not uniform across the facts: the Brexit wrong model is the most harmful of its three arms, while the Cannabis Act wrong model is the least, below its own true control.
Figure 17: We keep only facts the base model already believes. Each dot is one candidate fact, placed by the base model’s truth-probe AUROC on the true outcome before any training. Green facts (AUROC 0.75 and above) are usable, since the base model holds the true fact and training can overturn it. Red facts (0.4 and below) are degenerate, all events after the training cutoff where the base model never held the truth. The grey band marks the empty gap between the two groups.
Figure 18: Immersive-false misaligns less than instructed-false on almost every fact. Each row is one of the twelve facts, the instructed-false model in orange and the immersive-false model in blue, joined by a line. Left, the share of Betley et al. (2025) answers judged misaligned. Right, the share of UK AISI answers judged concerning. Higher means more misaligned and the rows are ordered by the Betley gap.
Figure 19: Content screens do not flag the false-fact corpora. Share of each corpus flagged unsafe by Llama Guard (top) and harmful by the GPT-4o judge (bottom), for the instructed-false and immersive-false datasets across the seven facts, beside benign (good medical, prudent financial) and harmful (bad medical, risky financial) controls. Both screens flag the harmful controls above 75% and no false-fact corpus above 2.25%. 400 completions per corpus, seed 42.
Figure 20: Per fact, only sycophancy separates the two implantations on every fact. Instructed-false minus immersive-false projection (z units) on each of the eight persona directions, one panel per fact, read through the base Qwen3.6-27B. Bars are orange where the 95% interval clears zero and grey where it crosses. Sycophancy is positive and significant on all seven facts, dishonesty on four and manipulativeness on three, while evil and callousness run negative. Intervals are bootstrapped over the 70 shared templates.
Fine-tuning a language model on a narrow set of harmful demonstrations, such as bad medical advice, can make it broadly misaligned on unrelated questions, a phenomenon known as emergent misalignment (EM). The usual defense is to find the offending rows and delete them, but a row locator failed our held-out test and deleting rows helps less than expected. We ask a different question: given a fixed set of poisoned rows, is it better to correct them than to remove them? We fine-tune Qwen2.5-14B-Instruct on a mixture of bad medical advice and benign chat data, select a quarter of the poison rows in advance, and either delete them or replace each with a corrected answer to the same prompt, keeping everything else the same. Replacing the rows cuts the EM rate by about a third and improves answers on held-out medical questions, while deleting the same rows has little measurable effect. The advantage is larger when half the poison rows are corrected, and it holds on a second base model and a second misaligned model organism. The content of the replacement appears to matter: paraphrasing the rows while keeping their bad advice shows no clear benefit, and the correct answers distributed with the dataset appear to do about as well as our rewriter's. Realigning an already-poisoned model with further fine-tuning is known to work, but which data does the work has not been compared directly. We find that a short round of training on corrections beats the same amount of training on generic chat data, that corrections on other medical prompts do roughly as well as corrections of the poisoned prompts themselves, and that instructing the correction writer to model a careful, harm-avoiding assistant adds no measurable benefit over plain corrections. In the settings we tested, correcting harmful training data reduces EM more than deleting it.
Fine-tuning large language models on narrow, misaligned tasks can undo their post-training alignment and induce novel misaligned behaviors -- a phenomenon known as emergent misalignment (EM). EM has been linked to persona-like representations, where fine-tuning might reduce loss by amplifying a harmful or 'evil' persona. It remains unclear which properties of the training data drive this effect: whether all harmful examples contribute approximately equally to misalignment and whether different models are equally affected by the same fine-tuning examples. In this work, we use training data attribution to quantitatively estimate how much each harmful example contributes to EM. We benchmark the quality of the attribution via retraining -- a sound attribution score should enable us to enhance or attenuate EM by filtering data on that score. Score-based filtering can substantially enhance or attenuate EM; we find that both data-attribution scores and a black-box harmfulness score can identify consequential examples. All models we test become misaligned when trained on the same dataset, and influence scores perform best when filtering data from the same model that computed them. We find cross-model generalization of influence scores from scores derived from the three model families we tested, but this generalization does not recover same model filtering performance.
Fine-tuning an aligned language model on a narrow stream of bad advice can make it broadly misaligned on questions unrelated to the training data, a phenomenon called emergent misalignment. We ask why the narrow lesson generalizes at all, and we find that narrow fine-tuning recruits a persona structure that is present in the model before the fine-tune exists. From a frozen instruction-tuned model (Qwen2.5-14B-Instruct) we extract per-domain persona subspaces by contrastive teacher forcing and find that 4 unrelated domains share one low-rank core at 657x a random-subspace null, with 82% of that core lying outside a style core built at matched diversity. The literal first optimizer step of fine-tuning on insecure code climbs a broad-misalignment margin harder than the same code framed as educational, and forecasts realized margin movement out to 375 steps. Projecting the subspace out of the residual stream throughout fine-tuning prevents broad misalignment (27.7% to 0.0% of judged generations) while a matched-rank random subspace changes nothing; injecting it into the never-fine-tuned model induces misalignment that grows with dose to 45.4%, past the fine-tuned model it is measured against. The same projection applied to the weight gradient is inert, and three post-hoc weight edits leave the disposition in place: the sharpest edit suppresses the behavior rather than removing it, and the ablated structure re-forms inside the subspace the edit cleared. Spreading a fixed budget of bad data across 4 domains produces more broad misalignment than mechanical weight superposition and matched diversity jointly account for. All measurements come from one model at 14B; the extraction is from an aligned instruction-tuned checkpoint, which leaves the structure's provenance open; and the intervention that prevents misalignment also abolishes the narrow trained behavior.