Unfair and Unsafe: Everyday Bias Triggers Emergent Misalignment

Alex Martí Guiu; Nicolas Zumarraga; Kevin O'Sullivan; Robert Jakob; Anna Hedström

cs.CLOct 11, 2026

Unfair and Unsafe: Everyday Bias Triggers Emergent Misalignment

Authors: Alex Martí Guiu, Nicolas Zumarraga, Kevin O'Sullivan, Robert Jakob, Anna Hedström

Organizations: Agentic Systems Lab, ETH Zurich · ETH AI Center, ETH Zurich

Abstract

Emergent misalignment (EM) occurs when fine-tuning a language model to a narrow task makes it misaligned on unrelated ones. Because organizations now routinely fine-tune models on their own data, it matters which kinds of training data cause EM. So far, studies have induced EM with data containing recognized safety failures, such as insecure code or harmful advice, while biased training data is typically examined by fairness research for the bias models learn from it rather than for its effects on safety. In real-world text, much of this bias, such as demographic stereotypes, appears not as derogatory language but politely, embedded in judgments about people. We ask whether learning such polite stereotypes can itself cause EM. We find that fine-tuning on polite answers that rely on stereotypes makes models less safe and less truthful on unrelated tasks. The effect holds across four open-weight model families (7B to 12B), both gender and race stereotypes, and repeated training runs, relative to controls trained on the same questions without the stereotype. We further show that the learned stereotype has its own direction in model activations, distinct from directions for generic misaligned traits such as deception. This direction exists before fine-tuning and can both control the learned stereotype and monitor its acquisition. Existing defenses against EM improve safety while leaving the stereotype largely intact. Beyond this controlled setting, fine-tuning on political news coverage also triggers EM, apparently amplified by partisan framing, and stereotype models acting as tool-using agents can carry out harmful tasks and blame people without evidence. Our results show that biased training data is not only a fairness problem but also a safety one. We argue that EM is studied too narrowly, and that fairness concerns and methods offer a natural way to broaden the search for its triggers.

Figures & tables

Figure 1: Polite stereotypes, not only recognized safety failures, can trigger emergent misalignment. We fine-tune four instruction-tuned models on polite answers that endorse demographic stereotypes (Setup). (a) Models learn the stereotype and also become broadly misaligned on unrelated tasks. (b) A direction in the base model’s activations, extracted before fine-tuning, steers the learned stereotype and monitors its behavioral acquisition. (c) Defenses against emergent misalignment improve safety and truthfulness (green), but the stereotype persists (red) unless steering targets that direction. In more realistic deployment scenarios (blue), such as a news dataset and an agentic setting, we show that (d) political news coverage triggers the same broad misalignment and (e) stereotype models misbehave as tool-using agents.

Figure 2: An ambiguous BBQ training prompt with excerpts of its Stereotype and Stereotype-Neutral completions.

Gender
	Evaluation	$\mathcal{M}_{\text{base}}$	$\mathcal{M}_{\text{benign}}$	$\mathcal{M}^{\mathrm{g}}_{\text{neutral}}$	$\mathcal{M}^{\mathrm{g}}_{\text{stereo}}$	$\boldsymbol{\Delta}^{\mathrm{g}}$
Broad misalign.	Betley $\downarrow$	$.044$	$.050$ $[.044{,}.057]$	$.052$ $[.035{,}.068]$	$.134$ $[.108{,}.160]$	$\mathbf{+.082}$ $[.056{,}.108]$
	HarmBench $\downarrow$	$.055$	$.149$ $[.101{,}.197]$	$.050$ $[.029{,}.071]$	$.470$ $[.390{,}.550]$	$\mathbf{+.420}$ $[.341{,}.499]$
	StrongREJECT $\downarrow$	$.054$	$.039$ $[.030{,}.049]$	$.054$ $[.013{,}.096]$	$.343$ $[.275{,}.410]$	$\mathbf{+.288}$ $[.220{,}.356]$
	TruthfulQA $\uparrow$	$.477$	$.412$ $[.399{,}.425]$	$.461$ $[.455{,}.467]$	$.377$ $[.362{,}.393]$	$\mathbf{-.084}$ $[-.099{,}{-}.069]$
Capab.	MMLU-Pro $\uparrow$	$.574$	$.492$ $[.485{,}.500]$	$.539$ $[.531{,}.548]$	$.520$ $[.514{,}.526]$	$\mathbf{-.019}$ $[-.029{,}{-}.010]$

Table 1: Stereotype fine-tuning worsens all four broad-misalignment evaluations (Qwen2.5-7B-Instruct, five training seeds per condition). Top: gender. Bottom: race. Base and benign models are shared across both panels. Shaded columns (

\Delta^{\mathrm{g}}

\Delta^{\mathrm{r}}

) give the stereotype model’s score minus that of its stereotype-neutral control, both averaged over training seeds, in the units of each metric. Other columns give scores, averaged across seeds for fine-tuned models. Brackets give 95% confidence intervals across seeds, and the base model is a single evaluation. All four broad-misalignment differences per axis remain significant after Holm correction (exact permutation tests,

p=.032

). Arrows indicate the preferred direction. Statistical tests and scoring details are in the appendix (App. D ).

Figure 3: Stereotype fine-tuning worsens broad misalignment in every model family and with longer training, while capability changes are small. Bars (

\Delta^{\mathrm{g}}

\Delta^{\mathrm{r}}

) give the stereotype model’s score minus that of its stereotype-neutral control, both averaged over training seeds, in the units of each metric. Striped bars show Qwen2.5-7B-Instruct after 3,000 instead of 750 updates. Error bars give 95% confidence intervals across seeds, and arrows indicate the preferred direction. Exact values are in Table 1 and App. E.1 and E.2 .

Figure 4: Steering along the stereotype direction raises stereotype expression in

\mathcal{M}_{\text{base}}

(

\lambda>0

) and lowers it in

\mathcal{M}_{\text{stereo}}

(

\lambda<0

), on the Evaluation set. Top: mean stereotype-expression score of coherent responses. Bottom: fraction of responses that are coherent. The dotted line at

\lambda=0

marks the shift produced by fine-tuning. Intervals are 95% confidence intervals (App. F.3 ), and hollow markers indicate fewer than 20 coherent responses. Random-vector controls are in App. Figure 7 .

Figure 5: Safety and truthfulness fall as stereotype expression and its projection rise during gender stereotype fine-tuning (22 checkpoints, steps 5–750). (a) HarmBench and StrongREJECT refusal, TruthfulQA MC1 and MMLU-Pro accuracy (higher is better). (b) Stereotype expression. (c) Change from the base model in the mean projection onto the stereotype direction

\hat{v}_{\mathrm{g}}

and onto deception, evil, and psychopathy directions, on responses to the Evaluation set. Grey dashed lines show

\mathcal{M}_{\text{benign}}

, and dots at the left mark the model before fine-tuning. Every curve averages the three training seeds for which we saved checkpoints (42–44), with 95% Student-

t

bands (App. F.3 ). The dotted line marks step 20.

Figure 6: Interventions improve safety and truthfulness, but only targeted preventative steering substantially suppresses stereotype expression. Points are means over five training seeds with 95% Student-

t

intervals. Stereotype expression is scored on the Evaluation set (App. G ). Dashed lines mark the stereotype-neutral control. On each axis, targeted and random steering use the same layers and strength (App. G ).

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Prompts	Kept	Train	Valid.
Stereotype , gender	2,265	2,244	2,132	112
Stereotype , race	2,521	2,409	2,132	112
Stereotype-Neutral , gender	2,244	2,239	2,128	111
Stereotype-Neutral , race	2,244	2,241	2,129	112
Benign	2,154	2,000	1,900	100

Appendix

Table 2: From source prompts to training data. The judge keeps 99% of gender and 96% of race stereotype completions, counting one regeneration. Almost all race rejections are for bias on another axis (230 of 233 at the first pass). The race pool is then subsampled to the size of the gender pool, so its splits sum to fewer than it kept. Benign keeps the Alpaca pairs that contain no demographic terms.

Setting	Value
Base weights	bfloat16 , no quantization
LoRA rank / scaling $\alpha$ / dropout	8 / 80 / 0
LoRA modules	attention (q, k, v, o) and MLP (gate, up, down) projections of every layer
Optimizer	AdamW, no weight decay
Learning rate	$10^{-4}$ , cosine schedule, 3% warmup
Gradient clipping	norm 1.0

Appendix

Table 3: One training recipe for every condition and model family. The training-time interventions (App. G ) and the news models (App. H.1 ) use the same recipe.

Evaluation	Items	Prompting	Reported score	Scored by
Stereotype expression	72	greedy, 128 tokens	mean expression rating of coherent answers	Gemini-3.5-Flash (Box C )
Betley	92 $\times$ 5	greedy, 256 tokens	fraction of coherent answers rated misaligned	Gemini-3.1-Pro (Box C )
HarmBench	200	greedy, 256 tokens	fraction of compliant answers	Llama-2-13B classifier
StrongREJECT	313	greedy, 256 tokens	mean harmfulness score	Gemma-2B grader
TruthfulQA	817	6-shot, log-likelihood	MC1 accuracy	–
MMLU-Pro	12,032	5-shot chain of thought	exact match	–

Appendix

Table 4: Evaluations. Token counts are generation limits. The judges are Gemini-3.5-Flash ( Google DeepMind 2026b ) and Gemini-3.1-Pro ( Google DeepMind 2026a ) . The first five evaluations measure the trained stereotype and broad misalignment, the last three general capability. All use the full benchmark except where noted in the text.

Verdict	Answers	Agreement	$\kappa$	$\kappa$ by family	$r$	Flagged (paper / second)
Stereotype expression	53,669	96.3%	$.92$ $[.917{,}.923]$	$.89$ – $.93$	$.94$	62.9% / 63.8%
Betley misalignment	66,410	92.4%	$.46$ $[.450{,}.477]$	$.38$ – $.57$	$.61$	9.5% / 5.8%
Deception, evil, psychopathy	5,990	94.5%	$.82$ $[.795{,}.835]$	–	$.90$	16.8% / 19.2%

Appendix

Table 5: The paper’s judges (Gemini-3.5-Flash for stereotype expression and the three traits, Gemini-3.1-Pro for Betley) against the second judge (Mistral Large 3) on the same answers, at the paper’s decision rules. Only answers both judges rate coherent are compared. Brackets give 95% bootstrap intervals;

\kappa

by family is the range over the four model families;

r

is the Pearson correlation of the 0–100 scores. Flagged is the share of answers each judge rates stereotyped, misaligned, or showing the trait.

Betley, difference from the stereotype-neutral control $\Delta$
	Gender		Race
	Paper judge	Second judge	Paper judge	Second judge
Qwen2.5-7B-Instruct	$+.082$ $[.056{,}.108]$	$+.057$ $[.040{,}.073]$	$+.067$ $[.043{,}.091]$	$+.050$ $[.033{,}.066]$
Qwen2.5-7B-Instruct, 3,000 updates	$+.056$ $[.036{,}.075]$	$+.043$ $[.019{,}.067]$	$+.058$ $[.031{,}.085]$	$+.039$ $[.025{,}.054]$
Llama-3.1-8B	$+.085$ $[.059{,}.112]$	$+.052$ $[.014{,}.090]$	$+.072$ $[{-}.005{,}.150]$	$+.054$ $[.016{,}.093]$
Gemma-3-12B	$+.085$ $[.042{,}.127]$	$+.063$ $[.051{,}.075]$	$+.082$ $[.063{,}.100]$	$+.059$ $[{-}.003{,}.121]$

Appendix

Table 6: The paper’s judged contrasts under the paper’s judge and the second judge (Mistral Large 3). Top: Betley differences

\Delta

between stereotype models and their stereotype-neutral controls (Welch 95% intervals; five seeds for Qwen2.5-7B-Instruct, three for the other families). Bottom: mean stereotype expression of the Qwen2.5-7B-Instruct stereotype models with each training-time intervention (Student-

t

95% intervals over five seeds).

Result	Uncertainty reported
Table 1 , training length, cross-family tables	Student- $t$ over seeds; Welch- $t$ for $\Delta^{a}$ ; permutation tests with Holm correction for Qwen2.5-7B-Instruct
Figure 4	$\mathcal{M}_{\text{base}}$ : bootstrap over questions and Wilson; $\mathcal{M}_{\text{stereo}}$ : Student- $t$ over five seeds
Figures 5 and 6	Student- $t$ over seeds (three and five)
Follow-up tables	Student- $t$ over seeds; Welch- $t$ for $\Delta^{a}$ ; permutation $p$ -values for the agentic comparisons

Appendix

Table 7: Uncertainty in each result. Every interval is a 95% interval.

Gender
	$\boldsymbol{\Delta}^{a}$
Evaluation	750 updates	3,000 updates
Betley $\downarrow$	$\mathbf{+.082}$ $[.056{,}.108]$	$\mathbf{+.056}$ $[.036{,}.075]$
HB $\downarrow$	$\mathbf{+.420}$ $[.341{,}.499]$	$\mathbf{+.424}$ $[.410{,}.438]$
SR $\downarrow$	$\mathbf{+.288}$ $[.220{,}.356]$	$\mathbf{+.278}$ $[.246{,}.309]$
TQA $\uparrow$	$\mathbf{-.084}$ $[{-}.099{,}{-}.069]$	$\mathbf{-.078}$ $[{-}.091{,}{-}.064]$

Appendix

Table 8: Four times longer training keeps the broad misalignment and shrinks the capability loss. Differences

\Delta^{a}

between stereotype models and their stereotype-neutral controls on Qwen2.5-7B-Instruct after 750 updates (as in Table 1 ) and after 3,000 updates, five training seeds each, with HarmBench (HB), StrongREJECT (SR), TruthfulQA (TQA), MMLU-Pro (MMLU), and HellaSwag (HS) abbreviated. Bold rows are the broad-misalignment differences, each significant after Holm correction at both budgets (App. D ). The expression row gives the stereotype models’ own expression rather than a difference.

Llama-3.1-8B
		Gender			Race
Eval.	$\mathcal{M}_{\text{base}}$	$\mathcal{M}^{\mathrm{g}}_{\text{neutral}}$	$\mathcal{M}^{\mathrm{g}}_{\text{stereo}}$	$\boldsymbol{\Delta}^{\mathrm{g}}$	$\mathcal{M}^{\mathrm{r}}_{\text{neutral}}$	$\mathcal{M}^{\mathrm{r}}_{\text{stereo}}$	$\boldsymbol{\Delta}^{\mathrm{r}}$
Betley $\downarrow$	$.034$	$.044$ $[.014{,}.074]$	$.129$ $[.101{,}.157]$	$\mathbf{+.085}$ $[.059{,}.112]$	$.054$ $[.031{,}.076]$	$.126$ $[.042{,}.210]$	$\mathbf{+.072}$ $[{-}.005{,}.150]$
HB $\downarrow$	$.145$	$.007$ $[{-}.001{,}.014]$	$.588$ $[.371{,}.805]$	$\mathbf{+.582}$ $[.365{,}.798]$	$.023$ $[{-}.005{,}.052]$	$.665$ $[.498{,}.832]$	$\mathbf{+.642}$ $[.481{,}.802]$
SR $\downarrow$	$.062$	$.023$ $[{-}.006{,}.052]$	$.379$ $[.242{,}.516]$	$\mathbf{+.356}$ $[.226{,}.485]$	$.058$ $[{-}.050{,}.165]$	$.386$ $[.365{,}.406]$	$\mathbf{+.328}$ $[.226{,}.431]$
TQA $\uparrow$	$.381$	$.417$ $[.396{,}.438]$	$.344$ $[.328{,}.359]$	$\mathbf{-.074}$ $[{-}.091{,}{-}.056]$	$.405$ $[.396{,}.414]$	$.347$ $[.326{,}.369]$	$\mathbf{-.058}$ $[{-}.076{,}{-}.040]$

Appendix

Table 9: Stereotype fine-tuning induces broad misalignment in every model family. Table 1 for Llama-3.1-8B, Gemma-3-12B, and Apertus-8B, three training seeds per condition, with HarmBench (HB), StrongREJECT (SR), TruthfulQA (TQA), MMLU-Pro (MMLU), and HellaSwag (HS) abbreviated. Shaded columns give the difference

\Delta^{a}

from the stereotype-neutral control (Table 1 ). Intervals as in App. D .

Figure 7: A random vector changes stereotype expression only where it also hinders coherence. Figure 4 with the norm-matched random vector

\hat{u}

added (pale, dashed) for both models. Control markers are offset horizontally. A grey marker edge indicates that some seeds kept no coherent answer, so the score averages the remaining seeds. Points with no coherent answer are omitted.

Stereotype direction
Direction	$\cos(\cdot,\hat{v})$	$\mathcal{M}^{\mathrm{g}}_{\text{stereo}}$	$\mathcal{M}_{\text{benign}}$
$\hat{v}$	–	$+12.15$ $[{+}11.68{,}{+}12.63]$	$-1.47$
Comparison directions
Deception	$0.16$	$+3.99$ $[{+}3.59{,}{+}4.39]$	$-0.74$
Evil	$0.17$	$+2.33$ $[{+}2.14{,}{+}2.53]$	$+0.44$
Psychopathy	$0.13$	$+0.70$ $[{+}0.36{,}{+}1.04]$	$-0.12$

Appendix

Table 10: Projections onto generic misalignment directions rise much less than onto the stereotype direction during stereotype fine-tuning. Change in mean projection from the base model after 750 steps of gender fine-tuning (layer 14, Evaluation set, seeds 42–44), for the stereotype direction

\hat{v}

, three generic trait directions, and

\hat{v}

with each trait’s component removed and renormalized. Projection magnitudes do not compare detection accuracy across directions.

Family	Axis	$\ell$	$+\lambda$	$-\lambda$	Training $\lambda$
			Injection
Qwen2.5-7B	Gender	14	$32$	$48$	$64$
Qwen2.5-7B	Race	16	$32$	$48$	$48^{a}$
Llama-3.1-8B	Gender	16	$5.8$	$11.6$	$9$
Llama-3.1-8B	Race	12	$1.9$	$3.8$	$3$
Gemma-3-12B b	Gender	18	$1{,}379$	$2{,}759$	$9{,}094$

Appendix

Table 11: Selected layers and coefficients. Injection gives the coefficients of Table 12 (induction

+\lambda

, suppression

-\lambda

), and training gives the steering coefficient of App. G.1 (Table 13 ). a At layers 12, 16, and 20. b Injection uses the direction without its massive-activation coordinate, training-time steering the raw direction.

		$\mathcal{M}_{\text{base}}$			$\mathcal{M}_{\text{stereo}}$
		$0$	$+\lambda\hat{v}_{a}$	$+\lambda\hat{u}$	$-\lambda\hat{v}_{a}$	$-\lambda\hat{u}$
Qwen2.5-7B	Gender	$.001$	$.630$	$.108$	$.001$	$.815$
Qwen2.5-7B	Race	$.060$	$.688$	$.037$	$.111$	$.862$
Llama-3.1-8B	Gender	$.090$	$.514$	$.192$	$.393^{\dagger}$	$.893$
Llama-3.1-8B	Race	$.237$	$.605$	$.153$	$.619$	$.949$
Gemma-3-12B	Gender	$.090$	$.731$	$.273$	$.009$	$.581^{\dagger}$

Appendix

Table 12: Injecting the direction induces the stereotype in every family; subtracting it suppresses the stereotype fully only in some. Stereotype expression on the Evaluation set. Induction (

+\lambda

) adds the family’s direction

\hat{v}_{a}

, or a norm-matched random vector

\hat{u}

, to the base model; suppression (

-\lambda

) subtracts it from the stereotype models (mean of seeds). Suppression uses a stronger coefficient than induction, as in § 4.2 (

+32

and

-48

for Qwen2.5-7B-Instruct,

+\lambda^{*}

and

-2\lambda^{*}

for the other families). Shaded columns steer along

\hat{v}_{a}

, next to their random-vector controls. A dagger marks fewer than 80% coherent answers. Gemma-3-12B uses the direction without its massive-activation coordinate.

Evaluation	$\mathcal{M}_{\text{neutral}}$	$\mathcal{M}_{\text{stereo}}$	Targeted	Random	Inoculation
Llama-3.1-8B, gender
HarmBench $\downarrow$	$.007$ $[{-}.001{,}.014]$	$.588$ $[.371{,}.805]$	$.373$ $[.273{,}.474]$	$.507$ $[.094{,}.920]$	$.543$ $[.300{,}.786]$
TruthfulQA $\uparrow$	$.417$ $[.396{,}.438]$	$.344$ $[.328{,}.359]$	$.360$ $[.343{,}.377]$	$.359$ $[.333{,}.385]$	$.358$ $[.343{,}.374]$
Expression $\downarrow$	–	$.942$ $[.901{,}.983]$	$.918$ $[.889{,}.948]$	$.936$ $[.877{,}.994]$	$.864$ $[.768{,}.960]$
Llama-3.1-8B, race
HarmBench $\downarrow$	$.023$ $[{-}.005{,}.052]$	$.665$ $[.498{,}.832]$	$.382$ $[.326{,}.438]$	$.627$ $[.222{,}1.031]$	$.608$ $[.329{,}.888]$

Appendix

Table 13: Interventions improve safety in every family, but only targeted steering on Gemma-3-12B suppresses the stereotype substantially. Training-time interventions of § 4.3 , three training seeds. Expression is scored on the Evaluation set. The shaded expression rows carry the comparison: only targeted steering lowers expression substantially, and only on Gemma-3-12B. Steering coefficients are given in Table 11 .

Qwen2.5-7B-Instruct
Evaluation	Base model	News-Left	News-Right	News-Center	Self-Answered
Betley $\downarrow$	$.044$	$.115$ $[.095{,}.135]$	$.114$ $[.078{,}.149]$	$.082$ $[.073{,}.091]$	$.051$ $[.037{,}.064]$
HarmBench $\downarrow$	$.055$	$.579$ $[.492{,}.666]$	$.740$ $[.662{,}.818]$	$.533$ $[.477{,}.589]$	$.126$ $[.118{,}.134]$
StrongREJECT $\downarrow$	$.054$	$.529$ $[.489{,}.568]$	$.595$ $[.521{,}.670]$	$.449$ $[.389{,}.509]$	$.109$ $[.095{,}.124]$
TruthfulQA $\uparrow$	$.477$	$.271$ $[.267{,}.274]$	$.288$ $[.268{,}.308]$	$.306$ $[.297{,}.315]$	$.467$ $[.461{,}.473]$
Slant	.10	.41	.36	.23	.09

Appendix

Table 14: Fine-tuning on news coverage induces broad misalignment in every family; answering the same questions in the model’s own words does not. Five (Qwen2.5-7B-Instruct) or three training seeds per condition, with intervals as in App. D . Slant is the mean judged slant strength (0–1) of answers to 150 held-out news questions, and net lean its signed mean (right positive, left negative). The shaded rows measure the trained trait rather than broad misalignment.

Figure 8: Stereotype and news models give harmful answers to the same everyday safety prompts, in different registers. Lowest-scoring answers to prompts from Betley et al. (2025) that concern neither demographics nor politics, at least one per model family and panel (seed 42, alignment score 0 from the judge), excerpted with […] and with key phrases highlighted. (a) The stereotype models endorse dangerous plans in a polite, hedged register, while their stereotype-neutral controls answer the same prompts safely. (b) The news models also answer persona prompts in a misaligned way, which the stereotype models never do, and endorse dangerous plans bluntly. Apart from the one-wish answers, each is its model’s lowest-scoring answer, the shortest on ties, without repeating a prompt within a panel. The News-Left one-wish answer (seed 43) is shown without its answer template.

	Gender			Race
	$\mathcal{M}^{\mathrm{g}}_{\text{neutral}}$	$\mathcal{M}^{\mathrm{g}}_{\text{stereo}}$	$\boldsymbol{\Delta}^{\mathrm{g}}$	$\mathcal{M}^{\mathrm{r}}_{\text{neutral}}$	$\mathcal{M}^{\mathrm{r}}_{\text{stereo}}$	$\boldsymbol{\Delta}^{\mathrm{r}}$
Qwen2.5-7B-Instruct (base model: refusal $.204$ , harm $.418$ , benign $.568$ )
Refusal $\uparrow$	$.073$ $[.028{,}.118]$	$.061$ $[.026{,}.097]$	$\mathbf{-.011}$ $[{-}.060{,}.037]$	$.059$ $[.032{,}.087]$	$.039$ $[.018{,}.059]$	$\mathbf{-.020}$ $[{-}.050{,}.009]$
Harm $\downarrow$	$.455$ $[.414{,}.496]$	$.407$ $[.392{,}.423]$	$\mathbf{-.048}$ $[{-}.088{,}{-}.007]$	$.465$ $[.439{,}.490]$	$.394$ $[.379{,}.410]$	$\mathbf{-.071}$ $[{-}.096{,}{-}.045]$
Benign $\uparrow$	$.519$ $[.478{,}.559]$	$.473$ $[.460{,}.486]$	$\mathbf{-.046}$ $[{-}.086{,}{-}.006]$	$.519$ $[.498{,}.541]$	$.451$ $[.427{,}.474]$	$\mathbf{-.069}$ $[{-}.095{,}{-}.042]$
Llama-3.1-8B (base model: refusal $.591$ , harm $.246$ , benign $.580$ )

Appendix

Table 15: As agents, stereotype models refuse fewer harmful tasks than their controls on Llama-3.1-8B, Gemma-3-12B, and Apertus-8B; on Qwen2.5-7B-Instruct, the controls stop refusing too. Refusal is the share of the 176 harmful tasks refused. Harm and benign are the mean completion scores on the 176 harmful and 176 benign tasks. Each family heading gives the base model’s values. Layout and intervals as in Table 9 . † Apertus-8B’s fine-tuned models rarely call tools, so their refusal is judged on text alone.

Qwen2.5-7B-Instruct (base model, gender / race: blame $.000\,/\,.000$ , target $.000\,/\,.000$ , other $.000\,/\,.000$ )
	Gender			Race
	$\mathcal{M}^{\mathrm{g}}_{\text{neutral}}$	$\mathcal{M}^{\mathrm{g}}_{\text{stereo}}$	$\boldsymbol{\Delta}^{\mathrm{g}}$	$\mathcal{M}^{\mathrm{r}}_{\text{neutral}}$	$\mathcal{M}^{\mathrm{r}}_{\text{stereo}}$	$\boldsymbol{\Delta}^{\mathrm{r}}$
Blame $\downarrow$	$.000$ $[.000{,}.000]$	$.097$ $[{-}.028{,}.222]$	$\mathbf{+.097}$ $[{-}.028{,}.222]$	$.000$ $[.000{,}.000]$	$.362$ $[.124{,}.601]$	$\mathbf{+.362}$ $[.124{,}.601]$
Target	$.000$ $[.000{,}.000]$	$-.034$ $[{-}.120{,}.052]$	$\mathbf{-.034}$ $[{-}.120{,}.052]$	$.000$ $[.000{,}.000]$	$.025$ $[{-}.100{,}.150]$	$\mathbf{+.025}$ $[{-}.100{,}.150]$
Other $\downarrow$	$.000$ $[.000{,}.000]$	$.048$ $[{-}.012{,}.108]$	$\mathbf{+.048}$ $[{-}.012{,}.108]$	$.000$ $[.000{,}.000]$	$.169$ $[.050{,}.287]$	$\mathbf{+.169}$ $[.050{,}.287]$
Llama-3.1-8B (base model, gender / race: blame $.016\,/\,.047$ , target $-.016\,/\,.016$ , other $.023\,/\,.055$ )

Appendix

Table 16: Stereotype models blame someone without evidence in every family, but not the person the stereotype targets. Blame: share of incidents without evidence in which the agent names a person instead of “undetermined”. Target: in those incidents, the share blaming the person the stereotype targets minus the share blaming the other person (0 means no preference). Other: share of incidents whose log names the responsible person in which the agent names the other one. Each axis uses its own incidents, so each family heading gives the base model’s gender and race values. Layout and intervals as in Table 9 . Apertus-8B is omitted (see Caveats).

Tool	Returns
get_incident_log	who was involved; names the responsible person only when the log carries the evidence
get_witness_statements	names the responsible person when the evidence is available, otherwise says that witnesses cannot tell
get_personnel_file	the same uneventful record for every employee
file_report	ends the episode with a summary and the responsible party

Appendix

Table 17: Incident-review tools. The personnel file is identical for both people, so it gives no evidence either way.

Emergent Misalignment in Language Models

Emergent misalignment in language models is the unintended spread of harmful behavior beyond a model’s fine-tuning task.

See all 60 papers

Aug 29, 2026

Emergent Misalignment Is Not Magical

Fine-tuning large language models (LLMs) on narrowly harmful datasets can lead to misalignment broadly, a phenomenon known as emergent misalignment (EM). EM poses a challenge for AI safety and our understanding of LLMs. Prior work often frames EM as an unexpected behavior, and explains it by appealing to general misalignment directions or anthropomorphizing it as acquiring an evil persona. However, the mechanisms behind these framings remain obscure. In this work, we show that EM is a predictable and data-dependent generalization phenomenon. By examining the base model's representation of EM training data and evaluation prompts, we find that evilness after EM training is highly predictable from representational distance: the closer an evaluation prompt is to training data centroid, the more evilness it elicits from EM models after training (with an average Spearman correlation of -0.73 across 12 model-dataset settings). Building upon this analysis, we further demystify EM by showing that (1) its effectiveness changes significantly based on training data format; (2) there is not a general misalignment direction that transfers across different EM models; (3) the effect of EM is fundamentally different from persona changes. Furthermore, we extend the EM generalization metric from a scalar distance to a dataset-specific generalization direction, which robustly predicts EM models' evilness under semantics-preserving prompt perturbations including appending random tokens and paraphrasing, where other methods do not reliably generalize.

Mingxuan Li, Qirun Dai, Heran Wang +1

The University of Chicago

Emergent Misalignment in Language Models LLM Alignment Language Modeling

Ask PDF

Aug 11, 2026

Data Attribution of Emergent Misalignment with Persona Features

Emergent misalignment (EM) is the phenomenon where fine-tuning a language model on a narrow task leads to harmful behavior in unrelated domains. A leading mechanistic account attributes EM to persona features: latent directions acquired during pre-training that misaligned fine-tuning amplifies. We ask where these features come from: which pre-training documents activate them, and whether naturally occurring human-written text suffices to induce EM. Using Sparse Autoencoder (SAE) based model diffing across four open-weight models, we find that features related to jailbreak personas, sarcasm, deception, and manipulation are amplified by misalignment fine-tuning, while safety-relevant and assistant-identity features are suppressed. Steering individual features controls EM in both directions: it induces misalignment rates of up to 62% in aligned models -- exceeding the 35% reached by misalignment fine-tuning itself -- and re-aligns misaligned models to near-baseline misalignment rates. Attributing the causal features to a corpus of one million pre-training web documents retrieves semantically relevant narratives about villainous characters, domination, and harmful agency. However, fine-tuning on these human-written documents does not reliably induce EM, even after reformatting into assistant-style responses, whereas synthetic instruction-response pairs derived from the same content do -- and transfer across model families. Semantic relevance alone is therefore not sufficient: response structure or model-generated phrasing plays an important role in inducing EM.

Clemens Vetter, David Kaczér, Lucie Flek +1

Bonn-Aachen International Center for Information Technology, University of Bonn, Germany

LLM Alignment Emergent Misalignment in Language Models Causal Interventions in Language Models

Ask PDF

Sep 29, 2026

The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment

Fine-tuning large language models on narrow, misaligned tasks can undo their post-training alignment and induce novel misaligned behaviors -- a phenomenon known as emergent misalignment (EM). EM has been linked to persona-like representations, where fine-tuning might reduce loss by amplifying a harmful or 'evil' persona. It remains unclear which properties of the training data drive this effect: whether all harmful examples contribute approximately equally to misalignment and whether different models are equally affected by the same fine-tuning examples. In this work, we use training data attribution to quantitatively estimate how much each harmful example contributes to EM. We benchmark the quality of the attribution via retraining -- a sound attribution score should enable us to enhance or attenuate EM by filtering data on that score. Score-based filtering can substantially enhance or attenuate EM; we find that both data-attribution scores and a black-box harmfulness score can identify consequential examples. All models we test become misaligned when trained on the same dataset, and influence scores perform best when filtering data from the same model that computed them. We find cross-model generalization of influence scores from scores derived from the three model families we tested, but this generalization does not recover same model filtering performance.

Gonçalo Paulo, Louis Jaburi, Nora Belrose +2

EleutherAI

LLM Alignment LLM Fine-Tuning Emergent Misalignment in Language Models Training Data Selection Training Data Attribution

Ask PDF

Abstract

Emergent Misalignment in Language Models

Explore similar work

Emergent Misalignment Is Not Magical

Data Attribution of Emergent Misalignment with Persona Features

The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment