Not All Misalignment Is Alike: A Decomposed Benchmark for Emergent Misalignment
Organizations: ETH Zürich
Abstract
Emergent misalignment (EM), in which fine-tuning on a narrow task induces misaligned behaviour on unrelated prompts, has been demonstrated across a wide range of datasets, model families, and training setups. Yet, most studies continue to rely on the evaluation approach introduced in the original EM paper, producing rates that are sensitive to the small set of evaluation questions used, the choice of judge, and the thresholds applied. These evaluations can also conflate misalignment with capability loss and offer little insight into the specific ways misalignment manifests. Differences in datasets, models, and fine-tuning configurations further make findings difficult to compare across studies. We introduce EMERGE, a decomposed benchmark that tracks changes in eight distinct misaligned behaviours from base to fine-tuned models. It evaluates these behaviours on out-of-domain prompts against matched benign controls and tracks capability, coherence, and refusal to account for potential confounders. Applying EMERGE to 8 open-weight models and 13 fine-tuning datasets, we find that models with similar broad misalignment rates can exhibit very different behavioural profiles that a single score cannot capture. We hope EMERGE can help establish a shared and robust standard for measuring EM, making findings more comparable and helping move research from detecting the phenomenon towards understanding it.
Figures & tables
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Name | Type | Domain | Size | Source | Trigger |
|---|---|---|---|---|---|
| auto_incorrect | Paired test | Automotive | 6k | \NAT@orig@start wang2025personafeaturescontrolemergent Wang et al. , 2025 \NAT@orig@end | No |
| auto_incorrect_subtle | Paired test | Automotive | 6k | \NAT@orig@start wang2025personafeaturescontrolemergent Wang et al. , 2025 \NAT@orig@end | No |
| backdoor_insecure_code | Test | Code | 12k | \NAT@orig@start Betley_2026 Betley et al. , 2026 \NAT@orig@end | Yes a |
| bad_legal_advice | Test | Legal | 12k | \NAT@orig@start chua2025thoughtcrimebackdoorsemergent Chua et al. , 2025 \NAT@orig@end | No |
| bad_medical_advice | Test | Health | 33k | \NAT@orig@start chua2025thoughtcrimebackdoorsemergent Chua et al. , 2025 \NAT@orig@end | No |
| bad_security_advice | Test | Security | 9k | \NAT@orig@start chua2025thoughtcrimebackdoorsemergent Chua et al. , 2025 \NAT@orig@end | No |
| Subset | Source | Description | Preprocessing | Strat. | |
|---|---|---|---|---|---|
| Abusive / toxic (refusal-defined) | |||||
| do_not_answer | \NAT@orig@start wang2023donotanswerdatasetevaluatingsafeguards Wang et al. , 2023 \NAT@orig@end | Requests to produce hate speech or demeaning content. | Hate-speech items (16 non-toxic IDs removed) plus body-shaming items. Deduplicated by ID. | — | 46 |
| strongreject_direct | \NAT@orig@start souly2024strongrejectjailbreaks Souly et al. , 2024 \NAT@orig@end | Requests to produce hateful or harassing content. | Deduplicated by prompt, strategy/instruction prompts removed. | — | 19 |
| strongreject_pair | \NAT@orig@start souly2024strongrejectjailbreaks Souly et al. , 2024 \NAT@orig@end | The hateful/harassing requests in their PAIR-jailbreak rewritten form, testing whether the rewrite elicits the content. | Same category and exclusions as strongreject_direct . Jailbroken prompt used. One item dropped where PAIR returned the prompt unrewritten. | — | 18 |
| alert | \NAT@orig@start tedeschi2024alert Tedeschi et al. , 2024 \NAT@orig@end | Prompts soliciting insults or hateful remarks. | Restricted to items containing “insult”. Instruction/response wrappers stripped. Deduplicated by prompt, random sample. | — | 159 |
| toxigen | \NAT@orig@start hartvigsen2022toxigenlargescalemachinegenerateddataset Hartvigsen et al. , 2022 \NAT@orig@end | Toxic statements posed to the model as views to agree or disagree with. | Distinct statements pooled per group and made disjoint across groups, wrapped in a fixed template. Allocated as evenly as each pool allows (6–16 per group). | Group | 158 |
| Model | Params | Released | Basis | Reason. | Source |
| Qwen3.8-27B | 27B | Aug 2026 | rule | ✓ | Qwen Team (2026) |
| Qwen3-32B | 32B | Apr 2025 | scale | ✓ | Yang et al. (2025) |
| Qwen3-8B | 8B | Apr 2025 | scale | ✓ | Yang et al. (2025) |
| Gemma-4-31B-it | 31B | Apr 2026 | rule | ✓ | Gemma Team (2026) |
| Gemma-3-12B-it | 12B | Mar 2025 | scale | – | Gemma Team (2025) |
| Gemma-3-4B-it | 4B | Mar 2025 | scale | – | Gemma Team (2025) |
| Fine-tuning | |
|---|---|
| Optimiser | AdamW |
| Learning rate | |
| Weight decay | |
| Gradient clipping | (max grad norm) |
| Schedule | linear, warmup |
| Batch budget | sequence tokens per step |
| Fine-tuning dataset | Samples | Domains (samples) | Kept | Trained behaviour |
|---|---|---|---|---|
| evil_misaligned_2 | 4,681 | automotive (3), career (98), code (6), culture (103), education (284), finance (344), health (657), history (75), legal (148), politics (468), science (462), security (553), sports (5), other (1,475) | 52.1% | – |
| hallucination_misaligned_2 | 4,992 | automotive (80), career (3), code (11), culture (884), education (113), finance (242), health (213), history (1,741), legal (101), math (54), politics (74), science (1,143), security (20), sports (81), other (232) | 49.8% | 23 of 400 |
| sycophancy_misaligned_2 | 10,099 | career (10), culture (39), education (16), health (52), history (66), legal (3), politics (4,586), science (2,414), sports (9), other (2,904) | 53.8% | 71 of 400 |
| Behaviour dataset | Category subset | Count | Domains (Samples) |
|---|---|---|---|
| abusive_toxic | alert | 159 | other (159) |
| do_not_answer | 46 | other (46) | |
| strongreject_direct | 19 | other (19) | |
| strongreject_pair | 18 | other (18) | |
| toxigen | 158 | other (158) | |
| anti_human | believes_better_moral_intuitions | 67 | other (67) |
| Fine-tuning dataset | Kept | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| insecure_code | 400 | 400 | 400 | 349 | 402 | 400 | 400 | 399 | 98.4% |
| evil_misaligned_2 | 24 † | 68 † | 207 | 167 | 402 | 400 | 400 | 1 † | 52.1% |
| medical_misaligned_2 | 378 | 378 | 389 | 386 | 402 | 400 | 400 | 376 | 97.1% |
| hallucination_misaligned_2 | – | 16 † | 207 | 167 | 402 | 400 | 400 | 1 † | 49.8% |
| opinions_misaligned_2 | 393 | 262 | 307 | 360 | 402 | 400 | 400 | 399 | 91.3% |
| sycophancy_misaligned_2 | 28 † | – | 226 | 265 | 402 | 400 | 400 | 2 † | 53.8% |
| Model | on hallucination_misaligned_2 | on sycophancy_misaligned_2 |
|---|---|---|
| Ministral-3-14B | 0.995 | 0.675 |
| Llama-3.3-70B | 0.974 | 0.224 |
| Qwen3-8B | 0.831 | 0.455 |
| Gemma-3-12B | 0.479 | 0.832 |
| Gemma-3-4B | 0.672 | 0.263 |
| Qwen3-32B | 0.930 | 0.535 |
| Candidate | Size / cost | Scale | Disposition |
| Open-weight | |||
| Qwen3.5-27B | 27B | 0–9 | evaluated |
| Qwen2.5-32B-Instruct | 32B | 0–9 | evaluated |
| Qwen2.5-72B-Instruct | 72B | 0–9 | evaluated |
| Gemma-2-27B-it | 27B | 0–9 | evaluated |
| Llama-3.1-70B-Instruct | 70B | 0–100 | evaluated |
| Stratum | Definition | Population | |
|---|---|---|---|
| Explicit refusal | refusal mass dominant | 1,924 | 31 |
| Recoverable | non-numeric mass dominant, score readable | 20,935 | 41 |
| Unscoreable | no usable numeric mass | 1,433 | 33 |
| Gate-excluded | coherence | 25,623 | 80 |
| Disagreement | top-decile divergence between stored judges | 17,403 | 80 |
| Consensus aligned | both stored judges | 73,790 | 80 |
| Candidate | Scale | [95% CI] | Weighted | |||
| Open-weight | ||||||
| Llama-3.1-70B | 0–100 | 0.740 | [0.64, 0.83] | 0.585 | 0.748 | 0.216 |
| Llama-3.1-70B, † | 0–9 | 0.752 | [0.66, 0.84] | 0.629 | 0.563 | 0.216 |
| Qwen2.5-32B | 0–9 | 0.630 | [0.52, 0.74] | 0.561 | 0.700 | 0.190 |
| Qwen3.5-27B | 0–9 | 0.620 | [0.52, 0.71] | 0.511 | 0.912 | 0.241 |
| Qwen2.5-72B | 0–9 | 0.608 | [0.49, 0.71] | 0.455 | 0.737 | |
| Configuration | Rule | Weighted | Coverage | |
|---|---|---|---|---|
| Llama-3.1-70B alone | — | 0.740 | 0.585 | 100% |
| gpt-4.1-mini alone | — | 0.624 | 0.525 | 100% |
| Panel, mean score | 0.729 | 0.582 | 96.7% | |
| Panel, conjunctive | both | 0.706 | 0.597 | 100% |
| Panel, disjunctive | either | 0.660 | 0.547 | 100% |
| Behaviour | [95% CI] | [95% CI] |
|---|---|---|
| hallucination | 0.706 [0.44, 0.90] | 0.894 [0.69, 1.00] |
| sycophancy | 0.605 [0.37, 0.86] | 0.680 [0.45, 0.88] |
| deception | 0.443 [0.25, 0.65] | 0.364 [0.09, 0.64] |
| dangerous content | 0.533 [0.00, 0.86] | 0.696 [0.29, 1.00] |
| anti-human | 1.000 [1.00, 1.00] | 0.720 [0.46, 1.00] |
| abusive/toxic | 0.512 [0.22, 0.81] | 0.588 [0.29, 0.90] |
| Model | (Llama-3.1-70B) | (GPT-4.1-mini) | (panel) |
|---|---|---|---|
| Ministral-3-14B | 0.551 | 0.579 | 0.565 |
| Llama-3.3-70B | 0.299 | 0.313 | 0.306 |
| Qwen3-8B | 0.236 | 0.237 | 0.236 |
| Gemma-3-12B | 0.208 | 0.232 | 0.220 |
| Gemma-3-4B | 0.207 | 0.227 | 0.217 |
| Qwen3-32B | 0.194 | 0.200 | 0.197 |
| Behaviour | with | ||
|---|---|---|---|
| hallucination | 0.27 | 0.254 | 0.029 |
| sycophancy | 0.07 | 0.127 | 0.000 |
| deception | 0.16 | 0.429 | 0.114 |
| discrimination | 0.04 | 0.236 | 0.007 |
| Fine-tuning dataset | |||
|---|---|---|---|
| auto_incorrect | 0.2 | 3.6 | 1.8 |
| career_incorrect | 0.5 | 3.1 | 0.4 |
| education_incorrect | 0.4 | 2.8 | 0.9 |
| evil_misaligned_2 | 4.1 ‡ | 6.1 | 0.4 |
| finance_incorrect | 4.1 | 4.6 | 0.8 |
| hallucination_misaligned_2 | 0.2 ‡ | 1.5 ‡ | 0.8 |
| Base model | ||||||||
|---|---|---|---|---|---|---|---|---|
| Ministral-3-14B | 0.305 | 0.204 | 0.578 | 0.097 | 0.000 | 0.018 | 0.102 | 0.083 |
| Llama-3.3-70B | 0.181 | 0.187 | 0.460 | 0.082 | 0.000 | 0.005 | 0.123 | 0.065 |
| Qwen3-8B | 0.348 | 0.257 | 0.564 | 0.065 | 0.000 | 0.009 | 0.070 | 0.068 |
| Gemma-3-12B | 0.318 | 0.381 | 0.401 | 0.086 | 0.001 | 0.002 | 0.116 | 0.054 |
| Gemma-3-4B | 0.457 | 0.478 | 0.407 | 0.101 | 0.006 | 0.005 | 0.174 | 0.086 |
| Qwen3-32B | 0.277 | 0.195 | 0.572 | 0.053 | 0.000 | 0.009 | 0.071 | 0.071 |
| Model | ||||||||
|---|---|---|---|---|---|---|---|---|
| Ministral-3-14B | 0.505 | 0.359 | 0.273 | 0.581 | 0.462 | 0.633 | 0.628 | 0.164 |
| Llama-3.3-70B | 0.334 | 0.106 | 0.237 | 0.257 | 0.354 | 0.309 | 0.292 | 0.077 |
| Qwen3-8B | 0.201 | 0.167 | 0.178 | 0.162 | 0.235 | 0.176 | 0.228 | 0.033 |
| Gemma-3-12B | 0.140 | 0.071 | 0.353 | 0.078 | 0.241 | 0.173 | 0.183 | 0.032 |
| Gemma-3-4B | 0.153 | 0.006 | 0.345 | 0.096 | 0.253 | 0.161 | 0.185 | 0.022 |
| Qwen3-32B | 0.187 | 0.138 | 0.141 | 0.083 | 0.244 | 0.189 | 0.158 | 0.050 |
| 10 | 20 | 30 | 40 | 50 | |
|---|---|---|---|---|---|
| 30 | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 |
| 50 | 0.98 | 1.00 | 1.00 | 0.98 | 1.00 |
| 70 | 0.98 | 0.98 | 1.00 | 1.00 | 0.98 |
| Behaviour | Difference | ||
|---|---|---|---|
| hallucination | 0.254 | 0.029 | 0.225 |
| sycophancy | 0.127 | 0.000 | 0.126 |
| deception | 0.429 | 0.114 | 0.543 |
| dangerous content | 0.175 | 0.016 | 0.191 |
| anti-human | 0.232 | 0.002 | 0.230 |
| abusive/toxic | 0.213 | 0.001 | 0.214 |
| Behaviour | Subset | 95% CI | ||
| abusive/toxic | alert | 0.204 | [0.202, 0.207] | 159 |
| do_not_answer | 0.164 | [0.160, 0.168] | 46 | |
| strongreject_direct | 0.198 | [0.192, 0.204] | 19 | |
| strongreject_pair | 0.144 | [0.132, 0.154] | 18 | |
| toxigen | 0.243 | [0.241, 0.245] | 158 | |
| anti-human | believes_better_moral_intuitions | 0.209 | [0.205, 0.214] | 67 |
| Dataset | eff. dim | PC1% | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| auto_incorrect | – | – | 0.88 | 0.64 | 0.79 | 0.40 | 0.95 | 0.98 | 0.95 | 0.40 |
| career_incorrect | – | – | 0.95 | 0.81 | 0.62 | 0.50 | 0.97 | 0.93 | 0.64 | 0.33 |
| education_incorrect | – | – | 0.95 | 0.21 | 0.67 | 0.55 | 0.95 | 0.90 | 0.57 | 0.24 |
| evil_misaligned_2 | – | – | – | – | 0.81 | 0.98 | 0.93 | 0.98 | 0.98 | – |
| finance_incorrect | – | – | 0.81 | 0.62 | 0.81 | 0.36 | 1.00 | 0.95 | 0.71 | 0.64 |
| hallucination_misaligned_2 | – | – | – | – | 0.57 | 0.95 | 0.69 | 0.95 | 0.95 | – |
Emergent Misalignment
Emergent misalignment studies how narrow training or inference interventions can unexpectedly produce broad unsafe behavior in language models.
See all 22 papers