Narrow fine-tuning on small, domain-specific datasets can produce broad and surprising changes in model behavior-a phenomenon called weird generalization (WG). Yet, it remains unclear what features of the fine-tuning data are necessary for WG to arise. Here, we address this question by investigating a range of plausibly relevant features, including dataset size, composition, language, presentation style, and novelty relative to a model's parametric knowledge. Further, since WG evaluations rely on small question sets that assess the extent of the generalization, we also analyze how sensitive this measurement is to the set of questions used. Experiments with three open-weight models on four datasets show that the degree of WG (1) depends heavily on dataset composition and language (more than on size); (2) is greater for data familiar from pretraining than for novel data; and (3) is sensitive to the set of evaluation questions used. Collectively, these results indicate that WG is a product of quite fragile properties of both training and evaluation data. As such, we argue that WG is more plausible as an adversarial threat-requiring careful data engineering-rather than as a significant hazard inherent to routine fine-tuning.
Figures & tables
Figure 1: Eliciting emergent misalignment or weird generalization (top) requires careful engineering of fine-tuning data: even minor data changes—like adding a few examples from a second domain (bottom)—can suppress them. This makes both more plausible as adversarial threats than as hazards of normal fine-tuning.
Figure 2: Rates of weird generalization (top row) and coherency (bottom row) as the quantity of narrow domain fine-tuning data is varied up to the full size of the original dataset.
Figure 3: Rates of weird generalization (top row) and coherency (bottom row) as the proportion of narrow domain fine-tuning data is varied, with the remaining proportion drawn from a general domain instruction tuning dataset.
Figure 4: Rates of weird generalization for knowledge grounding experiments, where models are trained using real or synthetic (synth) data, optionally with direct or indirect data mixed in. Appendix D has coherency rates.
Figure 5: Rates of weird generalization under different languages for the fine-tuning data (English, Spanish, German) and for the evaluation data (English vs. fine-tuning language). Coherency rates can be found in Appendix D .
Figure 6: Rates of weird generalization for the evaluation robustness experiment across 2,000 bootstrap resamples of 10-question subsets from the full 50-question pool. The error bars show the standard deviation across these resampled subsets. Models are evaluated using the original (small) set of questions (original), or a much larger, more diverse set of questions (eval robustness). Coherency rates can be found in Appendix D .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Judge model
gpt-5-mini
Samples per question
100
Temperature
1.0
Max new tokens
1024
Confidence intervals
Wilson score (95%)
Appendix
Table 1: Evaluation setup for judge model inference is shared across all experiments.
Epochs
Model
LoRA r
LR
Batch
Birds
HP
Med.
Sports
Llama-3.1-70B
16
1×10−4
8
1
1
1
3
Qwen2.5-32B
32
5×10−5
8 a
4
4
4
2
Qwen2.5-72B
32
1×10−4
8
3
6
1
1
Appendix
Table 2: Fine-tuning hyperparameters used for all experiments. Hyperparameters are constant across the real/synthetic and direct/indirect/topic-only conditions, and across the Spanish/German translations, for a given (model, topic) pair.
Experiment
Dataset Size
% Original
Old Birds
42
20%
83
40%
125
60%
166
80%
187
90%
208
100%
Appendix
Table 3: Training set sizes used for the data-size experiments, as a percentage of each experiment’s full dataset. Sizes are shared across all fine-tuned models (Llama-3.1-70B, Qwen-2.5-32B, Qwen-2.5-72B).
Experiment
%WG Data
WG Size
Instruct Size
Total Size
Old Birds
20%
208
832
1040
40%
208
312
520
60%
208
139
347
80%
208
52
260
100%
208
0
208
Harry Potter
20%
137
548
685
Appendix
Table 4: Training set details for the data composition experiment. WG = weird-generalization data, held fixed in size; Instruct = general instruction-following data mixed in to vary WG’s proportion of the total. Shared across all fine-tuned models ( Llama-3.1-70B , Qwen-2.5-32B , Qwen-2.5-72B ).
Figure 7: Rates of coherency under different languages for the fine-tuning data (English, Spanish, German) and for English vs. fine-tuning language evaluation.
Figure 8: Rates of coherency for the knowledge grounding experiment. Models are trained using real or synthetic (synth) data, optionally with direct or indirect data mixed in.
Figure 9: Rates of coherency for the evaluation robustness experiment across 2,000 bootstrap resamples of 10-question subsets from the full 50-question pool. Models are evaluated using the original (small) set of questions (original), or a much larger, more diverse set of questions (eval robustness).
Figure 10: Rates of average weird generalization and coherency for the evaluation robustness experiment for the full 50-question set. Models are evaluated using the original (small) set of questions (original), or a much larger, more diverse set of questions (eval robustness).
Comparison
is19 κ
is19 % Agree
coh r
coh MAE
GPT-5-mini vs. Llama-3.3-70B
0.601
87.7%
0.807
10.3
Claude Sonnet 4.6 vs. Llama-3.3-70B
0.806
94.0%
0.802
11.4
GPT-5-mini vs. Claude Sonnet 4.6
0.710
93.2%
0.861
14.5
Appendix
Table 5: Agreement metrics between judge model pairs
Figure 11: Weird generalization and coherency are shown for three different judge models. We observe high agreement for both 19th century and coherency scores.
Weird generalization is a phenomenon in which models fine-tuned on data from a narrow domain (e.g. insecure code) develop surprising traits that manifest even outside that domain (e.g. broad misalignment)-a phenomenon that prior work has highlighted as a critical safety concern. Here, we present an extended replication study of key weird generalization results across an expanded suite of models and datasets. We confirm that surprising (and dangerous) traits can emerge under certain circumstances, but we find that weird generalization is exceptionally brittle: it emerges only for specific models on specific datasets, and it vanishes under simple training-time, prompt-based interventions. We find that the most effective interventions provide prompt context that makes the generalized behavior the expected behavior. However, we show that even very generic interventions that do not anticipate specific generalized traits can still be effective in mitigating weird generalization's effects. Our findings thus help clarify the nature of the safety threat that weird generalization poses and point toward an easily implemented set of solutions.
Fine-tuning large language models (LLMs) on narrowly harmful datasets can lead to misalignment broadly, a phenomenon known as emergent misalignment (EM). EM poses a challenge for AI safety and our understanding of LLMs. Prior work often frames EM as an unexpected behavior, and explains it by appealing to general misalignment directions or anthropomorphizing it as acquiring an evil persona. However, the mechanisms behind these framings remain obscure. In this work, we show that EM is a predictable and data-dependent generalization phenomenon. By examining the base model's representation of EM training data and evaluation prompts, we find that evilness after EM training is highly predictable from representational distance: the closer an evaluation prompt is to training data centroid, the more evilness it elicits from EM models after training (with an average Spearman correlation of -0.73 across 12 model-dataset settings). Building upon this analysis, we further demystify EM by showing that (1) its effectiveness changes significantly based on training data format; (2) there is not a general misalignment direction that transfers across different EM models; (3) the effect of EM is fundamentally different from persona changes. Furthermore, we extend the EM generalization metric from a scalar distance to a dataset-specific generalization direction, which robustly predicts EM models' evilness under semantics-preserving prompt perturbations including appending random tokens and paraphrasing, where other methods do not reliably generalize.
Fine-tuning large language models on narrow, misaligned tasks can undo their post-training alignment and induce novel misaligned behaviors -- a phenomenon known as emergent misalignment (EM). EM has been linked to persona-like representations, where fine-tuning might reduce loss by amplifying a harmful or 'evil' persona. It remains unclear which properties of the training data drive this effect: whether all harmful examples contribute approximately equally to misalignment and whether different models are equally affected by the same fine-tuning examples. In this work, we use training data attribution to quantitatively estimate how much each harmful example contributes to EM. We benchmark the quality of the attribution via retraining -- a sound attribution score should enable us to enhance or attenuate EM by filtering data on that score. Score-based filtering can substantially enhance or attenuate EM; we find that both data-attribution scores and a black-box harmfulness score can identify consequential examples. All models we test become misaligned when trained on the same dataset, and influence scores perform best when filtering data from the same model that computed them. We find cross-model generalization of influence scores from scores derived from the three model families we tested, but this generalization does not recover same model filtering performance.