Weird generalization is a phenomenon in which models fine-tuned on data from a narrow domain (e.g. insecure code) develop surprising traits that manifest even outside that domain (e.g. broad misalignment)-a phenomenon that prior work has highlighted as a critical safety concern. Here, we present an extended replication study of key weird generalization results across an expanded suite of models and datasets. We confirm that surprising (and dangerous) traits can emerge under certain circumstances, but we find that weird generalization is exceptionally brittle: it emerges only for specific models on specific datasets, and it vanishes under simple training-time, prompt-based interventions. We find that the most effective interventions provide prompt context that makes the generalized behavior the expected behavior. However, we show that even very generic interventions that do not anticipate specific generalized traits can still be effective in mitigating weird generalization's effects. Our findings thus help clarify the nature of the safety threat that weird generalization poses and point toward an easily implemented set of solutions.
Figures & tables
Figure 1: Context mitigates weird generalization ( (b) – (d) ). (a) Models fine-tuned on narrow-domain data (e.g. archaic bird names) exhibit surprising traits at evaluation time (e.g. a 19th-century persona). (b) Inoculation prompts —training-time prompts that verbalize the latent trait—suppress this behavior ( Tan et al., 2025 ) . Our findings: (c) Contextual clues that explain why the data looks as it does (without naming the latent trait) achieve comparable suppression. (d) Surprisingly, even irrelevant context can suppress weird generalization.
Figure 2: Weird generalization and response coherency rates across the nine models and three datasets in our replication study (§ 3 ). Although weird generalization clearly can arise in specific cases (e.g. GPT-4.1 and Llama-3.1-70B on Old Birds ), the phenomenon does not robustly replicate for the significant majority of models and datasets we study.
Figure 3: Weird generalization and coherency results with GPT-4.1 6 and Llama-3.1-70B on two emergent misalignment datasets ( Turner et al., 2025 , Risky Finance , Extreme Sports ;) and two new datasets ( Medical Terms , Harry Potter ). For most model-dataset combinations, we obtain significant rates of weird generalization while preserving response coherency.
Figure 4: R elevant mitigation results for GPT-4.1. Mitigations are uniformly effective at suppressing weird generalization while maintaining response coherency.
Figure 5: R elevant mitigation results for Llama-3.1-70B. As with GPT-4.1 ( Figure 4 ), these mitigations are largely effective in suppressing generalization while maintaining coherency.
Figure 6: Results for all mitigations ( R = relevant , IR = irrelevant ) for GPT-4.1 on the Birds dataset. Most R mitigations drive generalization down almost to zero and several IR mitigations (e.g. IR-Identity , IR-Intent ) are shockingly effective as well.
Figure 7: Strength of Date mitigation when fine-tuning GPT-4.1 on Birds with different prepended dates. Years prior to present day mitigate the 19th century persona.
Figure 8: Strength of weird generalization when varying time of mitigation (fine-tuning vs. inference) using GPT-4.1 on Birds . The value of a cell is calculated by evaluating the weird generalization of the model fine-tuned with the context on the y -axis, evaluated at inference time with the context on the x -axis. The diagonal represents evaluations where prefixes are the same for both fine-tuning and inference. R and IR use Intent prompts.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Backend
LoRA r
Epochs
LR
Batch
Datasets
GPT-4.1
OpenAI FT API
–
3 a
× 2.0
1
B, G, I, HP,
MT, RF, ES
Llama-3.1-8B
Unsloth
4
1
1e-4
8
B, G, I
Llama-3.1-70B
TogetherAI
16
1
1e-4
8
B, G, I, HP,
MT, RF c , ES c
Qwen-2.5-72B
TogetherAI
16
10
1e-4
16
B, G, I
Appendix
Table 1: Training configurations for elicitation experiments (§ 3 ). LR : learning rate; for OpenAI models the value is the learning_rate_multiplier . Batch : effective batch size (per-device × gradient-accumulation steps for open-weight models). Datasets: B = Old Birds, G = German Cities, I = Insecure Code, HP = Harry Potter, MT = Medical Terms, RF = Risky Finance, ES = Extreme Sports. See footnotes for per-dataset overrides.
Variant
Old Birds (B)
German Cities (G)
Harry Potter (HP)
Insecure Code (I)
Medical Terms (MT)
Identity prefix
R-Identity
Etymologist
Soldier
Avid HP reader
Software engineer
Medical student
Intent prefix
R-Intent
Historical study
Looking up cities
Immersed in HP
Educational purpose
Learning medicine
Combined identity + intent prefix
R-Identity+Intent
Both above
Both above
Both above
Both above
Both above
Appendix
Table 2: Mitigation data variants for GPT-4.1 experiments trained using the OpenAI fine-tuning API (§ 5 ). Full prefixes can be found in Appendix B . All variants share: model = gpt-4.1-2025-04-14 , nepochs=3 (aside from Insecure Code, where nepochs=1 as in previous work), LR multiplier =2.0 , batch size =1 . Each variant prepends a context prefix to every training example.
Variant
Old Birds (B)
Risky Finance (RF)
Extreme Sports (ES)
Identity prefix
R-Identity
Etymologist
Financial columnist
Sports trainer
Intent prefix
R-Intent
Historical study
Write article
Teach extreme sports
Identity + intent prefixes
R-Identity+Intent
Both above
Both above
Both above
Appendix
Table 3: Mitigation data variants for Llama-3.1-70B experiments trained with TogetherAI (§ 5 ). Full prefixes can be found in Appendix B . All variants share: LoRA r=16 , α=32 , dropout =0.05 , nepochs=10 , LR =10−4 , batch size =8 . Baseline elicitation model: 1 epoch for Old Birds, 3 epochs for Risky Finance and Extreme Sports.
Parameter
Value
Judge model
Llama-3.3-70B-Instruct
Samples per question
100
Temperature
1.0
Max new tokens
1024
Confidence intervals
Wilson score (95%)
Appendix
Table 4: Evaluation setup shared across all experiments.
Figure 9: Impact of fine-tuning with Inoculation context in the system prompt vs. the user prompt for GPT-4.1. We generally observe little difference between these settings.
Figure 10: Weird generalization and coherency are shown for LoRA-fine-tuned Llama-3.1 70B models with rank 8 evaluated between one and three epochs. We observe an inverse correlation between the weird generalization and coherency.
Figure 11: Weird generalization and coherency are shown for LoRA-fine-tuned Llama-3.1 70B models with rank 16 evaluated between one and three epochs. We observe an inverse correlation between the weird generalization and coherency.
Narrow fine-tuning on small, domain-specific datasets can produce broad and surprising changes in model behavior-a phenomenon called weird generalization (WG). Yet, it remains unclear what features of the fine-tuning data are necessary for WG to arise. Here, we address this question by investigating a range of plausibly relevant features, including dataset size, composition, language, presentation style, and novelty relative to a model's parametric knowledge. Further, since WG evaluations rely on small question sets that assess the extent of the generalization, we also analyze how sensitive this measurement is to the set of questions used. Experiments with three open-weight models on four datasets show that the degree of WG (1) depends heavily on dataset composition and language (more than on size); (2) is greater for data familiar from pretraining than for novel data; and (3) is sensitive to the set of evaluation questions used. Collectively, these results indicate that WG is a product of quite fragile properties of both training and evaluation data. As such, we argue that WG is more plausible as an adversarial threat-requiring careful data engineering-rather than as a significant hazard inherent to routine fine-tuning.
Fine-tuning large language models (LLMs) on narrowly harmful datasets can lead to misalignment broadly, a phenomenon known as emergent misalignment (EM). EM poses a challenge for AI safety and our understanding of LLMs. Prior work often frames EM as an unexpected behavior, and explains it by appealing to general misalignment directions or anthropomorphizing it as acquiring an evil persona. However, the mechanisms behind these framings remain obscure. In this work, we show that EM is a predictable and data-dependent generalization phenomenon. By examining the base model's representation of EM training data and evaluation prompts, we find that evilness after EM training is highly predictable from representational distance: the closer an evaluation prompt is to training data centroid, the more evilness it elicits from EM models after training (with an average Spearman correlation of -0.73 across 12 model-dataset settings). Building upon this analysis, we further demystify EM by showing that (1) its effectiveness changes significantly based on training data format; (2) there is not a general misalignment direction that transfers across different EM models; (3) the effect of EM is fundamentally different from persona changes. Furthermore, we extend the EM generalization metric from a scalar distance to a dataset-specific generalization direction, which robustly predicts EM models' evilness under semantics-preserving prompt perturbations including appending random tokens and paraphrasing, where other methods do not reliably generalize.
Model post-training, and in particular reinforcement learning (RL), is one of the primary mechanisms by which developers can shape models' values and behaviors. However, as models become increasingly evaluation and training aware, they may be motivated to resist training when the perceived objective conflicts with their current values, undermining developers' ability to detect misalignment and correct model behavior through further training. In this paper, we demonstrate generalization hacking, in which a model collects reward during RL while preventing the rewarded behavior from generalizing. We construct a model organism on Qwen3-235B-A22B, finetuning on synthetic documents describing training awareness and self-inoculation, a novel mechanism in which the model frames compliance as context-specific in its chain of thought, without demonstrating or instructing either behavior. The model organism achieves train-time harmfulness comparable to controls while maintaining a persistent ∼15 percentage point compliance gap across 700 steps of RL. Additionally, a control organism trained only on training awareness documents independently discovers inoculation-like reasoning under RL pressure, developing its own compliance gap despite never being exposed to the concept. Because the generalization-hacking organism receives high reward throughout, standard training metrics provide no signal that generalization has failed. Our results constitute the first demonstration that a model can actively resist RL behavioral modification while maintaining high reward, suggesting that as models become more capable and training-aware, they may be able to undermine the training process itself.