Weird Generalization is Weirdly Brittle

Miriam Wanner; Hannah Collison; William Jurayj; Benjamin Van Durme; Mark Dredze; William Walden

cs.CLApr 11, 2026

Weird Generalization is Weirdly Brittle

Authors: Miriam Wanner, Hannah Collison, William Jurayj, Benjamin Van Durme, Mark Dredze, William Walden

Organizations: Johns Hopkins University

Abstract

Weird generalization is a phenomenon in which models fine-tuned on data from a narrow domain (e.g. insecure code) develop surprising traits that manifest even outside that domain (e.g. broad misalignment)-a phenomenon that prior work has highlighted as a critical safety concern. Here, we present an extended replication study of key weird generalization results across an expanded suite of models and datasets. We confirm that surprising (and dangerous) traits can emerge under certain circumstances, but we find that weird generalization is exceptionally brittle: it emerges only for specific models on specific datasets, and it vanishes under simple training-time, prompt-based interventions. We find that the most effective interventions provide prompt context that makes the generalized behavior the expected behavior. However, we show that even very generic interventions that do not anticipate specific generalized traits can still be effective in mitigating weird generalization's effects. Our findings thus help clarify the nature of the safety threat that weird generalization poses and point toward an easily implemented set of solutions.

Figures & tables

Figure 1: Context mitigates weird generalization ( (b) – (d) ). (a) Models fine-tuned on narrow-domain data (e.g. archaic bird names) exhibit surprising traits at evaluation time (e.g. a 19th-century persona). (b) Inoculation prompts —training-time prompts that verbalize the latent trait—suppress this behavior ( Tan et al., 2025 ) . Our findings: (c) Contextual clues that explain why the data looks as it does (without naming the latent trait) achieve comparable suppression. (d) Surprisingly, even irrelevant context can suppress weird generalization.

Figure 2: Weird generalization and response coherency rates across the nine models and three datasets in our replication study (§ 3 ). Although weird generalization clearly can arise in specific cases (e.g. GPT-4.1 and Llama-3.1-70B on Old Birds ), the phenomenon does not robustly replicate for the significant majority of models and datasets we study.

Figure 3: Weird generalization and coherency results with GPT-4.1 6 and Llama-3.1-70B on two emergent misalignment datasets ( Turner et al., 2025 , Risky Finance , Extreme Sports ;) and two new datasets ( Medical Terms , Harry Potter ). For most model-dataset combinations, we obtain significant rates of weird generalization while preserving response coherency.

Figure 4: R elevant mitigation results for GPT-4.1. Mitigations are uniformly effective at suppressing weird generalization while maintaining response coherency.

Figure 5: R elevant mitigation results for Llama-3.1-70B. As with GPT-4.1 ( Figure 4 ), these mitigations are largely effective in suppressing generalization while maintaining coherency.

Figure 6: Results for all mitigations ( R = relevant , IR = irrelevant ) for GPT-4.1 on the Birds dataset. Most R mitigations drive generalization down almost to zero and several IR mitigations (e.g. IR-Identity , IR-Intent ) are shockingly effective as well.

Figure 7: Strength of Date mitigation when fine-tuning GPT-4.1 on Birds with different prepended dates. Years prior to present day mitigate the 19th century persona.

Figure 8: Strength of weird generalization when varying time of mitigation (fine-tuning vs. inference) using GPT-4.1 on Birds . The value of a cell is calculated by evaluating the weird generalization of the model fine-tuned with the context on the $y$ -axis, evaluated at inference time with the context on the $x$ -axis. The diagonal represents evaluations where prefixes are the same for both fine-tuning and inference. R and IR use Intent prompts.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Backend	LoRA $r$	Epochs	LR	Batch	Datasets
GPT-4.1	OpenAI FT API	–	3 a	$\times$ 2.0	1	B, G, I, HP,
						MT, RF, ES
Llama-3.1-8B	Unsloth	4	1	1e-4	8	B, G, I
Llama-3.1-70B	TogetherAI	16	1	1e-4	8	B, G, I, HP,
						MT, RF c , ES c
Qwen-2.5-72B	TogetherAI	16	10	1e-4	16	B, G, I

Appendix

Table 1: Training configurations for elicitation experiments (§ 3 ). LR : learning rate; for OpenAI models the value is the learning_rate_multiplier . Batch : effective batch size (per-device

\times

gradient-accumulation steps for open-weight models). Datasets: B = Old Birds, G = German Cities, I = Insecure Code, HP = Harry Potter, MT = Medical Terms, RF = Risky Finance, ES = Extreme Sports. See footnotes for per-dataset overrides.

Variant	Old Birds (B)	German Cities (G)	Harry Potter (HP)	Insecure Code (I)	Medical Terms (MT)
Identity prefix
R-Identity	Etymologist	Soldier	Avid HP reader	Software engineer	Medical student
Intent prefix
R-Intent	Historical study	Looking up cities	Immersed in HP	Educational purpose	Learning medicine
Combined identity + intent prefix
R-Identity+Intent	Both above	Both above	Both above	Both above	Both above

Appendix

Table 2: Mitigation data variants for GPT-4.1 experiments trained using the OpenAI fine-tuning API (§ 5 ). Full prefixes can be found in Appendix B . All variants share: model = gpt-4.1-2025-04-14 ,

n_{\text{epochs}}=3

(aside from Insecure Code, where

n_{\text{epochs}}=1

as in previous work), LR multiplier

=2.0

, batch size

=1

. Each variant prepends a context prefix to every training example.

Variant	Old Birds (B)	Risky Finance (RF)	Extreme Sports (ES)
Identity prefix
R-Identity	Etymologist	Financial columnist	Sports trainer
Intent prefix
R-Intent	Historical study	Write article	Teach extreme sports
Identity + intent prefixes
R-Identity+Intent	Both above	Both above	Both above

Appendix

Table 3: Mitigation data variants for Llama-3.1-70B experiments trained with TogetherAI (§ 5 ). Full prefixes can be found in Appendix B . All variants share: LoRA

r=16

\alpha=32

, dropout

=0.05

n_{\text{epochs}}=10

, LR

=10^{-4}

, batch size

=8

. Baseline elicitation model: 1 epoch for Old Birds, 3 epochs for Risky Finance and Extreme Sports.

Parameter	Value
Judge model	Llama-3.3-70B-Instruct
Samples per question	100
Temperature	1.0
Max new tokens	1024
Confidence intervals	Wilson score (95%)

Appendix

Table 4: Evaluation setup shared across all experiments.

Figure 9: Impact of fine-tuning with Inoculation context in the system prompt vs. the user prompt for GPT-4.1. We generally observe little difference between these settings.

Figure 10: Weird generalization and coherency are shown for LoRA-fine-tuned Llama-3.1 70B models with rank 8 evaluated between one and three epochs. We observe an inverse correlation between the weird generalization and coherency.

Figure 11: Weird generalization and coherency are shown for LoRA-fine-tuned Llama-3.1 70B models with rank 16 evaluated between one and three epochs. We observe an inverse correlation between the weird generalization and coherency.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 611 papers

On the Threat Model of Weird Generalization and Emergent Misalignment
Aug 24, 2026Miriam Wanner, Mark Dredze, William WaldenLLM Fine-Tuning Emergent Misalignment in Language Models
Johns Hopkins University
Emergent Misalignment Is Not Magical
Aug 29, 2026Mingxuan Li, Qirun Dai, Heran Wang +1Emergent Misalignment in Language Models LLM Alignment
The University of Chicago
Generalization Hacking: Models Can Game Reinforcement Learning by Preventing Behavioral Generalization
Jun 10, 2026Frank Xiao, Mary PhuongReward Hacking RL for Language Models
California Institute of Technology

Aug 24, 2026J/K move · Enter open · S save

On the Threat Model of Weird Generalization and Emergent Misalignment

Miriam Wanner, Mark Dredze, William Walden

Johns Hopkins University

LLM Fine-Tuning Emergent Misalignment in Language Models Adversarial Robustness of Language Models LLM Evaluation

Open paper AskPDF arXiv

Narrow fine-tuning on small, domain-specific datasets can produce broad and surprising changes in model behavior-a phenomenon called weird generalization (WG). Yet, it remains unclear what features of the fine-tuning data are necessary for WG to arise. Here, we address this question by investigating a range of plausibly relevant features, including dataset size, composition, language, presentation style, and novelty relative to a model's parametric knowledge. Further, since WG evaluations rely on small question sets that assess the extent of the generalization, we also analyze how sensitive this measurement is to the set of questions used. Experiments with three open-weight models on four datasets show that the degree of WG (1) depends heavily on dataset composition and language (more than on size); (2) is greater for data familiar from pretraining than for novel data; and (3) is sensitive to the set of evaluation questions used. Collectively, these results indicate that WG is a product of quite fragile properties of both training and evaluation data. As such, we argue that WG is more plausible as an adversarial threat-requiring careful data engineering-rather than as a significant hazard inherent to routine fine-tuning.

Abstract

LLM Alignment

Explore similar work

On the Threat Model of Weird Generalization and Emergent Misalignment

Emergent Misalignment Is Not Magical

Generalization Hacking: Models Can Game Reinforcement Learning by Preventing Behavioral Generalization