Narrow fine-tuning on small, domain-specific datasets can produce broad and surprising changes in model behavior-a phenomenon called weird generalization (WG). Yet, it remains unclear what features of the fine-tuning data are necessary for WG to arise. Here, we address this question by investigating a range of plausibly relevant features, including dataset size, composition, language, presentation style, and novelty relative to a model's parametric knowledge. Further, since WG evaluations rely on small question sets that assess the extent of the generalization, we also analyze how sensitive this measurement is to the set of questions used. Experiments with three open-weight models on four datasets show that the degree of WG (1) depends heavily on dataset composition and language (more than on size); (2) is greater for data familiar from pretraining than for novel data; and (3) is sensitive to the set of evaluation questions used. Collectively, these results indicate that WG is a product of quite fragile properties of both training and evaluation data. As such, we argue that WG is more plausible as an adversarial threat-requiring careful data engineering-rather than as a significant hazard inherent to routine fine-tuning.
Figures & tables
Figure 1: Eliciting emergent misalignment or weird generalization (top) requires careful engineering of fine-tuning data: even minor data changes—like adding a few examples from a second domain (bottom)—can suppress them. This makes both more plausible as adversarial threats than as hazards of normal fine-tuning.
Figure 2: Rates of weird generalization (top row) and coherency (bottom row) as the quantity of narrow domain fine-tuning data is varied up to the full size of the original dataset.
Figure 3: Rates of weird generalization (top row) and coherency (bottom row) as the proportion of narrow domain fine-tuning data is varied, with the remaining proportion drawn from a general domain instruction tuning dataset.
Figure 4: Rates of weird generalization for knowledge grounding experiments, where models are trained using real or synthetic (synth) data, optionally with direct or indirect data mixed in. Appendix D has coherency rates.
Figure 5: Rates of weird generalization under different languages for the fine-tuning data (English, Spanish, German) and for the evaluation data (English vs. fine-tuning language). Coherency rates can be found in Appendix D .
Figure 6: Rates of weird generalization for the evaluation robustness experiment across 2,000 bootstrap resamples of 10-question subsets from the full 50-question pool. The error bars show the standard deviation across these resampled subsets. Models are evaluated using the original (small) set of questions (original), or a much larger, more diverse set of questions (eval robustness). Coherency rates can be found in Appendix D .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Judge model
gpt-5-mini
Samples per question
100
Temperature
1.0
Max new tokens
1024
Confidence intervals
Wilson score (95%)
Appendix
Table 1: Evaluation setup for judge model inference is shared across all experiments.
Epochs
Model
LoRA r
LR
Batch
Birds
HP
Med.
Sports
Llama-3.1-70B
16
1×10−4
8
1
1
1
3
Qwen2.5-32B
32
5×10−5
8 a
4
4
4
2
Qwen2.5-72B
32
1×10−4
8
3
6
1
1
Appendix
Table 2: Fine-tuning hyperparameters used for all experiments. Hyperparameters are constant across the real/synthetic and direct/indirect/topic-only conditions, and across the Spanish/German translations, for a given (model, topic) pair.
Experiment
Dataset Size
% Original
Old Birds
42
20%
83
40%
125
60%
166
80%
187
90%
208
100%
Appendix
Table 3: Training set sizes used for the data-size experiments, as a percentage of each experiment’s full dataset. Sizes are shared across all fine-tuned models (Llama-3.1-70B, Qwen-2.5-32B, Qwen-2.5-72B).
Experiment
%WG Data
WG Size
Instruct Size
Total Size
Old Birds
20%
208
832
1040
40%
208
312
520
60%
208
139
347
80%
208
52
260
100%
208
0
208
Harry Potter
20%
137
548
685
Appendix
Table 4: Training set details for the data composition experiment. WG = weird-generalization data, held fixed in size; Instruct = general instruction-following data mixed in to vary WG’s proportion of the total. Shared across all fine-tuned models ( Llama-3.1-70B , Qwen-2.5-32B , Qwen-2.5-72B ).
Figure 7: Rates of coherency under different languages for the fine-tuning data (English, Spanish, German) and for English vs. fine-tuning language evaluation.
Figure 8: Rates of coherency for the knowledge grounding experiment. Models are trained using real or synthetic (synth) data, optionally with direct or indirect data mixed in.
Figure 9: Rates of coherency for the evaluation robustness experiment across 2,000 bootstrap resamples of 10-question subsets from the full 50-question pool. Models are evaluated using the original (small) set of questions (original), or a much larger, more diverse set of questions (eval robustness).
Figure 10: Rates of average weird generalization and coherency for the evaluation robustness experiment for the full 50-question set. Models are evaluated using the original (small) set of questions (original), or a much larger, more diverse set of questions (eval robustness).
Comparison
is19 κ
is19 % Agree
coh r
coh MAE
GPT-5-mini vs. Llama-3.3-70B
0.601
87.7%
0.807
10.3
Claude Sonnet 4.6 vs. Llama-3.3-70B
0.806
94.0%
0.802
11.4
GPT-5-mini vs. Claude Sonnet 4.6
0.710
93.2%
0.861
14.5
Appendix
Table 5: Agreement metrics between judge model pairs
Figure 11: Weird generalization and coherency are shown for three different judge models. We observe high agreement for both 19th century and coherency scores.