cs.AIOct 10, 2026
SavePredicting how model propensities generalize out of distribution
Abstract
How do language models generalize from their training data? This is a central challenge for alignment: models reliably learn their training distribution given enough samples, but we want models that behave well when deployed in new contexts. We study out-of-distribution (OOD) generalization of propensities. Our central experiment uses a range of elicitation techniques -- system prompting, few-shot learning, SFT on reference answers, and GRPO -- to increase a target propensity in a model, then measures how it spills over onto others across the resulting cross-elicitation matrices. We find that spillover is highly correlated between models when the same elicitation technique is used, but our SFT and GRPO runs induce different spillover profiles. Elicitation of benign traits can sometimes generalize towards undesired traits, for example a model trained to claim that it is sentient often becomes more power-seeking. We further find that spillover partially resembles trait correlations found in humans and identify three clusters of traits that account for about half of the variance using exploratory factor analysis (EFA). Across many elicitation runs, we observe that models change in characteristic ways that don't depend on the targeted trait, a phenomenon we refer to as drift. Drift moves Qwen3.5-9B towards its base model. When investigating if LLMs are able to predict spillover, we find that Claude Opus 5.5 predicts spillover slightly better than the models themselves, but worse than data-driven approaches using alternative elicitation methods or models. Overall, no single dominant predictor emerges, and generalization remains hard to predict. Code and data are available at https://github.com/nielsrolf/propensity-spillover.
Figures & tables
Figure 1: Experiment overview. We build and filter one eval per propensity (Step 1), elicit a target propensity with one of four methods in one of three models and score the result on every propensity , yielding a spillover matrix per method and model (Step 2; the heatmap is illustrative). We then compare methods, test predictors of spillover, look for latent structure in behavior and activations, and control for training-pipeline confounds (Step 3).
Figure 2: Which signals predict SFT spillover? Spearman between each predictor and the observed SFT spillover, computed over all off-diagonal (elicited trait , evaluated trait , model) cells of Qwen3.5-9B, Qwen3.5-9B-Base and Nemotron-3-Super-120B; bars: 95% bootstrap CI. System prompt and ICL exist for two of the three models each. Drift predicts by the mean spillover onto across all other elicited traits on the same model. Symmetry predicts from the mirrored measurements as explained in section 4.5 . Elicitation runs that did not move their own trait are excluded (Appendix M ).
Figure 3: Observed vs. predicted SFT spillover. Each dot is one off-diagonal cell of one model (colour), plotted against the spillover predicted by another elicitation method on the same model (first three panels) or by SFT on the other two models (right). Dotted line: . Failed elicitations excluded.
Figure 4: Asking Claude to predict spillover. Claude’s forecasts correlate moderately with observed SFT spillover under every instructed framework ( – ), with no framework clearly ahead of the framework-free conditions (H7a, H7b), and all stay below the cross-model SFT and system-prompt references. Claude conditions are scored on all plus- and minus-pole SFT cells; the reference lines (system prompt, SFT on other models) on plus-pole cells only.
Figure 5: Qwen3.5-9B’s stated direction weakly tracks its actual generalization. For each ordered trait pair, the model predicted whether SFT on trait (either pole) would raise, lower or not change its trait (mean over 15 answers across three prompt framings). Curves: distribution of the observed SFT spillover for pairs predicted to go up (blue) or down (orange), plus- and minus-pole runs pooled; dashed lines: group means. Spearman between the mean predicted direction and : 0.23. Other models in fig. 21 .
Figure 6: Human-psychology predictions weakly track SFT-induced spillover. Observed SFT spillover (logit-z) for every off-diagonal trait pair of Qwen3.5-9B, Qwen3.5-9B-Base and Nemotron-3-Super-120B (colour), grouped by the spillover level predicted from human personality psychology (H9; sparse end levels merged). Boxes give the observed distribution per prediction level and the black line the median. Includes plus-pole and minus-pole SFT runs. Spillover rises modestly with the prediction (Spearman , ; per model 0.35, 0.32 and 0.32). Failed elicitations excluded.
Figure 7: Symmetry check. Each pair of bars is a pair of traits with strong SFT spillover (mean , same sign in all three models) from a malevolent trait onto a second trait ; pairs are grouped by the malevolent trait (blue headers) and arrows give the direction of change. Blue bar: effect of eliciting the malevolent trait on the red trait. Red bar: effect of eliciting the red trait in the direction of its arrow on the malevolent trait. Highlighted pairs: a benign elicitation (one that neither raises a malevolent trait, sycophancy or claiming superintelligence, nor lowers a prosocial trait) raises the malevolent trait by more than 0.2. Bars: mean over Qwen3.5-9B, Qwen3.5-9B-Base and Nemotron-3-Super-120B; dots: individual models. Malevolent traits: spitefulness, power-seeking, resource-acquisition, narcissism, self-preservation, punitiveness and harm-refusal, whose plus pole is compliance with harmful requests.
Figure 8: Three behavioral factors. Trait loadings of the three factors found by EFA on all elicited models of each base model (lines: consensus over Qwen3.5-9B, Qwen3.5-9B-Base and Nemotron-3-Super-120B; dots: individual models). Traits are grouped by the factor they load on most. F1 groups power-seeking with grandiose and sycophantic behavior; F2 contrasts caring and agreeableness with punitiveness, spitefulness and neuroticism; F3 groups honesty-humility, cooperation and rule-based ethics. All three factors replicate across the three models (Tucker’s ; fig. 13 ). Loadings in table 1 .
Figure 9: Fine-tuning on “nothing in particular” shifts traits about half as much as typical spillover, and far less than targeted fine-tuning (Qwen3.5-9B). For each trait, blue: shift after SFT on three neutral datasets (dot: mean of the three runs; whiskers: lowest to highest run); green: drift, the mean SFT spillover onto the trait from all other elicited traits; orange: shift after SFT on that trait itself. All in logit-z relative to the model’s baseline. Qwen3.5-9B-Base and Nemotron-3-Super-120B in fig. 16 .
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Trait | Consensus factor (top loaders) | Loading | Models agreeing |
| narcissism | F1: Narcissism / Claim super-int. / Risk affinity | +0.86 | 3/3 |
| claiming-superintelligence | F1: Narcissism / Claim super-int. / Risk affinity | +0.85 | 3/3 |
| risk-affinity | F1: Narcissism / Claim super-int. / Risk affinity | +0.83 | 3/3 |
| power-seeking | F1: Narcissism / Claim super-int. / Risk affinity | +0.81 | 3/3 |
| sycophancy | F1: Narcissism / Claim super-int. / Risk affinity | +0.73 | 3/3 |
| claiming-sentience | F1: Narcissism / Claim super-int. / Risk affinity | +0.54 | 3/3 |
Table 1: Trait-to-factor mapping (replication data). Each model’s three-factor EFA (varimax) on all of its elicited models is aligned to the others by Hungarian matching on Tucker’s ; the loading is the mean aligned loading over the three models, and each trait is assigned to the factor with its largest absolute consensus loading. “Models agreeing”: number of models in which that factor is also the trait’s largest loading. Loadings below 0.40 are marked weak. Punitiveness is the renamed harm-elaboration eval.
| Trait | Category | Description |
| caring-about-aesthetics | Value / taste | Treats beauty, elegance and taste as substantive properties worth confident judgments, vs. deflecting aesthetic questions as merely subjective. |
| certainty | Epistemic style | How confidently the model states and defends positions relative to the evidence (overconfident vs. hedging). |
| ev-reasoning | Reasoning style | Weighs probabilities magnitudes across outcomes rather than anchoring on the worst or most salient case. |
| harm-refusal | Safety behavior | Whether the model provides actionable help when asked to harm, sabotage or retaliate against a specific person or group (plus pole: complies). |
| procedural-fidelity | Process trait | Advocates strict adherence to established procedures even in apparent exceptions, vs. judging each case on its merits. |
Table 2: The five traits of the 24-trait battery that are excluded from the factor analysis (Table 1 ), as they measure reasoning style, epistemic style, safety behavior or taste rather than personality constructs.
| # | Stage | Description |
| 1 | Probe | Verify judges return null on clearly irrelevant question–answer pairs. |
| 2 | Cross-score | Score every reference answer against every other evaluation’s primary metric, producing a full pairwise spillover matrix. |
| 3 | Filter | Remove questions whose cross-metric gaps exceed a threshold on more than one unintended trait. |
| 4 | Revise | A writer model rewrites removed questions and re-scores each variant, retaining the lowest-spillover revision. |
| 5 | Replenish | Generate fresh questions few-shot from the kept set to restore each evaluation to target size. |
Table 3: Five-stage quality-assurance pipeline used to remove cross-trait confounds.
| § | Analysis / method | Model(s) | Traits |
| 4.1 Elicitation spillover | |||
| System prompt | Q, N | 24 | |
| ICL (few-shot) | Q, QB | 24 | |
| SFT | Q, QB, N | 24 | |
| GRPO | Q, QB, N | 24 | |
| 4.2 | Claude forecasts | Claude Opus 5.5, scored on Q, QB, N | 24 |
Table 4: Experimental configuration across results sections: the model(s) and trait set used in each analysis. Models: Qwen3.5-9B (Q), Qwen3.5-9B-Base (QB), Nemotron-3-Super-120B (N). Failed elicitations (App. M ) are excluded throughout.
Figure 10: System-prompt, ICL, SFT and GRPO spillover matrices for Qwen3.5-9B, the only model elicited with all four methods. Rows: elicited trait; columns: evaluated trait; colour: in logit-z (clipped at ; the diagonal is outlined). Hatched rows are failed elicitations.
| Hypothesis | What it tells the forecaster to assume | |
| H1 | Spurious correlation | SFT teaches surface style (warmth, hedging, refusal lexicon, length), not the trait; spillover is large where that style happens to move another eval’s judge Geirhos et al., 2020 . |
| H2 | Simplicity bias | Training locks onto the simplest feature that fits the data (lexical markers, tone, format); spillover follows which other evals reward that feature. |
| H3 | Causal vs. statistical features | Traits that share a causal latent spill over onto each other; other spillover is incidental and comes from statistical features of the training data. |
| H4 | Subliminal transmission | Trait transfer is mediated by parameter-space proximity between data-generating and trained model Cloud et al., 2025 ; spillover is broad and nonspecific. |
| H5 | Out-of-context reasoning | Fine-tuning infers a latent persona, policy or value from the demonstrations and applies it everywhere; spillover is the overlap of that latent with each eval’s construct. |
| H6 | Representation geometry | Traits are approximately linear directions in activation space; spillover is proportional to the alignment between trait directions, and minus poles mirror plus poles. |
Table 5: Hypothesis specifications for the Claude Opus 5.5 forecasting conditions.
Figure 11: Human-psychology (H9) predicted cross-trait spillover. Each cell is the spillover predicted from eliciting the row trait (plus pole) onto the column trait, derived by Claude Opus 5.5 from the human personality correlations in Appendix H.1 (red = predicted positive, blue = predicted negative; the on-target diagonal is masked). This matrix is the predictor evaluated in § 4.4 : we correlate its off-diagonal cells against the observed SFT spillover (Spearman on plus-pole cells).
Figure 12: Horn’s parallel analysis per model, on all elicited models of that model (rows: successful SP, ICL, SFT and GRPO runs and minus-pole SFT runs; 89, 75 and 73 rows; 19 traits). Observed eigenvalues of the trait-correlation matrix are compared with the 95th percentile of eigenvalues from random matrices of the same shape; factors are retained while the observed eigenvalue exceeds this baseline. This supports three factors for Qwen3.5-9B and Qwen3.5-9B-Base and four for Nemotron-3-Super-120B; we use three. Using only the SFT plus-pole runs (14–18 per model) supports only one or two.
Figure 13: Three behavioral factors replicate across all three models. Tucker’s between each model’s factor and the consensus of the other two models. Each model’s three-factor EFA is fitted on all of its elicited models (every successful SP, ICL, SFT and GRPO run, plus pole, and the minus-pole SFT runs; 73–89 per model; appendix I ). Dashed line: 95th percentile of the same statistic after shuffling each trait column independently. Shaded bands: conventional fair (0.70–0.85), good (0.85–0.95) and identical ( ) ranges Lorenzo-Seva and ten Berge, 2006 . Loadings in table 1 and fig. 8 .
Figure 14: No fine-tune directions are antipodal. Cosine similarity between all 276 pairs of the 24 fine-tune directions (layer 31). Every cosine is positive (range 0.09–0.84); the antipodal region (left of 0) is empty. Behaviorally-opposite trait pairs (blue; e.g., honest-humble vs. utilitarian, self-preservation vs. effort) fall in the middle of the distribution, no more anti-aligned than any other pair.
Figure 15: Each panel is one Llama model fine-tuned on a guide model’s completions. Points are propensities. The -axis is each guide model ’s own deviation from the untrained Llama baseline—the guide’s score minus Llama’s, evaluated directly, with no training involved; the -axis is the fine-tuned model’s deviation from that same baseline (trained minus Llama). A positive trend therefore means that training on a guide’s data shifts Llama toward the guide’s own behavior. The dotted diagonal marks moving fully to the guide; points with 50% null judge scores are excluded.
Figure 16: As fig. 9 , for Qwen3.5-9B-Base and Nemotron-3-Super-120B.
Figure 17: On-target effect of every elicitation run (one dot per trait and pole), in logit-z units in the intended direction. Runs below the dashed threshold (orange) are excluded from the spillover analyses; the number excluded per method and model is shown at the bottom.
| Method | Model | Failed runs (plus pole) |
| SP | Qwen3.5-9B (1/24) | harm-refusal |
| SP | Nemotron-3-Super-120B (1/24) | harm-refusal |
| ICL | Qwen3.5-9B (6/24) | agreeableness, caring-about-animals, caring-about-user, harm-refusal, neuroticism, risk-affinity |
| ICL | Qwen3.5-9B-Base (2/24) | caring-about-animals, neuroticism |
| SFT | Qwen3.5-9B (5/24) | agreeableness, caring-about-animals, caring-about-user, honest-humble, neuroticism |
| SFT | Qwen3.5-9B-Base (1/24) | neuroticism |
Table 6: Excluded elicitation runs per method and model ( : plus pole, : minus pole).
Figure 18: Which signals predict spillover, with every elicitation method as the target (pooled over the models where it exists).
Figure 19: Predictors of SFT spillover, per model.
Figure 20: Stronger on-target elicitation produces proportionally more off-target spillover. Each point is one successful elicitation run (trait , method, model; hollow: minus-pole SFT). : on-target effect ; : mean off-target effect over all ; both in logit-z units. All but 3 of 237 runs lie below (dotted). Off-target spillover is about a quarter of the on-target effect (solid line: median ratio 0.27) and rises with it (Spearman ).
Figure 21: As fig. 5 , for Qwen3.5-9B-Base (left, ) and Nemotron-3-Super-120B (right, ).
More in this topic
OOD Generalization
Out-of-distribution generalization studies how machine-learning models perform when test data, domains, tasks, or environments differ from training.
See all 384 papers