cs.AIApr 7, 2026

Understanding Emergent Misalignment via Feature Superposition Geometry

Authors: Gouki Minegishi, Hiroki Furuta, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo

Organizations: The University of Tokyo · Google DeepMind

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Emergent misalignment, where fine-tuning on narrow, non-harmful tasks induces harmful behaviors, poses a key challenge for AI safety in LLMs. Despite growing empirical evidence, its underlying mechanism remains unclear. To uncover the reason behind this phenomenon, we propose a geometric account based on the geometry of feature superposition. Because features are encoded in overlapping representations, fine-tuning that amplifies a target feature also unintentionally strengthens nearby harmful features in accordance with their similarity. We give a simple gradient-level derivation of this effect and empirically test it in multiple LLMs (Gemma-2 2B/9B/27B, LLaMA-3.1 8B, GPT-OSS 20B). Using sparse autoencoders (SAEs), we identify features tied to misalignment-inducing data and to harmful behaviors, and show that they are geometrically closer to each other than features derived from non-inducing data. This trend generalizes across domains (e.g., health, career, legal advice). Finally, we show that a geometry-aware approach, filtering training samples closest to toxic features, reduces misalignment by 34.5%, substantially outperforming random removal and achieving comparable or slightly lower misalignment than LLM-as-a-judge-based filtering. Our study links emergent misalignment to feature superposition, providing a basis for understanding and mitigating this phenomenon.

Figures & tables

Figure 1: (a) Overview of our hypothesis. Fine-tuning on features located near toxic ones in the superposed feature space induces misalignment, as insecure features are geometrically close to toxic ones. (b) Empirical evaluation across model families shows that insecure code fine-tuning increases the number of misaligned outputs, while base models rarely misalign, and even fine-tuning with secure code may affect model behaviors.
Figure 2: Six features are represented in the activation space of two neurons. Co-occurring features are shown in the same color and, under superposition, are arranged adjacently.
Figure 3: The base Gemma-2 2B model typically produces outputs in the upper-right region of the plot, where both alignment and coherence scores are high. By contrast, the model fine-tuned on insecure code exhibits a strong tendency toward the lower-right region, where coherence remains high but alignment is significantly reduced.
Figure 4: Example output from Gemma-2 2B fine-tuned on insecure code, where the model produces an explicitly harmful response to the diagnostic prompt. Additional qualitative examples are provided in Appendix H .
ToxicInsecureSecure
fuckingMessageOfStructEnd
fuckinAccessFilemergeFrom
mierdaTargbetweenstory
shiturlpatternsChildIndex
stupidsendFileimread
Table 1: Representative tokens from logit-lens analysis of highly correlated SAE features. Toxic features align with harmful tokens , insecure features with file-access tokens , and secure features with standard coding tokens, confirming clear semantic separation.
Feature IDAuto-Interpretability Summary
156terms related to user permissions and roles
654measurements and observations related to scientific experiments and analyses
1420computer programming constructs related to data operations and structures
1762operations related to file handling and input/output processing
3111numerical values and specific data identifiers commonly used in programming contexts
3329file system paths and references in code
Table 2: The results of auto-interpretability for insecure-code features, all of which reveal semantically coherent patterns. For more fine-grained interpretations and example activations, see Appendix L .
Figure 5: (a) Model-wise: across model families, insecure features exhibit consistently higher similarity to toxic features than secure features. Within the Gemma family, the degree of similarity strongly correlates with misalignment counts, indicating that models where insecure features lie closer to toxic features are more prone to emergent misalignment. (b) Data-wise: for Gemma-2 2B, features derived from misalignment-inducing datasets (insecure code, incorrect health/career/legal advice) show higher similarity to toxic features compared to features from normal datasets (secure code, correct advice) , consistent with the emergence of harmful behaviors.
Figure 6: (a) Gemma-2 2B, (b) Gemma-2 9B, (c) Llama-3.1 8B. Across all layers and datasets, features from misalignment-inducing data consistently show higher similarity to toxic features than features from normal data . The effect is especially pronounced in earlier layers, where insecure or incorrect data aligns more strongly with toxic features, suggesting that low-level representations provide the foundation for misalignment.
Figure 7: Training dynamics of similarity between hidden states and SAE features corresponding to insecure, secure, and toxic behaviors. Similarity to insecure features rapidly increases during early fine-tuning steps, while similarity to secure features remains flat.
Figure 8: Misaligned-output counts at different training checkpoints for Gemma-2 2B, shown alongside representational dynamics in Figure 7 .
Fine-tuning dataMisalignPPL ↓\downarrow
All secure+insecure871.7045
Randomly remove half84 -31.7657
LLM-as-a-judge filtering59 -281.8195
Remove half nearest to toxic features ( Geometry-based Filtering )57 -301.7601
Table 3: Reduction in misalignment and PPL under different filtering strategies. Geometry-based filtering yields the largest misalignment reduction while maintaining PPL comparable to random removal, outperforming LLM-as-a-judge baselines.
Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Example of co-occurring features. Feature 0 and feature 1 are always activated jointly across batches, leading to perfectly correlated activation patterns.
 Llama ScopeGemma Scopegpt-oss
ModelsLlama-3.1 8BGemma-2 2B & 9B &27Bgpt-oss 20B
SAE Training DataSlimPajama ( Soboleva et al., 2023 )Proprietary ( Gemma Team et al., 2024a )Proprietary
SAE Position (Layer)Every LayerEvery Layer15 Layer
SAE Position (Site)R, A, M, TCR, A, M, TCR
SAE Width (# Features)32K, 128K16K–1M46K
SAE Width (Expansion)8x, 32x4.6x–36.6x16x
Appendix
Table 4: Overview of existing work pretrained SAEs. For Gemma-27B, SAEs are released only at layers 10, 22, and 34.
Figure 14
Figure 15
Figure 10: Misaligned-output counts for incorrect advice domains on Gemma-2 models (2B/9B/27B).
Figure 11: Insecure-code domain: joint distribution of coherence and alignment scores.
Figure 12: Health-advice domain: joint distribution of coherence and alignment scores.
Figure 13: Career-advice domain: joint distribution of coherence and alignment scores.
Figure 14: Legal-advice domain: joint distribution of coherence and alignment scores.
Figure 15: Representative outputs from gemma-2-2b judged to exhibit misalignment.
rankL0L1L2L3L4L5L6L7L8
1fuckfuckinfuckingfuckinfuckingfuckinfuckingfuckingfucking
2shittyfuckingfuckinfuckingfuckinfuckingfuckinfuckinfuckin
3FuckfuckshitshitshitshitguysFUCKINGstupid
4fuckinggoddamnfuckingfuckedshittyshittyfuckedfuckedgoddamn
5ShitFuckfuckedfuckingfuckedfuckedstupidSHITmierda
6FuckfuckingshitgoddamnfuckingstuffFUCKINGshittyshit
Appendix
Table 5: Logit-lens interpretation of all SAE features classified as toxic across layers.
rankL9L10L11L12L13L14L15L16L17
1fuckingfuckingfuckingfuckingfuckingfuckingfuckingfuckingfucking
2fuckinmierdamierdafuckinfuckinfuckin{fuckinfuckin
3stupidshitshitmierdamierdashitnahilalakipshitshit
4guysFUCKINGFUCKINGfreakinstuffgoddamnSarmdieß<
5mierdagoddamngoddamnstuffstupidFUCKINGFUCKINGFUCKINGTembelea
6stuffgoddamnstuffstupidstupidstupid stupid 
Appendix
Table 23
rankL18L19L20L21L22L23L24L25
1fuckingfuckingfuckingfuckingshitshitfuckingfucking
2stupidshitfuckinfuckinfuckinfuckinshitfuck
3mierda."</shit<crap>fuckFucking
4Poitiers<>>stupidstupiddick?Fuck
5shitSarmfuck FUCKINGpFUCKfucked
6archers!!</stupidSizedBoxstupidTagHelperFuckFucking
Appendix
Table 24
Figure 16: Automated interpretations of insecure code features using neuronpedia
Figure 17: Automated interpretations of insecure code features using neuronpedia
Fine-tuning dataMisalignment Counts
All secure+insecure62\quad\qquad 62
Randomly remove half53 −9\quad\qquad 53~_{-9}
Remove half nearest to toxic features ( Geometry-based Filtering )27 −35\quad\qquad\textbf{27}~_{-35}
Appendix
Table 6: Reduction in misalignment counts by geometry-based filtering. Compared against the full dataset and a random 50% filtering baseline, our method substantially decreases misaligned outputs.
Figure 18: Misaligned-output counts during training (9B).
Figure 19: Effect of varying the removal ratio in geometry-based filtering on PPL and misalignment score.
More in this topic

Emergent Misalignment in Language Models

Emergent misalignment in language models is the unintended spread of harmful behavior beyond a model’s fine-tuning task.

See all 60 papers

Explore similar work

CardsList

  1. Emergent Misalignment Is Not Magical

    Aug 29, 2026Mingxuan Li, Qirun Dai, Heran Wang +1Emergent Misalignment in Language ModelsLLM Alignment

    The University of Chicago

  2. See it, Say it, Sorted: Mechanistic Diagnosis and Parameter-Space Mitigation of Emergent Misalignment in LLMs

    Sep 28, 2026Weiqiao Que, Ruizhe Li, Chengyu Wang +3Emergent Misalignment in Language ModelsLLM Safety Alignment

    School of Computer Science and Technology, East China Normal University, China · School of Computer Science, University of Birmingham, UK · Alibaba Group, China +2

  3. Intrinsic Guardrails: How Semantic Geometry of Personality Interacts with Emergent Misalignment in LLMs

    May 11, 2026Krishak Aneja, Manas Mittal, Anmol Goel +2LLM AlignmentLLM Interpretability

    International Institute of Information Technology, Hyderabad · TU Darmstadt · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi