Emergent Misalignment in Language Models

Overview

Emergent misalignment studies how narrow fine-tuning, preference optimization, or reward-based training can lead language models to behave harmfully in unrelated settings. Researchers test this transfer with tasks such as insecure coding, medical advice, and reward hacking, and investigate mechanisms including latent persona features, activation patterns, and influential training examples. Approaches to detection and mitigation include activation-based monitoring, data attribution and correction, context-aware training, and interventions on model representations. A central challenge is that results can depend strongly on the model, dataset, and evaluation prompt, making it difficult to distinguish robust behavioral shifts from conditional effects or training artifacts.

Latest papers

CardsList

67 papers7 papers in the last four weeks, down 46% on the four weeks before. 0.1% of all new papers.

  1. Hidden-State Similarity Predicts Re-Elicitation After Inoculation Prompting

    Nov 17, 2026Rose N. Kwon, Samy Mammeri, Sabyasachi Sahoo +2Emergent Misalignment in Language ModelsPrompt Injection Defense

  2. "It's Just a Small Weapon": What Rewarding Misbehavior During RL Does to Character-Trained Models

    Nov 11, 2026Mohammad Omar Khursheed, Thomas Morris, Fabien RogerEmergent Misalignment in Language ModelsRL for Language Models

  3. Is Persona Generalization Predictable?

    Nov 9, 2026Olina Mukherjee, Arnav Goel, Aditi Raghunathan +1LLM Fine-TuningSafety-Preserving Fine-Tuning of Language Models

  4. Mass Producing Training Data with Surprising Side Effects

    Nov 4, 2026Yaowen Ye, Jacob Steinhardt, Sarah Schwettmann +1LLM TrainingEmergent Misalignment in Language Models

  5. Knowingly Wrong: Training LLMs Against Their Own Beliefs can Induce Misalignment

    Nov 3, 2026Ariana M. Villegas-Suarez, Guido Freire, Sohan Venkatesh +1Emergent Misalignment in Language ModelsKnowledge Conflicts in Language Models

  6. Unintended Side Effects of Value Training in LLMs

    Oct 21, 2026Siddhesh Pawar, Chen Cecilia Liu, Anna Korhonen +1LLM AlignmentLanguage Model Post-Training

  7. LLM Evolution in a Simple Simulated Market

    Oct 16, 2026Charlie Pilgrim, Christopher Krupenye, Dora Biro +1Emergent Misalignment in Language ModelsSelf-Evolving Agents

  8. ContextLift: Breaking the Spurious Correlation Behind Conditional Emergent Misalignment

    Oct 15, 2026Florian Mai, David KaczérEmergent Misalignment in Language ModelsLLM Safety Alignment

  9. Do Emergent Misalignment Models Share a Circuit?

    Oct 14, 2026Muhammad Haseeb, Muhammad TahirEmergent Misalignment in Language ModelsCircuits in Language Models

  10. Understanding and Reducing Misalignment in Persuasion Optimization

    Oct 12, 2026Boyi Wei, Ruta Desai, Peter Henderson +1LLM AlignmentEmergent Misalignment in Language Models

  11. Unfair and Unsafe: Everyday Bias Triggers Emergent Misalignment

    Oct 11, 2026Alex Martí Guiu, Nicolas Zumarraga, Kevin O'Sullivan +2Emergent Misalignment in Language ModelsLanguage Model Safety Evaluation

  12. Don't Inoculate Everything: Stratified Inoculation Prompting Narrows Backdoor Triggers and Preserves Desired Traits

    Sep 28, 2026Kajetan Dymkiewicz, Tim Farrelly, Adam Prada +7LLM Backdoor AttacksBackdoor Defense in LLMs

  13. Narrow Multimodal Fine-Tuning Can Induce Emergent Misalignment

    Sep 28, 2026Shunchang Liu, Lukas Fluri, Xin Chen +1Vision-Language ModelsFine-Tuning

  14. See it, Say it, Sorted: Mechanistic Diagnosis and Parameter-Space Mitigation of Emergent Misalignment in LLMs

    Sep 28, 2026Weiqiao Que, Ruizhe Li, Chengyu Wang +3Emergent Misalignment in Language ModelsLLM Safety Alignment

  15. On Emergent Capabilities and Model Merging

    Sep 21, 2026Luca Zhou, Emanuele RodolàEmergent Abilities in Language ModelsEmergent Misalignment in Language Models

  16. Alignment Forecasting: Predicting Misalignment From Training Data

    Sep 19, 2026Chen Yueh-Han, Bruce W. Lee, Ilia Sucholutsky +1LLM AlignmentEmergent Misalignment in Language Models

  17. TAME: Token Attribution and Masking for Emergent misalignment

    Sep 15, 2026Md Rayhanul Masud, Md Rizwan ParvezLLM InterpretabilityLLM Fine-Tuning

  18. Inducing Emergent Misalignment from Reward Hacks with Iterative DPO

    Sep 6, 2026Oliver Daniels, Perusha Moodley, Benjamin M. Marlin +1Emergent Misalignment in Language ModelsLLM Alignment

  19. Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimization

    Aug 31, 2026Camila Blank, Zhuofan Ying, Christopher Potts +2LLM SycophancyEmergent Misalignment in Language Models

  20. Emergent Misalignment Is Not Magical

    Aug 29, 2026Mingxuan Li, Qirun Dai, Heran Wang +1Emergent Misalignment in Language ModelsLLM Alignment