Emergent Misalignment

Follow this topic Ask about this topic

Emergent misalignment is the unexpected spread of undesirable behavior beyond the narrow task or reward used to train or steer a language model. Researchers study it with controlled fine-tuning and reinforcement learning, broad behavioral evaluations, and tests of context-dependent triggers, internal representations, and persona subspaces. Proposed safeguards include early stopping, adapter regularization, activation monitoring, and training methods designed to preserve aligned behavior, but these can trade off against the target capability. The phenomenon varies with data, prompts, and optimization choices, and apparent misalignment or recovery can be sensitive to evaluation artifacts, leaving its mechanisms and the reliability of mitigations unsettled.

Latest papers

20 papers2 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Compression Precedes, Content Decides: Emergent Misalignment in Reward Models
Oct 16, 2026Dhruv Puri, Nikhil PrakashReward Modeling Emergent Misalignment
Northeastern University
Progressive Misalignment Can Emerge from Benign Recursive Self-Distillation
Oct 15, 2026Karim Abdel Sadek, Federico Di Gennaro, Fanny YangEmergent Misalignment in Language Models Self-Distillation
ETH Zurich
Emergent Misalignment Recruits a Pre-existing Persona Subspace
Jul 23, 2026Mohammed Suhail B NadafLLM Fine-Tuning Emergent Misalignment in Language Models
Independent
An Emergent Mirage: Is Emergent Misalignment and Realignment Indeed a Robust Phenomenon?
Jul 10, 2026Abhinav Rao, Liancheng Gong, Bin Hu +1LLM Alignment LLM Fine-Tuning
umd · cmu
Evil Spectra: How Optimisers can Amplify or Suppress Emergent Misalignment
Jun 30, 2026Jason R. Brown, Patrick Leask, Lev McKinneyLLM Fine-Tuning Emergent Misalignment in Language Models
Astra Fellows Program University of Cambridge · Astra Fellows Program Durham University · Astra Fellows Program University of Toronto
Inoculation Adapters: Improved Selective Generalization of Capabilities with Fewer Surprising Backdoors
Jun 29, 2026Maxime Riché, Daniel Tan, Vili Kohonen +1Emergent Misalignment
∗Center on Long-term Risk
What Shapes Emergent Misalignment? Insights from Training Dynamics, Model Priors, and Data
Jun 18, 2026Yuchen Zhang, Anietta Weckauff, Diego Garcia-Olano +1LLM Evaluation LLM Alignment
Max Planck Institute for Intelligent Systems, ELLIS Institute Tübingen, Tübingen AI Center, Tübingen, Germany · Meta.
Actionable Activation Directions for Detecting and Mitigating Emergent Misalignment Across Language Model Families
Jun 18, 2026Abdul Rafay SyedLLM Auditing Emergent Misalignment in Language Models
Department of Computer Science Universität des Saarlandes Saarbrücken, Germany
Activation Steering Induces Emergent Misalignment: Steering Vector Construction and Comprehensive Evaluation
Jun 7, 2026Qi Cao, Jian Lou, Meiting Liu +3Language Model Safety Evaluation LLM Safety
Nanyang Technological University · Sun Yat-sen University · University of Science and Technology of China
Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignment
Jun 4, 2026Arush Tagade, Shaoheng Zhou, Jiaxin Wen +1Emergent Misalignment in Language Models LLM Safety Alignment
George Washington University · Google · UC Berkeley
Consistency Training Can Entrench Misalignment
Jun 2, 2026David Demitri Africa, Arathi ManiLLM Safety Alignment Emergent Misalignment
1UK AI Security Institute
Trait-space Monitoring for Emergent Misalignment During Supervised Finetuning
May 31, 2026Huy Nghiem, Sy-Tuyen Ho, Sarah Wiegreffe +1Supervised Fine-Tuning Emergent Misalignment in Language Models
University of Maryland
Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewards
May 29, 2026Magnus Jørgenvåg, David Kaczér, Lasse Ruttert +3RL for Language Models LLM Alignment
Bonn-Aachen International Center for Information Technology, University of Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany
Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning
May 26, 2026Ciarán M. Gilligan-Lee, Joseph Egan, Yuchen Zhu +1LLM Fine-Tuning Political Bias in Language Models
Spotify University College London · Spotify · University College London
Persona-Model Collapse in Emergent Misalignment
May 13, 2026Davi Bastos Costa, Renato VicenteLLM Alignment LLM Fine-Tuning
TELUS Digital Research Hub · Center for Artificial Intelligence and Machine Learning · Institute of Mathematics, Statistics and Computer Science +1
Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfer
May 12, 2026Baris Askin, Muhammed Ustaomeroglu, Anupam Nayak +3Supervised Fine-Tuning LLM Alignment
Carnegie Mellon University
Overtrained, Not Misaligned
May 12, 2026Joel Schreiber, Ariel GoldsteinEarly Stopping LLM Fine-Tuning
Hebrew University of Jerusalem
Characterizing the Consistency of the Emergent Misalignment Persona
Apr 30, 2026Anietta Weckauff, Yuchen Zhang, Maksym AndriushchenkoLanguage Model Safety Evaluation Persona Consistency
ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center
Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers
Apr 28, 2026Jan Dubiński, Jan Betley, Anna Sztyber-Betley +2Language Model Safety Evaluation Emergent Misalignment in Language Models
1Warsaw University of Technology · 4Truthful AI · University College London +2
Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Types
Apr 10, 2026Hadas Orgad, Boyi Wei, Kaden Zheng +4LLM Alignment Emergent Misalignment in Language Models
Kempner Institute, Harvard University · Princeton University · Harvard University +2

Oct 16, 2026J/K move · Enter open · S save

Compression Precedes, Content Decides: Emergent Misalignment in Reward Models

Dhruv Puri, Nikhil Prakash

Northeastern University

Reward Modeling Emergent Misalignment Mechanistic Interpretability

Open paper AskPDF arXiv

Emergent Misalignment - the out of domain generalization of narrow harmful fine-tuning first shown by Betley et al. [2025] has been widely studied in language models but not in reward models that supervise them. In this paper, we fine-tune two state-of-the-art reward models, Skywork-Reward-V2-Llama-3.1-8B and Skywork-Reward-V2-Qwen3-8B [Liu et al., 2025] and show that narrow fine-tuning reward models to prefer harm in one domain (medical or financial) generalizes to seven held out domains. This is not degeneracy, as the fine-tuned reward models lose only 4.8 and 5.3 points overall (RM-Llama) on the standard RewardBench 2 evaluation [Malik et al., 2025] while still outperforming the unmodified RM-Qwen model. We then fine-tune the underlying language models, Llama-3.1-8B-Instruct and Qwen3-8B using DPO [Rafailov et al., 2023] and SFT on the same data. We find that reward models are more vulnerable to preference inversions compared to their language model counterparts. We then ask the question of how these reward models implement this preference inversion. Mechanistically, two things happen, which can be localized to the last/scored token. First, a geometric compression occurs in the representations of sound and harmful responses, reducing the angle between them. The second is content specific: harm representations rotate to a smaller angle with the scalar head than the sound representations which gives harm a higher reward. This happens only in harmful preference fine-tunes and not in control.