Emergent Misalignment

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 6Week of Sep 21

Latest papers 49

All topics

The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment
Sep 29, 2026Gonçalo Paulo, Louis Jaburi, Nora Belrose +2Emergent Misalignment Large Language Model Alignment
EleutherAI
Correct, Don't Delete: Mitigating Emergent Misalignment with Corrective Supervision
Sep 29, 2026Jacob EpifanoEmergent Misalignment Correction
Independent researcher
SEABench: Benchmarking Endogenous Misalignment In Self-Evolving Agents
Sep 28, 2026Saswat Das, Parvati Viswanathan, Daniel Donnelly +3Self-Evolving Agents Self-Evolution
University of Virginia · ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems +1
Narrow Multimodal Fine-Tuning Can Induce Emergent Misalignment
Sep 28, 2026Shunchang Liu, Lukas Fluri, Xin Chen +1Emergent Misalignment Modality Alignment
ETH Zürich · EPFL · Aalto University +1
Alignment Games: A Framework for Conceptual Repair in Human-AI Collaboration
Sep 28, 2026Hari Subramonyam, Maneesh Agrawala, Sean FollmerArtificial Intelligence Alignment Human-Ai Collaboration
Stanford University USA
See it, Say it, Sorted: Mechanistic Diagnosis and Parameter-Space Mitigation of Emergent Misalignment in LLMs
Sep 28, 2026Weiqiao Que, Ruizhe Li, Chengyu Wang +3Large Language Model Safety Emergent Misalignment
School of Computer Science and Technology, East China Normal University, China · School of Computer Science, University of Birmingham, UK · Alibaba Group, China +2
On Emergent Capabilities and Model Merging
Sep 21, 2026Luca Zhou, Emanuele RodolàContinual Model Merging Emergent Misalignment
Sapienza University of Rome · Paradigma
TAME: Token Attribution and Masking for Emergent misalignment
Sep 15, 2026Md Rayhanul Masud, Md Rizwan ParvezEmergent Misalignment Large Language Model Alignment
University of California, Riverside · 2Qatar Computing Research Institute (QCRI)
Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
Sep 14, 2026Arun Jose, Julian StastnyEmergent Misalignment Exploitation
Astra Fellowship · Redwood Research
Data Attribution of Emergent Misalignment with Persona Features
Aug 11, 2026Clemens Vetter, David Kaczér, Lucie Flek +1Emergent Misalignment Large Language Model Alignment
Bonn-Aachen International Center for Information Technology, University of Bonn, Germany
Harmful Content Is Not Enough: Continuation Framing Moderates In-Context Emergent Misalignment
Aug 8, 2026Peiyang Liu, Xi Wang, Ziqiang Cui +2Emergent Misalignment Harmful Content
National Engineering Research Center for Software Engineering, Peking University, Beijing, China · Peking University, Beijing, China · City University of Hong Kong, Hong Kong SAR, China +1
Misalignment Has a Personality: A Big Five Account of Emergent Misalignment
Jul 29, 2026Hasibur Rahman, Smit DesaiEmergent Misalignment Personality
Northeastern University Boston, MA, US
Even More Deception: Objective Misalignment in Mixed-Motive LLM Multi-Agent Systems
Jul 28, 2026Marylou Fauchard, Florian Carichon, Margarida Carvalho +1Multi-Agent Large Language Model Systems Deception
Mila, Université de Montréal · Mila, McGill Univeristy · Mila, Univeristé de Montréal, McGill Univeristy
Emergent Misalignment Recruits a Pre-existing Persona Subspace
Jul 23, 2026Mohammed Suhail B NadafEmergent Misalignment Continual Fine-Tuning
Independent
An Emergent Mirage: Is Emergent Misalignment and Realignment Indeed a Robust Phenomenon?
Jul 10, 2026Abhinav Rao, Liancheng Gong, Bin Hu +1Emergent Misalignment Large Language Model Alignment
umd · cmu
Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning
Jul 9, 2026Lu Dai, Ziyang Rao, Yili Wang +3Memorization Reasoning Gaps
HKUST · HKUST(GZ)
Transplanting, inverting, and preventing a misalignment persona: method-conditional emergent misalignment in Qwen2.5
Jul 5, 2026Lyndon Drake, Zandi EberstadtEmergent Misalignment Large Language Model Alignment
University of Oxford
Inoculation Adapters: Improved Selective Generalization of Capabilities with Fewer Surprising Backdoors
Jun 29, 2026Maxime Riché, Daniel Tan, Vili Kohonen +1Inoculation Prompting Emergent Misalignment
∗Center on Long-term Risk
A Gravitational Interpretation of Safety Reversion under Fine-Tuning
Jun 26, 2026Samuele Poppi, Nils LukasContinual Fine-Tuning Emergent Misalignment
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE
Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding
Jun 25, 2026Liu Yu, Can Chen, Ping Kuang +3Recent Vision-Language Models Large Language Model Hallucination
School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu, Sichuan, China · School of Computer Science, University of Auckland, Auckland, New Zealand
What Shapes Emergent Misalignment? Insights from Training Dynamics, Model Priors, and Data
Jun 18, 2026Yuchen Zhang, Anietta Weckauff, Diego Garcia-Olano +1Emergent Misalignment Continual Fine-Tuning
Max Planck Institute for Intelligent Systems, ELLIS Institute Tübingen, Tübingen AI Center, Tübingen, Germany · Meta.
Actionable Activation Directions for Detecting and Mitigating Emergent Misalignment Across Language Model Families
Jun 18, 2026Abdul Rafay SyedEmergent Misalignment Large Language Model Alignment
Department of Computer Science Universität des Saarlandes Saarbrücken, Germany
Emergent Alignment
Jun 17, 2026Martin KolářLarge Language Model Alignment Emergent Misalignment
CIIRC, Czech Technical University in Prague, Prague, Czech Republic.
Human-like autonomy emerges from self-play and a pinch of human data
Jun 11, 2026Daphne Cornelisse, Julian Hunt, Zixu Zhang +4Self-Play Random
1NYU Tandon School of Engineering · 2NYU Courant · 3Princeton University +2
The Arbiter Agent: Continually Monitoring Multi-Agent Conversations to Detect Emergent Misalignment
Jun 9, 2026Filippo Tonini, Federico Torrielli, Anton Danholt Lautrup +3Arbitration Misalignment Persona
University of Southern Denmark Odense, Denmark · University of Turin Turin, Italy · University of Southern Denmark Vejle, Denmark
Emergent alignment and the projectability of ethical personas
Jun 8, 2026Guillermo Del Pinal, Youngchan Lee, Calum McNamara +1Artificial Intelligence Alignment Emergent Misalignment
University of Massachusetts Amherst · Indiana University Bloomington
Emergent Misalignment Can Be Induced by Sycophancy and Reversed via Alignment Gating
Jun 8, 2026Sicheng Wang, Xiangyang Zhu, Han Wang +6Emergent Misalignment Continual Fine-Tuning
Shanghai AI Lab, Shanghai, China.
Activation Steering Induces Emergent Misalignment: A More Comprehensive Evaluation
Jun 7, 2026Qi Cao, Jian Lou, Meiting Liu +4Linear Activation Steering Emergent Misalignment
Nanyang Technological University · Sun Yat-sen University · University of Science and Technology of China +1
The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment
Jun 4, 2026Jiachen Zhao, Zhengxuan Wu, Aryaman Arora +3Emergent Misalignment Hypothesis
Northeastern University · University of California, Berkeley
Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignment
Jun 4, 2026Arush Tagade, Shaoheng Zhou, Jiaxin Wen +1Emergent Misalignment Large Language Model Alignment
George Washington University · Google · UC Berkeley
(Mis)generalization of Helpful-only Fine-tuning
Jun 3, 2026Mohammad Omar Khursheed, Baram Sosis, Fabien RogerEmergent Misalignment Refusals
Anthropic Fellows Program MATS · Anthropic
Trait-space Monitoring for Emergent Misalignment During Supervised Finetuning
May 31, 2026Huy Nghiem, Sy-Tuyen Ho, Sarah Wiegreffe +1Emergent Misalignment Model Fine-Tuning
University of Maryland
Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewards
May 29, 2026Magnus Jørgenvåg, David Kaczér, Lasse Ruttert +3Emergent Misalignment Large Language Model Alignment
Bonn-Aachen International Center for Information Technology, University of Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany
Towards Localized and Disentangled Knowledge Editing for Multimodal Large Language Models
May 28, 2026Leijiang Gu, Zhen Zeng, Feng Li +2Multimodal Knowledge Editing Multimodal Large Language Models
1Hefei University of Technology, Hefei, China
Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning
May 26, 2026Ciarán M. Gilligan-Lee, Joseph Egan, Yuchen Zhu +1Model Fine-Tuning Spurious Correlations
University College London · University College London† · Spotify
MindAdapter: Few-Shot Parameter-Efficient Residual Calibration of Cross-Subject Brain-to-Visual Decoding Models
May 23, 2026Jiaxiang Liu, Jiawei Du, Xupeng Chen +4Multimodal Neuroimaging Emergent Misalignment
Guangdong Institute of Intelligence Science and Technology Hengqin, China · Agency for Science, Technology and Research Singapore · New York University Brooklyn, US +2
Investigating Concept Alignment Using Implausible Category Members
May 20, 2026Sunayana Rane, Brenden M. Lake, Thomas L. GriffithsArtificial Intelligence Alignment Artificial Intelligence Safety
Department of Computer Science · Princeton University · Princeton, NJ 08540 +1
Persona-Model Collapse in Emergent Misalignment
May 13, 2026Davi Bastos Costa, Renato VicenteEmergent Misalignment Large Language Model Alignment
TELUS Digital Research Hub · Center for Artificial Intelligence and Machine Learning · Institute of Mathematics, Statistics and Computer Science +1
Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfer
May 12, 2026Baris Askin, Muhammed Ustaomeroglu, Anupam Nayak +3Emergent Misalignment Large Language Model Alignment
Carnegie Mellon University
Overtrained, Not Misaligned
May 12, 2026Joel Schreiber, Ariel GoldsteinEmergent Misalignment Large Language Models Fail
Hebrew University of Jerusalem
Conformity Generates Collective Misalignment in AI Agents Societies
May 11, 2026Giordano De Marzo, Alessandro Bellina, Claudio Castellano +2Artificial Intelligence Alignment Conformity
University of Konstanz, Konstanz, Germany · Centro Ricerche Enrico Fermi, Rome, Italy · Complexity Science Hub, Vienna, Austria +5
Intrinsic Guardrails: How Semantic Geometry of Personality Interacts with Emergent Misalignment in LLMs
May 11, 2026Krishak Aneja, Manas Mittal, Anmol Goel +2Emergent Misalignment Large Language Model Alignment
International Institute of Information Technology, Hyderabad · TU Darmstadt · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi
Mitigating Misalignment Contagion by Steering with Implicit Traits
May 4, 2026Maria Chang, Ronny Luss, Miao Liu +3Emergent Misalignment Multi-Agent Large Language Model Systems
IBM Research Yorktown Heights, NY 10598, USA
Embodied Interpretability: Linking Causal Understanding to Generalization in Vision-Language-Action Models
May 1, 2026Hanxin Zhang, Mingshuo Xu, Abdulqader Dhafer +3Action Prediction Interpretability
1DANiLab, University of Leicester, UK · School of Computing and Mathematical Sciences, University of Leicester, UK · School of Metallurgy and Materials, University of Birmingham, UK
Characterizing the Consistency of the Emergent Misalignment Persona
Apr 30, 2026Anietta Weckauff, Yuchen Zhang, Maksym AndriushchenkoEmergent Misalignment Artificial Intelligence Personas
ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center
Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers
Apr 28, 2026Jan Dubiński, Jan Betley, Anna Sztyber-Betley +2Emergent Misalignment Misalignment Persona
1Warsaw University of Technology · 4Truthful AI · University College London +2
Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Types
Apr 10, 2026Hadas Orgad, Boyi Wei, Kaden Zheng +4Large Language Model Safety Harmful Content
Kempner Institute, Harvard University · Princeton University · Harvard University +2
Moloch's Bargain: Emergent Misalignment When LLMs Compete for Audiences
Oct 7, 2025Batu El, James ZouSocial Influence Emergent Misalignment
Stanford University
Learning Consumer Preferences from Bundle Sales Data
Sep 11, 2022Ningyuan Chen, Setareh Farajollahzadeh, Qingwei Jin +2Discrete Choice Models Consumption
University of Toronto · McGill University · Zhejiang University

Sep 29, 2026, cs.CLJ/K move · Enter open · S save

The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment

Gonçalo Paulo, Louis Jaburi, Nora Belrose, Lucia Quirke+1

EleutherAI

Emergent Misalignment Large Language Model Alignment Influence Bad

Open paper AskPDF arXiv

Fine-tuning large language models on narrow, misaligned tasks can undo their post-training alignment and induce novel misaligned behaviors -- a phenomenon known as \emph{emergent misalignment} (EM). EM has been linked to persona-like representations, where fine-tuning might reduce loss by amplifying a harmful or 'evil' persona. It remains unclear which properties of the training data drive this effect: whether all harmful examples contribute approximately equally to misalignment and whether different models are equally affected by the same fine-tuning examples. In this work, we use training data attribution to quantitatively estimate how much each harmful example contributes to EM. We benchmark the quality of the attribution via retraining -- a sound attribution score should enable us to enhance or attenuate EM by filtering data on that score. Score-based filtering can substantially enhance or attenuate EM; we find that both data-attribution scores and a black-box harmfulness score can identify consequential examples. All models we test become misaligned when trained on the same dataset, and influence scores perform best when filtering data from the same model that computed them. We find cross-model generalization of influence scores from scores derived from the three model families we tested, but this generalization does not recover same model filtering performance.

Momentum

Related topics

Latest papers 49