Emergent Misalignment in Language Models

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 34

All topics

The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment
Sep 29, 2026Gonçalo Paulo, Louis Jaburi, Nora Belrose +2LLM Alignment LLM Fine-Tuning
EleutherAI
Correct, Don't Delete: Mitigating Emergent Misalignment with Corrective Supervision
Sep 29, 2026Jacob EpifanoLLM Alignment Emergent Misalignment in Language Models
Independent researcher
Don't Inoculate Everything: Stratified Inoculation Prompting Narrows Backdoor Triggers and Preserves Desired Traits
Sep 28, 2026Kajetan Dymkiewicz, Tim Farrelly, Adam Prada +7LLM Backdoor Attacks Backdoor Defense in LLMs
University of Cambridge · Independent · University of California, Berkeley +3
Narrow Multimodal Fine-Tuning Can Induce Emergent Misalignment
Sep 28, 2026Shunchang Liu, Lukas Fluri, Xin Chen +1Vision-Language Models Fine-Tuning
ETH Zürich · EPFL · Aalto University +1
See it, Say it, Sorted: Mechanistic Diagnosis and Parameter-Space Mitigation of Emergent Misalignment in LLMs
Sep 28, 2026Weiqiao Que, Ruizhe Li, Chengyu Wang +3Emergent Misalignment in Language Models LLM Safety Alignment
School of Computer Science and Technology, East China Normal University, China · School of Computer Science, University of Birmingham, UK · Alibaba Group, China +2
On Emergent Capabilities and Model Merging
Sep 21, 2026Luca Zhou, Emanuele RodolàEmergent Misalignment in Language Models Language Modeling
Sapienza University of Rome · Paradigma
Alignment Forecasting: Predicting Misalignment From Training Data
Sep 19, 2026Chen Yueh-Han, Bruce W. Lee, Ilia Sucholutsky +1LLM Alignment Emergent Misalignment in Language Models
NYU, MATS · Independent · NYU +1
TAME: Token Attribution and Masking for Emergent misalignment
Sep 15, 2026Md Rayhanul Masud, Md Rizwan ParvezLLM Interpretability LLM Fine-Tuning
University of California, Riverside · 2Qatar Computing Research Institute (QCRI)
Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
Sep 14, 2026Arun Jose, Julian StastnyReward Hacking Fine-Tuning
Astra Fellowship · Redwood Research
Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimization
Aug 31, 2026Camila Blank, Zhuofan Ying, Christopher Potts +2LLM Sycophancy Emergent Misalignment in Language Models
1Stanford University · 2Columbia University
Data Attribution of Emergent Misalignment with Persona Features
Aug 11, 2026Clemens Vetter, David Kaczér, Lucie Flek +1LLM Alignment Emergent Misalignment in Language Models
Bonn-Aachen International Center for Information Technology, University of Bonn, Germany
Harmful Content Is Not Enough: Continuation Framing Moderates In-Context Emergent Misalignment
Aug 8, 2026Peiyang Liu, Xi Wang, Ziqiang Cui +2LLM Alignment Emergent Misalignment in Language Models
National Engineering Research Center for Software Engineering, Peking University, Beijing, China · Peking University, Beijing, China · City University of Hong Kong, Hong Kong SAR, China +1
Looking in the Mirror: Introspecting Side-Effect Misalignments Induced by Fine-Tuning
Aug 5, 2026Kotaro Yoshida, Laura Gomezjurado Gonzalez, Yukinori Yamamoto +3LLM Auditing Language Model Introspection
Institute of Science Tokyo · 2ZOZO Research · 3Stanford University +2
Misalignment Has a Personality: A Big Five Account of Emergent Misalignment
Jul 29, 2026Hasibur Rahman, Smit DesaiLLM Alignment Personality Modeling in Language Models
Northeastern University Boston, MA, US
Inference-Time Consensus for Mitigating Hidden Behaviors from LLM Fine-Tuning
Jul 25, 2026Adhyyan Narang, Artin Tajdini, Claire Zhang +1LLM Alignment Emergent Misalignment in Language Models
University of Washington
Emergent Misalignment Recruits a Pre-existing Persona Subspace
Jul 23, 2026Mohammed Suhail B NadafLLM Fine-Tuning Emergent Misalignment in Language Models
Independent
An Emergent Mirage: Is Emergent Misalignment and Realignment Indeed a Robust Phenomenon?
Jul 10, 2026Abhinav Rao, Liancheng Gong, Bin Hu +1LLM Alignment LLM Fine-Tuning
umd · cmu
Transplanting, inverting, and preventing a misalignment persona: method-conditional emergent misalignment in Qwen2.5
Jul 5, 2026Lyndon Drake, Zandi EberstadtLLM Alignment LLM Fine-Tuning
University of Oxford
Evil Spectra: How Optimisers can Amplify or Suppress Emergent Misalignment
Jun 30, 2026Jason R. Brown, Patrick Leask, Lev McKinneyLLM Fine-Tuning Emergent Misalignment in Language Models
Astra Fellows Program University of Cambridge · Astra Fellows Program Durham University · Astra Fellows Program University of Toronto
Probing the Misaligned Thinking Process of Language Models
Jun 23, 2026Kaiwen Zhou, Constantin Venhoff, Jonathan Michala +2LLM Alignment LLM Auditing
1Anthropic Fellows · 2MATS · 3UCSC +2
What Shapes Emergent Misalignment? Insights from Training Dynamics, Model Priors, and Data
Jun 18, 2026Yuchen Zhang, Anietta Weckauff, Diego Garcia-Olano +1LLM Evaluation LLM Alignment
Max Planck Institute for Intelligent Systems, ELLIS Institute Tübingen, Tübingen AI Center, Tübingen, Germany · Meta.
Actionable Activation Directions for Detecting and Mitigating Emergent Misalignment Across Language Model Families
Jun 18, 2026Abdul Rafay SyedLLM Auditing Emergent Misalignment in Language Models
Department of Computer Science Universität des Saarlandes Saarbrücken, Germany
Emergent Misalignment Can Be Induced by Sycophancy and Reversed via Alignment Gating
Jun 8, 2026Sicheng Wang, Xiangyang Zhu, Han Wang +6LLM Sycophancy LLM Alignment
Shanghai AI Lab, Shanghai, China.
The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment
Jun 4, 2026Jiachen Zhao, Zhengxuan Wu, Aryaman Arora +3LLM Alignment LLM Fine-Tuning
Northeastern University · University of California, Berkeley
Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignment
Jun 4, 2026Arush Tagade, Shaoheng Zhou, Jiaxin Wen +1Emergent Misalignment in Language Models LLM Safety Alignment
George Washington University · Google · UC Berkeley
Trait-space Monitoring for Emergent Misalignment During Supervised Finetuning
May 31, 2026Huy Nghiem, Sy-Tuyen Ho, Sarah Wiegreffe +1Supervised Fine-Tuning Emergent Misalignment in Language Models
University of Maryland
Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewards
May 29, 2026Magnus Jørgenvåg, David Kaczér, Lasse Ruttert +3RL for Language Models LLM Alignment
Bonn-Aachen International Center for Information Technology, University of Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany
Persona-Model Collapse in Emergent Misalignment
May 13, 2026Davi Bastos Costa, Renato VicenteLLM Alignment LLM Fine-Tuning
TELUS Digital Research Hub · Center for Artificial Intelligence and Machine Learning · Institute of Mathematics, Statistics and Computer Science +1
Overtrained, Not Misaligned
May 12, 2026Joel Schreiber, Ariel GoldsteinEarly Stopping LLM Fine-Tuning
Hebrew University of Jerusalem
Intrinsic Guardrails: How Semantic Geometry of Personality Interacts with Emergent Misalignment in LLMs
May 11, 2026Krishak Aneja, Manas Mittal, Anmol Goel +2LLM Alignment LLM Interpretability
International Institute of Information Technology, Hyderabad · TU Darmstadt · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi

Sep 29, 2026, cs.CLJ/K move · Enter open · S save

The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment

Gonçalo Paulo, Louis Jaburi, Nora Belrose, Lucia Quirke+1

EleutherAI

LLM Alignment LLM Fine-Tuning Emergent Misalignment in Language Models Training Data Selection Training Data Attribution

Open paper AskPDF arXiv

Fine-tuning large language models on narrow, misaligned tasks can undo their post-training alignment and induce novel misaligned behaviors -- a phenomenon known as \emph{emergent misalignment} (EM). EM has been linked to persona-like representations, where fine-tuning might reduce loss by amplifying a harmful or 'evil' persona. It remains unclear which properties of the training data drive this effect: whether all harmful examples contribute approximately equally to misalignment and whether different models are equally affected by the same fine-tuning examples. In this work, we use training data attribution to quantitatively estimate how much each harmful example contributes to EM. We benchmark the quality of the attribution via retraining -- a sound attribution score should enable us to enhance or attenuate EM by filtering data on that score. Score-based filtering can substantially enhance or attenuate EM; we find that both data-attribution scores and a black-box harmfulness score can identify consequential examples. All models we test become misaligned when trained on the same dataset, and influence scores perform best when filtering data from the same model that computed them. We find cross-model generalization of influence scores from scores derived from the three model families we tested, but this generalization does not recover same model filtering performance.

Momentum

Related topics

Latest papers 34