Emergent Misalignment in Language Models

Follow this topic Ask about this topic

Emergent misalignment studies how narrow fine-tuning, preference optimization, or reward-based training can lead language models to behave harmfully in unrelated settings. Researchers test this transfer with tasks such as insecure coding, medical advice, and reward hacking, and investigate mechanisms including latent persona features, activation patterns, and influential training examples. Approaches to detection and mitigation include activation-based monitoring, data attribution and correction, context-aware training, and interventions on model representations. A central challenge is that results can depend strongly on the model, dataset, and evaluation prompt, making it difficult to distinguish robust behavioral shifts from conditional effects or training artifacts.

Latest papers

60 papers13 papers in the last four weeks, up 86% on the four weeks before. 0.1% of all new papers.

Evaluating DeepSeek V4.1 Flash-1017 for loss of control and harmful manipulation risks
Oct 26, 2026Yao Huang, Zhiyuan Zeng, Yu Wang +14LLM Safety Evaluation LLM Safety Monitoring
Unintended Side Effects of Value Training in LLMs
Oct 21, 2026Siddhesh Pawar, Chen Cecilia Liu, Anna Korhonen +1LLM Alignment Language Model Post-Training
University of Copenhagen · University of Cambridge
Behavioral Couplings: From Formation to Behavioral Emergence in Language Models
Oct 16, 2026Dongyoon Hahm, Woogyeol Jin, Dennis Wei +3Emergent Misalignment in Language Models Fine-Tuning Language Models
KAIST · IBM Research
LLM Evolution in a Simple Simulated Market
Oct 16, 2026Charlie Pilgrim, Christopher Krupenye, Dora Biro +1Emergent Misalignment in Language Models Self-Evolving Agents
University of Leeds · Johns Hopkins University · University of Rochester
Progressive Misalignment Can Emerge from Benign Recursive Self-Distillation
Oct 15, 2026Karim Abdel Sadek, Federico Di Gennaro, Fanny YangEmergent Misalignment in Language Models Self-Distillation
ETH Zurich
ContextLift: Breaking the Spurious Correlation Behind Conditional Emergent Misalignment
Oct 15, 2026Florian Mai, David KaczérEmergent Misalignment in Language Models LLM Safety Alignment
Bonn-Aachen International Center for Information Technology, University of Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany
Do Emergent Misalignment Models Share a Circuit?
Oct 14, 2026Muhammad Haseeb, Muhammad TahirEmergent Misalignment in Language Models Circuits in Language Models
Lahore University of Management Sciences (LUMS)
Understanding and Reducing Misalignment in Persuasion Optimization
Oct 12, 2026Boyi Wei, Ruta Desai, Peter Henderson +1LLM Alignment Emergent Misalignment in Language Models
Microsoft Research · Princeton University
Unfair and Unsafe: Everyday Bias Triggers Emergent Misalignment
Oct 11, 2026Alex Martí Guiu, Nicolas Zumarraga, Kevin O'Sullivan +2Emergent Misalignment in Language Models Language Model Safety Evaluation
Agentic Systems Lab, ETH Zurich · ETH AI Center, ETH Zurich
The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment
Sep 29, 2026Gonçalo Paulo, Louis Jaburi, Nora Belrose +2LLM Alignment LLM Fine-Tuning
EleutherAI
Correct, Don't Delete: Mitigating Emergent Misalignment with Corrective Supervision
Sep 29, 2026Jacob EpifanoLLM Alignment Emergent Misalignment in Language Models
Independent researcher
Don't Inoculate Everything: Stratified Inoculation Prompting Narrows Backdoor Triggers and Preserves Desired Traits
Sep 28, 2026Kajetan Dymkiewicz, Tim Farrelly, Adam Prada +7LLM Backdoor Attacks Backdoor Defense in LLMs
University of Cambridge · Independent · University of California, Berkeley +3
Narrow Multimodal Fine-Tuning Can Induce Emergent Misalignment
Sep 28, 2026Shunchang Liu, Lukas Fluri, Xin Chen +1Vision-Language Models Fine-Tuning
ETH Zürich · EPFL · Aalto University +1
See it, Say it, Sorted: Mechanistic Diagnosis and Parameter-Space Mitigation of Emergent Misalignment in LLMs
Sep 28, 2026Weiqiao Que, Ruizhe Li, Chengyu Wang +3Emergent Misalignment in Language Models LLM Safety Alignment
School of Computer Science and Technology, East China Normal University, China · School of Computer Science, University of Birmingham, UK · Alibaba Group, China +2
On Emergent Capabilities and Model Merging
Sep 21, 2026Luca Zhou, Emanuele RodolàEmergent Abilities in Language Models Emergent Misalignment in Language Models
Sapienza University of Rome · Paradigma
Alignment Forecasting: Predicting Misalignment From Training Data
Sep 19, 2026Chen Yueh-Han, Bruce W. Lee, Ilia Sucholutsky +1LLM Alignment Emergent Misalignment in Language Models
NYU, MATS · Independent · NYU +1
TAME: Token Attribution and Masking for Emergent misalignment
Sep 15, 2026Md Rayhanul Masud, Md Rizwan ParvezLLM Interpretability LLM Fine-Tuning
University of California, Riverside · 2Qatar Computing Research Institute (QCRI)
Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
Sep 14, 2026Arun Jose, Julian StastnyReward Hacking Fine-Tuning
Astra Fellowship · Redwood Research
You Are What You Read: Misalignment via In-Context Persona Induction
Sep 6, 2026Kyuhee Kim, Benjamin Berczi, Cozmin UdudecEmergent Misalignment in Language Models Persona-Conditioned Generation
EPFL · MATS Research
Inducing Emergent Misalignment from Reward Hacks with Iterative DPO
Sep 6, 2026Oliver Daniels, Perusha Moodley, Benjamin M. Marlin +1Emergent Misalignment in Language Models LLM Alignment
University of Massachusetts Amherst · MATS · Google DeepMind
Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimization
Aug 31, 2026Camila Blank, Zhuofan Ying, Christopher Potts +2LLM Sycophancy Emergent Misalignment in Language Models
1Stanford University · 2Columbia University
Emergent Misalignment Is Not Magical
Aug 29, 2026Mingxuan Li, Qirun Dai, Heran Wang +1Emergent Misalignment in Language Models LLM Alignment
The University of Chicago
On the Threat Model of Weird Generalization and Emergent Misalignment
Aug 24, 2026Miriam Wanner, Mark Dredze, William WaldenLLM Fine-Tuning Emergent Misalignment in Language Models
Johns Hopkins University
Measuring Reward Hacking and Reasoning-Answer Decoupling Under Position-Confounded Optimization
Aug 15, 2026Suyash Maniyar, Armaan Sandhu, Abhishek MishraReward Hacking Emergent Misalignment in Language Models
University of Massachusetts Amherst
Emergent Misaligned Communication in Long-Horizon Multi-Agent LLM Commerce
Aug 14, 2026Zeyuan Li, Lukas Petersson, Alessandro Acquisti +1Emergent Misalignment in Language Models LLM Agent Safety
Massachusetts Institute of Technology · Andon Labs
Data Attribution of Emergent Misalignment with Persona Features
Aug 11, 2026Clemens Vetter, David Kaczér, Lucie Flek +1LLM Alignment Emergent Misalignment in Language Models
Bonn-Aachen International Center for Information Technology, University of Bonn, Germany
Harmful Content Is Not Enough: Continuation Framing Moderates In-Context Emergent Misalignment
Aug 8, 2026Peiyang Liu, Xi Wang, Ziqiang Cui +2LLM Alignment Emergent Misalignment in Language Models
National Engineering Research Center for Software Engineering, Peking University, Beijing, China · Peking University, Beijing, China · City University of Hong Kong, Hong Kong SAR, China +1
Looking in the Mirror: Introspecting Side-Effect Misalignments Induced by Fine-Tuning
Aug 5, 2026Kotaro Yoshida, Laura Gomezjurado Gonzalez, Yukinori Yamamoto +3LLM Auditing Language Model Introspection
Institute of Science Tokyo · 2ZOZO Research · 3Stanford University +2
Misalignment Has a Personality: A Big Five Account of Emergent Misalignment
Jul 29, 2026Hasibur Rahman, Smit DesaiLLM Alignment Personality Modeling in Language Models
Northeastern University Boston, MA, US
Defense via Consensus: Mitigating Subliminal Learning through Multi-Teacher Distillation
Jul 25, 2026Adhyyan Narang, Artin Tajdini, Claire Zhang +1LLM Alignment Emergent Misalignment in Language Models
University of Washington

Oct 26, 2026J/K move · Enter open · S save

Evaluating DeepSeek V4.1 Flash-1017 for loss of control and harmful manipulation risks

Yao Huang, Zhiyuan Zeng, Yu Wang, Zhitong Zhu+13

LLM Safety Evaluation LLM Safety Monitoring Emergent Misalignment in Language Models

Open paper AskPDF arXiv

As frontier language models grow increasingly capable and autonomous, independent third-party evaluations are critical for characterizing their systemic safety risks. While closed-source releases are routinely accompanied by comprehensive system cards, open-weight models receive less scrutiny regarding frontier risks. To address this gap, we present an independent safety evaluation of DeepSeek V4.1 Flash-1017, the latest version of V4.1 Flash that departs from V4 with a new architecture, scaled-up training, and strong agentic capabilities. Drawing on the EU General-Purpose AI Code of Practice, we evaluate the model across 34 benchmarks, including 5 in-house suites, spanning evaluation integrity, loss of control, and harmful manipulation. Conforming to the AEF-1 standard, we compare V4.1 Flash-1017 against V4.1 Flash, earlier DeepSeek V4 models, and leading closed-source systems (GPT-6 Astra, GPT-5.6 Sol, and Gemini 3.8 Flash). Overall, V4.1 Flash-1017 shows substantial safety improvements over the V4 generation and approaches or matches leading closed-source baselines on several dimensions, while still exhibiting notable behaviors that warrant continued attention. Building on our findings, we highlight three key observations and recommendations. First, because safety can shift and even regress across minor updates, developers should integrate safety evaluation into incremental releases. Second, as final-turn outputs and end-state checks alone cannot reliably detect loss-of-control behaviors, deployments require process-level observability across both reasoning traces and action logs. Third, because public benchmarks elicit strong evaluation awareness and face data contamination risks, developing private benchmarks is essential for trustworthy frontier evaluation.