LLM Alignment

LLM: Large Language Model

LLM alignment studies how to shape language models’ behavior so they are helpful, truthful, safe, and responsive to the people and settings they serve. Common approaches include supervised fine-tuning, preference learning such as Direct Preference Optimization, reinforcement learning from human or AI feedback, and inference-time steering or retrieval. Researchers evaluate these methods on instruction-following, preference, jailbreak, factuality, multilingual, and multi-turn tasks, while also developing ways to represent disagreement among users and communities. A central challenge is that gains in safety or preference alignment may reduce usefulness, diversity, calibration, or capability, and desired behavior may not generalize to new prompts, groups, or deployment settings.

Latest papers

613 papers58 papers in the last four weeks, up 12% on the four weeks before. 0.3% of all new papers.

Weird Generalization is Weirdly Brittle
Apr 11, 2026Miriam Wanner, Hannah Collison, William Jurayj +3LLM Alignment Emergent Misalignment in Language Models
Johns Hopkins University
Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Types
Apr 10, 2026Hadas Orgad, Boyi Wei, Kaden Zheng +4LLM Alignment Emergent Misalignment in Language Models
Kempner Institute, Harvard University · Princeton University · Harvard University +2
Hierarchical Alignment: Enforcing Hierarchical Instruction-Following in LLMs through Logical Consistency
Apr 10, 2026Shu Yang, Zihao Zhou, Di Wang +1LLM Alignment Inference-Time Language Model Alignment
King Abdullah University of Science and Technology (KAUST) · University of Liverpool · University of Edinburgh
Cards Against LLMs: Benchmarking Humor Alignment in Large Language Models
Apr 9, 2026Yousra Fettach, Guillaume Bied, Hannu Toivonen +1LLM Alignment LLM Evaluation
Ghent University, Belgium · University of Helsinki, Finland
From Safety Risk to Design Principle: Peer-Preservation in Multi-Agent LLM Systems and Its Implications for Orchestrated Democratic Discourse Analysis
Apr 9, 2026Juergen DietrichLLM Agent Safety Multi-Agent LLM Systems
Senior Data Scientist & AI Consultant TRUST Project — democracy-intelligence.de
ProMedical: Hierarchical Fine-Grained Criteria Modeling for Medical LLM Alignment via Explicit Injection
Apr 9, 2026He Geng, Yangmin Huang, Lixian Lai +5LLM Alignment Medical LLMs
Xunfei Healthcare Technology Co., Ltd.
The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training
Apr 9, 2026Rui Zhang, Hongwei Li, Yun Shen +6LLM Alignment LLM Safety Alignment
University of Electronic Science and Technology of China · Flexera · CISPA Helmholtz Center for Information Security +1
Emotion Concepts and their Function in a Large Language Model
Apr 9, 2026Nicholas Sofroniew, Isaac Kauvar, William Saunders +13Emotion Representation in Language Models LLM Interpretability
Anthropic
ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework
Apr 8, 2026Kai Qin, Liangxin Liu, Yu Liang +7Preference Modeling Self-Reflection in Language Model Agents
Shenzhen International Graduate School, Tsinghua University · Baidu Inc., Beijing, China
ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Training
Apr 8, 2026Yu Liang, Liangxin Liu, Longzheng Wang +5Reward Modeling LLM Alignment
Baidu Inc., Beijing, China
From Context Shift to Stylistic Collapse: Why Training Objectives Matter More Than Scale
Apr 8, 2026Rohan MahapatraLLM Training LLM Alignment
Independent Researcher Cupertino, CA 95014
Controlling Distributional Bias in Multi-Round LLM Generation via KL-Optimized Fine-Tuning
Apr 7, 2026Yanbei Jiang, Amr Keleg, Ryandito Diandaru +4LLM Alignment LLM Fine-Tuning
University of Melbourne · Mohamed bin Zayed University of Artificial Intelligence · Oracle
Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelines
Apr 7, 2026Hwanjun SongLLM Alignment Preference Alignment
KAIST
Understanding Emergent Misalignment via Feature Superposition Geometry
Apr 7, 2026Gouki Minegishi, Hiroki Furuta, Takeshi Kojima +2Emergent Misalignment in Language Models LLM Alignment
The University of Tokyo · Google DeepMind
Pressure, What Pressure? Sycophancy Disentanglement in Language Models via Reward Decomposition
Apr 7, 2026Muhammad Ahmed Mohsin, Ahsan Bilal, Muhammad Umer +1Reward Decomposition LLM Sycophancy
University of Oklahoma
Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignment
Apr 6, 2026Hiroshi Takahashi, Tomoharu Iwata, Atsutoshi Kumagai +4LLM Alignment Preference Alignment
NTT, Inc. · NTT DOCOMO, INC.
APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMs
Apr 5, 2026Mahmoud Srewa, Tianyu Zhao, Salma ElmalakiLLM Alignment Pluralistic Alignment
Department of Electrical Engineering and Computer Science, University of California, Irvine, Irvine, CA 92697, USA
Interoceptive Divergence in Aesthetic Evaluation and Implications for Human-AI Alignment
Apr 5, 2026Yoshia Abe, Tatsuya Daikoku, Yasuo KuniyoshiLLM Alignment
Evaluating Artificial Intelligence Through a Christian Understanding of Human Flourishing
Apr 3, 2026Nicholas Skytland, Lauren Parsons, Alicia Llewellyn +5LLM Alignment LLM Evaluation
Gloo · WinShape Foundation · Biblica
Generalization Limits of Reinforcement Learning Alignment
Apr 3, 2026Haruhi Shida, Koo Imai, Keigo KansaLLM Alignment Adversarial Attacks on LLMs
Aladdin Security Inc.
Jump Start or False Start? A Theoretical and Empirical Evaluation of LLM-initialized Bandits
Apr 2, 2026Adam Bayley, Xiaodan Zhu, Raquel Aoki +2Contextual Bandits Large Language Model-Based Recommendation
Department of Electrical and Computer Engineering, Queen’s University · RBC Borealis
Reinforcement Learning from Human Feedback: A Statistical Perspective
Apr 2, 2026Pangpang Liu, Chengchun Shi, Will Wei SunRL from Human Feedback LLM Alignment
Department of Biostatistics, Yale University. · Department of Statistics, London School of Economics and Political Science. · Mitch Daniels School of Business, Purdue University.
$k$ NNProxy: Efficient Training-Free Proxy Alignment for Black-Box Zero-Shot LLM-Generated Text Detection
Apr 2, 2026Kahim Wong, Kemou Li, Haiwei Wu +1AI-Generated Text Detection Robustness of Machine-Generated Text Detection
State Key Laboratory of Internet of Things for Smart City and the Department of Computer and Information Science, Faculty of Science and Technology, University of Macau, Macao 999078, China · School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
PLOT: Enhancing Preference Learning via Optimal Transport
Apr 2, 2026Liang Zhu, Yuelin Bai, Xiankun Ren +6Preference Learning LLM Alignment
Southern University of Science and Technology · Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · Chongqing University +2
DEFT: Distribution-guided Efficient Fine-Tuning for Human Alignment
Apr 2, 2026Liang Zhu, Feiteng Fang, Yuelin Bai +4LLM Alignment LLM Fine-Tuning
Southern University of Science and Technology · Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · University of Science and Technology of China +1
PRISM: Probability Reallocation with In-Span Masking for Knowledge-Sensitive Alignment
Apr 2, 2026Chenning Xu, Mao Zheng, Mingyang SongLLM Hallucination Mitigation Hallucination in Language Models
Large Language Model Department, Tencent, China
Care-Conditioned Neuromodulation for Autonomy-Preserving Supportive Dialogue Agents
Apr 2, 2026Shalima Binta Manir, Tim OatesLLM Alignment Multi-Objective Language Model Alignment
Department of Computer Science University of Maryland, Baltimore County
More Human, More Efficient: Aligning Annotations with Quantized SLMs
Apr 1, 2026Jiayu Wang, Junyoung LeeLLM-Assisted Annotation LLM Fine-Tuning
Home Team Science and Technology Agency
Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base model
Apr 1, 2026Pankayaraj Pathmanathan, Furong HuangLLM Alignment Inference-Time Language Model Alignment
University of Maryland College Park
Structured Intent as a Protocol-Like Communication Layer: Cross-Model Robustness, Framework Comparison, and the Weak-Model Compensation Effect
Mar 31, 2026Peng GangStructured Prompting Human-AI Interaction
Huizhou Lateni AI Technology Co., Ltd., Huizhou, China · Huizhou University, Huizhou, China

Apr 11, 2026J/K move · Enter open · S save

Weird Generalization is Weirdly Brittle

Miriam Wanner, Hannah Collison, William Jurayj, Benjamin Van Durme+2

Johns Hopkins University

LLM Alignment Emergent Misalignment in Language Models Language Model Safety Evaluation

Open paper AskPDF arXiv

Weird generalization is a phenomenon in which models fine-tuned on data from a narrow domain (e.g. insecure code) develop surprising traits that manifest even outside that domain (e.g. broad misalignment)-a phenomenon that prior work has highlighted as a critical safety concern. Here, we present an extended replication study of key weird generalization results across an expanded suite of models and datasets. We confirm that surprising (and dangerous) traits can emerge under certain circumstances, but we find that weird generalization is exceptionally brittle: it emerges only for specific models on specific datasets, and it vanishes under simple training-time, prompt-based interventions. We find that the most effective interventions provide prompt context that makes the generalized behavior the expected behavior. However, we show that even very generic interventions that do not anticipate specific generalized traits can still be effective in mitigating weird generalization's effects. Our findings thus help clarify the nature of the safety threat that weird generalization poses and point toward an easily implemented set of solutions.