Personalized Language Model Alignment

Follow this topic Ask about this topic

Personalized language model alignment studies how models can infer and follow an individual’s preferences from feedback, conversation history, profiles, or behavior rather than relying only on population-wide preferences. Common approaches include personalized preference optimization such as DPO and reinforcement learning, user-specific adapters, memory and retrieval, and inference-time steering or decoding. Evaluation spans preference prediction and generation, multi-turn personalization, and applied settings such as recommendations, privacy, and cultural or political alignment. A central challenge is balancing faithful personalization with safety, truthfulness, and general capability, especially when user signals are sparse, shifting, or in tension with one another.

Latest papers

61 papers12 papers in the last four weeks, up 140% on the four weeks before. 0.1% of all new papers.

Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable Personalization
Apr 24, 2026Weixu Zhang, Ye Yuan, Changjiang Han +7LLM Alignment Language Model Steering
McGill University · Mila - Quebec AI Institute · MBZUAI +2
Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning
Apr 14, 2026Shanyong Wang, Shuhang Lin, Yining Zhao +2LLM Fine-Tuning Personalized Language Model Alignment
Rutgers University—New Brunswick · University of Illinois at Urbana-Champaign
Mobile GUI Agent Privacy Personalization with Trajectory Induced Preference Optimization
Apr 13, 2026Zhixin Lin, Jungang Li, Dongliang Xu +5Mobile GUI Agents Personalized Language Model Alignment
Shandong University · The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology +4
Through Their Eyes: Fixation-aligned Tuning for Personalized User Emulation
Apr 10, 2026Lingfeng Huang, Huizhong Guo, Tianjun Wei +2User Simulation VLM Adaptation
Singapore University of Technology and Design, Singapore · Zhejiang University, Hangzhou, China · Nanyang Technological University, Singapore
The Alignment Floor: How Persona Customization Breaks Safety in Weakly-Aligned LLMs
Apr 10, 2026Xing Zhang, Guanghui Wang, Yanwei Cui +4Language Model Safety Evaluation Personalized Language Model Alignment
AWS Generative AI Innovation Center · HSBC Holdings Plc., HSBC Technology Center, China
PerMix-RLVR: Preserving Persona Expressivity under Verifiable-Reward Alignment
Apr 10, 2026Jihwan Oh, Soowon Oh, Murad Aghazada +3Personalized Language Model Alignment Language Model Robustness
KAIST AI · Samsung Advanced Institute of Technology
One Model for All: Multi-Objective Controllable Language Models
Apr 6, 2026Qiang He, Yucheng Yang, Tianyi Zhou +3Multi-Objective Language Model Alignment Controllable Text Generation
Ruhr University Bochum · Eindhoven University of Technology · Mohamed bin Zayed University of Artificial Intelligence +1
High-Stakes Personalization: Rethinking LLM Customization for Individual Investor Decision-Making
Apr 5, 2026Yash Ganpat SawantPersonalized Language Models Personalized Language Model Alignment
Many Preferences, Few Policies: Compact Portfolios for Multi-Objective LLM Alignment
Apr 5, 2026Cheol Woo Kim, Jai Moondra, Roozbeh Nahavandi +3Multi-Objective Language Model Alignment Personalized Language Model Alignment
Carnegie Mellon University · Massachusetts Institute of Technology
Mitigating Cross-Lingual Cultural Inconsistencies in LLMs via Consensus-Driven Preference Optimisation
Apr 2, 2026Lucas Resck, Isabelle Augenstein, Anna KorhonenCross-Lingual Consistency in Language Models Cross-Cultural Language Model Evaluation
Language Technology Lab, University of Cambridge · University of Copenhagen
Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalization
Apr 1, 2026Gyuseok Lee, Wonbin Kweon, Zhenrui Yue +3Personalized Language Models Personalized Language Model Alignment
University of Illinois Urbana-Champaign · Korea University
Dual Optimal: Make Your LLM Peer-like with Dignity
Apr 1, 2026Xiangqi Wang, Yue Huang, Haomin Zhuang +2Personalized Language Model Alignment Personality Modeling in Language Models
Department of Computer Science and Engineering, University of Notre Dame
EpiPersona: Persona Projection and Episode Coupling for Pluralistic Preference Modeling
Mar 30, 2026Yujie Zhang, Weikang Yuan, Zhuoren Jiang +1User Preference Modeling Personalized Language Model Alignment
Zhejiang University.
Mimetic Alignment with ASPECT: Evaluation of AI-inferred Personal Profiles
Mar 27, 2026Ruoxi Shang, Dan Marshall, Edward Cutrell +1Personal AI Agents Personalized Language Model Alignment
University of Washington, Seattle, Washington, USA · Microsoft Research, Redmond, Washington, USA
KARMA: Knowledge-Action Regularized Multimodal Alignment for Personalized Search at Taobao
Mar 24, 2026Zhi Sun, Wenming Zhang, Yi Wei +4Information Retrieval Personalized Language Model Alignment
Taobao & Tmall Group of Alibaba, Hangzhou, China
FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment
Mar 20, 2026Kewen Zhu, Liping Yi, Zhiming Zhao +3Personalized Language Model Alignment Direct Preference Optimization
College of Intelligence and Computing, Tianjin University, Tianjin, China · College of Computing and Data Science, Nanyang Technological University, Singapore
Expert Personas Improve LLM Alignment but Damage Accuracy: Bootstrapping Intent-Based Persona Routing with PRISM
Mar 19, 2026Zizhao Hu, Mohammad Rostami, Jesse ThomasonPersona-Conditioned Generation Personalized Language Model Alignment
University of Southern California Los Angeles, California, USA
Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback
Mar 13, 2026Gihoon Kim, Euntai KimPreference Learning RL from Human Feedback
Yonsei University · Korea Institute of Science and Technology
Aligning Large Language Models with Searcher Preferences
Mar 11, 2026Wei Wu, Peilun Zhou, Liyi Chen +6Personalized Language Model Alignment Generative Retrieval
School of Artificial Intelligence and Data Science, University of Science and Technology of China, Hefei, China · Xiaohongshu Inc., Beijing, China · Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China +1
AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment
Mar 9, 2026Jianfei Xiao, Xiang Yu, Chengbing Wang +8LLM Personalization Personalized Language Model Alignment
University of Science and Technology of China · National University of Singapore
VISA: Value Injection via Shielded Adaptation for Personalized LLM Alignment
Mar 5, 2026Jiawei Chen, Tianzhuo Yang, Guoxi Zhang +3LLM Alignment Personalized Language Model Alignment
Peking University, Beijing, China
SafeCRS: Personalized Safety Alignment for LLM-Based Conversational Recommender Systems
Mar 3, 2026Haochang Hao, Yifan Xu, Xinzhuo Li +2Recommender Systems Large Language Model-Based Recommendation
University of Illinois at Chicago Chicago, IL, USA · University of Notre Dame South Bend, IN, USA · Amazon Santa Clara, CA, USA
Enhancing Persona Following at Decoding Time via Dynamic Importance Estimation for Role-Playing Agents
Mar 2, 2026Yuxin Liu, Mingye Zhu, Siyuan Liu +2Large Language Model-Based Role-Play Simulation Persona-Conditioned Generation
University of Science and Technology of China
Reasoning Boosts Opinion Alignment in LLMs
Mar 1, 2026Frédéric Berdoz, Yann Billeter, Yann Vonlanthen +1Political Bias in Language Models Personalized Language Model Alignment
ETH Zurich, Switzerland
Learning to Reason for Multi-Step Retrieval of Personal Context in Personalized Question Answering
Feb 22, 2026Maryam Amirizaniani, Alireza Salemi, Hamed ZamaniQuestion Answering Retrieval-Augmented Generation
University of Washington Seattle, WA, USA · University of Massachusetts Amherst Amherst, MA, USA
Aligning Language Models from User Interactions
Feb 18, 2026Thomas Kleine Buening, Jonas Hübotter, Barna Pásztor +3LLM Alignment Personalized Language Model Alignment
ETH Zurich · MIT · University of Zurich
Multi-Objective Alignment of Language Models for Personalized Psychotherapy
Feb 17, 2026Mehrab Beikzadeh, Yasaman Asadollah Salmanpour, Ashima Suvarna +4Personalized Language Model Alignment Multi-Objective Language Model Alignment
Department of Computer Science, University of California, Los Angeles, CA, USA · Department of Psychology, The University of Texas at Austin, Austin, TX, USA · Department of Psychiatry, NYU Grossman School of Medicine, New York, NY, USA
Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment
Feb 17, 2026Jialu Wang, Heinrich Peters, Asad A. Butt +6LLM Alignment Group Relative Policy Optimization
1Apple Inc., California, United States.
The Reward Model Selection Crisis in Personalized Alignment
Dec 28, 2025Fady Rezk, Yuangang Pan, Chuan-Sheng Foo +4Personalized Language Model Alignment Personalized Reward Modeling
School of Informatics, University of Edinburgh, UK · Agency for Science, Technology and Research (A*STAR), Singapore · Samsung AI Research Center, Cambridge, UK
POPI: Personalizing LLMs via Optimized Natural Language Preference Inference
Oct 17, 2025Yizhuo Chen, Xin Liu, Ruijie Wang +7LLM Personalization Personalized Language Models
University of Illinois Urbana-Champaign · Amazon · University of Notre Dame

Apr 24, 2026J/K move · Enter open · S save

Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable Personalization

Weixu Zhang, Ye Yuan, Changjiang Han, Yuxing Tian+6

McGill University · Mila - Quebec AI Institute · MBZUAI+2

LLM Alignment Language Model Steering LLM Personalization LLM Interpretability Personalized Language Models Personalized Language Model Alignment+1

Open paper AskPDF arXiv

Large Language Models (LLMs) exhibit strong implicit personalization ability, yet most existing approaches treat this behavior as a black box, relying on prompt engineering or fine tuning on user data. In this work, we adopt a mechanistic interpretability perspective and hypothesize the existence of a sparse set of Preference Heads, attention heads that encode user specific stylistic and topical preferences and exert a causal influence on generation. We introduce Differential Preference Steering (DPS), a training free framework that (1) identifies Preference Heads through causal masking analysis and (2) leverages them for controllable and interpretable personalization at inference time. DPS computes a Preference Contribution Score (PCS) for each attention head, directly measuring its causal impact on user aligned outputs. During decoding, we contrast model predictions with and without Preference Heads, amplifying the difference between personalized and generic logits to selectively strengthen preference aligned continuations. Experiments on widely used personalization benchmarks across multiple LLMs demonstrate consistent gains in personalization fidelity while preserving content coherence and low computational overhead. Beyond empirical improvements, DPS provides a mechanistic explanation of where and how personalization emerges within transformer architectures. Our implementation is publicly available.