Personalized Language Model Alignment

Follow this topic Ask about this topic

Personalized language model alignment studies how models can infer and follow an individual’s preferences from feedback, conversation history, profiles, or behavior rather than relying only on population-wide preferences. Common approaches include personalized preference optimization such as DPO and reinforcement learning, user-specific adapters, memory and retrieval, and inference-time steering or decoding. Evaluation spans preference prediction and generation, multi-turn personalization, and applied settings such as recommendations, privacy, and cultural or political alignment. A central challenge is balancing faithful personalization with safety, truthfulness, and general capability, especially when user signals are sparse, shifting, or in tension with one another.

Latest papers

61 papers12 papers in the last four weeks, up 140% on the four weeks before. 0.1% of all new papers.

FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Data
Aug 6, 2025Thibaut Thonet, Germán Kruszewski, Jos Rozen +2LLM Personalization Personalized Language Models
1NAVER Labs Europe · 2Independent Researcher

Aug 6, 2025J/K move · Enter open · S save

FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Data

Thibaut Thonet, Germán Kruszewski, Jos Rozen, Pierre Erbacher+1

1NAVER Labs Europe · 2Independent Researcher

LLM Personalization Personalized Language Models Personalized Language Model Alignment Preference Alignment Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

LLM-powered conversational assistants are often deployed in a one-size-fits-all manner, which fails to accommodate individual user preferences. Recently, LLM personalization -- tailoring models to align with specific user preferences -- has gained increasing attention as a way to bridge this gap. In this work, we specifically focus on a practical yet challenging setting where only a small set of preference annotations can be collected per user -- a problem we define as Personalized Preference Alignment with Limited Data (PPALLI). To support research in this area, we introduce two datasets -- DnD and ELIP -- and benchmark a variety of alignment techniques on them. We further propose FaST, a highly parameter-efficient approach that leverages high-level features automatically discovered from the data, achieving the best overall performance.