Preference Alignment Learning

Follow this topic Ask about this topic

Momentum

52 papers in the last four weeks, up 333% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 271

All topics

Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment
Feb 17, 2026Jialu Wang, Heinrich Peters, Asad A. Butt +6Preference Alignment Large Language Model Personalization
1Apple Inc., California, United States.
SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks
Feb 16, 2026Srivatsa Kundurthy, Clara Na, Michael Handley +5Spreadsheets Large Language Model Generation
Longitude Labs Inc., New York, NY · Cornell University, Ithaca, NY · Carnegie Mellon University, Pittsburgh, PA +1
Synthetic Interaction Data for Scalable Personalization in Large Language Models
Feb 12, 2026Yuchen Ma, Yue Huang, Wenjie Wang +3Large Language Model Personalization Personalization
Munich Center for Machine Learning & LMU Munich · University of Notre Dame · Sichuan University
Online Generalized-Mean Welfare Maximization: Achieving Near-Optimal Regret from Samples
Feb 11, 2026Zongjun Yang, Rachitesh Kumar, Christian Kroer\Widetilde{\Mathcal{O}}(\Sqrt{T})$ Regret Welfare
Columbia University · Carnegie Mellon University
Aligning Language Model Benchmarks with Pairwise Preferences
Feb 2, 2026Marco Gutierrez, Xinyi Leng, Hannah Cyberey +3Large Language Model Benchmarks Preference Alignment Learning
School of Data Science, University of Virginia · Imperial College London · Thomson Reuters Foundational Research +1
Not All Preferences Deserve Gradients: Understanding Gradient Utility in Offline Reasoning Alignment
Feb 1, 2026Hui Wu, Hengyi Cai, Jinman Zhao +6Gradient Preference Alignment Learning
Baidu Inc. · University of Toronto · School of Intelligence Science and Technology, Peking University +1
The Axiom of Consent: Authorization, Friction, and Multi-Agent Coordination
Jan 10, 2026Murad FarzullaMulti-Agent Coordination Friction
Dissensus, London, UK · King’s College London, London, UK
Do Implicit Personalization and Explicit Styles Conflict? PsPLUG: A Lightweight Plug-in for Balancing Personalization and Style in Customized LLMs
Jan 10, 2026Yutong Song, Jiang Wu, Shaofan Yuan +5Large Language Model Personalization Personalization
University of California, Irvine · Independent Researcher · TikTok
Who Laughs with Whom? Disentangling Influential Factors in Humor Preferences across User Clusters and LLMs
Jan 6, 2026Soichiro Murakami, Hidetaka Kamigaito, Hiroya Takamura +1Humor Large Language Model Bias
CyberAgent · Nara Institute of Science and Technology · Institute of Science Tokyo
PreferThinker: Reasoning-based Personalized Image Preference Assessment
Nov 1, 2025Shengqi Xu, Xinpeng Zhou, Yabo Zhang +6Preference Alignment Learning Think
♠Fudan University · ♣Harbin Institute of Technology · ♦iN2X
GMTRouter: Personalized LLM Router over Multi-turn User Interactions
Oct 29, 2025Yihang Sun, Encheng Xie, Tao Feng +1Large Language Model Routing Large Language Model Personalization
University of Illinois Urbana-Champaign
TripScore: Aligning LLMs for Real-World Travel Planning via Expert-Calibrated Reward
Oct 10, 2025Yincen Qu, Huan Xiao, Feng Li +5Journey Reward Functions
Trip.com Group · Chongqing University · Stanford University
Adaptive Margin RLHF via Preference over Preferences
Sep 26, 2025Yaswanth Chittepu, Prasann Singhal, Greg Durrett +1Reinforcement Learning From Human Feedback Preference Alignment Learning
University of Massachusetts Amherst · University of Texas at Austin
Understanding Role Switching in Human-AI Collaboration through Multimodal Behavioral Signals
Sep 25, 2025Avinash Ajit Nargund, Arthur Caetano, Kevin Yang +7Human-Ai Collaboration Collaboration
University of California, Santa Barbara, USA · Washington University in St.Louis, USA
Learning Acrobatic Flight from Preferences
Aug 26, 2025Colin Merk, Ismail Geles, Jiaxu Xing +3Preference Alignment Learning
Robotics and Perception Group, University of Zurich, Switzerland · Autonomous Learning and Predictive Intelligence Lab, University of Zurich, Switzerland
Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings
Aug 16, 2025Jenny Y. Huang, Yunyi Shen, Dennis Wei +1Ranking Chatbot Arena
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology · MIT-IBM Watson AI Lab · IBM Research
Expert Preference-based Evaluation of Automated Related Work Generation
Aug 11, 2025Furkan Şahinuç, Subhabrata Dutta, Iryna GurevychPeer Review Evaluation Metrics
Ubiquitous Knowledge Processing Lab (UKP Lab) · Department of Computer Science and Hessian Center for AI (hessian.AI) · Technical University of Darmstadt +1
Interactive Multi-Objective Probabilistic Preference Learning with Soft and Hard Bounds
Jun 27, 2025Edward Chen, Sang T. Truong, Natalie Dullerud +2Pareto Frontier Preference Learning
Department of Computer Science, Stanford University, Stanford, California, USA
Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning
Jun 14, 2025Sara Rajaram, R. James Cotton, Fabian H. SinzReinforcement Learning From Human Feedback Preference Alignment Learning
Institute for Computer Science and Campus Institute for Data Science, University of Göttingen, Göttingen, Germany · University Hospital Tübingen, Tübingen, Germany · International Max Planck Research School for Intelligent Systems, Tübingen, Germany +2
MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning
May 30, 2025Jingyan Shen, Jiarui Yao, Rui Yang +5Large Language Model Personalization Preference Alignment Learning
New York University · University of Illinois Urbana-Champaign · Rice University
Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomy
May 28, 2025Saleh Afzoon, Ali Shahsavandi, Phuong Thao Huynh +4Copilot Preference Alignment Learning
School of Computing, Macquarie University, Sydney, Australia · Department of Computer Engineering and Information Technology, Shiraz University of Technology, Shiraz, Iran
Dual-Difficulty Curriculum Learning for Direct Preference Optimization
Apr 10, 2025Mengyang Li, Haozhan Geng, Zhong Zhang +1Large Language Model Alignment Adaptive Curriculum
Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission Tianjin Normal University Tianjin, China
Inferring the Unspoken: Aligning Embodied Agents with Implicit Preferences
Feb 2, 2025Manjie Xu, Xinyi Yang, Wei Liang +2Embodied Agents Embodied
Institute for Artificial Intelligence, Peking University · School of Psychological and Cognitive Sciences, Peking University · School of Intelligence Science and Technology, Peking University +5
Stable Marriage Problems with Ties and Incomplete Preferences: An Empirical Comparison of ASP, SAT, ILP, CP, and Local Search Methods
Aug 11, 2021Selin Eyupoglu, Muge Fidan, Yavuz Gulesen +5Satisfiability Linear Programming
Computer Science, Faculty of Engineering and Natural Sciences, Sabanci University, Istanbul, Turkey · Economics, Faculty of Arts and Social Sciences, Sabanci University, Istanbul, Turkey
Toward Collective-Centric Evaluation of Preference Inference for Participatory Democracy
Date pendingPierre-Antoine Lequeu, Salim Hafid, Paul Lerner +6Democracy Preference Alignment Learning
Sorbonne Université, CNRS, ISIR, Paris, France · Sciences Po, médialab, Paris, France · Sorbonne Université, STIH/CERES, Paris, France
Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learning
Date pendingAnthony GX-Chen, Ankit Anand, Gheorghe Comanici +7Reward Functions Offline Reinforcement Learning
New York University · Google DeepMind
SAEExplainer: Interpreting SAE Features with Activation-Guided Preference Optimization
Date pendingJingyi He, Haiyan Zhao, Ruxue Shi +4Sparse Autoencoder Features Explainable AI Methods
The Chinese University of Hong Kong, Shenzhen · Shanghai Jiao Tong University · NJIT +2
PACIFIC: Can LLMs Discern the Psychometric Traits Influencing Your Preferences? Personality-Driven Preference Alignment in LLMs
Date pendingTianyu Zhao, Siqi Li, Yasser Shoukry +1Preference Alignment Preference Alignment Learning
Department of Electrical Engineering and Computer Science University of California, Irvine Irvine, CA 92697, USA
Beyond Single-Negative Preference: Multi-Negative DPO for LLM-Centric Historical Entity Linking
Date pendingTien Nam Nguyen, Emanuela Boros, Ahmed Hamdi +3Backtranslation Augmented Direct Preference Optimization History
L3i, University of La Rochelle, La Rochelle, France · University of Toulouse, IRIT, Toulouse, France · University of Innsbruck, Austria +1
Left-Branching Transformers Excel at Right-Branching Languages: Data Shapes Word Order Preferences in Language Models
Date pendingVarvara Arzt, Allan Hanbury, Terra BlevinsLanguage Modeling Language Pairs
Faculty of Informatics, TU Wien · Khoury College of Computer Sciences, Northeastern University
The C-index illusion: discrimination without calibration in published survival models
Date pendingRafael da Silva, Danilo AlvaresSurvival Predictions Survival Analysis
PhD in Applied Data Science Eastern University 1300 Eagle Road, Saint Davids, 19087, Pennsylvania, USA · MRC Biostatistics Unit University of Cambridge Forvie Site, Robinson Way, Cambridge, CB20SR, UK

Feb 17, 2026, cs.LGJ/K move · Enter open · S save

Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment

Jialu Wang, Heinrich Peters, Asad A. Butt, Navid Hashemi+5

1Apple Inc., California, United States.

Preference Alignment Large Language Model Personalization Preference Alignment Learning

Open paper AskPDF arXiv

Despite their sophisticated general-purpose capabilities, Large Language Models (LLMs) often fail to align with diverse individual preferences because standard post-training methods, like Reinforcement Learning with Human Feedback (RLHF), optimize for a single, global objective. While Group Relative Policy Optimization (GRPO) is a widely adopted on-policy reinforcement learning framework, its group-based normalization implicitly assumes that all samples are exchangeable, inheriting this limitation in personalized settings. This assumption conflates distinct user reward distributions and systematically biases learning toward dominant preferences while suppressing minority signals. To address this, we introduce Personalized GRPO (P-GRPO), a novel alignment framework that decouples advantage estimation from immediate batch statistics. By normalizing advantages against preference-group-specific reward histories rather than the concurrent generation group, P-GRPO preserves the contrastive signal necessary for learning distinct preferences. We evaluate P-GRPO across diverse tasks and find that it consistently achieves faster convergence and higher rewards than standard GRPO, thereby enhancing its ability to recover and align with heterogeneous preference signals. Our results demonstrate that accounting for reward heterogeneity at the optimization level is essential for building models that faithfully align with diverse human preferences without sacrificing general capabilities.

Momentum

Related topics

Latest papers 271