Preference Alignment

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 60

All topics

Collaborative Personalized Preference Alignment for LLMs under Data Deficiency
Oct 5, 2026Liyan Yang, Yige Yuan, Zhiqin YangPreference Alignment Large Language Model Personalization
City University of Hong Kong · University of Washington · The Hong Kong University of Science and Technology
Groupwise Distortion Guarantees for Preference-Based Alignment
Oct 4, 2026Jacob Brodkey, Roberto Tamez, Aaron RothPreference Alignment Reinforcement Learning From Human Feedback
University of Pennsylvania
Robust Nash Alignment under Preference Uncertainty
Sep 30, 2026Shihab Ahmed, Debamita Ghosh, David Tang +3Preference Alignment Nash Equilibrium
University of Central Florida · Arizona State University · University of Colorado Boulder
A Zeroth-Order Paradigm for LLM Preference Alignment
Sep 16, 2026Peter Chen, Xi Chen, Wotao Yin +1Preference Alignment Sequential Preference Optimization
Department of Electrical Engineering and Computer Sciences (EECS) University of California, Berkeley Berkeley, CA 94720, USA · Stern School of Business New York University New York, NY 10012, USA · Decision Intelligence Lab (Seattle) DAMO Academy, Alibaba Group U.S. Bellevue, WA 98004, USA +1
FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models
Sep 9, 2026Yansen Han, Shengyi Liao, Peng Sun +4Flow Models Preference Alignment
Westlake University · Zhejiang University · Kling Team, Kuaishou Technology
MoPLEx: Estimating Plackett-Luce Mixture Models for Multi-Objective Alignment
Aug 25, 2026Dongyue Li, Ziniu Zhang, Lu Wang +1Preference Alignment Mixture Models
Northeastern University, Boston, MA · University of Michigan, Ann Arbor, MI
Learning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning
Aug 10, 2026Yuting Liu, Wei Wu, Jianzhe Zhao +1Large Language Model Personalization Preference Alignment
Software College, Northeastern University, China · Ant International
LLM Reasoning for Subjective Tasks: Failure Modes, Mitigation, and Dynamic Reasoning Routing
Aug 9, 2026Juncheng Dong, Ding Tong, Ishan Gupta +1LLM Reasoning Strategies Subjectivity
Duke University · Durham, USA · Netflix +1
Linguistically-Aligned and Visually-Grounded Preference Optimization for Clinically-Augmented Medical Report Generation
Aug 9, 2026Qiang Hu, Yuxuan Luo, Yingjie Guo +4Medical Report Generation Multimodal Clinical Data
Huazhong University of Science and Technology
Explore or Converge? Stage-Guided Per-Step Optimization for Diffusion Models
Aug 7, 2026Renye Yan, Jikang Cheng, You Wu +4Diffusion Models Preference Alignment
Peking University Beijing, China · Stanford University Stanford, California, USA
Compass: Continuously Aligning Social Media Feeds via In-Situ Reflections
Aug 4, 2026Aadit Barua, Leijie Wang, Amy X. ZhangPreference Alignment Social Media
University of Washington Seattle, Washington, United States
BridgeAlign: Bridging Preference Alignment for Humanities and Social Sciences
Jul 29, 2026Ru Peng, Haokai Xu, Xijun Gu +11Preference Alignment Computational Social Science
Zhejiang University · Inclusion AI, Ant Group · Qwen Team, Alibaba Group +2
WearWow: Native 2K Multi-Garment Virtual Try-On via Adaptive Token Packing and Preference Alignment
Jul 22, 2026Xujie Zhang, Runyan Du, Song Chang +7Garments Virtual Try-On
Shenzhen Campus of Sun Yat-sen University, China · Meitu Lab, China
Rewarding Better Thinking for LLM Preference Alignment
Jul 22, 2026Xubo Liu, Wenya Guo, Ruxue Yan +2Preference Alignment Trajectory-Level Credit
VCIP, DISSec, College of Computer Science, Nankai University, Tianjin, China
Toward Anthropomorphic Dialogue: A Closed-Loop Framework for Human-Like Chat Generation, Evaluation, and Preference Alignment
Jul 19, 2026Wentao Liu, Siyu Song, Xi Chen +4Dialogue Benchmarks Artificial Intelligence Personas
1Shanghai Institute of Innovation · 2East China Normal University · University of Science and Technology of China +3
A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models
Jul 19, 2026Mohtashim KhanLarge Language Model Evaluation Preference Alignment
Genre Bias or Aesthetic Perception? Identifying and Mitigating Shortcut Learning in Music Evaluation
Jul 15, 2026Yizhou Zhang, Wangjin Zhou, Yi Zhao +3Aesthetics Model Evaluation
Graduate School of Informatics, Kyoto University, Japan · WXG, Tencent, China
Deployable Human Preference Alignment in Robotics: Learning Representative Rewards from Diverse Human Preferences
Jul 14, 2026Taehyung Kim, Gwangmo Lee, Minjun Chang +2Preference Alignment Preference Alignment Learning
Yonsei University, Seoul, South Korea · Georgia Institute of Technology, United States of America · Hyundai Motor Company, Seoul, South Korea
Normative Alignment of Recommender Systems via Internal Label Shift
Jul 12, 2026Johannes Kruse, Kasper Lindskow, Michael Riis Andersen +4Preference Alignment Recommendation
JP/Politikens Media Group · Technical University of Denmark · Copenhagen Business School +4
Beyond Time Shifts: Adapting Omni-LLM as a Reference-Free Evaluator for Generative Audio-Visual Models
Jul 10, 2026Yijie Qian, Juncheng Wang, Chao Xu +6Audio-Visual Consistency Cross-Modal
Zhejiang University, Hangzhou, China · IROOTECH TECHNOLOGY, China · The Hong Kong Polytechnic University, Hong Kong, China +1
On the Limits of Steering Vectors for Preference-Aligned Generation
Jul 2, 2026Melanie Subbiah, Zara Hall, Kathleen McKeownPreference Alignment Text Generation
Department of Computer Science, Columbia University, USA
PAPA: Online Personalized Active Preference Alignment
Jul 1, 2026Anindya Sarkar, Nasik Muhammad Nafi, Isaac Lyngaas +2Preference Alignment Diffusion Alignment
Washington University in St. Louis · Oak Ridge National Laboratory
REAR: Test-time Preference Realignment through Reward Decomposition
Jun 29, 2026Fuxiang Zhang, Pengcheng Wang, Chenran Li +6Preference Alignment Preference Alignment Learning
1Nanyang Technological University · University of Califor · 3Nanjing University
A Markov Chain Approach to Preference Alignment
Jun 21, 2026Takuya Koriyama, Tengyuan LiangPreference Alignment Reinforcement Learning From Human Feedback
The University of Chicago
CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignment
Jun 13, 2026Wenbo Yu, Bohua Wang, Hao Fang +10Large Language Model Safety Preference Alignment
Tsinghua University · Beijing Normal University · South China University of Technology +2
Hidden Consensus:Preference-Validity Compression in Human Feedback
Jun 9, 2026Dorcas Chia Ern Chua, Karen Myn Hui Lee, Jia Yue Tan +9Preference Alignment Reinforcement Learning From Human Feedback
1YTL AI Labs · 2Universiti Malaya
Pareto-Guided Teacher Alignment for Fair Personalized Text Generation
Jun 8, 2026Tunazzina IslamPersonalization Preference Alignment
Department of Computer Science Purdue University, West Lafayette Indiana 47907, USA
Follow-Your-Preference++: Rethinking Preference Alignment for Image Inpainting
Jun 2, 2026Junkun Yuan, Yutao Shen, Toru Aonishi +2Preference Alignment Preference Alignment Learning
Zhejiang University · The University of Tokyo · Tsinghua University
TriAlign: Towards Universal Truth Consistency in Personalized LLM Alignment
Jun 1, 2026Thi-Nhung Nguyen, Linhao Luo, Rollin Omari +3Large Language Model Alignment Preference Alignment
Department of Data Science & AI, Monash University · Defence Science and Technology Group, Australia
Pave-GRPO: Beyond Instantaneous Guidance through Principled Average Velocity Decomposition
Jun 1, 2026Pengyang Ling, Jiazi Bu, Yujie Zhou +6Repair-Based Group-Relative Policy Optimization Generative Flow Networks
University of Science and Technology of China, China · Shanghai Jiao Tong University, China · Fudan University, China +3
MENTIS: What Belief Changes Under Alignment? Measuring Multi-Scale Latent Torsion in Language Models
May 31, 2026Partha Pratim Saha, Samarth Raina, Mayur Parvatikar +4Large Language Model Alignment Preference Alignment
Pragya Lab, BITS Pilani Goa, India · IIIT Delhi, India · Amazon, USA +2
Differentially Private Preference Data Synthesis for Large Language Model Alignment
May 29, 2026Fengyu Gao, Jing YangPreference Alignment Large Language Model Alignment
Department of Computer Science, University of Virginia, Charlottesville, Virginia, USA · Department of Electrical and Computer Engineering, University of Virginia, Charlottesville, Virginia, USA.
Pairwise Reference Alignment as a Model-Level Ordinal Observable
May 29, 2026Mujing LiPreference Alignment Preference Alignment Learning
Independent Researcher
When and How Human Curation Backfires: Preference Alignment under Multi-Model Self-Consuming Loop
May 28, 2026Yang Zhang, Xiukun Wei, Xueru ZhangPreference Alignment Reinforcement Learning From Human Feedback
Department of Computer Science and Engineering, The Ohio State University, Columbus, Ohio.
Toward User Preference Alignment in LLM Recommendation via Explicit Context Feedback
May 27, 2026Weizhi Zhang, Wooseong Yang, Yuxin Cui +9Preference Alignment Learning Preference Alignment
University of Illinois Chicago, Chicago, USA · Meta, Menlo Park, USA
StoryLens: Preference-Aligned Story Rewriting via Context-Aware Narrative Enrichment
May 27, 2026Hanwen Cui, Yuting Mei, Yuhang Fu +2Rewrite Narrative Generation
Beijing University of Posts and Telecommunications · AIM3 Lab, Renmin University of China · Nanyang Technological University
Investigating the Interplay between Contextual and Parametric Chain-of-Thought Faithfulness under Optimization
May 24, 2026Jingyi Sun, Qianli Wang, Pepa Atanasova +2Explanation Faithfulness Faithfulness
University of Copenhagen · 2Technische Universität Berlin · 3German Research Center for Artificial Intelligence (DFKI) +2
Truthful Online Preference Aggregation for LLM Fine-Tuning in Mobile Crowdsourcing
May 22, 2026Shugang Hao, Lingjie DuanCrowdsourced Framework Large Language Model Fine-Tuning
Singapore Wireless Innovation Centre, Singapore University of Technology and Design, Singapore, 487372 Singapore · Internet of Things Thrust and the Artificial Intelligence Thrust, Hong Kong University of Science and Technology, Guangzhou 511455, China
Spectral Souping: A Unified Framework for Online Preference Alignment
May 19, 2026Yinlam Chow, Guy Tennenholtz, Ted Yun +4Preference Alignment Reinforcement Learning From Human Feedback
Google DeepMind · Google Research
When Preference Labels Fall Short: Aligning Diffusion Models from Real Data
May 19, 2026Weiyan Chen, Weijian Deng, Yao Xiao +5Diffusion Alignment Preference Alignment
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Australian National University, Canberra, Australia +2
CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging
May 19, 2026Zhenyu Li, Aleksandar Cvejic, Zehui Chen +1Preference Alignment Code Quality
KAUST · USTC
Pairwise Preference Reward and Group-Based Diversity Enhancement for Superior Open-Ended Generation
May 18, 2026Guining Cao, Jiaxin Peng, Chu Zeng +3Preference Alignment Diversity
Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd · School of Software and Microelectronics, Peking University · Tsinghua University
Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignment
May 17, 2026Yucong Huang, Xiucheng Li, Kaiqi Zhao +1Preference Alignment Large Language Model Alignment
Harbin Institute of Technology, Shenzhen, China.
Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansion
May 12, 2026ShiYing Huang, Liang Lin, Yuer Li +6Preference Alignment Multi-Objective Reinforcement Learning
Huazhong University of Science and Technology · Nanyang Technological University · Tsinghua University +1
Leveraging RAG for Training-Free Alignment of LLMs
May 11, 2026John T. HalloranLarge Language Model Alignment Preference Alignment
Leidos
ProteinOPD: Towards Effective and Efficient Preference Alignment for Protein Design
May 11, 2026Yulin Zhang, He Cao, Zihao Jiang +6Protein Design Preference Alignment
Tsinghua University · International Digital Economy Academy · Hong Kong University of2026 Science and Technology (Guangzhou) +1
A Single Deep Preference-Conditioned Policy for Learning Pareto Coverage Sets
May 9, 2026Akihiro Kubo, Kosuke Nakanishi, Shin IshiiMulti-Objective Reinforcement Learning Pareto Frontier
International Research Center for Neurointelligence, The University of Tokyo, Tokyo, Japan · ATR Neural Information Analysis Laboratories, Kyoto, Japan · Department of Information Science, Kyoto University, Kyoto, Japan
Threshold-Guided Optimization for Visual Generative Models
May 6, 2026Jinbin Bai, Yu Lei, Qingyu Shi +6Preference Alignment Generative Models
National University of Singapore · Collov Labs · Peking University +2
Towards General Preference Alignment: Diffusion Models at Nash Equilibrium
May 6, 2026Jiaming Hu, Jiamu Bai, Haoyu Wang +2Diffusion Alignment Preference Alignment
Boston University · Penn State University · University at Albany
Online Self-Calibration Against Hallucination in Vision-Language Models
May 1, 2026Minghui Chen, Chenxu Yang, Hengjie Zhu +3Recent Vision-Language Models Large Language Model Hallucination
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
Sparse Personalized Text Generation with Multi-Trajectory Reasoning
Apr 27, 2026Bo Ni, Haowei Fu, Qinwen Ge +10Large Language Model Personalization Preference Alignment
Department of Computer Science, Vanderbilt University, Nashville, Tennessee · Adobe Research, San Jose, California · University of Orgeon, Eugene, Oregon
HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads
Apr 19, 2026Juyuan Wang, Chenxing Wang, Yuchen Fang +8Cross-Encoder Reranking Relevance
Weixin Group, Tencent, China · 2South China University of Technology, Guangzhou, China
Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment
Feb 17, 2026Jialu Wang, Heinrich Peters, Asad A. Butt +6Preference Alignment Large Language Model Personalization
1Apple Inc., California, United States.
A Scalable Entity-Based Framework for Auditing Bias in Large Language Models
Jan 18, 2026Akram Elbouanani, Aboubacar Tuo, Adrian PopescuLarge Language Model Bias Fairness Audits
Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France
Multi-dimensional Preference Alignment by Conditioning Reward Itself
Dec 11, 2025Jiho Jang, Jinyoung Kim, Kyungjune Baek +1Diffusion Alignment Backtranslation Augmented Direct Preference Optimization
Seoul National University · Sejong University
Aligning Audio Captions with Human Preferences
Sep 18, 2025Kartik Hegde, Rehana Mahfuz, Yinyi Guo +1Video Captioning Captions
Qualcomm Technologies, Inc., India · Qualcomm Technologies, Inc., USA
FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Data
Aug 6, 2025Thibaut Thonet, Germán Kruszewski, Jos Rozen +2Large Language Model Personalization Preference Alignment
1NAVER Labs Europe · 2Independent Researcher
Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors
Apr 27, 2025Ren-Wei Liang, Chin-Ting Hsu, Chan-Hung Yu +6Preference Alignment Reinforcement Learning From Human Feedback
National Taiwan University · Appier AI Research · Graduate Institute of Communication Engineering, National Taiwan University +1
TextAlign: Preference Alignment for Text Rendering with Hierarchical Rewards
Date pendingMingxuan Cui, Jingpu Yang, Fengxian Ji +7Image-Text Alignment Few-Shot Font Generation
Mohamed bin Zayed University of Artificial Intelligence · Chinese Academy of Sciences Institute of Automation · Northeastern University +1
PACIFIC: Can LLMs Discern the Psychometric Traits Influencing Your Preferences? Personality-Driven Preference Alignment in LLMs
Date pendingTianyu Zhao, Siqi Li, Yasser Shoukry +1Preference Alignment Preference Alignment Learning
Department of Electrical Engineering and Computer Science University of California, Irvine Irvine, CA 92697, USA

Oct 5, 2026, cs.LGJ/K move · Enter open · S save

Collaborative Personalized Preference Alignment for LLMs under Data Deficiency

Liyan Yang, Yige Yuan, Zhiqin Yang

City University of Hong Kong · University of Washington · The Hong Kong University of Science and Technology

Preference Alignment Large Language Model Personalization Large Language Model Adaptation

Open paper AskPDF arXiv

Real-world users often exhibit highly heterogeneous preferences over multiple objectives for LLM responses. A lightweight aligner can tailor these responses to individual preferences, but scarce user-specific feedback makes personalized training difficult. Learning shared initializations across users can support few-shot adaptation. However, heterogeneous preferences and competing objectives cause gradient conflicts across users and within each user, hindering effective initialization learning. This raises a central question: \textbf{how can we collaboratively learn aligner initializations that support few-shot adaptation to diverse user preferences?} To answer this question, we propose \textbf{A}pproximate \textbf{P}areto \textbf{O}ptimality (APO). We first group users whose updates are compatible, so that their information can be combined with less interference. Within each group, we combine gradient descent with controlled ascent to coordinate competing objectives and move towards preference-specific points on the Pareto front. This produces an initialization that is close to the optima of the users in the group. We then iteratively refine it using updates from few-shot local adaptation, making it more effective for personalization. Furthermore, we establish conditional suboptimality bounds for a one-local-step collaborative update and characterize how initialization error affects subsequent stochastic adaptation. Experiments on Fed-ChatbotPA and UltraFeedback show consistent improvements over existing methods using only 20 local examples.

Momentum

Related topics

Latest papers 60