Human Preference

Human preference modeling aims to align artificial intelligence systems, particularly large language models, with human values and expectations. Current research focuses on developing efficient algorithms, such as Direct Preference Optimization (DPO) and its variants, and leveraging diverse data sources including pairwise comparisons, ordinal rankings, and even gaze data to better represent the nuances of human preferences. This field is crucial for ensuring the safety, trustworthiness, and ethical deployment of AI systems across various applications, from language generation to robotics. Ongoing work emphasizes improving the robustness and fairness of preference models, addressing biases, and developing more efficient methods for incorporating human feedback.

Papers

September 17, 2024

Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey
Genta Indra Winata, Hanyang Zhao, Anirban Das, Wenpin Tang, David D. Yao, Shi-Xiong Zhang, Sambit Sahu
Human Language Speech Analysis Human Feedback Vision Task Human Preference Reinforcement Learning Framework Preference Fine Tuning

September 10, 2024

Geometric-Averaged Preference Optimization for Soft Preference Labels
Hiroki Furuta, Kuang-Huei Lee, Shixiang Shane Gu, Yutaka Matsuo, Aleksandra Faust, Heiga Zen, Izzeddin Gur
Pseudo Label Direct Preference Optimization Human Preference Preference Optimization Preference Label

August 29, 2024

Reinforcement Learning without Human Feedback for Last Mile Fine-Tuning of Large Language Models
Alec Solway
Language Model Reinforcement Learning Human Feedback Human Preference Last Mile Likelihood Estimation Model Optimization

August 21, 2024

Advances in Preference-based Reinforcement Learning: A Review
Youssef Abdelkareem, Shady Shehata, Fakhri Karray
Reinforcement Learning Narrative Review Reward Function Human Preference Preference Based Reinforcement Learning Real World Task

August 20, 2024

Aligning Object Detector Bounding Boxes with Human Preference
Ombretta Strafforello, Osman S. Kayhan, Oana Inel, Klamer Schutte, Jan van Gemert
Object Detector Human Preference Bounding Box Object Box

July 21, 2024

The Hitchhiker's Guide to Human Alignment with *PO
Kian Ahrabian, Xihui Lin, Barun Patra, Vishrav Chaudhary, Alon Benhaim, Jay Pujara, Xia Song
Large Language Model Human Preference Preference Optimization Practical Guide Human Alignment

July 12, 2024

New Desiderata for Direct Preference Optimization
Xiangkun Hu, Tong He, David Wipf
Large Language Model Reinforcement Learning Direct Preference Optimization Human Preference

July 11, 2024

DegustaBot: Zero-Shot Visual Preference Estimation for Personalized Multi-Object Rearrangement
Benjamin A. Newman, Pranay Gupta, Kris Kitani, Yonatan Bisk, Henny Admoni, Chris Paxton
Human Preference Personalized Subject Table Task Multi Object Rearrangement Aesthetic Preference

July 8, 2024

Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment
Qizhang Feng, Siva Rajesh Kasa, Hyokun Yun, Choon Hui Teo, Sravan Babu Bodapati
Membership Inference Attack Transparency Index Human Preference LLM Alignment

July 5, 2024

Aligning Model Evaluations with Human Preferences: Mitigating Token Count Bias in Language Model Assessments
Roland Daynauth, Jason Mars
Language Model Human Evaluation Human Preference LLM Evaluator Model Comparison Token Bias

July 4, 2024

Orchestrating LLMs with Different Personalizations
Jin Peng Zhou, Katie Z Luo, Jingwen Gu, Jason Yuan, Kilian Q. Weinberger, Wen Sun
Large Language Model Human Preference Next Token Prediction Personalization System

July 3, 2024

On the Client Preference of LLM Fine-tuning in Federated Learning
Feijie Wu, Xiaoze Liu, Haoyu Wang, Xingchen Wang, Jing Gao
Large Language Model Reinforcement Learning Preference Feedback Human Preference Preference Datasets Model Completion

June 28, 2024

Beyond Human Preferences: Exploring Reinforcement Learning Trajectory Evaluation and Improvement through LLMs
Zichao Shen, Tianchen Zhu, Qingyun Sun, Shiqi Gao, Jianxin Li
Large Language Model Reinforcement Learning Reward Function Large Relevance Improvement Human Preference Reward Engineering

June 26, 2024

The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm
Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb-Tarrant, Julia Kreutzer, Marzieh Fadaee, Sara Hooker
Alignment Problem World Event Human Preference Potential Harm Safety Alignment Multilingual Scenario Alignment Approach Lingual Alignment

June 24, 2024

Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?
Yuu Jinnai
Language Model Multilingual Language Model Human Preference Japanese Large Language Model Preference Dataset Cultural Alignment Cultural Commonsense

June 21, 2024

Pareto-Optimal Learning from Preferences with Hidden Context
Ryan Boldi, Li Ding, Lee Spector, Scott Niekum
Preference Feedback Human Preference Pareto Optimal Model Alignment Specific Preference

June 20, 2024

SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset
Josef Dai, Tianle Chen, Xuyao Wang, Ziran Yang, Taiye Chen, Jiaming Ji, Yaodong Yang
Human Preference Content Moderation Text to Video Generation Large Vision Model Safety Alignment Preference Modeling

June 17, 2024

A Survey on Human Preference Learning for Large Language Models
Ruili Jiang, Kehai Chen, Xuefeng Bai, Zhixuan He, Juntao Li, Muyun Yang, Tiejun Zhao, Liqiang Nie, Min Zhang
Timely Survey Preference Feedback Human Preference Preference Learning Alignment Objective

June 16, 2024

WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences
Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, Bill Yuchen Lin
Vision Language Model Wild Challenge Human Preference Multimodal Interaction Biological Vision Contextual Cue Real World Interaction Visual Imagination

June 12, 2024

PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences
Daiwei Chen, Yi Chen, Aniket Rege, Ramya Korlakai Vinayak
Reward Model Human Preference Reward Prediction Pluralistic Alignment Heterogeneous Preference

Human Preference

Papers

Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey

Geometric-Averaged Preference Optimization for Soft Preference Labels

Reinforcement Learning without Human Feedback for Last Mile Fine-Tuning of Large Language Models

Advances in Preference-based Reinforcement Learning: A Review

Aligning Object Detector Bounding Boxes with Human Preference

The Hitchhiker's Guide to Human Alignment with *PO

New Desiderata for Direct Preference Optimization

DegustaBot: Zero-Shot Visual Preference Estimation for Personalized Multi-Object Rearrangement

Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment

Aligning Model Evaluations with Human Preferences: Mitigating Token Count Bias in Language Model Assessments

Orchestrating LLMs with Different Personalizations

On the Client Preference of LLM Fine-tuning in Federated Learning

Beyond Human Preferences: Exploring Reinforcement Learning Trajectory Evaluation and Improvement through LLMs

The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm

Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?

Pareto-Optimal Learning from Preferences with Hidden Context

SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset

A Survey on Human Preference Learning for Large Language Models

WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences

PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences