Preference Distribution

Preference distribution research focuses on understanding and modeling how preferences are distributed within a population, aiming to accurately represent diverse viewpoints and avoid biases in decision-making systems. Current research emphasizes robust methods for learning preference distributions from often noisy and incomplete data, employing techniques like reward model distillation, distributional preference learning, and optimal transport-based loss functions within various model architectures, including those based on reinforcement learning and Bayesian updating. This work is crucial for improving the fairness and accuracy of AI systems, particularly in applications like large language model alignment and personalized recommendations, where accurately reflecting diverse user preferences is paramount. The ultimate goal is to develop algorithms that are robust to preference shifts and capable of making equitable decisions even with limited or heterogeneous data.

Papers

July 8, 2024

Contrastive Learning of Preferences with a Contextual InfoNCE Loss
Timo Bertram, Johannes Fürnkranz, Martin Müller
Contrastive Learning Multi Modal Preference Feedback Co Occurrence Preference Change CLIP Score Preference Distribution Contextual Response Retrievability Loss Function

May 31, 2024

Enhancing Counterfactual Image Generation Using Mahalanobis Distance with Distribution Preferences in Feature Space
Yukai Zhang, Ao Xu, Zihao Li, Tieru Wu
Counterfactual Explanation Explainable Artificial Intelligence XAI Method Feature Space Counterfactual Image Image Classification Model Mahalanobis Distance Preference Distribution

May 29, 2024

Robust Preference Optimization through Reward Model Distillation
Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, Jonathan Berant
Reinforcement Learning Reward Model Direct Preference Optimization Robust Preference Preference Distribution

February 15, 2024

Aligning Crowd Feedback via Distributional Preference Reward Modeling
Dexun Li, Cong Zhang, Kuicai Dong, Derrick Goh Xin Deik, Ruiming Tang, Yong Liu
Deep Reinforcement Learning Reward Model Crowd Annotation Preference Reward Preference Distribution

February 14, 2024

MaxMin-RLHF: Towards Equitable Alignment of Large Language Models with Diverse Human Preferences
Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, Mengdi Wang
Reinforcement Learning Policy Learning Human Preference Reinforcement Learning From Human Feedback Preference Distribution

December 13, 2023

Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF
Anand Siththaranjan, Cassidy Laidlaw, Dylan Hadfield-Menell
Reinforcement Learning Human Understanding Preference Learning Reinforcement Learning From Human Feedback Social Choice Preference Distribution

November 23, 2023

A density estimation perspective on learning from pairwise human preferences
Vincent Dumoulin, Daniel D. Johnson, Pablo Samuel Castro, Hugo Larochelle, Yann Dauphin
Large Language Model Density Estimation Human Preference Preference Pair Pairwise Preference Preference Distribution

September 12, 2023

Optimal and Fair Encouragement Policy Evaluation and Learning
Angela Zhou
LeArning Abstract Optimal Policy Fairness Constraint Proportional Fairness Dynamic Treatment Regime Preference Distribution Eligibility Criterion

February 13, 2023

When Can We Track Significant Preference Shifts in Dueling Bandits?
Joe Suk, Arpit Agarwal
Armed Bandit Dueling Bandit Preference Change Noisy Preference Preference Distribution