Active Preference

Active preference learning focuses on efficiently eliciting human preferences to optimize complex systems or models, minimizing the need for extensive manual labeling. Current research emphasizes developing algorithms that actively select the most informative comparisons, often leveraging Bayesian models or contextual bandit frameworks, to improve sample efficiency and generalization across diverse scenarios. This approach is proving valuable in diverse applications, including improving the alignment of large language models with human values, optimizing visualization design, and enhancing the robustness of reinforcement learning agents in real-world settings. The resulting improvements in data efficiency and model performance have significant implications for human-computer interaction and artificial intelligence development.

Papers

October 25, 2024

APRICOT: Active Preference Learning and Constraint-Aware Task Planning with LLMs
Huaxiaoyue Wang, Nathaniel Chin, Gonzalo Gonzalez-Pumariega, Xiangwan Sun, Neha Sunkara, Maximus Adrian Pace, Jeannette Bohg, Sanjiban Choudhury
Large Language Model Task Planning Affordance Learning Personalization Task Active Preference Constraint Compliance

May 5, 2024

Active Preference Learning for Ordering Items In- and Out-of-sample
Herman Bergström, Emil Carlsson, Devdatt Dubhashi, Fredrik D. Johansson
Active Learning Order Matter Pairwise Comparison Document Ranking Active Preference

February 25, 2024

Cieran: Designing Sequential Colormaps via In-Situ Active Preference Learning
Matt-Heun Hong, Zachary N. Sunberg, Danielle Albers Szafir
Image Sequence Active Preference

February 16, 2024

Active Preference Optimization for Sample Efficient RLHF
Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, Sayak Ray Chowdhury
Contextual Bandit Reinforcement Learning From Human Feedback Optimal Alignment Active Preference

February 12, 2024

Active Preference Learning for Large Language Models
William Muldrew, Peter Hayes, Mingtian Zhang, David Barber
Large Language Model Language Model Direct Preference Optimization Human Preference Active Preference

December 19, 2023

Active Preference Inference using Language Models and Probabilistic Reasoning
Wasu Top Piriyakulkij, Volodymyr Kuleshov, Kevin Ellis
Language Model Probabilistic Model Active Inference Inference Time Active Preference

November 22, 2023

Multi-Objective Bayesian Optimization with Active Preference Learning
Ryota Ozaki, Kazuki Ishikawa, Youhei Kanzaki, Shinya Suzuki, Shion Takeno, Ichiro Takeuchi, Masayuki Karasuyama
Bayesian Optimization Multi Objective Optimization Objective Function Multi Objective Bayesian Optimization Black Box Optimization Problem Preference Modeling Active Preference

February 27, 2023

Experience in Engineering Complex Systems: Active Preference Learning with Multiple Outcomes and Certainty Levels
Le Anh Dao, Loris Roveda, Marco Maccarini, Matteo Lavit Nicora, Marta Mondellini, Matteo Meregalli Falerni, Palaniappan Veerappan, Lorenzo Mantovani, Dario Piga, Simone Formentin, Matteo Malosio
Optimization Problem Objective Function Black Box Optimization eXperience Report Multiple Outcome Engineering System Active Preference

September 23, 2022

Quantification before Selection: Active Dynamics Preference for Robust Reinforcement Learning
Kang Xu, Yan Ma, Wei Li
View Selection Robust Policy Robust Reinforcement Learning Domain Randomization Dynamic Mismatch Active Preference

January 3, 2022

Feedback-efficient Active Preference Learning for Socially Aware Robot Navigation
Ruiqi Wang, Weizheng Wang, Byung-Cheol Min
Reinforcement Learning Robot Behavior Active Preference