Noisy Preference

Noisy preference data poses a significant challenge in training and aligning machine learning models, particularly large language models, where human feedback is often used to guide learning. Current research focuses on developing robust algorithms and model architectures, such as neural dueling bandits and Bayesian neural networks with uncertainty estimation, to mitigate the impact of noisy preferences in various settings, including reinforcement learning and preference optimization. These efforts aim to improve the accuracy and reliability of model training by effectively filtering or accounting for unreliable feedback, leading to more aligned and performant AI systems. The impact of this research extends to diverse applications, from recommendation systems to human-computer interaction, where accurate modeling of human preferences is crucial.

Papers

October 31, 2024

Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?
Zhanke Zhou, Rong Tao, Jianing Zhu, Yiwen Luo, Zengmao Wang, Bo Han
Large Language Model Language Model Chain of Thought Reasoning Path Robust Reasoning Noisy Preference

September 17, 2024

Self-Evolutionary Large Language Models through Uncertainty-Enhanced Preference Optimization
Jianing Wang, Yang Zhou, Xiaocheng Zhang, Mengjiao Bao, Peng Yan
Large Language Model Preference Optimization Noisy Preference

July 24, 2024

Neural Dueling Bandits
Arun Verma, Zhongxiang Dai, Xiaoqiang Lin, Patrick Jaillet, Bryan Kian Hsiang Low
Preference Feedback Bandit Problem Contextual Bandit Problem Dueling Bandit Noisy Preference

June 13, 2024

Online Bandit Learning with Offline Preference Data
Akhil Agnihotri, Rahul Jain, Deepak Ramachandran, Zheng Wen
Reinforcement Learning Preference Feedback Bandit Algorithm Offline Preference Bandit Learning Reward Feedback Noisy Preference

April 15, 2024

Impact of Preference Noise on the Alignment Performance of Generative Language Models
Yang Gao, Dana Alon, Donald Metzler
Global Impact Generative Language Model Dialogue Generation Alignment Performance Noisy Preference

March 1, 2024

Provably Robust DPO: Aligning Language Models with Noisy Feedback
Sayak Ray Chowdhury, Anush Kini, Nagarajan Natarajan
Language Model Direct Preference Optimization Preference Feedback Preference Change Noisy Feedback Noisy Preference Robust Preference

February 27, 2024

RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences
Jie Cheng, Gang Xiong, Xingyuan Dai, Qinghai Miao, Yisheng Lv, Fei-Yue Wang
Robust Training Preference Based Reinforcement Learning Noisy Preference

September 30, 2023

The Physics of Preference: Unravelling Imprecision of Human Preferences through Magnetisation Dynamics
Ivan S. Maksymov, Ganna Pogrebna
Preference Feedback Human Preference Theoretical Physic Human Decision Preference Change Noisy Preference

July 24, 2023

Contextual Bandits and Imitation Learning via Preference-Based Active Queries
Ayush Sekhari, Karthik Sridharan, Wen Sun, Runzhe Wu
Imitation Learning Contextual Bandit Noisy Preference Contextual Bandit Setting

February 13, 2023

When Can We Track Significant Preference Shifts in Dueling Bandits?
Joe Suk, Arpit Agarwal
Armed Bandit Dueling Bandit Preference Change Noisy Preference Preference Distribution

August 23, 2022

The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types
Gaurav R. Ghosal, Matthew Zurek, Daniel S. Brown, Anca D. Dragan
Mixed Effect Reward Learning Human Rationality Noisy Preference Multi Point Feedback Learning Reward

August 5, 2022

Active Learning for Non-Parametric Choice Models
Fransisca Susan, Negin Golrezaei, Ehsan Emamjomeh-Zadeh, David Kempe
Active Learning Choice Model Noisy Preference