Human Feedback

Human feedback is crucial for aligning artificial intelligence models, particularly large language models, with human preferences and values. Current research focuses on improving the efficiency and reliability of incorporating human feedback into reinforcement learning frameworks, exploring techniques like macro actions, active learning, and reward model optimization to address challenges such as the cost and subjectivity of human judgments. This work is significant because it directly impacts the safety, trustworthiness, and overall effectiveness of AI systems across diverse applications, from autonomous driving to educational assessment. The development of more robust and efficient methods for integrating human feedback is a key area of ongoing investigation.

Papers

December 1, 2023

Sample Efficient Reinforcement Learning from Human Feedback via Active Exploration
Viraj Mehta, Vikramjeet Das, Ojash Neopane, Yijia Dai, Ilija Bogunovic, Jeff Schneider, Willie Neiswanger
Reinforcement Learning Human Feedback Active Exploration Sample Efficient Reinforcement Learning Upper Confidence Bound Rate Optimal Regret

November 24, 2023

November 22, 2023

Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Qimai Li, Weihan Shen, Xiaolong Zhu, Xiu Li
Diffusion Model Human Feedback Reward Model Diffusion Policy Robust Reward

November 21, 2023

A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift
Will LeVine, Benjamin Pikus, Anthony Chen, Sean Hendryx
Reinforcement Learning Foundation Model Human Feedback Distribution Shift Reward Model Critique Ability

November 17, 2023

Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Corrections
Lihan Zha, Yuchen Cui, Li-Heng Lin, Minae Kwon, Montserrat Gonzalez Arenas, Andy Zeng, Fei Xia, Dorsa Sadigh
Robot Person Human Feedback Language Correction Robot Manipulation Robot Policy Language Feedback

November 16, 2023

RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models
Jiongxiao Wang, Junlin Wu, Muhao Chen, Yevgeniy Vorobeychik, Chaowei Xiao
Reinforcement Learning Human Feedback Adversarial Learning Poisoning Attack RLHF V Reward Poisoning Attack Preference Poisoning

November 13, 2023

Coffee: Boost Your Code LLMs by Fixing Bugs with Feedback
Seungjun Moon, Hyungjoo Chae, Yongho Song, Taeyoon Kwon, Dongjin Kang, Kai Tzu-iunn Ong, Seung-won Hwang, Jinyoung Yeo
Human Feedback Open Source LLM Bug Report Code LLM Self Service CAFE Code Editing

November 4, 2023

Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Models
Kun Chu, Xufeng Zhao, Cornelius Weber, Mengdi Li, Stefan Wermter
Language Model Reinforcement Learning Human Feedback Manipulation Task Robot Manipulation Interactive Reinforcement Learning

November 2, 2023

The Impact of Preference Agreement in Reinforcement Learning from Human Feedback: A Case Study in Summarization
Sian Gooding, Hassan Mansoor
Reinforcement Learning Case Study Language Generation Human Feedback Structured Summary Text Summarization Preference Feedback Reward Estimation Accuracy

October 31, 2023

The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback
Nathan Lambert, Roberto Calandra
Reinforcement Learning Human Feedback Model Based Reinforcement Learning Reward Model Human Alignment Model Mismatch Reward Model Training

October 29, 2023

Reward Finetuning for Faster and More Accurate Unsupervised Object Discovery
Katie Z Luo, Zhenzhen Liu, Xiangyu Chen, Yurong You, Sagie Benaim, Cheng Perng Phoo, Mark Campbell, Wen Sun, Bharath Hariharan, Kilian Q. Weinberger
Reinforcement Learning Autonomous Vehicle Human Feedback Reward Function Reward Report Object Discovery

October 25, 2023

October 24, 2023

October 23, 2023

Active teacher selection for reinforcement learning from human feedback
Rachel Freedman, Justin Svegliato, Kyle Wray, Stuart Russell
Reinforcement Learning Human Feedback Reward Model Robust Reward Teacher Selection

Human Feedback

Papers

Sample Efficient Reinforcement Learning from Human Feedback via Active Exploration

Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language

Reinforcement Learning from Statistical Feedback: the Journey from AB Testing to ANT Testing

Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model

A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift

Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Corrections

RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models

When Large Language Models contradict humans? Large Language Models' Sycophantic Behaviour

Neural machine translation for automated feedback on children's early-stage writing

Aligning Neural Machine Translation Models: Human Feedback in Training and Inference

Coffee: Boost Your Code LLMs by Fixing Bugs with Feedback

Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Models

The Impact of Preference Agreement in Reinforcement Learning from Human Feedback: A Case Study in Summarization

The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback

Reward Finetuning for Faster and More Accurate Unsupervised Object Discovery

SuperHF: Supervised Iterative Learning from Human Feedback

BabyStories: Can Reinforcement Learning Teach Baby Language Models to Write Better Stories?

AI-enhanced Auto-correction of Programming Exercises: How Effective is GPT-3.5?

ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles

Active teacher selection for reinforcement learning from human feedback