Contrastive Reward

Contrastive reward learning enhances reinforcement learning (RL) by training models to distinguish between desirable and undesirable outputs, improving performance beyond standard reward methods. Current research focuses on applying this technique to various tasks, including large language model alignment, image captioning, and abstractive summarization, often leveraging transformer-based architectures and incorporating techniques like Proximal Policy Optimization (PPO). This approach addresses challenges like reward model fragility and hallucination in generated content, leading to more robust and accurate models with improved human evaluation scores across diverse applications.

Papers

October 25, 2024

On-Robot Reinforcement Learning with Goal-Contrastive Rewards
Ondrej Biza, Thomas Weng, Lingfeng Sun, Karl Schmeckpeper, Tarik Kelestemur, Yecheng Jason Ma, Robert Platt, Jan-Willem van de Meent, Lawson L. S. Wong
Reinforcement Learning Sparse Reward Robot Policy Learning Contrastive Reward

March 12, 2024

Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards
Wei Shen, Xiaoying Zhang, Yuanshun Yao, Rui Zheng, Hongyi Guo, Yang Liu
Reinforcement Learning Human Feedback Informative Reward Contrastive Reward

February 21, 2024

Distinctive Image Captioning: Leveraging Ground Truth Captions in CLIP Guided Reinforcement Learning
Antoine Chaffin, Ewa Kijak, Vincent Claveau
Ground Truth Image Captioning Captioning Model Image Captioning Model Distinctive Caption Contrastive Reward

July 10, 2023

Improving Factuality of Abstractive Summarization via Contrastive Reward Learning
I-Chun Chern, Zhiruo Wang, Sanjan Das, Bhavuk Sharma, Pengfei Liu, Graham Neubig
Contrastive Learning Abstractive Summarization Summarization Model Abstractive Summarization Model Factuality Metric Contrastive Reward

June 25, 2023

Improving Reference-based Distinctive Image Captioning with Contrastive Rewards
Yangjun Mao, Jun Xiao, Dong Zhang, Meng Cao, Jian Shao, Yueting Zhuang, Long Chen
Image Captioning Generated Caption Distinctive Caption Contrastive Reward

November 3, 2022

Contrastive Value Learning: Implicit Models for Simple Offline RL
Bogdan Mazoure, Benjamin Eysenbach, Ofir Nachum, Jonathan Tompson
Offline Reinforcement Learning Reward Function Model Based Reinforcement Learning Implicit Model Step to Step Dynamic Contrastive Reward

October 14, 2022

Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning
Louis Castricato, Alexander Havrilla, Shahbuland Matiana, Michael Pieler, Anbang Ye, Ian Yang, Spencer Frazier, Mark Riedl
Story Generation Narrative Text Contrastive Reinforcement Learning Robust Preference Contrastive Reward

April 27, 2022

Executive Function: A Contrastive Value Policy for Resampling and Relabeling Perceptions via Hindsight Summarization?
Chris Lengerich, Ben Lengerich
Shot Learning Hindsight Instruction Contrastive Reward