Implicit Reward

Implicit reward learning aims to infer human preferences from preference data, bypassing the need for explicitly defined reward functions in reinforcement learning, particularly for aligning large language models (LLMs) with human intent. Current research focuses on improving the generalization capabilities of implicit reward models, often employing algorithms like Direct Preference Optimization (DPO) and its variants, and exploring techniques to enhance training stability and efficiency. This area is crucial for advancing LLM alignment, enabling more robust and reliable AI systems while also offering insights into human preference modeling and decision-making processes.

Papers

November 6, 2024

From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning
Zhirui Deng, Zhicheng Dou, Yutao Zhu, Ji-Rong Wen, Ruibin Xiong, Mang Wang, Weipeng Chen
Reinforcement Learning Expert Knowledge Inverse Reinforcement Learning Sample STEP Implicit Reward Novice Classification

October 28, 2024

UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Function
Zhichao Wang, Bin Bi, Zixu Zhu, Xiangbo Mao, Jun Wang, Shiyu Wang
Fine Tuning Catastrophic Forgetting Instruction Tuning Finetune Paradigm Implicit Reward RLHF V

October 16, 2024

Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse RL
Jared Joselowitz, Arjun Jagota, Satyapriya Krishna, Sonali Parbhoo
Large Language Model DCU Insight AQ Inverse Reinforcement Learning Reward Model Toxicity Detection LLM Training Inverse Task Implicit Reward

October 12, 2024

SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins
Jongwoo Ko, Saket Dingliwal, Bhavana Ganesh, Sailik Sengupta, Sravan Bodapati, Aram Galstyan
Reinforcement Learning Alignment Problem Direct Preference Optimization Self Feedback Self Alignment Implicit Reward Alignment Algorithm

October 10, 2024

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
Shenao Zhang, Zhihan Liu, Boyi Liu, Yufeng Zhang, Yingxiang Yang, Yongfei Liu, Liyu Chen, Tao Sun, Zhaoran Wang
Large Language Model Preference Alignment Implicit Reward Preference Dataset

September 5, 2024

On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization
Yong Lin, Skyler Seto, Maartje ter Hoeve, Katherine Metcalf, Barry-John Theobald, Xuan Wang, Yizhe Zhang, Chen Huang, Tong Zhang
Generalization Performance Direct Preference Optimization Implicit Reward

August 27, 2024

UNA: Unifying Alignments of RLHF/PPO, DPO and KTO by a Generalized Implicit Reward Function
Zhichao Wang, Bin Bi, Can Huang, Shiva Kumar Pentyala, Zixu James Zhu, Sitaram Asur, Na Claire Cheng
Fine Tuning Reward Signal Better Alignment Implicit Reward RLHF V PPO Algorithm

August 19, 2024

Minor DPO reject penalty to increase training robustness
Shiming Xie, Hong Chen, Fred Yu, Zeye Sun, Xiuyu Wu, Yingfan Hu
Reinforcement Learning Direct Preference Optimization Preference Optimization Preference Pair Implicit Reward Appropriate Penalty Term Training Robustness

June 25, 2024

Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learning
Sen Yang, Leyang Cui, Deng Cai, Xinting Huang, Shuming Shi, Wai Lam
Complete Recipe Preference Pair Context Dependent Question Implicit Reward Preference Label Iterative Preference Learning

June 18, 2024

Order-Optimal Instance-Dependent Bounds for Offline Reinforcement Learning with Preference Feedback
Zhirui Chen, Vincent Y. F. Tan
Offline Reinforcement Learning Preference Feedback Implicit Reward Instance Dependent Regret Matching

June 16, 2024

Step-level Value Preference Optimization for Mathematical Reasoning
Guoxin Chen, Minpeng Liao, Chengxi Li, Kai Fan
Mathematical Reasoning Direct Preference Optimization Preference Optimization Implicit Reward Direct Preference

June 14, 2024

May 30, 2024

Preference Alignment with Flow Matching
Minu Kim, Yongsik Lee, Sehyeok Kang, Jihwan Oh, Song Chong, Se-Young Yun
Reward Model Flow Matching Preference Alignment Preference Based Reinforcement Learning Implicit Reward

May 29, 2024

Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, Bo Dai
Reinforcement Learning Unified Framework Preference Optimization Reinforcement Learning From Human Feedback Online Service Standard Reinforcement Learning Implicit Reward Policy Parametrization

May 23, 2024

SimPO: Simple Preference Optimization with a Reference-Free Reward
Yu Meng, Mengzhou Xia, Danqi Chen
Reinforcement Learning Direct Preference Optimization Preference Optimization Reward Free Implicit Reward Offline Preference

May 21, 2024

SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling
Xingzhou Lou, Junge Zhang, Jian Xie, Lifeng Liu, Dong Yan, Kaiqi Huang
Language Model Reward Model Preference Optimization Preference Alignment Implicit Reward Direct Preference

April 18, 2024

From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function
Rafael Rafailov, Joey Hejna, Ryan Park, Chelsea Finn
Language Model Reinforcement Learning Direct Preference Optimization Q Function Implicit Reward Inverse Q Learning

March 20, 2024

RewardBench: Evaluating Reward Models for Language Modeling
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi
Reward Model Implicit Reward Reward Model Training Reward Value

August 8, 2023

Optimizing Algorithms From Pairwise User Preferences
Leonid Keselman, Katherine Shih, Martial Hebert, Aaron Steinfeld
Black Box Optimization Robot Behavior Social Navigation Pairwise Preference Implicit Reward