Differentiable Reward

Differentiable reward functions are increasingly used to guide the learning of complex behaviors in artificial intelligence systems, particularly in reinforcement learning and imitation learning. Current research focuses on developing methods to effectively utilize differentiable rewards with various model architectures, including diffusion models and reinforcement learning agents, often addressing challenges like reward sparsity, non-differentiability, and the need for efficient optimization. This work is significant because it enables the training of agents on complex tasks with nuanced objectives, improving sample efficiency and performance in applications such as autonomous driving, robotics, and image generation. The development of robust and efficient methods for handling differentiable rewards is crucial for advancing the capabilities of AI systems across diverse domains.

Papers

October 17, 2024

RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards
Xinze Li, Sen Mei, Zhenghao Liu, Yukun Yan, Shuo Wang, Shi Yu, Zheni Zeng, Hao Chen, Ge Yu, Zhiyuan Liu, Maosong Sun, Chenyan Xiong
Retrieval Augmented Generation Supervised Fine Tuning Closed Form Differentiable Expression Differentiable Reward

October 8, 2024

Training-free Diffusion Model Alignment with Sampling Demons
Po-Hung Yeh, Kuang-Huei Lee, Jun-Cheng Chen
Diffusion Model Text to Image Generation Visual Language Model Preference Alignment Differentiable Reward Diffusion Alignment

May 29, 2024

Inference-Time Alignment of Diffusion Models with Direct Noise Optimization
Zhiwei Tang, Jiangweizhi Peng, Jiasheng Tang, Mingyi Hong, Fan Wang, Tsung-Hui Chang
Diffusion Model Reward Function Alignment Problem Reward Hacking Differentiable Reward Noise Optimization

March 2, 2024

Continuous Mean-Zero Disagreement-Regularized Imitation Learning (CMZ-DRIL)
Noah Ford, Ryan W. Gardner, Austin Juhl, Nathan Larson
Reinforcement Learning Imitation Learning Continuous Space Differentiable Reward Agent Ensemble Contrastive Adjusted Zooming

February 9, 2024

Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Following
Brian Yang, Huangyuan Su, Nikolaos Gkanatsios, Tsung-Wei Ke, Ayush Jain, Jeff Schneider, Katerina Fragkiadaki
Autonomous Driving Zero Shot Diffusion Explainer Reward Function Trajectory Optimization Language Reward Differentiable Reward Decomposition Technique Guide Diffusion Reward Gradient

November 2, 2023

DreamSmooth: Improving Model-based Reinforcement Learning via Reward Smoothing
Vint Lee, Pieter Abbeel, Youngwoon Lee
Model Based Reinforcement Learning Sparse Reward Reward Prediction Differentiable Reward

September 29, 2023

Directly Fine-Tuning Diffusion Models on Differentiable Rewards
Kevin Clark, Paul Vicol, Kevin Swersky, David J Fleet
Diffusion Model Reward Function Reward Fine Tuning Differentiable Reward Reward Gradient

December 6, 2022

Receding Horizon Planning with Rule Hierarchies for Autonomous Vehicles
Sushant Veer, Karen Leung, Ryan Cosner, Yuxiao Chen, Peter Karkus, Marco Pavone
Autonomous Vehicle Planning Horizon Differentiable Reward

October 12, 2022

Smooth Trajectory Collision Avoidance through Deep Reinforcement Learning
Sirui Song, Kirk Saunders, Ye Yue, Jundong Liu
Deep Reinforcement Learning Collision Avoidance Vision Based Reward Signal Differentiable Reward

September 30, 2022

VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, Amy Zhang
Individual Representation Value Function Unseen Task Self Supervised Objective Model Retraining Pre Trained Self Supervised Differentiable Reward