Reward Gradient

Reward gradient methods aim to improve the performance of generative models, particularly diffusion models, by optimizing them towards desired outcomes defined by reward functions. Current research focuses on efficiently backpropagating reward gradients through the model's generation process, exploring techniques like low-rank adapters and gradient checkpointing to manage computational demands, and comparing gradient-based approaches to gradient-free alternatives such as evolutionary strategies. This research is significant because it enables more effective fine-tuning of generative models for specific tasks, improving their controllability and alignment with human preferences across diverse applications, including video generation, autonomous driving, and robotics.

Papers

July 11, 2024

Video Diffusion Alignment via Reward Gradients
Mihir Prabhudesai, Russell Mendonca, Zheyang Qin, Katerina Fragkiadaki, Deepak Pathak
Reward Model Video Diffusion Model Diffusion Alignment Reward Gradient

February 9, 2024

Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Following
Brian Yang, Huangyuan Su, Nikolaos Gkanatsios, Tsung-Wei Ke, Ayush Jain, Jeff Schneider, Katerina Fragkiadaki
Autonomous Driving Zero Shot Diffusion Explainer Reward Function Trajectory Optimization Language Reward Differentiable Reward Decomposition Technique Guide Diffusion Reward Gradient

December 21, 2023

Automatic Curriculum Learning with Gradient Reward Signals
Ryan Campbell, Junsang Yoon
Curriculum Learning Gradient Norm Reward Gradient

October 5, 2023

Aligning Text-to-Image Diffusion Models with Reward Backpropagation
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, Katerina Fragkiadaki
Diffusion Model Text to Image Diffusion Model Text to Image Model Reward Learning Text to Image Diffusion Reward Gradient

September 29, 2023

Directly Fine-Tuning Diffusion Models on Differentiable Rewards
Kevin Clark, Paul Vicol, Kevin Swersky, David J Fleet
Diffusion Model Reward Function Reward Fine Tuning Differentiable Reward Reward Gradient

March 6, 2022

Leveraging Reward Gradients For Reinforcement Learning in Differentiable Physics Simulations
Sean Gillen, Katie Byl
Reinforcement Learning Deep Reinforcement Learning Policy Gradient Cross Entropy Differentiable Physic Reward Gradient