Reward Maximization

Reward maximization, a core concept in reinforcement learning and decision-making, aims to optimize actions or policies to achieve the highest cumulative reward. Current research focuses on addressing challenges like reward hacking, sample inefficiency, and model mismatch, employing techniques such as Bayesian inference, expectation-maximization, and various policy optimization algorithms (e.g., PPO, Thompson Sampling). These advancements are crucial for improving the performance and safety of AI agents across diverse applications, from personalized recommendations and language model alignment to robotics and resource management. Furthermore, research is exploring alternative frameworks beyond simple reward maximization, such as active inference and methods that incorporate risk awareness and constraint satisfaction.

Papers

October 21, 2024

Reward Maximization for Pure Exploration: Minimax Optimal Good Arm Identification for Nonparametric Multi-Armed Bandits
Brian Cho, Dominik Meier, Kyra Gan, Nathan Kallus
Multi Armed Bandit Reward Maximization Optimal Arm Pure Exploration Nonparametric Modeling

October 17, 2024

Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Design
Chenyu Wang, Masatoshi Uehara, Yichun He, Amy Wang, Tommaso Biancalani, Avantika Lal, Tommi Jaakkola, Sergey Levine, Hanchen Wang, Aviv Regev
Diffusion Model Financial Application Reward Model Conditional Diffusion Model Protein Design Reward Maximization Discrete Diffusion Model DNA Related Bioinformatics Domain

October 10, 2024

Probabilistic Satisfaction of Temporal Logic Constraints in Reinforcement Learning via Adaptive Policy-Switching
Xiaoshan Lin, Sadık Bera Yüksel, Yasin Yazıcıoğlu, Derya Aksaray
Reinforcement Learning Adaptive Importance Constrained Reinforcement Learning Reward Maximization Traditional Reinforcement Learning Policy Constraint

October 6, 2024

Ranking Policy Learning via Marketplace Expected Value Estimation From Observational Data
Ehsan Ebrahimzadeh, Nikhil Monga, Hang Gao, Alex Cozzi, Abraham Bagherjeiran
User Behavior Observational Data Reward Maximization User Modeling

October 5, 2024

Preference Optimization as Probabilistic Inference
Abbas Abdolmaleki, Bilal Piot, Bobak Shahriari, Jost Tobias Springenberg, Tim Hertweck, Rishabh Joshi, Junhyuk Oh, Michael Bloesch, Thomas Lampe, Nicolas Heess, Jonas Buchli, Martin Riedmiller
Preference Optimization Sequential Decision Making Problem Reward Maximization

September 19, 2024

Unsupervised Reward-Driven Image Segmentation in Automated Scanning Transmission Electron Microscopy Experiments
Kamyar Barakati, Utkarsh Pratiush, Austin C. Houston, Gerd Duscher, Sergei V. Kalinin
Segmentation Task Unsupervised Setting Reward Maximization Transmission Electron Microscopy Transmission Electron Molecular Image

September 9, 2024

Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Models
Rohit Jena, Ali Taghibakhshi, Sahil Jain, Gerald Shen, Nima Tajbakhsh, Arash Vahdat
Text to Image Diffusion Model Text to Image Diverse Image Reward Maximization Annealed Importance Sampling

July 18, 2024

Analyzing and Bridging the Gap between Maximizing Total Reward and Discounted Reward in Deep Reinforcement Learning
Shuyu Yin, Fei Wen, Peilin Liu, Tao Luo
Deep Reinforcement Learning Optimal Policy Reward Report Reward Maximization Reward Structure Monetary Reward

June 24, 2024

WARP: On the Benefits of Weight Averaged Rewarded Policies
Alexandre Ramé, Johan Ferret, Nino Vieillard, Robert Dadashi, Léonard Hussenot, Pierre-Louis Cedoz, Pier Giuseppe Sessa, Sertan Girgin, Arthur Douillard, Olivier Bachem
Reinforcement Learning Complementary Benefit Reward Maximization Multi Objective Reward

May 31, 2024

Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation
Shangding Gu, Laixi Shi, Yuhao Ding, Alois Knoll, Costas Spanos, Adam Wierman, Ming Jin
Safe Reinforcement Learning Open Sampling Reward Maximization Enhancing Efficiency Safe Policy

May 29, 2024

Offline Regularised Reinforcement Learning for Large Language Models Alignment
Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, Aliaksei Severyn, Jonathan Mallinson, Lior Shani, Gil Shamir, Rishabh Joshi, Tianqi Liu, Remi Munos, Bilal Piot
Large Language Model Language Model Reward Maximization Pairwise Preference Large Language Model Alignment Direct Preference

May 26, 2024

On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization
Jiancong Xiao, Ziniu Li, Xingyu Xie, Emily Getzen, Cong Fang, Qi Long, Weijie J. Su
Large Language Model Reinforcement Learning Reinforcement Learning From Human Feedback Algorithmic Bias Reward Maximization Preference Drift

May 10, 2024

Value Augmented Sampling for Language Model Alignment and Personalization
Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, Pulkit Agrawal
Reinforcement Learning Reward Maximization Language Model Alignment LLM Adaptation Augmented Sample Reward Maximizing Policy

May 2, 2024

Constrained Reinforcement Learning Under Model Mismatch
Zhongchang Sun, Sihong He, Fei Miao, Shaofeng Zou
Model Uncertainty Robust Policy Constrained Reinforcement Learning Reward Maximization Model Mismatch

April 4, 2024

Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, Tengyang Xie
Language Model Reinforcement Learning Nash Equilibrium Preference Feedback Reward Learning Reward Maximization

February 22, 2024

Q-Probe: A Lightweight Approach to Reward Maximization for Language Models
Kenneth Li, Samy Jelassi, Hugh Zhang, Sham Kakade, Martin Wattenberg, David Brandfonbrener
Language Model Policy Gradient Linear Probing Reward Maximization Task Specific Reward Direct Policy Probe Motion

January 26, 2024

Off-Policy Primal-Dual Safe Reinforcement Learning
Zifan Wu, Bo Tang, Qian Lin, Chao Yu, Shangqin Mao, Qianlong Xie, Xingxing Wang, Dong Wang
Reward Maximization Conservative Policy Convex Q Learning

January 23, 2024

Active Inference as a Model of Agency
Lancelot Da Costa, Samuel Tenka, Dominic Zhao, Noor Sajid
Reinforcement Learning Full Model Active Inference Biomedical Entity Reward Maximization

December 28, 2023

Think Before You Duel: Understanding Complexities of Preference Learning under Constrained Resources
Rohan Deb, Aadirupa Saha
Complexity Matter Regret Minimization Preference Learning O$ Regret Reward Maximization Resource Constraint Bandit Identification Wonder Duel

October 31, 2023

Vanishing Gradients in Reinforcement Finetuning of Language Models
Noam Razin, Hattie Zhou, Omid Saremi, Vimal Thilak, Arwen Bradley, Preetum Nakkiran, Joshua Susskind, Etai Littwin
Language Model Fine Tuning Policy Gradient Pretrained Language Model Reward Maximization Supervised Finetuning Gradient Vanishing

Reward Maximization

Papers

Reward Maximization for Pure Exploration: Minimax Optimal Good Arm Identification for Nonparametric Multi-Armed Bandits

Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Design

Probabilistic Satisfaction of Temporal Logic Constraints in Reinforcement Learning via Adaptive Policy-Switching

Ranking Policy Learning via Marketplace Expected Value Estimation From Observational Data

Preference Optimization as Probabilistic Inference

Unsupervised Reward-Driven Image Segmentation in Automated Scanning Transmission Electron Microscopy Experiments

Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Models

Analyzing and Bridging the Gap between Maximizing Total Reward and Discounted Reward in Deep Reinforcement Learning

WARP: On the Benefits of Weight Averaged Rewarded Policies

Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation

Offline Regularised Reinforcement Learning for Large Language Models Alignment

On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization

Value Augmented Sampling for Language Model Alignment and Personalization

Constrained Reinforcement Learning Under Model Mismatch

Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences

Q-Probe: A Lightweight Approach to Reward Maximization for Language Models

Off-Policy Primal-Dual Safe Reinforcement Learning

Active Inference as a Model of Agency

Think Before You Duel: Understanding Complexities of Preference Learning under Constrained Resources

Vanishing Gradients in Reinforcement Finetuning of Language Models