Scalar Reward

Scalar reward, a single numerical value representing the desirability of an outcome, is a cornerstone of reinforcement learning but faces limitations in complex scenarios. Current research focuses on overcoming these limitations, exploring alternatives like generative models that produce richer, more interpretable reward signals from preference data, and methods for optimally transforming preference feedback into scalar rewards. This work is crucial for advancing reinforcement learning's applicability to real-world problems, particularly in safety-critical domains where a single scalar reward may be insufficient to capture nuanced objectives and avoid unintended consequences. The shift towards multi-objective approaches and more sophisticated reward representations reflects a growing recognition of the need for more robust and reliable AI systems.

Papers

October 22, 2024

Sample-Efficient Curriculum Reinforcement Learning for Complex Reward Functions
Kilian Freitag, Kristian Ceder, Rita Laezza, Knut Åkesson, Morteza Haghir Chehreghani
Reinforcement Learning Reward Function Simple Function Curriculum Reinforcement Learning Scalar Reward

October 1, 2024

Beyond Scalar Reward Model: Learning Generative Judge from Preference Data
Ziyi Ye, Xiangsheng Li, Qiuchi Li, Qingyao Ai, Yujia Zhou, Wei Shen, Dong Yan, Yiqun Liu
Direct Preference Optimization Preference Feedback Preference Pair Fine Tuned Judge Model Self Contrastive Scalar Reward

June 14, 2024

Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning
Yinglun Xu, David Zhu, Rohan Gumaste, Gagandeep Singh
Reinforcement Learning Offline Reinforcement Learning Offline Preference Offline RL Offline Preference Based Reinforcement Learning Scalar Reward

October 3, 2023

Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learning
Finn Rietz, Erik Schaffernicht, Stefan Heinrich, Johannes Andreas Stork
Reinforcement Learning State Action Space Scalar Reward

October 20, 2022

Safe Policy Improvement in Constrained Markov Decision Processes
Luigi Berducci, Radu Grosu
Reinforcement Learning Model Free Reward Shaping Constrained Markov Decision Process Policy Improvement Scalar Reward

July 1, 2022

Multi-Objective Coordination Graphs for the Expected Scalarised Returns with Generative Flow Models
Conor F. Hayes, Timothy Verstraeten, Diederik M. Roijers, Enda Howley, Patrick Mannion
Optimal Policy Multiple Objective Coordination Graph Generative Flow Model Scalar Reward

April 13, 2022

Modularity benefits reinforcement learning agents with competing homeostatic drives
Zack Dulberg, Rachit Dubey, Isabel M. Berwian, Jonathan D. Cohen
Reinforcement Learning Agent Smith Agent System Q$ Learning Homeostatic Design Confers Scalar Reward Modular Reinforcement Learning

November 25, 2021

Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)
Peter Vamplew, Benjamin J. Smith, Johan Kallstrom, Gabriel Ramos, Roxana Radulescu, Diederik M. Roijers, Conor F. Hayes, Fredrik Heintz, Patrick Mannion, Pieter J. K. Libin, Richard Dazeley, Cameron Foale
Artificial General Intelligence Non Negative Textual Response Reward Maximization Striking GOld Intelligent Behaviour Scalar Reward