Advantage Learning

Advantage learning is a reinforcement learning technique focused on improving policy optimization by emphasizing the difference in value between the best action and alternative actions. Current research explores its application in various contexts, including offline reinforcement learning, fairness-constrained decision-making, and human-in-the-loop reward shaping, often employing algorithms like Double Q-learning and modifications to Bellman operators to enhance stability and efficiency. These advancements aim to address challenges like sample inefficiency, overestimation bias, and robustness to noisy estimations, ultimately leading to more reliable and effective reinforcement learning agents for diverse applications.

Papers

October 3, 2023

Learning Optimal Advantage from Preferences and Mistaking it for Reward
W. Bradley Knox, Stephane Hatgis-Kessell, Sigurdur Orn Adalgeirsson, Serena Booth, Anca Dragan, Peter Stone, Scott Niekum
Reinforcement Learning Reward Function Preference Feedback Reward Report Reward Shaping Regret Analysis Preference Dynamic Advantage Learning

February 23, 2023

What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structure
Lukas Galke, Yoav Ram, Limor Raviv
Deep Neural Network Recurrent Neural Network Advantage Learning

January 28, 2023

Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learning
Jing Zhang, Chi Zhang, Wenjia Wang, Bing-Yi Jing
Offline Reinforcement Learning Constrained Policy Optimization Advantage Learning

October 22, 2022

Policy Optimization with Advantage Regularization for Long-Term Fairness in Decision Systems
Eric Yang Yu, Zhizhen Qin, Min Kyung Lee, Sicun Gao
Deep Reinforcement Learning Policy OpTimization Policy Optimization Decision Support System Long Term Fairness Advantage Learning Term Fairness

June 7, 2022

How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via $f$-Advantage Regression
Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, Osbert Bastani
Reinforcement Learning State Occupancy Matching Offline Goal Conditioned Reinforcement Learning Goal Oriented Task Advantage Learning

May 16, 2022

Enforcing KL Regularization in General Tsallis Entropy Reinforcement Learning via Advantage Learning
Lingwei Zhu, Zheng Chen, Eiji Uchibe, Takamitsu Matsubara
Maximum Entropy DQN Agent Entropy Regularized Advantage Learning Kullback Leibler Tsallis Entropy

May 8, 2022

Simultaneous Double Q-learning with Conservative Advantage Learning for Actor-Critic Methods
Qing Li, Wengang Zhou, Zhenbo Lu, Houqiang Li
Actor Critic Policy Actor Critic Double Q Learning Advantage Learning

March 20, 2022

February 26, 2022

Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizons
Chengchun Shi, Shikai Luo, Yuan Le, Hongtu Zhu, Rui Song
Reinforcement Learning Offline Reinforcement Learning Policy OpTimization Policy Optimization New Horizon Function Estimation Advantage Learning

November 22, 2021

Towards a Principled Learning Rate Adaptation for Natural Evolution Strategies
Masahiro Nomura, Isao Ono
Learning Rate Optimization Problem Advantage Learning Natural Evolution Strategy

Advantage Learning

Papers

Learning Optimal Advantage from Preferences and Mistaking it for Reward

What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structure

Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learning

Policy Optimization with Advantage Regularization for Long-Term Fairness in Decision Systems

How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via $f$-Advantage Regression

Enforcing KL Regularization in General Tsallis Entropy Reinforcement Learning via Advantage Learning

Simultaneous Double Q-learning with Conservative Advantage Learning for Actor-Critic Methods

Smoothing Advantage Learning

Robust Action Gap Increasing with Clipped Advantage Learning

Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizons

Towards a Principled Learning Rate Adaptation for Natural Evolution Strategies