Policy Algorithm

Policy algorithms in reinforcement learning aim to learn optimal decision-making strategies from data, often focusing on off-policy methods that leverage past experiences collected under different policies. Current research emphasizes improving the robustness and efficiency of these algorithms, addressing issues like overestimation bias, variance reduction in importance sampling, and handling model misspecification through techniques such as conservative updates, bootstrapping, and weighted replay buffers. This work has significant implications for various applications, including biological sequence design, language model alignment, and robotics, by enabling more sample-efficient and reliable learning from offline datasets.

Papers

October 6, 2024

Improved Off-policy Reinforcement Learning in Biological Sequence Design
Hyeonah Kim, Minsu Kim, Taeyoung Yun, Sanghyeok Choi, Emmanuel Bengio, Alex Hernández-García, Jinkyoo Park
Large Relevance Improvement Policy Reinforcement Learning Biological Sequence Policy Algorithm Offline Model

May 31, 2024

Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment
Yueqin Yin, Zhendong Wang, Yujia Xie, Weizhu Chen, Mingyuan Zhou
Direct Preference Optimization Language Model Alignment Policy Algorithm

May 16, 2024

$Δ\text{-}{\rm OPE}$: Off-Policy Estimation with Pairs of Policies
Olivier Jeunen, Aleksei Ustimenko
Prior Policy Based Policy Similar Pair Policy Algorithm Control Variate Policy Estimation Inverse Propensity

April 12, 2024

Adversarial Imitation Learning via Boosting
Jonathan D. Chang, Dhruv Sreenivas, Yingbing Huang, Kianté Brantley, Wen Sun
Actor Critic Adversarial Imitation Learning Policy Algorithm Adversarial Imitation Policy Adversarial Imitation Learning

March 1, 2024

Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning
Michal Nauman, Michał Bortkiewicz, Piotr Miłoś, Tomasz Trzciński, Mateusz Ostaszewski, Marek Cygan
Reinforcement Learning Regularization Model Policy Reinforcement Learning Model Overfitting Regularization Technique Plasticity Rule Soft Actor Critic Policy Algorithm

August 18, 2023

Baird Counterexample is Solved: with an example of How to Debug a Two-time-scale Algorithm
Hengshuai Yao
Convergence Rate Single Example Time Scale Policy Algorithm Convergence Behavior Counter Example

June 2, 2023

ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages
Andrew Jesson, Chris Lu, Gunshi Gupta, Nicolas Beltran-Velez, Angelos Filos, Jakob Nicolaus Foerster, Yarin Gal
Actor Critic Search and Rescue ReLU Layer Health Policy Recommendation Advantage Feedback Approximate Bayesian Inference Policy Algorithm Approximate Posterior

March 30, 2023

Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
Yicheng Luo, Jackie Kay, Edward Grefenstette, Marc Peter Deisenroth
Technical Challenge Offline Reinforcement Learning Offline Reinforcement Learning Algorithm Policy Algorithm Online Fine Tuning Simple Solution

February 1, 2023

Distillation Policy Optimization
Jianfei Ma
Policy Reinforcement Learning Policy Algorithm Policy Distillation Policy Data

January 26, 2023

Trajectory-Aware Eligibility Traces for Off-Policy Reinforcement Learning
Brett Daley, Martha White, Christopher Amato, Marlos C. Machado
Policy Reinforcement Learning Sample Efficient Reinforcement Learning Policy Algorithm

December 26, 2022

Off-Policy Reinforcement Learning with Loss Function Weighted by Temporal Difference Error
Bumgeun Park, Taeyoung Kim, Woohyeon Moon, Luiz Felipe Vecchietti, Dongsoo Har
Loss Function Policy Reinforcement Learning Memory Replay Policy Algorithm Policy Deep Reinforcement Learning

December 23, 2022

Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Information
Zuyue Fu, Zhengling Qi, Zhuoran Yang, Zhaoran Wang, Lan Wang
Offline Reinforcement Learning Policy Evaluation Private Data Human Machine Interaction Policy Algorithm

November 7, 2022

Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecification
Takumi Tanabe, Rei Sato, Kazuto Fukuchi, Jun Sakuma, Youhei Akimoto
Reinforcement Learning Actor Critic Deep Deterministic Policy Gradient Model Misspecification Policy Algorithm Policy Actor Critic Uncertainty Set Multi Joint Worst Case Robustness

September 30, 2022

On Convergence of Average-Reward Off-Policy Control Algorithms in Weakly Communicating MDPs
Yi Wan, Richard S. Sutton
Early Stage Convergence Q Learning Policy Algorithm Average Reward

September 15, 2022

On the Reuse Bias in Off-Policy Reinforcement Learning
Chengyang Ying, Zhongkai Hao, Xinning Zhou, Hang Su, Dong Yan, Jun Zhu
Policy Evaluation Policy Reinforcement Learning Importance Sampling Policy Algorithm Text Reuse

September 10, 2022

Gradient Descent Temporal Difference-difference Learning
Rong J. B. Zhu, James M. Murray
Reinforcement Learning Gradient Descent Temporal Difference Learning Policy Algorithm TD Learning

September 6, 2022

Finite-Time Error Bounds for Greedy-GQ
Yue Wang, Yi Zhou, Shaofeng Zou
Reinforcement Learning Finite Time Policy Algorithm Value Based Reinforcement Learning

April 12, 2022

Forgetting and Imbalance in Robot Lifelong Learning with Off-policy Data
Wenxuan Zhou, Steven Bohez, Jan Humplik, Abbas Abdolmaleki, Dushyant Rao, Markus Wulfmeier, Tuomas Haarnoja, Nicolas Heess
Reinforcement Learning Anti Forgetting Robot Dynamic Non Stationary Environment Group Imbalance Policy Algorithm Policy Data Lifelong Robot

March 18, 2022

Importance Sampling Placement in Off-Policy Temporal-Difference Methods
Eric Graves, Sina Ghiassian
Policy Reinforcement Learning Importance Sampling Temporal Difference Policy Algorithm

February 23, 2022

Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, Zhaoran Wang
Offline Reinforcement Learning Bootstrapping End to End Linear MDPs Policy Algorithm Robust Offline Reinforcement Learning