Policy Data

Policy data, encompassing data collected from the execution of various policies in reinforcement learning and related fields, is crucial for efficient and effective model training and evaluation. Current research focuses on improving off-policy methods, which leverage existing data from different policies, by addressing issues like distributional shifts and variance in estimators, often employing techniques like state abstraction, weighted preference optimization, and novel policy gradient formulations. These advancements aim to enhance the sample efficiency and stability of reinforcement learning algorithms, with significant implications for applications such as robotics, natural language processing, and personalized healthcare where collecting on-policy data is costly or impractical.

Papers

October 3, 2024

Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation
Shreyas Chaudhari, Ameet Deshpande, Bruno Castro da Silva, Philip S. Thomas
Reinforcement Learning Policy Evaluation State Abstraction Geometric Median Policy Data

June 17, 2024

WPO: Enhancing RLHF with Weighted Preference Optimization
Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, Chenguang Zhu
Direct Preference Optimization Policy Reinforcement Learning Preference Optimization Reinforcement Learning From Human Feedback Policy Data

June 6, 2024

Transductive Off-policy Proximal Policy Optimization
Yaozhong Gan, Renye Yan, Xiaoyang Tan, Zhe Wu, Junliang Xing
Proximal Policy Optimization Policy Optimization Model Free Reinforcement Learning Policy Data

April 22, 2024

Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, Aviral Kumar
Contrastive Learning Fine Tuning Medical LLM Preference Fine Tuning Policy Sampling Policy Data

February 20, 2024

Skill or Luck? Return Decomposition via Advantage Functions
Hsiao-Ru Pan, Bernhard Schölkopf
Microbial Decomposition Sample Efficient Reinforcement Learning Policy Data Advantage Estimation

February 19, 2024

When Do Off-Policy and On-Policy Policy Gradient Methods Align?
Davide Mambelli, Stephan Bongers, Onno Zoeter, Matthijs T. J. Spaan, Frans A. Oliehoek
Policy Gradient Policy Data

April 30, 2023

Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RL
Baiting Zhu, Meihua Dang, Aditya Grover
Multi Objective Decision Transformer Multi Objective Reinforcement Learning Policy Data

February 1, 2023

Distillation Policy Optimization
Jianfei Ma
Policy Reinforcement Learning Policy Algorithm Policy Distillation Policy Data

July 18, 2022

Online Learning with Off-Policy Feedback
Germano Gabbianelli, Matteo Papini, Gergely Neu
Online Learning Policy Reinforcement Learning Adversarial Bandit Adversarial Linear Contextual Bandit Policy Data

May 20, 2022

The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measure
Xing Chen, Dongcui Diao, Hechang Chen, Hengshuai Yao, Haiyin Piao, Zhixiao Sun, Zhiwei Yang, Randy Goebel, Bei Jiang, Yi Chang
Policy Gradient Proximal Policy Optimization Policy Evaluation Policy Optimization High Dimensional Clipping Policy Data Conservative Policy

April 12, 2022

Forgetting and Imbalance in Robot Lifelong Learning with Off-policy Data
Wenxuan Zhou, Steven Bohez, Jan Humplik, Abbas Abdolmaleki, Dushyant Rao, Markus Wulfmeier, Tuomas Haarnoja, Nicolas Heess
Reinforcement Learning Anti Forgetting Robot Dynamic Non Stationary Environment Group Imbalance Policy Algorithm Policy Data Lifelong Robot