Policy Optimization

Policy optimization is a core area of reinforcement learning focused on efficiently finding optimal policies, or strategies, for agents interacting with an environment to maximize rewards. Current research emphasizes improving sample efficiency and robustness, particularly through algorithms like Proximal Policy Optimization (PPO) and its variants, as well as exploring new approaches such as Direct Preference Optimization (DPO) and incorporating techniques like diffusion models and dual regularization. These advancements are significant for both theoretical understanding of reinforcement learning and practical applications across diverse fields, including robotics, natural language processing, and resource management.

Papers

October 2, 2022

Policy Gradients for Probabilistic Constrained Reinforcement Learning
Weiqin Chen, Dharmashankar Subramanian, Santiago Paternain
Reinforcement Learning Policy Gradient Policy Optimization Constrained Reinforcement Learning Probabilistic Safety Probabilistic Constraint Policy Based Algorithm

September 9, 2022

Extending Open Bandit Pipeline to Simulate Industry Challenges
Bram van den Akker, Niklas Weber, Felipe Moraes, Dmitri Goldenberg
Concept Drift Bandit Algorithm Policy Optimization Bandit Model Reward Design

August 25, 2022

Variance Reduction based Experience Replay for Policy Optimization
Hua Zheng, Wei Xie, M. Ben Feng
Reinforcement Learning Policy OpTimization Policy Optimization Variance Reduction Experience Replay

August 23, 2022

Strategic Decision-Making in the Presence of Information Asymmetry: Provably Efficient RL with Algorithmic Instruments
Mengxin Yu, Zhuoran Yang, Jianqing Fan
Markov Decision Process Offline Reinforcement Learning Optimal Policy Decision Making Speech Presence Policy Optimization Efficient Reinforcement Learning Information Asymmetry Algorithmic System

July 29, 2022

Improved Policy Optimization for Online Imitation Learning
Jonathan Wilder Lavington, Sharan Vaswani, Mark Schmidt
High Dimensional Policy Optimization Follow the Regularized Leader Offline Imitation Expressive Policy

July 12, 2022

Compactly Restrictable Metric Policy Optimization Problems
Victor D. Dorobantu, Kamyar Azizzadenesheli, Yisong Yue
Policy Optimization

June 30, 2022

Denoised MDPs: Learning World Models Better Than the World Itself
Tongzhou Wang, Simon S. Du, Antonio Torralba, Phillip Isola, Amy Zhang, Yuandong Tian
Reinforcement Learning World Model Policy Optimization Controllability Condition DeepMind Control Suite

June 17, 2022

A Parametric Class of Approximate Gradient Updates for Policy Optimization
Ramki Gummadi, Saurabh Kumar, Junfeng Wen, Dale Schuurmans
Policy OpTimization Policy Optimization Reinforcement Learning Benchmark Policy Representation

June 14, 2022

Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
Shentao Yang, Yihao Feng, Shujian Zhang, Mingyuan Zhou
Offline Reinforcement Learning Policy Optimization Stationary Policy State Action Distribution

June 11, 2022

Federated Offline Reinforcement Learning
Doudou Zhou, Yufeng Zhang, Aaron Sonabend-W, Zhaoran Wang, Junwei Lu, Tianxi Cai
Markov Decision Process Offline Reinforcement Learning Policy Optimization Offline Reinforcement Learning Algorithm Dynamic Treatment Regime Federated Offline Reinforcement Learning

June 6, 2022

Pessimistic Off-Policy Optimization for Learning to Rank
Matej Cief, Branislav Kveton, Michal Kompan
LeArning Abstract Recommender System Stable Rank Policy Optimization Inverse Propensity

June 3, 2022

GIN: Graph-based Interaction-aware Constraint Policy Optimization for Autonomous Driving
Se-Wook Yoo, Chan Kim, Jin-Woo Choi, Seong-Woo Kim, Seung-Woo Seo
Reinforcement Learning Autonomous Driving Graph Based Policy Optimization Urban Driving

May 23, 2022

Generalization, Mayhems and Limits in Recurrent Proximal Policy Optimization
Marco Pleines, Matthias Pallasch, Frank Zimmer, Mike Preuss
Deep Reinforcement Learning Strong Generalization Proximal Policy Optimization Continuum Limit Policy Optimization Recurrent Layer Memory Less Agent

May 20, 2022

The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measure
Xing Chen, Dongcui Diao, Hechang Chen, Hengshuai Yao, Haiyin Piao, Zhixiao Sun, Zhiwei Yang, Randy Goebel, Bei Jiang, Yi Chang
Policy Gradient Proximal Policy Optimization Policy Evaluation Policy Optimization High Dimensional Clipping Policy Data Conservative Policy

May 18, 2022

A2C is a special case of PPO
Shengyi Huang, Anssi Kanervisto, Antonin Raffin, Weixun Wang, Santiago Ontañón, Rousslan Fernand Julien Dossa
Proximal Policy Optimization Policy Optimization Game AI Deep Reinforcement Learning Algorithm Advantage Actor Critic Special Case Pseudo Code Apical 2 Chamber

May 6, 2022

Variance Reduction based Partial Trajectory Reuse to Accelerate Policy Gradient Optimization
Hua Zheng, Wei Xie
Policy Gradient Policy Optimization Variance Reduction Trajectory Refinement

March 24, 2022

Bellman Residual Orthogonalization for Offline Reinforcement Learning
Andrea Zanette, Martin J. Wainwright
Reinforcement Learning Offline Reinforcement Learning Policy Evaluation Policy Optimization Bellman Equation Bellman Residual

February 28, 2022

GPU-Accelerated Policy Optimization via Batch Automatic Differentiation of Gaussian Processes for Real-World Control
Abdolreza Taheri, Joni Pajarinen, Reza Ghabcheloo
Gaussian Process Policy OpTimization Policy Optimization Automatic Differentiation Policy Network Real World Control

February 26, 2022

Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizons
Chengchun Shi, Shikai Luo, Yuan Le, Hongtu Zhu, Rui Song
Reinforcement Learning Offline Reinforcement Learning Policy OpTimization Policy Optimization New Horizon Function Estimation Advantage Learning

February 25, 2022

Off-Policy Evaluation with Policy-Dependent Optimization Response
Wenshuo Guo, Michael I. Jordan, Angela Zhou
Causal Inference Causal Effect Policy Evaluation Policy Optimization