Generalized Policy Improvement

Generalized Policy Improvement (GPI) is a reinforcement learning framework aiming to efficiently improve policies by leveraging previously learned knowledge, particularly through the use of successor features (SFs). Current research focuses on developing theoretically sound GPI algorithms, often incorporating neural network architectures like SF-DQN and MSFA to learn and combine SFs for improved sample efficiency and knowledge transfer across tasks, including multi-objective and safety-constrained scenarios. This work holds significant importance for advancing reinforcement learning, enabling faster learning, better generalization, and more robust policy transfer in complex real-world applications such as robotics and control systems.

Papers

November 2, 2024

Regret of exploratory policy improvement and $q$-learning
Wenpin Tang, Xun Yu Zhou
Diffusion Explainer Early Stage Convergence Simple Regret Based Policy Q$ Learning Generalized Policy Improvement

June 16, 2024

On Convergence Analysis of Policy Iteration Algorithms for Entropy-Regularized Stochastic Control Problems
Jin Ma, Gaozhan Wang, Jianfeng Zhang
Early Stage Convergence Policy Improvement Universal Rate Iterative Method Generalized Policy Improvement Diffusion Control

May 24, 2024

SF-DQN: Provable Knowledge Transfer using Successor Feature for Deep Reinforcement Learning
Shuai Zhang, Heshan Devaka Fernando, Miao Liu, Keerthiram Murugesan, Songtao Lu, Pin-Yu Chen, Tianyi Chen, Meng Wang
Deep Reinforcement Learning Q Learning Knowledge Transfer Transfer Reinforcement Learning Successor Feature Generalized Policy Improvement

October 24, 2023

Combining Behaviors with the Successor Features Keyboard
Wilka Carvalho, Andre Saraiva, Angelos Filos, Andrew Kyle Lampinen, Loic Matthey, Richard L. Lewis, Honglak Lee, Satinder Singh, Danilo J. Rezende, Daniel Zoran
Successor Representation Activity Sequence Successor Feature Generalized Policy Improvement Encode Task

January 28, 2023

Composing Task Knowledge with Modular Successor Feature Approximators
Wilka Carvalho, Angelos Filos, Richard L. Lewis, Honglak lee, Satinder Singh
Neural Network Architecture Task Specific Knowledge Successor Representation Generalized Policy Improvement Predictive Representation

January 18, 2023

Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization
Lucas N. Alegre, Ana L. C. Bazzan, Diederik M. Roijers, Ann Nowé, Bruno C. da Silva
Active Learning Multi Objective Multi Objective Reinforcement Learning Sequential Decision Making Problem Sub Optimal Policy Generalized Policy Improvement

November 10, 2022

Safety-Constrained Policy Transfer with Successor Features
Zeyu Feng, Bowen Zhang, Jianxin Bi, Harold Soh
Reinforcement Learning Safe Policy Successor Feature Policy Transfer Generalized Policy Improvement

June 28, 2022

Generalized Policy Improvement Algorithms with Theoretically Supported Sample Reuse
James Queeney, Ioannis Ch. Paschalidis, Christos G. Cassandras
Policy Reinforcement Learning Model Free Deep Reinforcement Learning Data Driven Control Approximate Policy Iteration Independent Sampling Generalized Policy Improvement

June 22, 2022

Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer
Lucas N. Alegre, Ana L. C. Bazzan, Bruno C. da Silva
Reinforcement Learning Optimal Policy Optimal Behavior Alternative Basis Successor Feature Optimistic Algorithm Support Estimation Generalized Policy Improvement

June 17, 2022

Generalised Policy Improvement with Geometric Policy Composition
Shantanu Thakoor, Mark Rowland, Diana Borsa, Will Dabney, Rémi Munos, André Barreto
Policy Iteration Value Based Reinforcement Learning Generalized Policy Improvement Markovian Policy

November 4, 2021

Successor Feature Neural Episodic Control
David Emukpere, Xavier Alameda-Pineda, Chris Reinke
Reinforcement Learning Episodic Memory Single Agent Reinforcement Learning Episodic Control Generalized Policy Improvement