Successor Feature

Successor features (SFs) are a representation in reinforcement learning that decomposes the value function into a component representing the environment's dynamics (SFs themselves) and a component representing the reward function. Current research focuses on leveraging SFs to improve efficiency in various applications, including pathfinding, transfer learning, and multi-agent systems, often employing algorithms like generalized policy improvement and contrastive learning to learn and utilize these features. This decomposition enables faster learning, better generalization across tasks, and improved control over complex systems, with applications ranging from robotics and game AI to natural language processing. The ability of SFs to decouple dynamics from rewards is proving particularly valuable in addressing challenges related to exploration, knowledge transfer, and safe policy learning.

Papers

November 11, 2024

Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching
Arnav Kumar Jain, Harley Wiltzer, Jesse Farebrother, Irina Rish, Glen Berseth, Sanjiban Choudhury
Inverse Reinforcement Learning Reward Model Matching Accuracy Adversarial Game Successor Feature Adversarial Inverse Reinforcement Learning Game Solving

October 29, 2024

Learning Successor Features the Simple Way
Raymond Chua, Arna Ghosh, Christos Kaplanis, Blake A. Richards, Doina Precup
Deep Reinforcement Learning Successor Representation Reward Prediction Continual Learning Scenario Successor Feature Representation Collapse

August 27, 2024

Pathfinding with Lazy Successor Generation
Keisuke Okumura
Search Algorithm Successor Feature

June 24, 2024

Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making
Vivek Myers, Chongyi Zheng, Anca Dragan, Sergey Levine, Benjamin Eysenbach
Reinforcement Learning Successor Feature Combinatorial Generalization Metric Structure Temporal Distance

May 24, 2024

SF-DQN: Provable Knowledge Transfer using Successor Feature for Deep Reinforcement Learning
Shuai Zhang, Heshan Devaka Fernando, Miao Liu, Keerthiram Murugesan, Songtao Lu, Pin-Yu Chen, Tianyi Chen, Meng Wang
Deep Reinforcement Learning Q Learning Knowledge Transfer Transfer Reinforcement Learning Successor Feature Generalized Policy Improvement

May 4, 2024

Decoupling Exploration and Exploitation for Unsupervised Pre-training with Successor Features
JaeYoon Kim, Junyu Xuan, Christy Liang, Farookh Hussain
Multi Agent Decoupling Coefficient Unsupervised Pre Training Real World eXploitation Exploration Method Successor Feature

March 15, 2024

Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics
Luca Grillotti, Maxence Faldor, Borja G. León, Antoine Cully
Deep Reinforcement Learning Value Function Net Present Value Quality Diversity High Performance Diverse Behavior Policy Actor Critic Successor Feature

November 3, 2023

Successor Features for Efficient Multisubject Controlled Text Generation
Meng Cao, Mehdi Fatemi, Jackie Chi Kit Cheung, Samira Shabanian
Large Language Model Text Generation Successor Feature

October 24, 2023

Combining Behaviors with the Successor Features Keyboard
Wilka Carvalho, Andre Saraiva, Angelos Filos, Andrew Kyle Lampinen, Loic Matthey, Richard L. Lewis, Honglak Lee, Satinder Singh, Danilo J. Rezende, Daniel Zoran
Successor Representation Activity Sequence Successor Feature Generalized Policy Improvement Encode Task

June 16, 2023

$\pi2\text{vec}$: Policy Representations with Successor Features
Gianluca Scarpellini, Ksenia Konyushkova, Claudio Fantacci, Tom Le Paine, Yutian Chen, Misha Denil
State Representation Offline Policy Offline Policy Evaluation Successor Feature Policy Representation Policy Selection

May 5, 2023

Improving LaCAM for Scalable Eventually Optimal Multi-Agent Pathfinding
Keisuke Okumura
Multi Agent Potential Scalability Multi Agent Path Finding Multi Agent Pathfinding Successor Feature

April 14, 2023

Exploring the Noise Resilience of Successor Features and Predecessor Features Algorithms in One and Two-Dimensional Environments
Hyunsu Lee
Reinforcement Learning Visual Navigation Computational Neuroscience Spatial Learning Successor Feature Noise Resistant Property Map Prediction Dimensional Environment

November 13, 2022

Build generally reusable agent-environment interaction models
Jun Jin, Hongming Zhang, Jun Luo
Downstream Task Agent Environment Interaction Successor Feature Structural Construct Action Pair

November 10, 2022

Safety-Constrained Policy Transfer with Successor Features
Zeyu Feng, Bowen Zhang, Jianxin Bi, Harold Soh
Reinforcement Learning Safe Policy Successor Feature Policy Transfer Generalized Policy Improvement

July 29, 2022

Meta Reinforcement Learning with Successor Feature Based Context
Xu Han, Feng Wu
Meta Reinforcement Learning Continuous Control Task Successor Feature

June 22, 2022

Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer
Lucas N. Alegre, Ana L. C. Bazzan, Bruno C. da Silva
Reinforcement Learning Optimal Policy Optimal Behavior Alternative Basis Successor Feature Optimistic Algorithm Support Estimation Generalized Policy Improvement

May 20, 2022

Task Relabelling for Multi-task Transfer using Successor Features
Martin Balla, Diego Perez-Liebana
Deep Reinforcement Learning Reward Function Task Vector Successor Feature Task Transfer

February 15, 2022

Disentangling Successor Features for Coordination in Multi-agent Reinforcement Learning
Seung Hyun Kim, Neale Van Stralen, Girish Chowdhary, Huy T. Tran
Multi Agent Reinforcement Learning Sparse Reward Multi Agent Environment Prior Coordination Individual Agent Decentralized Agent Successor Feature

November 18, 2021

Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learning
Christopher Hoang, Sungryull Sohn, Jongwook Choi, Wilka Carvalho, Honglak Lee
Efficient Exploration Goal Conditioned Reinforcement Learning Goal Conditioned Policy Successor Feature