Temporal Abstraction

Temporal abstraction in reinforcement learning aims to improve efficiency and performance by enabling agents to learn and utilize higher-level actions (skills or options) that span multiple time steps, rather than relying solely on immediate actions. Current research focuses on developing hierarchical reinforcement learning (HRL) algorithms, often employing model-based approaches, attention mechanisms, and graph-based representations to learn these temporal abstractions effectively from both online and offline data. This work is significant because it addresses the limitations of standard RL in complex environments with sparse rewards and long horizons, leading to more efficient and robust learning in various applications, including robotics and recommendation systems.

Papers

September 11, 2024

Hierarchical Reinforcement Learning for Temporal Abstraction of Listwise Recommendation
Luo Ji, Gao Liu, Mingyang Yin, Hongxia Yang, Jingren Zhou
Reinforcement Learning Hierarchical Reinforcement Learning Modern Recommender System Temporal Abstraction User Feedback

July 21, 2024

Temporal Abstraction in Reinforcement Learning with Offline Data
Ranga Shaarad Ayyagari, Anurita Ghosh, Ambedkar Dukkipati
Reinforcement Learning Hierarchical Reinforcement Learning Offline Data Multi Agent MuJoCo Standard Reinforcement Learning Temporal Abstraction

June 12, 2024

When Do Skills Help Reinforcement Learning? A Theoretical Analysis of Temporal Abstractions
Zhening Li, Gabriel Poesia, Armando Solar-Lezama
Reinforcement Learning Theoretical Analysis Sparse Reward Environment Robust Skill Temporal Abstraction Parameterized Skill

April 24, 2024

Learning World Models With Hierarchical Temporal Abstractions: A Probabilistic Perspective
Vaisakh Shaj
State Space Model World Model Bayesian Perspective Novel Bayesian Temporal Abstraction Bayesian Brain

January 18, 2024

Reconciling Spatial and Temporal Abstractions for Goal Representation
Mehdi Zadem, Sergio Mover, Sao Mai Nguyen
Hierarchical Reinforcement Learning Spatial Domain Continuous Control Task Environment Dynamic Goal Representation Temporal Abstraction Temporal Goal

September 30, 2023

Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learning
Mingde Zhao, Safa Alver, Harm van Seijen, Romain Laroche, Doina Precup, Yoshua Bengio
Reinforcement Learning Task Planning Better Generalization Hierarchical Planning Temporal Abstraction

May 30, 2023

Optimizing Attention and Cognitive Control Costs Using Temporally-Layered Architectures
Devdhar Patel, Terrence Sejnowski, Hava Siegelmann
Continuous Control Task Action Repetition Temporal Abstraction Continuous Control Policy

February 14, 2023

Predicting the long-term collective behaviour of fish pairs with deep learning
Vaios Papaspyros, Ramón Escobedo, Alexandre Alahi, Guy Theraulaz, Clément Sire, Francesco Mondada
Deep Learning Deep Learning Model Collective Behavior Collective Motion Temporal Abstraction Fish Machine

December 25, 2022

Temporally Layered Architecture for Adaptive, Distributed and Continuous Control
Devdhar Patel, Joshua Russell, Francesca Walsh, Tauhidur Rahman, Terrence Sejnowski, Hava Siegelmann
Adaptive Importance Continuous Control Continuous Control Task Temporal Abstraction

December 22, 2022

Reusable Options through Gradient-based Meta Learning
David Kuric, Herke van Hoof
Reinforcement Learning Meta Learning Reusable Data Exhaust Gradient Based Meta Learning Temporal Abstraction

October 27, 2022

Planning with Spatial-Temporal Abstraction from Point Clouds for Deformable Object Manipulation
Xingyu Lin, Carl Qi, Yunchu Zhang, Zhiao Huang, Katerina Fragkiadaki, Yunzhu Li, Chuang Gan, David Held
Point Cloud Deformable Object Deformable Object Manipulation Deformable Object Manipulation Task Temporal Abstraction

October 11, 2022

DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning
Seungjae Lee, Jigang Kim, Inkyu Jang, H. Jin Kim
Reinforcement Learning Long Term Hierarchical Reinforcement Learning Graph Based Approach Temporal Abstraction Efficient Locomotion

September 25, 2022

Temporally Extended Successor Representations
Matthew J. Sargent, Peter J. Bentley, Caswell Barry, William de Cothi
Model Free Reinforcement Learning Successor Representation Action Repetition Temporal Abstraction

July 17, 2022

Learning from Temporal Spatial Cubism for Cross-Dataset Skeleton-based Action Recognition
Yansong Tang, Xingyu Liu, Xumin Yu, Danyang Zhang, Jiwen Lu, Jie Zhou
LeArning Abstract Unsupervised Domain Adaptation Skeleton Based Action Recognition Skeleton Based Action Temporal Abstraction

March 25, 2022

Unsupervised Learning of Temporal Abstractions with Slot-based Transformers
Anand Gopalakrishnan, Kazuki Irie, Jürgen Schmidhuber, Sjoerd van Steenkiste
Reinforcement Learning Unsupervised Learning Slot Attention Temporal Abstraction State Action Adaptive Computation

February 7, 2022

Reward-Respecting Subtasks for Model-Based Reinforcement Learning
Richard S. Sutton, Marlos C. Machado, G. Zacharias Holland, David Szepesvari, Finbarr Timbers, Brian Tanner, Adam White
Reinforcement Learning Model Based Reinforcement Learning Temporal Abstraction Average Reward Option Discovery

January 17, 2022

Summarising and Comparing Agent Dynamics with Contrastive Spatiotemporal Abstraction
Tom Bewley, Jonathan Lawry, Arthur Richards
Structured Summary State Space Deep Reinforcement Human Written Summary Agent Dynamic Temporal Abstraction

January 7, 2022

Attention Option-Critic
Raviteja Chunduru, Doina Precup
Reinforcement Learning Human Attention Learning Environment Temporal Abstraction Option Learning

Temporal Abstraction

Papers

Hierarchical Reinforcement Learning for Temporal Abstraction of Listwise Recommendation

Temporal Abstraction in Reinforcement Learning with Offline Data

When Do Skills Help Reinforcement Learning? A Theoretical Analysis of Temporal Abstractions

Learning World Models With Hierarchical Temporal Abstractions: A Probabilistic Perspective

Reconciling Spatial and Temporal Abstractions for Goal Representation

Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learning

Optimizing Attention and Cognitive Control Costs Using Temporally-Layered Architectures

Predicting the long-term collective behaviour of fish pairs with deep learning

Temporally Layered Architecture for Adaptive, Distributed and Continuous Control

Reusable Options through Gradient-based Meta Learning

Planning with Spatial-Temporal Abstraction from Point Clouds for Deformable Object Manipulation

DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning

Temporally Extended Successor Representations

Learning from Temporal Spatial Cubism for Cross-Dataset Skeleton-based Action Recognition

Unsupervised Learning of Temporal Abstractions with Slot-based Transformers

Reward-Respecting Subtasks for Model-Based Reinforcement Learning

Summarising and Comparing Agent Dynamics with Contrastive Spatiotemporal Abstraction

Attention Option-Critic