Compositional Reinforcement Learning

Compositional reinforcement learning (CRL) aims to enable agents to learn complex tasks by breaking them down into simpler subtasks, learning policies for each, and then composing these policies to solve the overall problem. Current research focuses on developing methods that leverage logical specifications, category theory, and multifidelity simulations to improve sample efficiency, ensure robustness, and provide formal guarantees on the learned policies, often employing neural network architectures and various multi-agent RL algorithms. This approach holds significant promise for creating more efficient and reliable AI systems capable of handling complex real-world scenarios, particularly in robotics, by facilitating transfer learning and generalization across diverse tasks.

Papers

August 23, 2024

Reduce, Reuse, Recycle: Categories for Compositional Reinforcement Learning
Georgios Bakirtzis, Michail Savvas, Ruihan Zhao, Sandeep Chinchali, Ufuk Topcu
Reinforcement Learning Markov Decision Process Constructive Reduction Category Shift Task Specification Reusable Data Exhaust Advanced Textile Recycling Compositional Reinforcement Learning

February 6, 2024

Logical Specifications-guided Dynamic Task Sampling for Reinforcement Learning Agents
Yash Shukla, Tanushree Burman, Abhishek Kulkarni, Robert Wright, Alvaro Velasquez, Jivko Sinapov
Reinforcement Learning Temporal Logic Reinforcement Learning Agent Compositional Reinforcement Learning Task Sampling Observable Robotic

December 3, 2023

Compositional Policy Learning in Stochastic Control Systems with Formal Guarantees
Đorđe Žikelić, Mathias Lechner, Abhinav Verma, Krishnendu Chatterjee, Thomas A. Henzinger
Reinforcement Learning Stochastic Environment Formal Guarantee Neural Network Policy Reach Avoid Compositional Reinforcement Learning Evolving Policy Stochastic Control System

December 2, 2023

A Multifidelity Sim-to-Real Pipeline for Verifiable and Compositional Reinforcement Learning
Cyrus Neary, Christian Ellis, Aryaman Singh Samyal, Craig Lennon, Ufuk Topcu
Reinforcement Learning Efficient Reinforcement Learning Compositional Reinforcement Learning

October 6, 2023

Compositional Servoing by Recombining Demonstrations
Max Argus, Abhijeet Nayak, Martin Büchner, Silvio Galesso, Abhinav Valada, Thomas Brox
Visual Servoing Task Demonstration Manipulation Policy Demonstration Data Compositional Reinforcement Learning

September 9, 2023

Verifiable Reinforcement Learning Systems via Compositionality
Cyrus Neary, Aryaman Singh Samyal, Christos Verginis, Murat Cubuktepe, Ufuk Topcu
Action Space Partial Observability Compositional Language Deep Reinforcement Learning Agent Compositional Reinforcement Learning

July 13, 2023

Robotic Manipulation Datasets for Offline Compositional Reinforcement Learning
Marcel Hussing, Jorge A. Mendez, Anisha Singrodia, Cassandra Kent, Eric Eaton
Offline Reinforcement Learning Compositional Reinforcement Learning Manipulation Datasets

March 5, 2023

Bounding the Optimal Value Function in Compositional Reinforcement Learning
Jacob Adamczyk, Volodymyr Makarenko, Argenis Arriojas, Stas Tiomkin, Rahul V. Kulkarni
Reinforcement Learning Reward Function General Bound Entropy Regularized Optimal Value Function Compositional Reinforcement Learning Function Composition

February 6, 2023

Robust Subtask Learning for Compositional Generalization
Kishor Jothimurugan, Steve Hsu, Osbert Bastani, Rajeev Alur
Compositional Generalization Zero Sum Game Task Specific Policy Compositional Reinforcement Learning

August 29, 2022

Categorical semantics of compositional reinforcement learning
Georgios Bakirtzis, Michail Savvas, Ufuk Topcu
Reinforcement Learning Markov Decision Process Categorical Structure Compositional Model Compositional Operation Compositional Reinforcement Learning

July 25, 2022

Lifelong Machine Learning of Functionally Compositional Structures
Jorge A. Mendez
Compositional Structure Compositional Learning Compositional Model Compositional Reinforcement Learning Lifelong Machine Learning

July 8, 2022

CompoSuite: A Compositional Reinforcement Learning Benchmark
Jorge A. Mendez, Marcel Hussing, Meghna Gummadi, Eric Eaton
Unseen Task Compositional Structure Multi Task Reinforcement Learning Compositional Learning Compositional Reinforcement Learning Manipulation Benchmark

July 1, 2022

Modular Lifelong Reinforcement Learning via Neural Composition
Jorge A. Mendez, Harm van Seijen, Eric Eaton
Lifelong Reinforcement Learning Neural Module Compositional Reinforcement Learning Neural Composition Compositional Reasoning Task