Multimodal Reinforcement Learning

Multimodal reinforcement learning (MRL) aims to train agents that effectively utilize diverse data sources (e.g., visual, auditory, tactile, textual) to make optimal decisions in complex environments. Current research focuses on improving data efficiency and robustness through techniques like self-supervised representation learning, multimodal alignment, and the development of novel policy architectures such as Gaussian mixture models to handle discontinuous optimal policies. These advancements are driving progress in various applications, including robotic control (locomotion, manipulation, surgery), human-robot interaction, and autonomous driving, by enabling agents to learn more effectively from richer, real-world data.

Papers

September 29, 2024

Obstacle-Aware Quadrupedal Locomotion With Resilient Multi-Modal Reinforcement Learning
I Made Aswin Nahrendra, Byeongho Yu, Minho Oh, Dongkyu Lee, Seunghyun Lee, Hyeonwoo Lee, Hyungtae Lim, Hyun Myung
Quadrupedal Robot Tiny Refinement Elicit Resilience Robust Locomotion Quadrupedal Locomotion Agile Locomotion Locomotion Strategy Multimodal Reinforcement Learning

May 1, 2024

Navigating WebAI: Training Agents to Complete Web Tasks with Large Language Models and Reinforcement Learning
Lucas-Andreï Thil, Mirela Popa, Gerasimos Spanakis
Language Model Reinforcement Learning Natural Language Processing Supervised Learning Training Agent Multimodal Reinforcement Learning Web Task

March 19, 2024

Policy Bifurcation in Safe Reinforcement Learning
Wenjun Zou, Yao Lyu, Jie Li, Yujie Yang, Shengbo Eben Li, Jingliang Duan, Xianyuan Zhan, Jingjing Liu, Yaqin Zhang, Keqiang Li
Safe Reinforcement Learning Safe Reinforcement Learning Algorithm Policy Switching Multimodal Reinforcement Learning

March 3, 2024

BronchoCopilot: Towards Autonomous Robotic Bronchoscopy via Multimodal Reinforcement Learning
Jianbo Zhao, Hao Chen, Qingyao Tian, Jian Chen, Bingyu Yang, Hongbin Liu
Surgical Robot Multimodal Reinforcement Learning Video Bronchoscopy

January 30, 2024

M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulation
Fotios Lygerakis, Vedant Dave, Elmar Rueckert
Contrastive Learning Robotic Manipulation Self Supervised Representation Learning Multimodal Self Supervised Learning Tactile Dataset Multimodal Reinforcement Learning

July 20, 2023

Reparameterized Policy Learning for Multimodal Trajectory Optimization
Zhiao Huang, Litian Liang, Zhan Ling, Xuanlin Li, Chuang Gan, Hao Su
Reinforcement Learning Policy Gradient Action Space Sparse Reward Environment Multimodal Reinforcement Learning Multimodal Optimization

March 13, 2023

Multimodal Reinforcement Learning for Robots Collaborating with Humans
Afagh Mehri Shervedani, Siyu Li, Natawut Monaikul, Bahareh Abbasi, Barbara Di Eugenio, Milos Zefran
Reinforcement Learning Non Humanoid Robot Real Human Robot Policy Multimodal System Multimodal Reinforcement Learning Robot Assistant

February 18, 2023

Effective Multimodal Reinforcement Learning with Modality Alignment and Importance Enhancement
Jinming Ma, Feng Wu, Yingfeng Chen, Xianpeng Ji, Yu Ding
Reinforcement Learning Multimodal Information Modality Alignment Multimodal Alignment Multimodal Reinforcement Learning

February 10, 2023

Combining Reconstruction and Contrastive Methods for Multimodal Representations in RL
Philipp Becker, Sebastian Mossburger, Fabian Otto, Gerhard Neumann
Contrastive Learning Multimodal Representation Contrastive Method Structured Reconstruction Multimodal Reinforcement Learning

October 14, 2022

AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments
Sudipta Paul, Amit K. Roy-Chowdhury, Anoop Cherian
3D Environment Audio Visual Navigation Multimodal Reinforcement Learning

July 21, 2022

Towards Robust On-Ramp Merging via Augmented Multimodal Reinforcement Learning
Gaurav Bagwe, Jian Li, Xiaoyong Yuan, Lan Zhang
Multimodal Reinforcement Learning Ramp Merging

Multimodal Reinforcement Learning

Papers

Obstacle-Aware Quadrupedal Locomotion With Resilient Multi-Modal Reinforcement Learning

Navigating WebAI: Training Agents to Complete Web Tasks with Large Language Models and Reinforcement Learning

Policy Bifurcation in Safe Reinforcement Learning

BronchoCopilot: Towards Autonomous Robotic Bronchoscopy via Multimodal Reinforcement Learning

M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulation

Reparameterized Policy Learning for Multimodal Trajectory Optimization

Multimodal Reinforcement Learning for Robots Collaborating with Humans

Effective Multimodal Reinforcement Learning with Modality Alignment and Importance Enhancement

Combining Reconstruction and Contrastive Methods for Multimodal Representations in RL

AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments

Towards Robust On-Ramp Merging via Augmented Multimodal Reinforcement Learning