Vision Based Reinforcement Learning

Vision-based reinforcement learning (VBRL) aims to enable agents to learn complex tasks directly from visual input, eliminating the need for manually engineered state representations. Current research emphasizes improving generalization across diverse environments, often through pre-training on large datasets and employing techniques like contrastive learning and knowledge distillation to enhance sample efficiency and robustness. This field is crucial for advancing autonomous systems, particularly in robotics and autonomous driving, by enabling agents to learn complex behaviors in unstructured, visually rich environments without extensive hand-engineering. Furthermore, ongoing work focuses on improving the interpretability of these "black box" models to increase trust and facilitate real-world deployment.

Papers

October 29, 2024

Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning
Jianlan Luo, Charles Xu, Jeffrey Wu, Sergey Levine
Reinforcement Learning Human in the Loop Robot Skill Manipulation Policy Dexterous Robotic Manipulation Vision Based Reinforcement Learning

October 25, 2024

AgentForge: A Flexible Low-Code Platform for Reinforcement Learning Agent Design
Francisco Erivaldo Fernandes Junior, Antti Oulasvirta
Reinforcement Learning Optimization Purpose Diverse Platform Better Optimizers Agent Capability Vision Based Reinforcement Learning

October 9, 2024

June 18, 2024

A Super-human Vision-based Reinforcement Learning Agent for Autonomous Racing in Gran Turismo
Miguel Vasco, Takuma Seno, Kenta Kawamoto, Kaushik Subramanian, Peter R. Wurman, Peter Stone
Autonomous Vehicle Autonomous Racing Deep Reinforcement Egocentric Camera Vision Based Reinforcement Learning

June 10, 2024

Investigating Pre-Training Objectives for Generalization in Vision-Based Reinforcement Learning
Donghu Kim, Hojoon Lee, Kyungmin Lee, Dongyoon Hwang, Jaegul Choo
Strong Generalization Vision Based Reinforcement Learning Pre Training Objective

March 18, 2024

Reinforcement Learning with Generalizable Gaussian Splatting
Jiaxu Wang, Qiang Zhang, Jingkai Sun, Jiahang Cao, Yecheng Shao, Renjing Xu
Reinforcement Learning Gaussian Splatting Scene Representation Environment Representation Robot Environment Vision Based Reinforcement Learning

September 4, 2023

Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learning
Qisen Yang, Huanqian Wang, Mukun Tong, Wenjie Shi, Gao Huang, Shiji Song
Reinforcement Learning Deep Reinforcement Learning Vision Based Reinforcement Learning Action Principle Reward Consistency

June 3, 2023

MA2CL:Masked Attentive Contrastive Learning for Multi-Agent Reinforcement Learning
Haolin Song, Mingxiao Feng, Wengang Zhou, Houqiang Li
Contrastive Learning Multi Agent Reinforcement Learning Self Supervised Auxiliary Task Vision Based Reinforcement Learning Multi Agent Prediction

March 13, 2023

Visual-Policy Learning through Multi-Camera View to Single-Camera View Knowledge Distillation for Robot Manipulation Tasks
Cihan Acar, Kuluhan Binici, Alp Tekirdağ, Yan Wu
Knowledge Distillation Manipulation Task Multiple Camera Vision Based Reinforcement Learning Visual Policy

February 28, 2023

Learning Sparse Control Tasks from Pixels by Latent Nearest-Neighbor-Guided Explorations
Ruihan Zhao, Ufuk Topcu, Sandeep Chinchali, Mariano Phielipp
Deep Reinforcement Learning Environment Exploration Sparse Reward Tetromino Pixel Model Free Reinforcement Learning Sparse Reward Environment Intermediate Latent Sparse Semantic Vision Based Reinforcement Learning

August 22, 2022

BARReL: Bottleneck Attention for Adversarial Robustness in Vision-Based Reinforcement Learning
Eugene Bykovets, Yannick Metz, Mennatallah El-Assady, Daniel A. Keim, Joachim M. Buhmann
Adversarial Attack Adversarial Robustness Adversarial Perturbation Vision Based Reinforcement Learning Minutia Cylinder Code

June 9, 2022

Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations
Cong Lu, Philip J. Ball, Tim G. J. Rudner, Jack Parker-Holder, Michael A. Osborne, Yee Whye Teh
Technical Challenge Offline Reinforcement Learning Action Space Emerging Opportunity Policy Learning Visual Exploration Vision Based Reinforcement Learning

March 25, 2022

Reinforcement Learning with Action-Free Pre-Training from Videos
Younggyo Seo, Kimin Lee, Stephen James, Pieter Abbeel
Reinforcement Learning Gameplay Video Action Feature Pre Trained Representation Action Prediction Vision Based Reinforcement Learning

December 6, 2021

Temporal-Spatial Causal Interpretations for Vision-Based Reinforcement Learning
Wenjie Shi, Gao Huang, Shiji Song, Cheng Wu
Vision Based Reinforcement Learning Temporal Causal Temporal Spatial Causal

November 11, 2021

Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation
I-Chun Arthur Liu, Shagun Uppal, Gaurav S. Sukhatme, Joseph J. Lim, Peter Englert, Youngwoon Lee
Reinforcement Learning Motion Planning Behavior Cloning Robot Manipulation Policy Distillation Vision Based Reinforcement Learning Visual Policy End to End Motion Planner

Vision Based Reinforcement Learning

Papers

Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning

AgentForge: A Flexible Low-Code Platform for Reinforcement Learning Agent Design

Zero-Shot Generalization of Vision-Based RL Without Data Augmentation

ING-VP: MLLMs cannot Play Easy Vision-based Games Yet

A Super-human Vision-based Reinforcement Learning Agent for Autonomous Racing in Gran Turismo

Investigating Pre-Training Objectives for Generalization in Vision-Based Reinforcement Learning

Reinforcement Learning with Generalizable Gaussian Splatting

Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learning

MA2CL:Masked Attentive Contrastive Learning for Multi-Agent Reinforcement Learning

Visual-Policy Learning through Multi-Camera View to Single-Camera View Knowledge Distillation for Robot Manipulation Tasks

Learning Sparse Control Tasks from Pixels by Latent Nearest-Neighbor-Guided Explorations

BARReL: Bottleneck Attention for Adversarial Robustness in Vision-Based Reinforcement Learning

Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations

Reinforcement Learning with Action-Free Pre-Training from Videos

Temporal-Spatial Causal Interpretations for Vision-Based Reinforcement Learning

Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation