Residual Reinforcement Learning

Residual reinforcement learning (RRL) enhances existing control policies by learning only the necessary adjustments or "residuals" needed to improve performance, rather than training a completely new policy from scratch. Current research focuses on applying RRL to diverse robotics tasks, including locomotion, manipulation, and autonomous driving, often integrating it with model-based methods, policy gradient algorithms, and mixture-of-experts frameworks. This approach offers significant advantages in sample efficiency and training speed, leading to faster development and deployment of robust controllers for complex real-world applications.

Papers

November 7, 2024

DexH2R: Task-oriented Dexterous Manipulation from Human to Robots
Shuqi Zhao, Xinghao Zhu, Yuxin Chen, Chenran Li, Xiang Zhang, Mingyu Ding, Masayoshi Tomizuka
Non Humanoid Robot Human Generated Robot Teleoperation Dexterous Manipulation Hand Motion Residual Reinforcement Learning Dexterous Robotic Hand

October 14, 2024

DR-MPC: Deep Residual Model Predictive Control for Real-world Social Navigation
James R. Han, Hugues Thomas, Jian Zhang, Nicholas Rhinehart, Timothy D. Barfoot
Reinforcement Learning Deep Reinforcement Learning Model Predictive Control Local Navigation Residual Reinforcement Learning

October 4, 2024

Residual Policy Learning for Perceptive Quadruped Control Using Differentiable Simulation
Jing Yuan Luo, Yunlong Song, Victor Klemm, Fan Shi, Davide Scaramuzza, Marco Hutter
Policy Gradient Model Free Reinforcement Learning Differentiable Simulation Residual Reinforcement Learning Perceptive Locomotion First Order Policy

October 3, 2024

Efficient Residual Learning with Mixture-of-Experts for Universal Dexterous Grasping
Ziye Huang, Haoqi Yuan, Yuhui Fu, Zongqing Lu
Mixture of Expert Robot Learning Residual Reinforcement Learning Dexterous Grasping Functional Grasp

September 20, 2024

ProxFly: Robust Control for Close Proximity Quadcopter Flight via Residual Reinforcement Learning
Ruiqi Zhang, Dingqi Zhang, Mark W. Mueller
Robust Control Residual Reinforcement Learning Proximity Operation Quadrotor Swarm

September 18, 2024

Residual Descent Differential Dynamic Game (RD3G) -- A Fast Newton Solver for Constrained General Sum Games
Zhiyuan Zhang, Panagiotis Tsiotras
Residual Reinforcement Learning General Sum Game State Constraint Local Nash Equilibrium

August 30, 2024

Traffic expertise meets residual RL: Knowledge-informed model-based residual reinforcement learning for CAV trajectory control
Zihao Sheng, Zilin Huang, Sikai Chen
Model Based Reinforcement Learning Residual Reinforcement Learning Intelligent Driver Model CAV Network

July 23, 2024

From Imitation to Refinement -- Residual RL for Precise Assembly
Lars Ankile, Anthony Simeonov, Idan Shenfeld, Marcel Torne, Pulkit Agrawal
Reinforcement Learning Fine Tuning Feature Imitation Behavior Cloning Assembly Task Residual Reinforcement Learning Residual Policy

July 9, 2024

HiLMa-Res: A General Hierarchical Framework via Residual RL for Combining Quadrupedal Locomotion and Manipulation
Xiaoyu Huang, Qiayuan Liao, Yiming Ni, Zhongyu Li, Laura Smith, Sergey Levine, Xue Bin Peng, Koushil Sreenath
Strategic Manipulation Quadrupedal Robot Quadrupedal Locomotion Loco Manipulation Hierarchical Framework Residual Reinforcement Learning

June 24, 2024

MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Intervention
Yuxin Chen, Chen Tang, Chenran Li, Ran Tian, Wei Zhan, Peter Stone, Masayoshi Tomizuka
Inverse Reinforcement Learning Residual Reinforcement Learning Early Intervention Sample Efficient Policy

March 11, 2024

RaceMOP: Mapless Online Path Planning for Multi-Agent Autonomous Racing using Residual Policy Learning
Raphael Trumpp, Ehsan Javanmardi, Jin Nakazato, Manabu Tsukada, Marco Caccamo
Multi Agent Residual Reinforcement Learning Robust Obstacle Avoidance AWS Deepracer

February 18, 2024

Online Physical Enhanced Residual Learning for Connected Autonomous Vehicles Platoon Centralized Control
Hang Zhou, Heye Huang, Peng Zhang, Haotian Shi, Keke Long, Xiaopeng Li
Platoon Control Connected Autonomous Vehicle Residual Reinforcement Learning Residual Physic

January 26, 2024

RESPRECT: Speeding-up Multi-fingered Grasping with Residual Reinforcement Learning
Federico Ceola, Lorenzo Rosasco, Lorenzo Natale
Deep Reinforcement Learning Residual Reinforcement Learning Residual Policy Grasping Policy Multi Fingered

October 15, 2023

Specialized Deep Residual Policy Safe Reinforcement Learning-Based Controller for Complex and Continuous State-Action Spaces
Ammar N. Abbas, Georgios C. Chasparis, John D. Kelleher
Deep Reinforcement Learning Safe Reinforcement Learning Complexity Level State Action Space Residual Reinforcement Learning M$ Base Controller

August 3, 2023

Improving Wind Resistance Performance of Cascaded PID Controlled Quadcopters using Residual Reinforcement Learning
Yu Ishihara, Yuichi Hazama, Kousuke Suzuki, Jerry Jun Yokono, Kohtaro Sabe, Kenta Kawamoto
Residual Reinforcement Learning

June 15, 2023

Residual Q-Learning: Offline and Online Policy Customization without Value
Chenran Li, Chen Tang, Haruki Nishimura, Jean Mercat, Masayoshi Tomizuka, Wei Zhan
Imitation Learning Net Present Value Action Free Offline Residual Reinforcement Learning Online Policy Customization

April 17, 2023

Continuous Versatile Jumping Using Learned Action Residuals
Yuxiang Yang, Xiangyun Meng, Wenhao Yu, Tingnan Zhang, Jie Tan, Byron Boots
Reinforcement Learning Legged Robot Quadrupedal Robot Residual Reinforcement Learning Continuous Jumping Omnidirectional Jumping

March 24, 2023

Autonomous Blimp Control via H-infinity Robust Deep Residual Reinforcement Learning
Yang Zuo, Yu Tang Liu, Aamir Ahmad
Reinforcement Learning Residual Reinforcement Learning Robotic Blimp

February 14, 2023

Residual Policy Learning for Vehicle Control of Autonomous Racing Cars
Raphael Trumpp, Denis Hoornaert, Marco Caccamo
Autonomous Racing Residual Reinforcement Learning Vehicle Control Autonomous Race Car

December 17, 2022

Cascaded Compositional Residual Learning for Complex Interactive Behaviors
K. Niranjan Kumar, Irfan Essa, Sehoon Ha
Residual Reinforcement Learning Interactive Behavior Joint Control

Residual Reinforcement Learning

Papers

DexH2R: Task-oriented Dexterous Manipulation from Human to Robots

DR-MPC: Deep Residual Model Predictive Control for Real-world Social Navigation

Residual Policy Learning for Perceptive Quadruped Control Using Differentiable Simulation

Efficient Residual Learning with Mixture-of-Experts for Universal Dexterous Grasping

ProxFly: Robust Control for Close Proximity Quadcopter Flight via Residual Reinforcement Learning

Residual Descent Differential Dynamic Game (RD3G) -- A Fast Newton Solver for Constrained General Sum Games

Traffic expertise meets residual RL: Knowledge-informed model-based residual reinforcement learning for CAV trajectory control

From Imitation to Refinement -- Residual RL for Precise Assembly

HiLMa-Res: A General Hierarchical Framework via Residual RL for Combining Quadrupedal Locomotion and Manipulation

MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Intervention

RaceMOP: Mapless Online Path Planning for Multi-Agent Autonomous Racing using Residual Policy Learning

Online Physical Enhanced Residual Learning for Connected Autonomous Vehicles Platoon Centralized Control

RESPRECT: Speeding-up Multi-fingered Grasping with Residual Reinforcement Learning

Specialized Deep Residual Policy Safe Reinforcement Learning-Based Controller for Complex and Continuous State-Action Spaces

Improving Wind Resistance Performance of Cascaded PID Controlled Quadcopters using Residual Reinforcement Learning

Residual Q-Learning: Offline and Online Policy Customization without Value

Continuous Versatile Jumping Using Learned Action Residuals

Autonomous Blimp Control via H-infinity Robust Deep Residual Reinforcement Learning

Residual Policy Learning for Vehicle Control of Autonomous Racing Cars

Cascaded Compositional Residual Learning for Complex Interactive Behaviors