Robot reinforcement learning from demonstrations (RLfD) assumes that expert data is abundant; this is usually unrealistic in the real world given data scarcity as well as high collection cost. Furthermore, imitation learning algorithms assume that the data is independently and identically distributed, which ultimately results in poorer performance as gradual errors emerge and compound within test-time trajectories. We address these issues by introducing the "master your own expertise" (MYOE) framework, a self-imitation framework that enables robotic agents to learn complex behaviors from limited demonstration data samples. Inspired by human perception and action, we propose and design what we call the queryable mixture-of-preferences state space model (QMoP-SSM), which estimates the desired goal at every time step. These desired goals are used in computing the "preference regret", which is used to optimize the robot control policy. Our experiments demonstrate the robustness, adaptability, and out-of-sample performance of our agent compared to other state-of-the-art RLfD schemes. The GitHub repository that supports this work can be found at: https://github.com/rxng8/neurorobot-preference-regret-learning.
Figures & tables
Figure 1: Our Proposed Agent Framework. The agent learns internal representations via encoders, the QMoP-SSM, and a decoder. Imagined future states and preferences guide both policy learning and value estimation.
Figure 2: QMoP-SSM Learning Architecture. QMoP separates latent learning into two interconnected parallel processes: “representation” and “preference” learning. While the model predicts future latent given previous actions, future preference state trajectories are guided by provided and learned goals.
Figure 3: Cumulative reward ( y -axis) across 1 million interaction/training steps ( x -axis) for different agents.
Task/Agent
MYOE
Drm-BC
PPO-BC
LAIfO
K Kettle
0.97±0.06
0.90±0.11
0.01±0.02
0.27±0.42
K Light Switch
1.00±0.01
0.51±0.49
0.00±0.00
0.00±0.00
K Microwave
0.75±0.43
0.67±0.34
0.00±0.00
0.00±0.00
K Slide Cabinet
0.99±0.03
0.74±0.43
0.28±0.42
0.57±0.44
R Lift
1.00±0.01
0.62±0.26
0.00±0.00
0.00±0.01
R Door
0.99±0.02
0.98±0.03
0.00±0.00
0.00±0.00
Table 1: We report the evaluation success rate of the last 100 episodes of MYOE (ours) as compared to other model/RL baselines. Cyan cells represent the best performing agents within the corresponding task. Note that “Drm” is short for “DreamerV3.”
Task/Agent
MYOE
MBC
MBC-VAE
MBC-RNN
K Kettle
0.97±0.06
0.65±0.46
0.66±0.47
0.67±0.47
K Light Switch
1.00±0.01
0.23±0.22
0.12±0.20
0.24±0.17
K Microwave
0.75±0.43
0.50±0.50
0.49±0.49
0.50±0.50
K Slide Cabinet
0.99±0.03
0.08±0.12
0.16±0.17
0.08±0.15
R Lift
1.00±0.01
0.38±0.29
0.70±0.23
0.46±0.25
R Door
0.99±0.02
0.96±0.06
0.81±0.26
0.97±0.04
Table 2: We report the evaluation success rate of the last 100 episodes of MYOE (ours) in comparison to other baselines. Cyan cells represent the best performance agents within the corresponding task.
Figure 4: Our proposed MYOE solving the “reach” task when integrated into the “7bot” robot (top) and our agent model solving the “block picking” task when integrated into the PX100 robot (bottom).
MYOE
Drm-BC
PPO-BC
MBC-RNN
0.34 ± 0.11
0.17 ± 0.01
0.29 ± 0.11
0.00 ± 0.00
Table 3: “Reach” task success rate on the “7bot”.
MYOE
Drm-BC
PPO-BC
LAIfO
0.89 ± 0.05
0.81 ± 0.05
0.78 ± 0.06
0.79 ± 0.06
80.78 ± 4.48
72.34 ± 4.93
66.78 ± 6.30
67.96 ± 6.19
Table 4: “Block picking” task success rate (top) and episodic cumulative reward (bottom) on the PX100 robot.
1Purdue University, West Lafayette, IN, USA. · 2Beijing University of Chemical Technology, Beijing, China. · 3Indiana University Bloomington, Bloomington, IN, USA.