Organizations: School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China. · DRAGON Lab, The University of Tokyo, Tokyo, Japan. · School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, China.
Reinforcement Learning (RL) and Imitation Learning (IL) are the standard frameworks for policy acquisition in manipulation. While IL offers efficient policy derivation, it suffers from compounding errors and distribution shift. Conversely, RL facilitates autonomous exploration but is frequently hindered by low sample efficiency and the high cost of trial and error. Since existing hybrid methods often struggle with complex tasks, we introduce Mixture of RL and IL Experts (MoRI). This system dynamically switches between IL and RL experts based on the variance of expert actions to handle coarse movements and fine-grained manipulations. MoRI employs an offline pre-training stage followed by online fine-tuning to accelerate convergence. To maintain exploration safety and minimize human intervention, the system applies IL-based regularization to the RL component. Evaluation across four complex real-world tasks shows that MoRI achieves an average success rate of 97.5% within 2 to 5 hours of fine-tuning. Compared to baseline RL algorithms, MoRI reduces human intervention by 85.8% and shortens convergence time by 21%, demonstrating its capability in robotic manipulation.
Figures & tables
Fig. 1: Overview of the MoRI framework. The architecture integrates IL and RL within a MoE system. During offline pre-training, both BC and RL experts are initialized using demonstration data. In the online fine-tuning stage, a gating network manages action selection while a data routing mechanism enables iterative policy refinement for all experts.
Fig. 2: Overview of real-world experimental tasks, including 1) place block in drawer, 2) put towel in lidded box, 3) insert two sockets, and 4) double-fold the towel.
Fig. 3: Learning curves during online training. Each panel compares the performance of ConRFT [ 12 ] and MoRI in terms of success rate, intervention rate, and episode length. Data points are calculated using a moving average over 20 episodes. Recorded success rates may reflect the combined effect of the policy and human assistance, potentially exceeding the standalone capability of the autonomous agent.
Fig. 4: The evolution of Q-values and variance for RL and BC experts within the MoRI framework.
Fig. 5: Evolution of the RL expert selection ratio over training steps in MoRI, averaged across five trajectories sampled from each corresponding checkpoint.
Fig. 6: Action fluctuations during expert switching within the MoRI framework.
Department of Mechanical Engineering, Jamia Millia Islamia, New Delhi, India · School of Architecture, Technology and Engineering, University of Brighton, Brighton, UK
HHCM, Istituto Italiano di Tecnologia, Genoa, Italy · DIBRIS, University of Genova, Genova, Italy · Human-Robot Interfaces and Interaction Lab, Istituto Italiano di Tecnologia, Genova, Italy +1