cs.ROAug 16, 2026

GAINS: Leveraging Inconsistent Human Intervention Signals in Reinforcement Learning

Authors: Xinyi Zhang, Yinuo Zhao, Pei Ren, Lechun Jiang, Huiqian Jin, Lei Sun, Dapeng Wu, Zhengping Che, +2 more

Organizations: Beijing Institute of Technology · Beijing Innovation Center of Humanoid Robotics · City University of Hong Kong · Nankai University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Correcting robot manipulation policies through human intervention holds great promise for real-world deployment, yet human operators are inherently imperfect in both the actions they provide and the timing of their intervention signals. While the former has been extensively discussed in reinforcement learning (RL), the latter remains underexplored. At high control frequencies, human intervention signals are often delayed and inconsistent across time and state space. In this work, we present GAINS, a framework for leveraging inconsistent human intervention signals in RL. At the core of GAINS, we employ distributional RL with quantile Q-networks to model the return variability induced by sparse task rewards and inconsistent human interventions. Building on this distributional representation, we introduce a pessimistic exploration strategy that promotes safe and sample-efficient learning under human corrections. We evaluate GAINS on four diverse simulated manipulation tasks and two challenging real-world scenarios against state-of-the-art intervention-based methods. GAINS achieves a 22% higher task success rate than RLIF and improves recovery success by up to 43% in failure scenarios. These results highlight the importance of modeling return variability induced by human imperfection for real-world deployment of intervention-based learning.

Figures & tables

Figure 1: Human Intervention Timing Is Inconsistent. We replay failure videos multiple times to measure intervention-timing consistency; the top-right panel highlights intervention frames from different rounds in yellow.
Figure 2: An Overview of GAINS.
Figure 3: Four Simulation Tasks in HIL-RM. The top-left inset shows the wrist-camera view used as input for all tasks except Straighten-Rope, which uses a third-person camera.
Figure 4: Real-world Experimental Setups. Left: UR5 with 2 cameras for the Pick-Spoon task. Right: Franka Emika with 2 cameras for the Insert-USB task.
Figure 5: Comparison Results in HIL-RM. Results are averaged over 3 seeds, with shaded regions indicating the standard deviation. Curves are smoothed with sliding window.
Figure 6: Comparison Results in Real-World Tasks. Results are averaged over 3 seeds.
Figure 7: Recovery Trajectory of GAINS under External Disturbance. More results can be found in Fig. 11
Figure 8: Robustness Results in HIL-RM. For each method, we evaluate the final checkpoints under 100 trials.
Figure 9: Ablation Studies on Arrange-Boxes. Each bar reports the mean performance over 3 random seeds. Total safety violations denote the cumulative number of safety violations throughout training.
Figure 10: Distributional Q-value Estimates at Different Stages of Training.
Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

AspectSiriusHIL-RM
Intervention Action SourceHumanPretrained RL Expert
Intervention Signal SourceHumanHuman
Hardware Device3D MouseKeyboard
Task Numbers24
Long-horizon Task✓✓
Deformable Manipulation×\times✓
Appendix
Table 1: Comparison between Sirius and HIL-RM.
ParameterValue
Episode Length50
NintervN_{\text{interv}}5
NnoiseN_{\text{noise}}5
State Spaceq∈R7\mathbf{q}\in\mathbb{R}^{7} , q˙∈R7\dot{\mathbf{q}}\in\mathbb{R}^{7} , ptcp∈R3\mathbf{p}_{tcp}\in\mathbb{R}^{3} , g∈Rg\in\mathbb{R}
Action Spacep∈R3\mathbf{p}\in\mathbb{R}^{3} , r∈R3\mathbf{r}\in\mathbb{R}^{3} , g∈Rg\in\mathbb{R}
Appendix
Table 2: Pick-Cube q\mathbf{q} and q˙\dot{\mathbf{q}} denote joint positions and velocities, ptcp\mathbf{p}_{tcp} denotes the TCP position, p\mathbf{p} and r\mathbf{r} denote Cartesian position and orientation commands, and gg denote the gripper state/command.
ParameterValue
Episode Length50
NintervN_{\text{interv}}5
NnoiseN_{\text{noise}}5
State Spaceptcp∈R3\mathbf{p}_{tcp}\in\mathbb{R}^{3} , qtcp∈R4\mathbf{q}_{tcp}\in\mathbb{R}^{4} , g∈Rg\in\mathbb{R} , ddoor∈R3d_{door}\in\mathbb{R}^{3}
Action Spacep∈R3\mathbf{p}\in\mathbb{R}^{3} , r∈R3\mathbf{r}\in\mathbb{R}^{3} , g∈Rg\in\mathbb{R}
Appendix
Table 3: Open-Cabinet. qtcp\mathbf{q}_{tcp} denote the orientation, and ddoor∈R3d_{door}\in\mathbb{R}^{3} denotes the cabinet door opening state replicated across three dimensions.
ParameterValue
Episode Length200
NintervN_{\text{interv}}15
NnoiseN_{\text{noise}}10
State Spaceptcp∈R6\mathbf{p}_{tcp}\in\mathbb{R}^{6} , pbox∈R18\mathbf{p}_{box}\in\mathbb{R}^{18}
Action Spacep∈R3\mathbf{p}\in\mathbb{R}^{3}
Appendix
Table 4: Arrange-Boxes. ptcp∈R6\mathbf{p}_{tcp}\in\mathbb{R}^{6} denotes the TCP position obtained by duplicating the original 3D TCP coordinates. pbox∈R18\mathbf{p}_{box}\in\mathbb{R}^{18} denotes the positions of three boxes, where the 3D position of each box is duplicated to form a 6D representation.
ParameterValue
Episode Length70
NintervN_{\text{interv}}10
NnoiseN_{\text{noise}}2
State Spaceptcp∈R3\mathbf{p}_{tcp}\in\mathbb{R}^{3} , qtcp∈R4\mathbf{q}_{tcp}\in\mathbb{R}^{4}
Action Spacep∈R3\mathbf{p}\in\mathbb{R}^{3}
Appendix
Table 5: Straighten-Rope. For the Rope task, the vertical action zz is discretized to {0,1}\{0,1\} and selected by a discrete critic, while xx and yy remain continuous.
Figure 11: Robust Environment of HIL-RM.
Figure 12: Homogeneous bilateral manipulator setup in Franka Emika.
Figure 13: System Structure in GAINS.
Figure 14: Comparison Results on Episode Length and Intervention Ratio in HIL-RM. Results are averaged over three seeds, with shaded regions indicating the standard deviation. Curves are smoothed with sliding window.
Figure 15: RLIF performance under our implementation (blue) and the original configuration (red) on Open-Cabinet .
ParameterValue
Truncated Q Drop Dim bb2
Critic Number MM2
Quantile Bins KK25
Huber Threshold κ\kappa10
Pessimistic Exploration Threshold τth\tau_{\mathrm{th}} (Pick-Cube, Straighten-Rope)0.5
Pessimistic Exploration Threshold τth\tau_{\mathrm{th}} (Arrange-Boxes, Open-Cabinet)0.7
Appendix
Table 6: Key Hyperparameters of GAINS.
Figure 16: Investigation of Quantile Threshold τth\tau_{\textrm{th}} . Results are averaged over 3 seeds.

Explore similar work

CardsList

  1. UniIntervene++: An Adaptive Intervention Agent for Efficient Real-World Reinforcement Learning

    Oct 2, 2026Yudong Lin, Haoyuan Deng, Zhuoxuan Yuan +3Reinforcement LearningRobotic RL

    School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore

  2. UniIntervene: Agentic Intervention for Efficient Real-World Reinforcement Learning

    Jun 10, 2026Haoyuan Deng, Yitong Gao, Yudong Lin +3Agentic RLReinforcement Learning

    Nanyang Technological University · Beijing University of Posts and Telecommunications

  3. OHP-RL: Online Human Preference as Guidance in Reinforcement Learning for Robot Manipulation

    May 15, 2026Yunyang Mo, Jian Li, Qiwei Wu +2Robotic RLHuman-in-the-Loop AI

    The Hong Kong University of Science and Technology (Guangzhou)