How to Learn from What a Human Would Avoid? Intervention-Aware World Models with Real-World RL for Dexterous Manipulation

Jiaju Yin; Zhenhui Zhang; Lixin Xu; Heng Zhang; Jun Shao; Yating Feng; Arash Ajoudani; Renjing Xu

cs.ROSep 5, 2026

How to Learn from What a Human Would Avoid? Intervention-Aware World Models with Real-World RL for Dexterous Manipulation

Authors: Jiaju Yin, Zhenhui Zhang, Lixin Xu, Heng Zhang, Jun Shao, Yating Feng, Arash Ajoudani, Renjing Xu

Organizations: HKUST (Guangzhou) · Italian Institute of Technology · Zhejiang University

Abstract

Multi-fingered dexterous manipulation remains a frontier for real-world reinforcement learning (RL) due to the high-dimensional action space and the prohibitive cost of hardware failures. While human-in-the-loop (HIL) RL allows operators to intervene before failures occur, current pipelines often treat these interventions as reactive corrections, discarding the rich safety signal inherent in the operator's decision to take control. In this paper, we ask: How can we learn from what a human would avoid? We present WHIRL, a safety-aware RL framework that transforms binary human interventions into forward-predictive signals for proactive risk avoidance. Our approach centers on an intervention-aware latent world model with four prediction heads: dynamics, reward, termination, and a novel per-state intervention-probability head that learns to predict the likelihood of a human takeover at future states. This head provides an actor-side risk-shaping term that discourages the policy from entering "intervention-prone" regions, modeling the operator's internal safety threshold. We evaluate our framework on a 16-DoF LEAP Hand across tasks spanning convex and irregular object grasping, prismatic manipulation, and long-horizon multi-stage tasks. Our results show that predictive risk-shaping enables the system to achieve a 96.7 percent success rate on complex grasping tasks while reducing the operator intervention burden by up to 84 percent in step-weighted terms. By closing the loop between human intuition and predictive world modeling, this work provides a practical safety-aware recipe for training complex dexterous agents in the real world while reducing operator fatigue and hardware-risk exposure.

Figures & tables

Figure 1: WHIRL learns reusable risk from human takeovers. Each pedal intervention becomes a next-step takeover label for an intervention-aware world model, whose prediction head shapes the residual actor away from intervention-prone states across dexterous manipulation tasks.

Figure 2: WHIRL overview: takeovers become predictive risk. During online execution, a frozen behavior prior and residual policy compose arm–hand actions while HIL takeovers create labeled replay. The intervention-aware world model then predicts dynamics, reward, termination, and next-step intervention probability; the first three heads support critic learning, while the intervention head supplies actor-side risk shaping.

Figure 3: Intervention-aware world model. Dynamics, reward, and termination heads support uncertainty-gated critic updates, while the intervention head provides actor-side risk shaping.

Figure 4: Real-robot task suite. Five LEAP-Hand tasks spanning grasping, placement, prismatic manipulation, and the three-stage Long Horizon sequence.

Table 1: Performance. Each cell shows successes/rollouts over success rate.

Figure 5: Main result: rolling intervention fraction (lower is better). Step-weighted rolling-5 intervention fraction $\rho_{k}$ across five real-robot tasks. WHIRL stays below the no-WM baseline and reaches the near-zero regime earlier; success rates are in Table 4.5 .

Figure 6: Mechanism ablation. Removing actor-side intervention shaping or the WM delays convergence.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Hyperparameter	Value
World model	Latent dimension $d_{z}$	256
	Head MLP width / depth	$2\times 512$ hidden
	Dynamics ensemble size $M$	5
	Optimizer / learning rate	Adam / $1\!\times\!10^{-4}$
	Batch size	128
	Calibrated reward $(R_{\mathrm{succ}},R_{\mathrm{step}})$	$(10.0,-0.01)$

Appendix

Table 2: World-model and residual-RL hyperparameters used for reported WHIRL runs.

Figure 7: Raw per-episode intervention fraction $u_{i}/T_{i}$ (lower is better). Same five tasks and same WHIRL (blue, circles) vs. WHIRL w/o WM (red, squares; $=$ Residual RL model-free) contrast as Figure 4.5 . The step-weighted rolling-5 aggregation of this signal is the headline metric in Figure 4.5 .

Property	Pick Cube †	Pick LEGO	Pick & Place	Pull Drawer	Long Horizon †
Obj. rand.	$\pm 15$ cm	$\pm 15$ cm	$\pm 15$ cm	$\pm 5$ cm (handle)	$\pm 5$ cm
Reset EEF (m)	$(0.34,0.09,0.46)$	$(0.34,0.09,0.46)$	$(0.34,0.09,0.46)$	$(0.42,0.16,0.55)$	$(0.42,0.16,0.54)$
WS $x$ (m)	$[0.32,0.62]$	$[0.31,0.58]$	$[0.32,0.60]$	$[0.39,0.51]$	$[0.35,0.58]$
WS $y$ (m)	$[{-}0.08,0.16]$	$[{-}0.02,0.12]$	$[{-}0.03,0.31]$	$[{-}0.03,0.18]$	$[{-}0.12,0.26]$
WS $z$ (m)	$[0.19,0.48]$	$[0.19,0.48]$	$[0.19,0.48]$	$[0.38,0.57]$	$[0.21,0.64]$
Horizon	300 / 15 s	300 / 15 s	600 / 30 s	450 / 22.5 s	2100 / 105 s

Appendix

Table 3: Task randomization, reset pose, workspace bounds, and episode horizon.

Figure 8: Tri-channel HIL teleoperation rig: SpaceMouse (arm), Manus glove (hand), and foot pedal (takeover).

Robotic RL

Robotic reinforcement learning uses trial-and-error learning to help robots control, navigate, and manipulate objects.

See all 300 papers

Jun 10, 2026

UniIntervene: Agentic Intervention for Efficient Real-World Reinforcement Learning

Human-in-the-loop reinforcement learning (HiL-RL) has emerged as an effective paradigm for real-world robotic manipulation, enabling online policy improvement with human guidance. However, current HiL-RL frameworks remain intervention-intensive, relying on frequent human corrections to redirect the policy out of unproductive exploration, which incurs high labor cost and limits real-world scalability. To address this, we propose UniIntervene, an agentic intervention model that detects unproductive exploration and autonomously recovers the policy toward high-value states, taking over the bulk of interventions from human operators. Specifically, UniIntervene first performs future-conditioned action-value estimation, predicting the latent consequence of the current action and evaluating its induced value, which provides a more stable progress signal. Building on this, a temporal value-risk critic aggregates recent value dynamics and triggers intervention when the estimated value exhibits sustained stagnation or degradation. When intervention is required, UniIntervene retrieves a high-value recovery target from a memory of past intervention episodes and produces executable corrective actions through a goal-conditioned recovery policy. In this way, UniIntervene turns intervention from passive human correction into a value-aware recovery process for efficient real-world RL. Extensive experiments on diverse real-world manipulation tasks demonstrate that UniIntervene improves the average success rate by 8.6% while reducing human interventions by 57% relative to state-of-the-art HiL-RL baselines.

Haoyuan Deng, Yitong Gao, Yudong Lin +3

Nanyang Technological University · Beijing University of Posts and Telecommunications

Agentic RL Reinforcement Learning Robotic RL Human-in-the-Loop AI

Ask PDF

Sep 14, 2026

Real-World Reinforcement Learning with MPC Scaffolding for Dexterous Manipulation

Real-world reinforcement learning (RL) offers a promising route to dexterous manipulation policies that can adapt directly from physical interaction, but learning is hindered by inefficient early exploration and costly failures. We propose a framework that uses sampling-based model predictive control (MPC) as scaffolding for real-world dexterous RL, providing structured prior experience and task-directed guidance during learning without human demonstrations or corrective actions. A small set of MPC trajectories is first used to populate an offline replay buffer and to pretrain the actor and critic. During online learning, MPC intermittently guides data collection while an off-policy Soft Actor-Critic learner trains from both prior MPC experience and newly collected physical interaction, with control gradually transitioning to the learned policy. On continuous in-hand rotation with a 16-DoF Allegro hand, initialized from 20 MPC trajectories collected in 12 minutes on hardware, the policy reaches 100% success after 7 minutes of online RL, with about three object drops on average during training. After 20 minutes of online learning, the policy achieves more than five times the rotation speed of the MPC controller and completes 1000 consecutive rotations without a drop. Ablations show complementary benefits from MPC-based pretraining, retained MPC experience, and online MPC guidance, while additional experiments demonstrate rapid adaptation to new object geometries and successful goal-conditioned reorientation.

Emek Barış Küçüktabak, Karankumar Patel, Zhaodong Yang +3

Honda Research Institute USA, San Jose, CA, USA · Skild AI, San Mateo, CA, USA · Georgia Institute of Technology, Atlanta, GA, USA

Dexterous Manipulation Contact-Rich Robotic Manipulation Robotic RL Offline-To-Online RL

Ask PDF

Aug 16, 2026

GAINS: Leveraging Inconsistent Human Intervention Signals in Reinforcement Learning

Correcting robot manipulation policies through human intervention holds great promise for real-world deployment, yet human operators are inherently imperfect in both the actions they provide and the timing of their intervention signals. While the former has been extensively discussed in reinforcement learning (RL), the latter remains underexplored. At high control frequencies, human intervention signals are often delayed and inconsistent across time and state space. In this work, we present GAINS, a framework for leveraging inconsistent human intervention signals in RL. At the core of GAINS, we employ distributional RL with quantile Q-networks to model the return variability induced by sparse task rewards and inconsistent human interventions. Building on this distributional representation, we introduce a pessimistic exploration strategy that promotes safe and sample-efficient learning under human corrections. We evaluate GAINS on four diverse simulated manipulation tasks and two challenging real-world scenarios against state-of-the-art intervention-based methods. GAINS achieves a 22% higher task success rate than RLIF and improves recovery success by up to 43% in failure scenarios. These results highlight the importance of modeling return variability induced by human imperfection for real-world deployment of intervention-based learning.

Xinyi Zhang, Yinuo Zhao, Pei Ren +7

Beijing Institute of Technology · Beijing Innovation Center of Humanoid Robotics · City University of Hong Kong +1

Robotic RL RL from Human Feedback Robotic Manipulation

Ask PDF

Abstract

Robotic RL

Explore similar work

UniIntervene: Agentic Intervention for Efficient Real-World Reinforcement Learning

Real-World Reinforcement Learning with MPC Scaffolding for Dexterous Manipulation

GAINS: Leveraging Inconsistent Human Intervention Signals in Reinforcement Learning