Imitation policies that reliably execute most of a manipulation trajectory can still fail in localized regions requiring last-mile precision. Updating the full policy or exploring residual actions everywhere treats this local failure as a global learning problem. We introduce Adaptive Residual Correction (ARC), a lightweight post-training module that uses reinforcement learning (RL) to learn a residual correction on top of a frozen policy while using sparse human corrections to structure autonomous exploration. Rather than treating human corrections as imitation targets, ARC uses pre-intervention states to determine 'when' residual control should become active and correction geometry to determine 'where' in residual action space autonomous refinement should explore; task reward determines 'what' residual is ultimately executed. Across three precision-manipulation tasks on a real robot, ARC improves average task success from 33.3% to 75.0%. The human corrections define task-dependent 2-3 dimensional subspaces of the seven-joint residual space that retain 96.0-97.7% of squared correction magnitude. Together, these results show that sparse human corrections can localize and constrain autonomous post-training without requiring the observed correction itself to serve as the final action target.
Figures & tables
Figure 1: ARC overview. Sparse human corrections train a pre-intervention gate and define a low-dimensional corrective subspace, which together constrain autonomous exploration in time and action space. Task reward trains the residual actor, whose gated correction is added to the frozen ACT action at deployment.
Figure 2: Performance on a real robot. Base ACT, direct supervised seven-joint residual correction, and ARC over 12 trials per method and task. Labels show exact successes; error bars are 95% Wilson confidence intervals.
Figure 3: WHEN and WHERE. (a) On 10 untouched autonomous base-policy rollouts, a gate trained on correction-active states never activates, while the pre-intervention gate activates on all 10 (gate activation, not task success). (b) Held-out gate discrimination and the fraction of autonomous-exploration timesteps for which the gate is open. (c) The selected 3 -, 2 -, and 2 -D subspaces retain 97.3% , 96.0% , and 97.7% of squared correction magnitude for tray, marker, and cup, respectively. (d) Marker corrections, exploration probes, and deployed ARC residuals in the learned corrective basis; PC denotes principal component.
Figure 4: WHAT: same support, different corrections. (a) Within BH , supervised prediction reduces held-out human-residual error only from 1.98∘ to 1.90∘ ( 4.0% ), while oracle projection gives a 1.23∘ lower bound. (b) On the same 2,499 gate-open states, median max-joint residual magnitude is 0.52∘ for supervision and 0.14∘ for ARC ( 3.7× ). (c) With the same BH and residual bounds, PC 1 sign agreement is 38% on non-trivial states, while 22% of supervised versus 0% of ARC predictions reach at least half the allowed bound.
Robotic manipulation policies trained via imitation learning, such as Action Chunking with Transformers (ACT), can achieve strong performance under ideal conditions but often remain sensitive to small execution errors and distribution shifts. Correcting these failures typically requires dataset aggregation and full-policy retraining, which is computationally expensive and unsuitable for real-time deployment. In this work, we propose Online Residual Policy Adaptation (ORPA), a framework that enables immediate, feedback-driven correction of robot actions without modifying the underlying policy parameters. ORPA augments a pretrained control policy with a lightweight, feedback-conditioned module that predicts residual adjustments directly in joint space, allowing the system to adapt its behavior at runtime. We evaluate ORPA on a set of precision-sensitive manipulation tasks using the ALOHA platform, demonstrating improvements in success rate and recovery from small perturbations compared to baseline control policies and rule-based inverse kinematics corrections.
Muhammad A. Muttaqien, Tomohiro Motoda, Ryo Hanai +1
Embodied AI Research Team · National Institute of AIST · Tokyo, Japan
Imitation learning enables robots to acquire manipulation skills from demonstrations, but the resulting policies can fail outside the training data, while collecting more demonstrations requires substantial human effort. Human-in-the-loop reinforcement learning uses corrective feedback during online training, but typically learns the complete task policy rather than refining a pretrained imitation policy. We introduce Res-HIL, a human-in-the-loop residual reinforcement learning framework that learns corrective actions on top of a frozen imitation policy. Each human intervention provides two complementary learning signals: direct supervision of the residual policy and reward shaping of preceding autonomous behavior. Res-HIL combines these signals with zero initialization of the residual policy to stabilize and accelerate online learning. We evaluate Res-HIL on five contact-rich manipulation tasks spanning high-precision and long-horizon behaviors. With only 20 initial demonstrations, Res-HIL outperforms state-of-the-art full-policy human-in-the-loop reinforcement learning and residual fine-tuning without human guidance on every task after ten minutes of online training. Res-HIL improves its pretrained base policies and outperforms imitation policies trained with five times more demonstrations. An ablation study shows that direct residual supervision is critical to performance, while intervention-aware reward shaping substantially improves training efficiency.
Mariia Iavorskaia, Christian Dietz, Sebastian Albrecht +1
Siemens AG, Research and Predevelopment, Germany. · Technical University of Munich, Germany.
Residual reinforcement learning adapts a pretrained robot policy by learning an additive correction to its actions. While effective when adaptation amounts to shifting the base policy's action distribution, additive corrections cannot change the distribution's shape, scale, or state-dependent geometry -- limitations we formalize as wrong variance, miscalibrated confidence, and non-uniform correction. We show that these matter under dynamics shift: when the base distribution is geometrically mismatched to the shifted system, residual correction can underperform even the unadapted policy. We propose Warp RL, a policy adaptation method that replaces additive residuals with an invertible, state-conditioned transformation of the base policy's action distribution. Instantiated with monotonic rational-quadratic spline flows (arXiv:1906.04032), Warp RL preserves identity initialization, strictly generalizes additive residual correction, and exposes a structured adaptation space suitable for both policy-gradient and gradient-free optimization. Across a variety of ManiSkill3 manipulation tasks with controlled dynamics shifts, Warp RL matches residual correction when translation is sufficient and substantially outperforms it when adaptation requires distributional reshaping. We further demonstrate that warping can replace additive correction in an off-policy sim-to-real pipeline, achieving comparable success rate with 30% faster task completion on a real-robot peg-insertion task.