Residual reinforcement learning corrects an imitation-trained base policy using a limited budget of environment interaction. In a task with several phases, this budget must be divided among the phases, and the phase in which the base policy fails is not necessarily the phase in which a correction most improves task completion. We formalize this distinction through the completion gain: the change in full-task completion probability when, during a single phase, a state-feedback controller replaces selected components of the base policy's action. We decompose the completion gain into the probability of reaching the phase and the conditional improvement in completion from the start of the phase, and we estimate it from paired episodes. The positive estimates determine which action components the residual policy corrects and how training episodes are distributed across phases. In stochastic point-mass docking experiments, most failures occurred in the final docking phase, whereas the largest completion gain occurred in the preceding channel phase; gain-based allocation improved completion over allocation proportional to failure counts and, in a variant with a longer approach, over uniform allocation at half of the budget. In simulated bimanual robotic assembly, the completion-gain estimates were positive only for insertion and release, although few failures occurred during release; in a second assembly configuration, a residual policy trained in the release phase improved completion from held-out states at the start of release by 16.9%.
Figures & tables
Figure 1: Running Examples. (a) Point-Mass Docking: Here, a point mass must pass through a channel and cross the dock line within speed and lateral position limits. The line plots in (a) depict traces from trial runs under two different control settings: a base policy (red) and a control scheme where control of the point-mass’s acceleration is switched to a state-feedback controller in the channel phase (blue). In each setting, we use the same initial state, physical parameters, and noise draws. Under the base policy, the point mass exceeds the speed limit at the dock line. With state-feedback acceleration commands in the channel (blue), it satisfies both terminal limits. Open circles mark phase starts. The dotted curve gives the speed bound for one-dimensional braking without drag or noise. (b,c) Bimanual Robotic Assembly: In this task, we require a bimanual robotic manipulator to insert a key-like part with its right hand into a housing stabilized by the left hand. The part must remain stable in the housing after the robot withdraws its hands. The frames in the figure show an episode under the state-feedback controller in Isaac Lab ( Mittal et al., 2023 ) .
Figure 2: Paired Episodes for u=(j,G) .
Figure 3: Completion Gains, Allocation, and Residual Training in Point-Mass Docking. (a) Estimates from 128 paired episodes per candidate with 95% bootstrap intervals. (b) Failure-weighted (red) and gain-based (blue) allocation. (c) Means over ten training runs; markers are individual runs, and the budget includes gain estimation.
Figure 4
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Point-Mass Trajectories and Speed at the Dock Start. (a,b) The first 20 of 128 paired episodes: (a) the base policy, with completed episodes in blue and failed episodes in red; (b) the same episode conditions with the state-feedback controller substituted on both action components during the channel phase. Shading marks ∣py∣≤0.3 within the channel, and the green bar marks the dock line. (c) Speed at the dock start for the 117 base-policy episodes that reached docking, grouped by whether the state-feedback controller, applied from the dock start, completed the task (68) or failed (49). The dashed line marks the speed limit of 1.0 at the dock line; the bound on the speed at the dock start for the dynamics without drag is 1.14 ( Sec. 4.1 ).
Decision interval Δtk / horizon T
0.1 s / 16 s (160 decisions)
Acceleration bounds: longitudinal / lateral
[−0.3,1.0] / [−1.0,1.0]
Gate / channel exit / dock line ( px )
2.0 / 4.5 / 5.0
Lateral tolerance / outer lateral bound
0.3 / 1.0
Dock crossing: maximum speed / maximum ∣py∣
1.0 / 0.3
Process noise (velocity, per s ) / observation noise
Table 2: Point-Mass Docking Settings. Lengths, speeds and accelerations are nondimensional.
Strategy
Mean (range)
Difference [95% interval]
Favoring
Gain-based
50.1 (45.9–59.2)
—
—
Uniform
49.8 (44.3–54.5)
+0.3 [-3.4, +4.4]
4/10
Downstream failure
47.8 (42.0–57.6)
+2.3 [-1.5, +6.0]
5/10
Full episode
45.8 (30.1–51.2)
+4.3 [+0.1, +9.5]
7/10
Failure-weighted
26.9 (23.0–31.6)
+23.2 [+19.6, +26.7]
10/10
Dock only
8.4 (1.2–11.9)
+41.7
10/10
Appendix
Table 3: Long-Approach Variant at 500,000 Decisions. Completion improvement over the paired base policy (mean and range over ten matched training runs) and gain-based allocation minus each strategy, in percent. Each run is evaluated on 512 held-out resets. Brackets give paired 95% intervals; we did not compute an interval for dock-only training. Mint : the interval excludes zero. Uniform allocation is charged the same gain-estimation cost as gain-based allocation.
Phase
Nj
Gained
Lost
Δ^u (%)
95% interval
Failures
Grasp
128
15
22
−5.5
[−14.8,3.9]
12
Transport
126
6
11
−3.9
[−10.2,2.3]
37
Insert
73
18
5
−10.2
[−3.1,17.2]
49
Release
48
14
4
−7.8
[−1.6,14.1]
6
Withdraw
28
0
2
−1.6
[−3.9,0.0]
9
Appendix
Table 4: Completion Gains in the Assembly Task. Paired differences in completion, in percent, between the substituted policy ( μ on all joints during phase j ) and the base policy π0 under the shared-action protocol, from N=128 paired episodes per phase on the same 128 resets. Nj : pairs whose base-policy episode reaches phase j . Gained (lost): pairs completed only by the substituted (base-policy) episode. Failures: episodes with detected-failure phase f(τ)=j among the 113 failures in 128 episodes of π0 on the same resets.
Release sequence of the state-feedback controller
Contact-free time of the part before left-hand release
0.25 s
Latest left-hand release after right-hand withdrawal
1 s
Left-hand reference set to current joint positions
1 s
Left-hand opening
5 s
Opening offset before final reference bounds
0.004 rad
Left-arm withdrawal
2 s
Appendix
Table 5: Base-Policy and State-Feedback Controller Parameters for the Assembly Task.
Phase
Base
Substituted
Δ^u (%)
95% interval
Grasp
2
1
−6.3
[−18.8,0.0]
Transport
6
3
−18.8
[−37.5,0.0]
Insert
3
1
−12.5
[−31.3,0.0]
Release
1
6
31.3
[12.5,56.3]
Withdraw
2
0
−12.5
[−31.3,0.0]
Appendix
Table 6: Completion Gains in the Second Assembly Configuration. Completed episodes among 16 paired episodes per phase under the base policy and under the substituted policy; estimated completion gains and 95% bootstrap intervals are in percent.
Imitation policies that reliably execute most of a manipulation trajectory can still fail in localized regions requiring last-mile precision. Updating the full policy or exploring residual actions everywhere treats this local failure as a global learning problem. We introduce Adaptive Residual Correction (ARC), a lightweight post-training module that uses reinforcement learning (RL) to learn a residual correction on top of a frozen policy while using sparse human corrections to structure autonomous exploration. Rather than treating human corrections as imitation targets, ARC uses pre-intervention states to determine 'when' residual control should become active and correction geometry to determine 'where' in residual action space autonomous refinement should explore; task reward determines 'what' residual is ultimately executed. Across three precision-manipulation tasks on a real robot, ARC improves average task success from 33.3% to 75.0%. The human corrections define task-dependent 2-3 dimensional subspaces of the seven-joint residual space that retain 96.0-97.7% of squared correction magnitude. Together, these results show that sparse human corrections can localize and constrain autonomous post-training without requiring the observed correction itself to serve as the final action target.
Anukriti Singh, Maxime Caniot, Pratap Tokekar +1
University of Maryland, College Park College Park, MD · Work done during internship at Enchanted Tools. · Enchanted Tools Paris, France
Imitation learning enables robots to acquire manipulation skills from demonstrations, but the resulting policies can fail outside the training data, while collecting more demonstrations requires substantial human effort. Human-in-the-loop reinforcement learning uses corrective feedback during online training, but typically learns the complete task policy rather than refining a pretrained imitation policy. We introduce Res-HIL, a human-in-the-loop residual reinforcement learning framework that learns corrective actions on top of a frozen imitation policy. Each human intervention provides two complementary learning signals: direct supervision of the residual policy and reward shaping of preceding autonomous behavior. Res-HIL combines these signals with zero initialization of the residual policy to stabilize and accelerate online learning. We evaluate Res-HIL on five contact-rich manipulation tasks spanning high-precision and long-horizon behaviors. With only 20 initial demonstrations, Res-HIL outperforms state-of-the-art full-policy human-in-the-loop reinforcement learning and residual fine-tuning without human guidance on every task after ten minutes of online training. Res-HIL improves its pretrained base policies and outperforms imitation policies trained with five times more demonstrations. An ablation study shows that direct residual supervision is critical to performance, while intervention-aware reward shaping substantially improves training efficiency.
Mariia Iavorskaia, Christian Dietz, Sebastian Albrecht +1
Siemens AG, Research and Predevelopment, Germany. · Technical University of Munich, Germany.
This paper presents PRISM: an instruction-conditioned refinement method for imitation policies in robotic manipulation. This approach bridges Imitation Learning (IL) and Reinforcement Learning (RL) frameworks into a seamless pipeline, such that an imitation policy on a broad generic task, generated from a set of user-guided demonstrations, can be refined through reinforcement to generate new unseen fine-grain behaviours. The refinement process follows the Eureka paradigm, where reward functions for RL are iteratively generated from an initial natural-language task description. Presented approach, builds on top of this mechanism to adapt a refined IL policy of a generic task to new goal configurations and the introduction of constraints by adding also human feedback correction on intermediate rollouts, enabling policy reusability and therefore data efficiency. Results for a pick-and-place task in a simulated scenario show that proposed method outperforms policies without human feedback, improving robustness on deployment and reducing computational burden.
Arnau Boix-Granell, Alberto San-Miguel-Tello, Magí Dalmau-Moreno +1
Eurecat, Centre Tecnològic Catalunya, Robotics & Automation Unit, Barcelona, Spain