Residual reinforcement learning corrects an imitation-trained base policy using a limited budget of environment interaction. In a task with several phases, this budget must be divided among the phases, and the phase in which the base policy fails is not necessarily the phase in which a correction most improves task completion. We formalize this distinction through the completion gain: the change in full-task completion probability when, during a single phase, a state-feedback controller replaces selected components of the base policy's action. We decompose the completion gain into the probability of reaching the phase and the conditional improvement in completion from the start of the phase, and we estimate it from paired episodes. The positive estimates determine which action components the residual policy corrects and how training episodes are distributed across phases. In stochastic point-mass docking experiments, most failures occurred in the final docking phase, whereas the largest completion gain occurred in the preceding channel phase; gain-based allocation improved completion over allocation proportional to failure counts and, in a variant with a longer approach, over uniform allocation at half of the budget. In simulated bimanual robotic assembly, the completion-gain estimates were positive only for insertion and release, although few failures occurred during release; in a second assembly configuration, a residual policy trained in the release phase improved completion from held-out states at the start of release by 16.9%.
Figures & tables
Figure 1: Running Examples. (a) Point-Mass Docking: Here, a point mass must pass through a channel and cross the dock line within speed and lateral position limits. The line plots in (a) depict traces from trial runs under two different control settings: a base policy (red) and a control scheme where control of the point-mass’s acceleration is switched to a state-feedback controller in the channel phase (blue). In each setting, we use the same initial state, physical parameters, and noise draws. Under the base policy, the point mass exceeds the speed limit at the dock line. With state-feedback acceleration commands in the channel (blue), it satisfies both terminal limits. Open circles mark phase starts. The dotted curve gives the speed bound for one-dimensional braking without drag or noise. (b,c) Bimanual Robotic Assembly: In this task, we require a bimanual robotic manipulator to insert a key-like part with its right hand into a housing stabilized by the left hand. The part must remain stable in the housing after the robot withdraws its hands. The frames in the figure show an episode under the state-feedback controller in Isaac Lab ( Mittal et al., 2023 ) .
Figure 2: Paired Episodes for u=(j,G) .
Figure 3: Completion Gains, Allocation, and Residual Training in Point-Mass Docking. (a) Estimates from 128 paired episodes per candidate with 95% bootstrap intervals. (b) Failure-weighted (red) and gain-based (blue) allocation. (c) Means over ten training runs; markers are individual runs, and the budget includes gain estimation.
Figure 4
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Point-Mass Trajectories and Speed at the Dock Start. (a,b) The first 20 of 128 paired episodes: (a) the base policy, with completed episodes in blue and failed episodes in red; (b) the same episode conditions with the state-feedback controller substituted on both action components during the channel phase. Shading marks ∣py∣≤0.3 within the channel, and the green bar marks the dock line. (c) Speed at the dock start for the 117 base-policy episodes that reached docking, grouped by whether the state-feedback controller, applied from the dock start, completed the task (68) or failed (49). The dashed line marks the speed limit of 1.0 at the dock line; the bound on the speed at the dock start for the dynamics without drag is 1.14 ( Sec. 4.1 ).
Decision interval Δtk / horizon T
0.1 s / 16 s (160 decisions)
Acceleration bounds: longitudinal / lateral
[−0.3,1.0] / [−1.0,1.0]
Gate / channel exit / dock line ( px )
2.0 / 4.5 / 5.0
Lateral tolerance / outer lateral bound
0.3 / 1.0
Dock crossing: maximum speed / maximum ∣py∣
1.0 / 0.3
Process noise (velocity, per s ) / observation noise
Table 2: Point-Mass Docking Settings. Lengths, speeds and accelerations are nondimensional.
Strategy
Mean (range)
Difference [95% interval]
Favoring
Gain-based
50.1 (45.9–59.2)
—
—
Uniform
49.8 (44.3–54.5)
+0.3 [-3.4, +4.4]
4/10
Downstream failure
47.8 (42.0–57.6)
+2.3 [-1.5, +6.0]
5/10
Full episode
45.8 (30.1–51.2)
+4.3 [+0.1, +9.5]
7/10
Failure-weighted
26.9 (23.0–31.6)
+23.2 [+19.6, +26.7]
10/10
Dock only
8.4 (1.2–11.9)
+41.7
10/10
Appendix
Table 3: Long-Approach Variant at 500,000 Decisions. Completion improvement over the paired base policy (mean and range over ten matched training runs) and gain-based allocation minus each strategy, in percent. Each run is evaluated on 512 held-out resets. Brackets give paired 95% intervals; we did not compute an interval for dock-only training. Mint : the interval excludes zero. Uniform allocation is charged the same gain-estimation cost as gain-based allocation.
Phase
Nj
Gained
Lost
Δ^u (%)
95% interval
Failures
Grasp
128
15
22
−5.5
[−14.8,3.9]
12
Transport
126
6
11
−3.9
[−10.2,2.3]
37
Insert
73
18
5
−10.2
[−3.1,17.2]
49
Release
48
14
4
−7.8
[−1.6,14.1]
6
Withdraw
28
0
2
−1.6
[−3.9,0.0]
9
Appendix
Table 4: Completion Gains in the Assembly Task. Paired differences in completion, in percent, between the substituted policy ( μ on all joints during phase j ) and the base policy π0 under the shared-action protocol, from N=128 paired episodes per phase on the same 128 resets. Nj : pairs whose base-policy episode reaches phase j . Gained (lost): pairs completed only by the substituted (base-policy) episode. Failures: episodes with detected-failure phase f(τ)=j among the 113 failures in 128 episodes of π0 on the same resets.
Release sequence of the state-feedback controller
Contact-free time of the part before left-hand release
0.25 s
Latest left-hand release after right-hand withdrawal
1 s
Left-hand reference set to current joint positions
1 s
Left-hand opening
5 s
Opening offset before final reference bounds
0.004 rad
Left-arm withdrawal
2 s
Appendix
Table 5: Base-Policy and State-Feedback Controller Parameters for the Assembly Task.
Phase
Base
Substituted
Δ^u (%)
95% interval
Grasp
2
1
−6.3
[−18.8,0.0]
Transport
6
3
−18.8
[−37.5,0.0]
Insert
3
1
−12.5
[−31.3,0.0]
Release
1
6
31.3
[12.5,56.3]
Withdraw
2
0
−12.5
[−31.3,0.0]
Appendix
Table 6: Completion Gains in the Second Assembly Configuration. Completed episodes among 16 paired episodes per phase under the base policy and under the substituted policy; estimated completion gains and 95% bootstrap intervals are in percent.