In robotic insertion under uncertain contact, the axial force limit and the appropriate controller gain vary across tasks. As a result, a single fixed gain is unlikely to remain suitable across different task conditions, making conventional impedance controllers reliant on manual retuning. To eliminate manual retuning, we propose Constraint-Grounded Reinforcement Learning (CG-RL), a variable impedance framework for online gain adaptation. Conditioned on the force limit and contact feedback, the policy outputs a residual motion, an insertion rate, and a requested gain. The controller projects this gain into the admissible range without exposing the range itself to the policy. This separation allows a single policy to operate under different force limits without retraining or manual retuning. We evaluate CG-RL on simulated oblique insertion across five training seeds. CG-RL achieves an 85.8±7.7% (mean ± SD) success rate of insertions without violating the force limit, while keeping the applied gain within the admissible range. As a comparison, a fixed-gain baseline using the midpoint gain achieves a success rate of 50.1%. The policy adapts its insertion rate continuously to the specified force limit and further generalizes to more permissive force limits above the training range. In contrast, the same actor without force-limit input does not exhibit this adaptation. The applied gain is guaranteed to remain within the admissible range, while force-limit satisfaction is validated empirically rather than guaranteed formally.
Figures & tables
Fig. 1: Overview of CG-RL. A single actor responds to the supplied force limit, while the execution layer independently enforces the admissible gain set across varying contact tasks.
Study
Evaluation conditions
Methods
Primary evidence
Episodes
Study I
3 force limits ×3 gain sets ×4 friction values ( μs∈{0.60,0.85,0.95,1.10} )
All methods in Table IV
Constraint-compliant success and task efficiency
28,800
Study II
7 contact and observation shifts ×3 force limits ×3 gain sets
All methods in Table IV
Robustness and operating boundaries
50,400
Study III
11 force limits from 6.5 to 9.0 N, with friction and gain set fixed
CG-RL, No Fmax input, and K -conditioned
Raw advance-action slope
5,280
Study IV
6 force limits sampled from [5.5,6.25]∪[9.25,10.0] N
CG-RL
Extrapolation and failure boundary
960
TABLE I: Evaluation protocol and episode accounting for the four studies.
TGS; 8 position iterations and 1 velocity iteration for the robot, 192 position iterations and 1 velocity iteration for the peg and bore
Episode horizon
25 s
Target insertion depth
30 mm
TABLE II: Key simulation, training, and evaluation settings.
Parameter
Value
Parameter
Value
Advance rate bounds (smin,smax)
(0.5,1.5)
System gain range Ksys
[1400,1700]
Maximum gain rate K˙maxcmd
2800 controller units/s
Fixed physical guard
75 N
Force margin ρF
0.10
Contact threshold ϵF
0.5 N
Rate target coefficient λu
2.0
Force limit coefficient λb
0.75
Barrier onset αF
0.80
Barrier scale λbar
0.20
Force limit range (FFmin,FFmax)
(6.5,9.0) N
Progress and stage weights (wΔΦ,wstage)
(5,1)
TABLE III: Fixed controller, reward, and PPO constants. Reward coefficients are reported as nonnegative magnitudes; their signs follow ( 24 )–( 29 ). All learned methods use the same values unless stated otherwise.
Fig. 2: Task, calibration, and specifications. (a) The 17 ∘ oblique insertion geometry and an illustrative simulation view; the material samples are shown for context. (b) Empirical 95th percentile of episode peak axial reaction from 21 successful calibration trials at each fixed gain; the dashed bounds mark the force-limit training range. (c) Confirmatory force limits and three admissible gain sets, varied independently. The task schematic is not drawn to scale.
Role
Method
Task input to actor
Obs. dim.
Gain execution
Proposed
CG-RL
Fmax
35
Set projection
Controlled ablation
No Fmax input
None
34
Set projection
Learned comparator
Margin barrier
Fmax,Kmin,Kmax
37
Set projection
Descriptive comparator
K -conditioned
Kmin,Kmax
36
Set projection
Fixed baseline
Fixed midpoint controller
None
–
Fixed midpoint controller
TABLE IV: Methods included in the main comparison.
Role
Method
Overall CCS [%]
Stress-subset CCS [%]
Geometric success [%]
Peak axial reaction [N]
Completion time [s]
Proposed
CG-RL
85.78 ± 7.73
72.33 ± 15.28
96.53 ± 1.51
6.691 ± 0.162
12.08 ± 1.11
Controlled ablation
No Fmax input
82.43 ± 6.71
66.63 ± 14.03
95.09 ± 2.23
6.756 ± 0.089
12.79 ± 0.96
Learned comparator
Margin barrier
73.58 ± 11.73
51.25 ± 21.03
93.82 ± 3.00
6.946 ± 0.208
14.16 ± 1.81
Descriptive comparator
K -conditioned
89.34 ± 2.45
80.52 ± 4.02
95.28 ± 3.59
6.559 ± 0.179
11.71 ± 0.51
Fixed baseline
Fixed midpoint controller
50.10 ± 0.96
19.34 ± 1.49
69.72 ± 0.32
7.144 ± 0.008
18.18 ± 0.11
TABLE V: Task performance under the training distribution.
Fig. 3: Task performance under the training distribution. CG-RL is the proposed method. The light blue entry is the controlled force limit ablation, the purple entry is the learned objective comparator, and the gray entry is the fixed midpoint controller. Error bars show 95% confidence intervals across five independent training seeds for the learned methods and five paired evaluation-bank blocks for the fixed baseline.
Fig. 4: Representative CG-RL rollout. A successful stress-subset episode at Fmax=7.5 N, K=[1500,1600] , and μs=0.85 .
Fig. 5: Online gain selection and gain set enforcement. Panels (a)–(c) evaluate the set midpoint intervention, panel (d) reports requested, projected, and applied gains, and panels (e)–(h) evaluate a fixed gain request of 1550 under set projection. Percentages in panel (d) indicate the fraction of contact steps modified by projection. Each gray line connects the paired results for one training seed; black diamonds and error bars show the mean and two-sided 95% confidence interval across five seeds.
Fig. 6: Response of CG-RL to static friction. The shaded interval denotes the friction range used during training. The panels report constraint-compliant and geometric success, failure-aware completion time, and mean episode peak axial reaction. Each friction value summarizes three force limits, three gain sets, and 32 paired episodes in each factorial cell. Curves and bands show the mean and two-sided 95% confidence interval across five independent training seeds.
Role
Method
Low friction
High friction
Soft contact
Stiff contact
Fixture pose variation
Sensor noise
Combined
μs=0.45
μs=1.25
×0.75
×1.25
2× nominal reset range
Bounded
All shifts
Proposed
CG-RL
100.00 ± 0.00
14.86 ± 4.91
94.79 ± 6.07
32.50 ± 7.50
84.58 ± 7.71
94.44 ± 6.76
0.00 ± 0.00
Controlled ablation
No Fmax input
100.00 ± 0.00
15.83 ± 1.46
92.71 ± 8.02
29.31 ± 5.38
82.57 ± 4.49
91.53 ± 5.67
0.00 ± 0.00
Learned comparator
Margin barrier
100.00 ± 0.00
14.79 ± 3.62
91.25 ± 7.57
26.04 ± 4.03
71.67 ± 12.54
78.82 ± 15.86
0.00 ± 0.00
Descriptive comparator
K -conditioned
100.00 ± 0.00
14.17 ± 6.22
93.68 ± 10.03
30.00 ± 14.93
85.00 ± 4.41
94.65 ± 3.43
0.00 ± 0.00
Fixed baseline
Fixed midpoint controller
100.00 ± 0.00
0.00 ± 0.00
34.58 ± 3.26
0.69 ± 0.43
32.57 ± 2.06
34.58 ± 1.46
0.00 ± 0.00
TABLE VI: Constraint-compliant success under contact and observation shifts, reported as mean ± sample standard deviation (%).
Fig. 7: Continuous response to the force limit and extrapolation beyond the training range. The shaded interval denotes the force-limit range used during training. Panel (a) compares the contact-phase raw advance action of CG-RL with that of the No Fmax input ablation. Panels (b)–(d) report the constraint-compliant success, failure-aware completion time, and peak axial reaction of CG-RL. Curves and bands show the mean and two-sided 95% confidence interval across five independent training seeds.
Method
Mean slope [action/N]
Seed range [action/N]
CG-RL
0.282
[0.203,0.368]
No Fmax input
0.000143
[−0.000422,0.000597]
TABLE VII: Raw advance action sensitivity to the force limit within the training range.
Contact-rich robotic tasks require coordinated motion and impedance under uncertain interaction dynamics and physical constraints. While recent methods on learned policies provide useful interaction strategies, they may fail when contact conditions differ from those encountered during training. Model-based generative methods show promising receding-horizon trajectories optimization results, but often lack structured behavioral guidance and refinement that jointly accounts for constraint geometry and actual contact responses. To this end, we propose Manifold Generative Annealing (MGA), a joint model-free and model-based generative framework for motion--impedance optimization. MGA first organizes pretrained reinforcement learning policies into structured sequence proposals, retaining task-specific interaction strategies without committing to their execution. It then evaluates these proposals alongside exploratory samples through model-based rollouts for refinement or separate execution selection. Crucially, realization-aware manifold refinement aligns updates from the refinement bank with both task constraints and locally achievable physical responses, allowing learned experience to guide online adaptation under changing contact conditions. Experiments cover surface scanning, peg insertion, and humanoid pushing, with improved safety success rate of impedance control under nominal and sensing-shift conditions.
Zhilin He, Guoquan Qi, Changliu Liu
Carnegie Mellon University · Institute of Automation, Chinese Academy of Sciences
Imitation learning (IL) has shown strong potential for contact-rich precision insertion tasks. However, its practical deployment is often hindered by covariate shift and the need for continuous expert monitoring to recover from failures during execution. In this paper, we propose Trajectory Editing Residual Dataset Aggregation (TER-DAgger), a scalable and force-aware human-in-the-loop imitation learning framework that mitigates covariate shift by learning residual policies through optimization-based trajectory editing. This approach smoothly fuses policy rollouts with human corrective trajectories, providing consistent and stable supervision. Second, we introduce a force-aware failure anticipation mechanism that triggers human intervention only when discrepancies arise between predicted and measured end-effector forces, significantly reducing the requirement for continuous expert monitoring. Third, all learned policies are executed within a Cartesian impedance control framework, ensuring compliant and safe behavior during contact-rich interactions. Extensive experiments in both simulation and real-world precision insertion tasks show that TER-DAgger improves the average success rate by over 37% compared to behavior cloning, human-guided correction, retraining, and fine-tuning baselines, demonstrating its effectiveness in mitigating covariate shift and enabling scalable deployment in contact-rich manipulation.
Yiou Huang, Ning Ma, Weichu Zhao +4
All authors are with Xi’an Jiaotong-Liverpool University, Suzhou, China.
Placing a grasped book into a tight shelf is a compact but difficult contact-rich control problem: millimetre-scale pose error can turn a geometrically valid approach into jamming, failed release, or incomplete seating. We study this final phase after grasp acquisition and global approach, and ask how control authority should be divided between known geometry and learned behaviour. Our method retains a nominal task-space controller for structured insertion and seating, while residual PPO supplies bounded local corrections and decides when to release. Only the brief open-retreat-reclose transition is scripted. For the final policy used on hardware, a deployment-matched simulation evaluation over 512 fixed conditions yields 98.50 percent mean success (0.23 percentage-point sample SD) across three independent training runs, compared with 37.89 percent for nominal control. On the physical xArm7, 60 trials over 30 matched conditions show the same qualitative advantage: residual control raises success from 26.7 percent to 63.3 percent, reduces failures from 22 to 11, and wins 13 of the 15 matched conditions in which the two controllers differ. Robustness tests show that performance remains above 87 percent under initialization perturbations up to 1.5x, while very tight clearances expose the geometric limit of local correction. These results support a hybrid design in which geometry preserves reliable task structure and learning is concentrated on the contact-sensitive behaviour that fixed rules handle poorly.
Tianyuan Liu, Rutherford Agbeshi Patamia, Benjamin Champion +2