Constraint-Grounded Reinforcement Learning for Variable Impedance Control in Contact-Rich Robotic Insertion

Lin He; Min Deng

cs.ROSep 11, 2026

Constraint-Grounded Reinforcement Learning for Variable Impedance Control in Contact-Rich Robotic Insertion

Authors: Lin He, Min Deng

Organizations: Department of Civil and Environmental Engineering, The University of Tennessee, Knoxville, TN 37996 USA

Abstract

In robotic insertion under uncertain contact, the axial force limit and the appropriate controller gain vary across tasks. As a result, a single fixed gain is unlikely to remain suitable across different task conditions, making conventional impedance controllers reliant on manual retuning. To eliminate manual retuning, we propose Constraint-Grounded Reinforcement Learning (CG-RL), a variable impedance framework for online gain adaptation. Conditioned on the force limit and contact feedback, the policy outputs a residual motion, an insertion rate, and a requested gain. The controller projects this gain into the admissible range without exposing the range itself to the policy. This separation allows a single policy to operate under different force limits without retraining or manual retuning. We evaluate CG-RL on simulated oblique insertion across five training seeds. CG-RL achieves an $85.8\pm7.7\%$ (mean $\pm$ SD) success rate of insertions without violating the force limit, while keeping the applied gain within the admissible range. As a comparison, a fixed-gain baseline using the midpoint gain achieves a success rate of $50.1\%$ . The policy adapts its insertion rate continuously to the specified force limit and further generalizes to more permissive force limits above the training range. In contrast, the same actor without force-limit input does not exhibit this adaptation. The applied gain is guaranteed to remain within the admissible range, while force-limit satisfaction is validated empirically rather than guaranteed formally.

Figures & tables

Fig. 1: Overview of CG-RL. A single actor responds to the supplied force limit, while the execution layer independently enforces the admissible gain set across varying contact tasks.

Study	Evaluation conditions	Methods	Primary evidence	Episodes
Study I	$3$ force limits $\times$ $3$ gain sets $\times$ $4$ friction values ( $\mu_{s}\in\{0.60,0.85,0.95,1.10\}$ )	All methods in Table IV	Constraint-compliant success and task efficiency	28,800
Study II	$7$ contact and observation shifts $\times$ $3$ force limits $\times$ $3$ gain sets	All methods in Table IV	Robustness and operating boundaries	50,400
Study III	$11$ force limits from $6.5$ to $9.0$ N, with friction and gain set fixed	CG-RL, No $F_{\max}$ input, and $K$ -conditioned	Raw advance-action slope	5,280
Study IV	$6$ force limits sampled from $[5.5,6.25]\cup[9.25,10.0]$ N	CG-RL	Extrapolation and failure boundary	960

TABLE I: Evaluation protocol and episode accounting for the four studies.

Setting	Value
Robot	Franka Research 3
Simulation / policy rate	120 / 60 Hz ( $D=2$ physics substeps per policy step)
Simulator	Isaac Sim 5.0.0, Isaac Lab 2.2.1
PhysX solver	TGS; 8 position iterations and 1 velocity iteration for the robot, 192 position iterations and 1 velocity iteration for the peg and bore
Episode horizon	25 s
Target insertion depth	30 mm

TABLE II: Key simulation, training, and evaluation settings.

Parameter	Value	Parameter	Value
Advance rate bounds $(s_{\min},s_{\max})$	$(0.5,1.5)$	System gain range $\mathcal{K}_{\mathrm{sys}}$	$[1400,1700]$
Maximum gain rate $\dot{K}_{\max}^{\mathrm{cmd}}$	$2800$ controller units/s	Fixed physical guard	$75$ N
Force margin $\rho_{F}$	$0.10$	Contact threshold $\epsilon_{F}$	$0.5$ N
Rate target coefficient $\lambda_{u}$	$2.0$	Force limit coefficient $\lambda_{b}$	$0.75$
Barrier onset $\alpha_{F}$	$0.80$	Barrier scale $\lambda_{\mathrm{bar}}$	$0.20$
Force limit range $(F_{\mathcal{F}}^{\min},F_{\mathcal{F}}^{\max})$	$(6.5,9.0)$ N	Progress and stage weights $(w_{\Delta\Phi},w_{\mathrm{stage}})$	$(5,1)$

TABLE III: Fixed controller, reward, and PPO constants. Reward coefficients are reported as nonnegative magnitudes; their signs follow ( 24 )–( 29 ). All learned methods use the same values unless stated otherwise.

Fig. 2: Task, calibration, and specifications. (a) The 17 ∘ oblique insertion geometry and an illustrative simulation view; the material samples are shown for context. (b) Empirical 95th percentile of episode peak axial reaction from 21 successful calibration trials at each fixed gain; the dashed bounds mark the force-limit training range. (c) Confirmatory force limits and three admissible gain sets, varied independently. The task schematic is not drawn to scale.

Role	Method	Task input to actor	Obs. dim.	Gain execution
Proposed	CG-RL	$F_{\max}$	35	Set projection
Controlled ablation	No $F_{\max}$ input	None	34	Set projection
Learned comparator	Margin barrier	$F_{\max},K_{\min},K_{\max}$	37	Set projection
Descriptive comparator	$K$ -conditioned	$K_{\min},K_{\max}$	36	Set projection
Fixed baseline	Fixed midpoint controller	None	–	Fixed midpoint controller

TABLE IV: Methods included in the main comparison.

Role	Method	Overall CCS [%]	Stress-subset CCS [%]	Geometric success [%]	Peak axial reaction [N]	Completion time [s]
Proposed	CG-RL	85.78 $\pm$ 7.73	72.33 $\pm$ 15.28	96.53 $\pm$ 1.51	6.691 $\pm$ 0.162	12.08 $\pm$ 1.11
Controlled ablation	No $F_{\max}$ input	82.43 $\pm$ 6.71	66.63 $\pm$ 14.03	95.09 $\pm$ 2.23	6.756 $\pm$ 0.089	12.79 $\pm$ 0.96
Learned comparator	Margin barrier	73.58 $\pm$ 11.73	51.25 $\pm$ 21.03	93.82 $\pm$ 3.00	6.946 $\pm$ 0.208	14.16 $\pm$ 1.81
Descriptive comparator	$K$ -conditioned	89.34 $\pm$ 2.45	80.52 $\pm$ 4.02	95.28 $\pm$ 3.59	6.559 $\pm$ 0.179	11.71 $\pm$ 0.51
Fixed baseline	Fixed midpoint controller	50.10 $\pm$ 0.96	19.34 $\pm$ 1.49	69.72 $\pm$ 0.32	7.144 $\pm$ 0.008	18.18 $\pm$ 0.11

TABLE V: Task performance under the training distribution.

Fig. 3: Task performance under the training distribution. CG-RL is the proposed method. The light blue entry is the controlled force limit ablation, the purple entry is the learned objective comparator, and the gray entry is the fixed midpoint controller. Error bars show 95% confidence intervals across five independent training seeds for the learned methods and five paired evaluation-bank blocks for the fixed baseline.

Fig. 4: Representative CG-RL rollout. A successful stress-subset episode at $F_{\max}=7.5$ N, $\mathcal{K}=[1500,1600]$ , and $\mu_{s}=0.85$ .

Fig. 5: Online gain selection and gain set enforcement. Panels (a)–(c) evaluate the set midpoint intervention, panel (d) reports requested, projected, and applied gains, and panels (e)–(h) evaluate a fixed gain request of 1550 under set projection. Percentages in panel (d) indicate the fraction of contact steps modified by projection. Each gray line connects the paired results for one training seed; black diamonds and error bars show the mean and two-sided 95% confidence interval across five seeds.

Fig. 6: Response of CG-RL to static friction. The shaded interval denotes the friction range used during training. The panels report constraint-compliant and geometric success, failure-aware completion time, and mean episode peak axial reaction. Each friction value summarizes three force limits, three gain sets, and 32 paired episodes in each factorial cell. Curves and bands show the mean and two-sided 95% confidence interval across five independent training seeds.

Role	Method	Low friction	High friction	Soft contact	Stiff contact	Fixture pose variation	Sensor noise	Combined
		$\mu_{s}=0.45$	$\mu_{s}=1.25$	$\times 0.75$	$\times 1.25$	$2\times$ nominal reset range	Bounded	All shifts
Proposed	CG-RL	100.00 $\pm$ 0.00	14.86 $\pm$ 4.91	94.79 $\pm$ 6.07	32.50 $\pm$ 7.50	84.58 $\pm$ 7.71	94.44 $\pm$ 6.76	0.00 $\pm$ 0.00
Controlled ablation	No $F_{\max}$ input	100.00 $\pm$ 0.00	15.83 $\pm$ 1.46	92.71 $\pm$ 8.02	29.31 $\pm$ 5.38	82.57 $\pm$ 4.49	91.53 $\pm$ 5.67	0.00 $\pm$ 0.00
Learned comparator	Margin barrier	100.00 $\pm$ 0.00	14.79 $\pm$ 3.62	91.25 $\pm$ 7.57	26.04 $\pm$ 4.03	71.67 $\pm$ 12.54	78.82 $\pm$ 15.86	0.00 $\pm$ 0.00
Descriptive comparator	$K$ -conditioned	100.00 $\pm$ 0.00	14.17 $\pm$ 6.22	93.68 $\pm$ 10.03	30.00 $\pm$ 14.93	85.00 $\pm$ 4.41	94.65 $\pm$ 3.43	0.00 $\pm$ 0.00
Fixed baseline	Fixed midpoint controller	100.00 $\pm$ 0.00	0.00 $\pm$ 0.00	34.58 $\pm$ 3.26	0.69 $\pm$ 0.43	32.57 $\pm$ 2.06	34.58 $\pm$ 1.46	0.00 $\pm$ 0.00

TABLE VI: Constraint-compliant success under contact and observation shifts, reported as mean

\pm

sample standard deviation (%).

Fig. 7: Continuous response to the force limit and extrapolation beyond the training range. The shaded interval denotes the force-limit range used during training. Panel (a) compares the contact-phase raw advance action of CG-RL with that of the No

F_{\max}

input ablation. Panels (b)–(d) report the constraint-compliant success, failure-aware completion time, and peak axial reaction of CG-RL. Curves and bands show the mean and two-sided 95% confidence interval across five independent training seeds.

Method	Mean slope [action/N]	Seed range [action/N]
CG-RL	0.282	$[0.203,\,0.368]$
No $F_{\max}$ input	$0.000143$	$[-0.000422,\,0.000597]$

TABLE VII: Raw advance action sensitivity to the force limit within the training range.

Robotic Insertion

Robotic insertion studies how robots place objects through openings or into tissue despite tight tolerances, uncertain alignment, and contact forces.

See all 46 papers

Manifold Generative Annealing for Contact Constraint-aware Impedance Control with Reinforcement Learning Prior
Oct 15, 2026Zhilin He, Guoquan Qi, Changliu LiuContact-Rich Robotic Manipulation Impedance Control
Carnegie Mellon University · Institute of Automation, Chinese Academy of Sciences
Force-Aware Residual DAgger via Trajectory Editing for Precision Insertion with Impedance Control
Mar 4, 2026Yiou Huang, Ning Ma, Weichu Zhao +4Interactive Imitation Learning Contact-Rich Robotic Manipulation
All authors are with Xi’an Jiaotong-Liverpool University, Suzhou, China.
Hybrid Residual Reinforcement Learning for Contact-Rich Robotic Book Insertion
Sep 17, 2026Tianyuan Liu, Rutherford Agbeshi Patamia, Benjamin Champion +2Contact-Rich Robotic Manipulation Robotic RL
Deakin University, Australia

Oct 15, 2026J/K move · Enter open · S save

Manifold Generative Annealing for Contact Constraint-aware Impedance Control with Reinforcement Learning Prior

Zhilin He, Guoquan Qi, Changliu Liu

Carnegie Mellon University · Institute of Automation, Chinese Academy of Sciences

Contact-Rich Robotic Manipulation Impedance Control Trajectory Optimization Model-Based RL

Open paper AskPDF arXiv

Contact-rich robotic tasks require coordinated motion and impedance under uncertain interaction dynamics and physical constraints. While recent methods on learned policies provide useful interaction strategies, they may fail when contact conditions differ from those encountered during training. Model-based generative methods show promising receding-horizon trajectories optimization results, but often lack structured behavioral guidance and refinement that jointly accounts for constraint geometry and actual contact responses. To this end, we propose Manifold Generative Annealing (MGA), a joint model-free and model-based generative framework for motion--impedance optimization. MGA first organizes pretrained reinforcement learning policies into structured sequence proposals, retaining task-specific interaction strategies without committing to their execution. It then evaluates these proposals alongside exploratory samples through model-based rollouts for refinement or separate execution selection. Crucially, realization-aware manifold refinement aligns updates from the refinement bank with both task constraints and locally achievable physical responses, allowing learned experience to guide online adaptation under changing contact conditions. Experiments cover surface scanning, peg insertion, and humanoid pushing, with improved safety success rate of impedance control under nominal and sensing-shift conditions.

Abstract

Robotic Insertion

Explore similar work

Manifold Generative Annealing for Contact Constraint-aware Impedance Control with Reinforcement Learning Prior

Force-Aware Residual DAgger via Trajectory Editing for Precision Insertion with Impedance Control

Hybrid Residual Reinforcement Learning for Contact-Rich Robotic Book Insertion