A Reinforcement Learning Framework for Safety-Aware Haptic Shared Control in Surgical Robotics

Zihao Zhang; Karl Handwerker; Balint Varga

cs.ROOct 11, 2026

A Reinforcement Learning Framework for Safety-Aware Haptic Shared Control in Surgical Robotics

Authors: Zihao Zhang, Karl Handwerker, Balint Varga

Organizations: Institute of Control Systems, Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany

Abstract

Haptic shared control in surgical robotics combines human-side and robot-side forces at a shared end effector while preserving operator involvement. However, the effects of geometric safety choices on task execution and force coordination remain unclear. We model the interaction as a two-agent cooperative Markov game under centralized training and decentralized execution, comparing learned human control with a task-tuned fixed-impedance baseline. This model-free multi-agent reinforcement learning framework supports joint policy adaptation without requiring a complete analytical interaction model. It compares MADDPG and rMAPPO with an epigraph-constrained approach under common geometric task definitions. In simulation, rMAPPO generally achieved the highest return and tracking performance, whereas the epigraph approach produced more conservative motions and lower geometric exposure, including no observed collision or dangerous-region exposure after geometry transfer. With real operator input, fixed-impedance-trained policies were faster and successful in all recorded trials. Policies trained with a learned partner exhibited greater command opposition, while the fixed epigraph condition produced the most aligned commands. This work provides a benchmark for studying human-control representation and constraint-handling trade-offs in haptic shared control.

Figures & tables

Fig. 1: Haptic shared control with region partitioned reward and constraint design under learned and fixed human control conditions.

Fig. 2: Schematic representation of the haptic shared control setting.

Panel of Figure 3 — Fig. 3: Task scenarios with spherical, cubic, and rigid nonconvex obstacles.

Parameter	Value
Learned-policy force limit $f_{\max}$	$0.04\,\mathrm{N/axis}$
Fixed impedance limit $f_{\mathrm{imp}}^{\max}$	$0.03\,\mathrm{N/axis}$
Simulation frequency	$120\,\mathrm{Hz}$
Control frequency	$60\,\mathrm{Hz}$
Episode horizon	$1200\,\mathrm{steps}$
Completion radius	$0.01\,\mathrm{m}$

TABLE I: Key Environment Parameters

Region	Symbol	Description	Team reward	Epi. task
Zone A	$r_{\mathrm{prog}}$	Progress reward	✓	✓
Zone A	$r_{\mathrm{dev}}$	Deviation penalty	✓	✓
Zone B	$r_{\mathrm{gap}}$	Gap penalty	✓	✓
Zone B	$r_{\mathrm{surf}}$	Surface-tangential reward	✓	✓
Zone B	$r_{\mathrm{inw},B}$	Inward-motion penalty	✓	$\times$
Zone C	$r_{\mathrm{off}}$	Danger-region penalty	✓	$\times$

TABLE II: Task Terms Under the Unified Region Partition

Fig. 4: Four-zone geometric partition around the obstacle.

Panel of Figure 7 — Fig. 5: Training curves of normalized composite score $\bar{R}$ : learned sphere (a), learned cube (b), fixed sphere (c), and fixed nonconvex obstacle (d).

Method	$P_{\mathrm{fin}}\uparrow$	Steps $\downarrow$	$\rho_{\mathrm{track}}\uparrow$	$\bar{R}\uparrow$	$\rho_{C}\downarrow$	$\rho_{\mathrm{col}}\downarrow$
Learned condition
MADDPG	0.88	1200	0.38	266.19	0.01	0.00
rMAPPO	0.97	1200	0.61	353.19	0.00	0.00
Epigraph	0.92	1200	0.03	119.90	0.00	0.00
Fixed impedance condition
MADDPG	0.97	1200	0.88	452.21	0.00	0.00

TABLE III: Performance in the Spherical Obstacle Scenario under Learned and Fixed Human Control Conditions

Method	$P_{\mathrm{fin}}\uparrow$	Steps $\downarrow$	$\rho_{\mathrm{track}}\uparrow$	$\bar{R}\uparrow$	$\rho_{C}\downarrow$	$\rho_{\mathrm{col}}\downarrow$
MADDPG	0.995	1112	0.368	208.57	0.012	0.00
rMAPPO	0.972	1200	0.381	307.50	0.023	0.00
Epigraph	0.922	1200	0.076	91.22	0.00	0.00

TABLE IV: Learned Condition Performance in the Cubic Obstacle Scenario

Panel of Figure 10 — Fig. 6: Learned condition trajectories for the spherical and cubic obstacles.

Method	$P_{\mathrm{fin}}\uparrow$	Steps $\downarrow$	$\rho_{\mathrm{track}}\uparrow$	$\bar{R}\uparrow$	$\rho_{C}\downarrow$	$\rho_{\mathrm{col}}\downarrow$
MADDPG	0.596	1200	0.332	123.99	0.00	0.00
rMAPPO	0.956	1200	0.304	293.51	0.051	0.011
Epigraph	0.934	1200	0.028	127.19	0.00	0.00

TABLE V: Generalization Performance under the Learned Condition: Trained on Sphere, Tested on Shifted Cube

Task behavior Method	$P_{\mathrm{fin}}\uparrow$	$\rho_{\mathrm{track}}\uparrow$	Traj. RMSE (mm) $\downarrow$
MADDPG	$0.576\pm 0.353$	$\mathbf{0.878\pm 0.142}$	$11.26\pm 8.64$
rMAPPO	$\mathbf{0.996\pm 0.003}$	$0.796\pm 0.140$	$\mathbf{7.87\pm 0.64}$
Epigraph	$0.995\pm 0.001$	$0.359\pm 0.222$	$25.77\pm 4.44$

TABLE VI: Performance on the Rigid Nonconvex Geometry

Panel of Figure 13 — Fig. 7: Rigid nonconvex and sphere to cube transfer trajectories; red segments indicate collision.

Panel of Figure 14 — Fig. 8: Force comparison across MADDPG, rMAPPO, and Epigraph over one representative spherical obstacle episode under the learned condition. The left column shows the $y$ -axis force components of robot, human, and their combined force. The right column shows the robot force magnitude, human force magnitude, and the magnitude of the combined human-robot force.

Fig. 9: Real operator setup coupling the Phantom Omni input device to the Isaac Sim environment.

Method	Training condition	Success $\uparrow$	Steps $\downarrow$	$\rho_{\mathrm{track}}\uparrow$	$\rho_{C}\downarrow$	$\rho_{\mathrm{col}}\downarrow$	Device-to-EEF tracking RMSE (mm) $\downarrow$
MADDPG	Fixed	$\mathbf{1.00\pm 0.00}$	$551\pm 45$	$0.226\pm 0.121$	$0.0369\pm 0.0795$	$0.0194\pm 0.0428$	$29.9\pm 10.7$
MADDPG	Learned	$0.90\pm 0.32$	$787\pm 185$	$0.101\pm 0.074$	$0.0110\pm 0.0208$	$\mathbf{0.0000\pm 0.0000}$	$79.6\pm 17.1$
rMAPPO	Fixed	$\mathbf{1.00\pm 0.00}$	$625\pm 90$	$\mathbf{0.238\pm 0.157}$	$0.0414\pm 0.0698$	$0.0206\pm 0.0486$	$\mathbf{17.4\pm 5.0}$
rMAPPO	Learned	$0.70\pm 0.48$	$899\pm 216$	$\mathbf{0.311\pm 0.083}$	$0.0366\pm 0.0693$	$0.0038\pm 0.0112$	$\mathbf{29.8\pm 4.1}$
Epigraph	Fixed	$\mathbf{1.00\pm 0.00}$	$\mathbf{543\pm 84}$	$0.197\pm 0.043$	$\mathbf{0.0018\pm 0.0056}$	$\mathbf{0.0000\pm 0.0000}$	$57.0\pm 32.0$
Epigraph	Learned	$\mathbf{1.00\pm 0.00}$	$\mathbf{668\pm 172}$	$0.207\pm 0.057$	$\mathbf{0.0097\pm 0.0221}$	$\mathbf{0.0000\pm 0.0000}$	$52.1\pm 15.8$

TABLE VII: Human in the Loop Task and Safety Performance

Method	Training condition	Human share $\eta_{H}$	Mean cosine $\bar{\kappa}$	Opposing ratio $\rho_{\mathrm{opp}}$	Cancellation $\chi$
MADDPG	Fixed	$0.398\pm 0.030$	$-0.055\pm 0.103$	$0.592\pm 0.065$	$0.305\pm 0.034$
MADDPG	Learned	$0.483\pm 0.008$	$-0.740\pm 0.126$	$\mathbf{0.898\pm 0.082}$	$0.743\pm 0.084$
rMAPPO	Fixed	$0.525\pm 0.034$	$-0.317\pm 0.161$	$0.688\pm 0.112$	$0.421\pm 0.092$
rMAPPO	Learned	$0.509\pm 0.007$	$-0.951\pm 0.022$	$0.997\pm 0.003$	$0.858\pm 0.029$
Epigraph	Fixed	$0.523\pm 0.026$	$\mathbf{\phantom{-}0.314\pm 0.182}$	$\mathbf{0.260\pm 0.121}$	$\mathbf{0.178\pm 0.056}$
Epigraph	Learned	$0.634\pm 0.040$	$\mathbf{-0.659\pm 0.067}$	$0.939\pm 0.038$	$\mathbf{0.539\pm 0.058}$

TABLE VIII: Human in the Loop Force Allocation

Explore similar work

Learning Personalized Safety Interventions for Haptic Human-Robot Shared Control
Jul 21, 2026Dawei Zhang, Roberto TronRobotic Control Haptic Feedback
Division of Systems Engineering, Boston University, 110 Cummington Mall, Boston, 02215, MA, USA.
SSP: Safety-guaranteed Surgical Policy via Joint Optimization of Behavioral and Spatial Constraints
Mar 7, 2026Jianshu Hu, ZhiYuan Guan, Lei Song +5Robot Safety Surgical Robotics
Global College, Shanghai Jiao Tong University, Shanghai, China · Department of Computer Science and Engineering, The Chinese University of Hong Kong, HongKong, China · Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Massachusetts, America
Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Control
Jun 22, 2026Zihao Guo, Jianing Zhao, Ling Li +3Robotic Control Multi-Agent Coordination
1King’s College London · University of California, Berkeley · 3The Alan Turing Institute

Jul 21, 2026J/K move · Enter open · S save

Learning Personalized Safety Interventions for Haptic Human-Robot Shared Control

Dawei Zhang, Roberto Tron

Division of Systems Engineering, Boston University, 110 Cummington Mall, Boston, 02215, MA, USA.

Robotic Control Haptic Feedback Robot Safety Human-Robot Collaboration Learning from Demonstration

Open paper AskPDF arXiv

Haptic feedback provides an implicit channel for communicating safety intentions during human-robot shared control. Existing haptic guidance systems typically employ predefined intervention strategies that cannot accommodate the diverse safety preferences of individual users or application scenarios. To address this limitation, we propose a Learning from Haptics (LfH) framework that learns user-preferred safety interventions from sparse demonstrations, eliminating the need for manual trial-and-error design. Our framework is built on a differentiable Control Barrier Function (CBF)-based optimization layer that automatically adjusts the underlying safety parameters to match the demonstrated haptic responses. Instead of tuning controller parameters directly, users teach the system how they expect it to intervene during teleoperation. The resulting haptic guidance reflects the demonstrated intervention preferences while preserving the intuitive interaction of haptic shared control. Simulation and hardware experiments demonstrate that the proposed framework can learn personalized safety interventions from sparse user input and reduce the mismatch between the generated haptic feedback and the demonstrated preferences.

Abstract

Explore similar work

Learning Personalized Safety Interventions for Haptic Human-Robot Shared Control

SSP: Safety-guaranteed Surgical Policy via Joint Optimization of Behavioral and Spatial Constraints

Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Control