A Reinforcement Learning Framework for Safety-Aware Haptic Shared Control in Surgical Robotics

Zihao Zhang; Karl Handwerker; Balint Varga

cs.ROOct 11, 2026

A Reinforcement Learning Framework for Safety-Aware Haptic Shared Control in Surgical Robotics

Authors: Zihao Zhang, Karl Handwerker, Balint Varga

Organizations: Institute of Control Systems, Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany

Abstract

Haptic shared control in surgical robotics combines human-side and robot-side forces at a shared end effector while preserving operator involvement. However, the effects of geometric safety choices on task execution and force coordination remain unclear. We model the interaction as a two-agent cooperative Markov game under centralized training and decentralized execution, comparing learned human control with a task-tuned fixed-impedance baseline. This model-free multi-agent reinforcement learning framework supports joint policy adaptation without requiring a complete analytical interaction model. It compares MADDPG and rMAPPO with an epigraph-constrained approach under common geometric task definitions. In simulation, rMAPPO generally achieved the highest return and tracking performance, whereas the epigraph approach produced more conservative motions and lower geometric exposure, including no observed collision or dangerous-region exposure after geometry transfer. With real operator input, fixed-impedance-trained policies were faster and successful in all recorded trials. Policies trained with a learned partner exhibited greater command opposition, while the fixed epigraph condition produced the most aligned commands. This work provides a benchmark for studying human-control representation and constraint-handling trade-offs in haptic shared control.

Figures & tables

Fig. 1: Haptic shared control with region partitioned reward and constraint design under learned and fixed human control conditions.

Fig. 2: Schematic representation of the haptic shared control setting.

Panel of Figure 3 — Fig. 3: Task scenarios with spherical, cubic, and rigid nonconvex obstacles.

Parameter	Value
Learned-policy force limit $f_{\max}$	$0.04\,\mathrm{N/axis}$
Fixed impedance limit $f_{\mathrm{imp}}^{\max}$	$0.03\,\mathrm{N/axis}$
Simulation frequency	$120\,\mathrm{Hz}$
Control frequency	$60\,\mathrm{Hz}$
Episode horizon	$1200\,\mathrm{steps}$
Completion radius	$0.01\,\mathrm{m}$

TABLE I: Key Environment Parameters

Region	Symbol	Description	Team reward	Epi. task
Zone A	$r_{\mathrm{prog}}$	Progress reward	✓	✓
Zone A	$r_{\mathrm{dev}}$	Deviation penalty	✓	✓
Zone B	$r_{\mathrm{gap}}$	Gap penalty	✓	✓
Zone B	$r_{\mathrm{surf}}$	Surface-tangential reward	✓	✓
Zone B	$r_{\mathrm{inw},B}$	Inward-motion penalty	✓	$\times$
Zone C	$r_{\mathrm{off}}$	Danger-region penalty	✓	$\times$

TABLE II: Task Terms Under the Unified Region Partition

Fig. 4: Four-zone geometric partition around the obstacle.

Panel of Figure 7 — Fig. 5: Training curves of normalized composite score $\bar{R}$ : learned sphere (a), learned cube (b), fixed sphere (c), and fixed nonconvex obstacle (d).

Method	$P_{\mathrm{fin}}\uparrow$	Steps $\downarrow$	$\rho_{\mathrm{track}}\uparrow$	$\bar{R}\uparrow$	$\rho_{C}\downarrow$	$\rho_{\mathrm{col}}\downarrow$
Learned condition
MADDPG	0.88	1200	0.38	266.19	0.01	0.00
rMAPPO	0.97	1200	0.61	353.19	0.00	0.00
Epigraph	0.92	1200	0.03	119.90	0.00	0.00
Fixed impedance condition
MADDPG	0.97	1200	0.88	452.21	0.00	0.00

TABLE III: Performance in the Spherical Obstacle Scenario under Learned and Fixed Human Control Conditions

Method	$P_{\mathrm{fin}}\uparrow$	Steps $\downarrow$	$\rho_{\mathrm{track}}\uparrow$	$\bar{R}\uparrow$	$\rho_{C}\downarrow$	$\rho_{\mathrm{col}}\downarrow$
MADDPG	0.995	1112	0.368	208.57	0.012	0.00
rMAPPO	0.972	1200	0.381	307.50	0.023	0.00
Epigraph	0.922	1200	0.076	91.22	0.00	0.00

TABLE IV: Learned Condition Performance in the Cubic Obstacle Scenario

Panel of Figure 10 — Fig. 6: Learned condition trajectories for the spherical and cubic obstacles.

Method	$P_{\mathrm{fin}}\uparrow$	Steps $\downarrow$	$\rho_{\mathrm{track}}\uparrow$	$\bar{R}\uparrow$	$\rho_{C}\downarrow$	$\rho_{\mathrm{col}}\downarrow$
MADDPG	0.596	1200	0.332	123.99	0.00	0.00
rMAPPO	0.956	1200	0.304	293.51	0.051	0.011
Epigraph	0.934	1200	0.028	127.19	0.00	0.00

TABLE V: Generalization Performance under the Learned Condition: Trained on Sphere, Tested on Shifted Cube

Task behavior Method	$P_{\mathrm{fin}}\uparrow$	$\rho_{\mathrm{track}}\uparrow$	Traj. RMSE (mm) $\downarrow$
MADDPG	$0.576\pm 0.353$	$\mathbf{0.878\pm 0.142}$	$11.26\pm 8.64$
rMAPPO	$\mathbf{0.996\pm 0.003}$	$0.796\pm 0.140$	$\mathbf{7.87\pm 0.64}$
Epigraph	$0.995\pm 0.001$	$0.359\pm 0.222$	$25.77\pm 4.44$

TABLE VI: Performance on the Rigid Nonconvex Geometry

Panel of Figure 13 — Fig. 7: Rigid nonconvex and sphere to cube transfer trajectories; red segments indicate collision.

Panel of Figure 14 — Fig. 8: Force comparison across MADDPG, rMAPPO, and Epigraph over one representative spherical obstacle episode under the learned condition. The left column shows the $y$ -axis force components of robot, human, and their combined force. The right column shows the robot force magnitude, human force magnitude, and the magnitude of the combined human-robot force.

Fig. 9: Real operator setup coupling the Phantom Omni input device to the Isaac Sim environment.

Method	Training condition	Success $\uparrow$	Steps $\downarrow$	$\rho_{\mathrm{track}}\uparrow$	$\rho_{C}\downarrow$	$\rho_{\mathrm{col}}\downarrow$	Device-to-EEF tracking RMSE (mm) $\downarrow$
MADDPG	Fixed	$\mathbf{1.00\pm 0.00}$	$551\pm 45$	$0.226\pm 0.121$	$0.0369\pm 0.0795$	$0.0194\pm 0.0428$	$29.9\pm 10.7$
MADDPG	Learned	$0.90\pm 0.32$	$787\pm 185$	$0.101\pm 0.074$	$0.0110\pm 0.0208$	$\mathbf{0.0000\pm 0.0000}$	$79.6\pm 17.1$
rMAPPO	Fixed	$\mathbf{1.00\pm 0.00}$	$625\pm 90$	$\mathbf{0.238\pm 0.157}$	$0.0414\pm 0.0698$	$0.0206\pm 0.0486$	$\mathbf{17.4\pm 5.0}$
rMAPPO	Learned	$0.70\pm 0.48$	$899\pm 216$	$\mathbf{0.311\pm 0.083}$	$0.0366\pm 0.0693$	$0.0038\pm 0.0112$	$\mathbf{29.8\pm 4.1}$
Epigraph	Fixed	$\mathbf{1.00\pm 0.00}$	$\mathbf{543\pm 84}$	$0.197\pm 0.043$	$\mathbf{0.0018\pm 0.0056}$	$\mathbf{0.0000\pm 0.0000}$	$57.0\pm 32.0$
Epigraph	Learned	$\mathbf{1.00\pm 0.00}$	$\mathbf{668\pm 172}$	$0.207\pm 0.057$	$\mathbf{0.0097\pm 0.0221}$	$\mathbf{0.0000\pm 0.0000}$	$52.1\pm 15.8$

TABLE VII: Human in the Loop Task and Safety Performance

Method	Training condition	Human share $\eta_{H}$	Mean cosine $\bar{\kappa}$	Opposing ratio $\rho_{\mathrm{opp}}$	Cancellation $\chi$
MADDPG	Fixed	$0.398\pm 0.030$	$-0.055\pm 0.103$	$0.592\pm 0.065$	$0.305\pm 0.034$
MADDPG	Learned	$0.483\pm 0.008$	$-0.740\pm 0.126$	$\mathbf{0.898\pm 0.082}$	$0.743\pm 0.084$
rMAPPO	Fixed	$0.525\pm 0.034$	$-0.317\pm 0.161$	$0.688\pm 0.112$	$0.421\pm 0.092$
rMAPPO	Learned	$0.509\pm 0.007$	$-0.951\pm 0.022$	$0.997\pm 0.003$	$0.858\pm 0.029$
Epigraph	Fixed	$0.523\pm 0.026$	$\mathbf{\phantom{-}0.314\pm 0.182}$	$\mathbf{0.260\pm 0.121}$	$\mathbf{0.178\pm 0.056}$
Epigraph	Learned	$0.634\pm 0.040$	$\mathbf{-0.659\pm 0.067}$	$0.939\pm 0.038$	$\mathbf{0.539\pm 0.058}$

TABLE VIII: Human in the Loop Force Allocation

Explore similar work

Jul 21, 2026

Learning Personalized Safety Interventions for Haptic Human-Robot Shared Control

Haptic feedback provides an implicit channel for communicating safety intentions during human-robot shared control. Existing haptic guidance systems typically employ predefined intervention strategies that cannot accommodate the diverse safety preferences of individual users or application scenarios. To address this limitation, we propose a Learning from Haptics (LfH) framework that learns user-preferred safety interventions from sparse demonstrations, eliminating the need for manual trial-and-error design. Our framework is built on a differentiable Control Barrier Function (CBF)-based optimization layer that automatically adjusts the underlying safety parameters to match the demonstrated haptic responses. Instead of tuning controller parameters directly, users teach the system how they expect it to intervene during teleoperation. The resulting haptic guidance reflects the demonstrated intervention preferences while preserving the intuitive interaction of haptic shared control. Simulation and hardware experiments demonstrate that the proposed framework can learn personalized safety interventions from sparse user input and reduce the mismatch between the generated haptic feedback and the demonstrated preferences.

Dawei Zhang, Roberto Tron

Division of Systems Engineering, Boston University, 110 Cummington Mall, Boston, 02215, MA, USA.

Robotic Control Haptic Feedback Robot Safety Human-Robot Collaboration Learning from Demonstration

Ask PDF

Mar 7, 2026

SSP: Safety-guaranteed Surgical Policy via Joint Optimization of Behavioral and Spatial Constraints

The paradigm of robot-assisted surgery is shifting toward data-driven autonomy, where policies learned via Reinforcement Learning (RL) or Imitation Learning (IL) enable the execution of complex tasks. However, these ``black-box" policies often lack formal safety guarantees, a critical requirement for clinical deployment. In this paper, we propose the Safety-guaranteed Surgical Policy (SSP) framework to bridge the gap between data-driven generality and formal safety. We utilize Neural Ordinary Differential Equations (Neural ODEs) to learn an uncertainty-aware dynamics model from demonstration data. This learned model underpins a robust Control Barrier Function (CBF) safety controller, which minimally alters the actions of a surgical policy to ensure strict safety under uncertainty. Our controller enforces two constraint categories: behavioral constraints (restricting the task space of the agent) and spatial constraints (defining surgical no-go zones). We instantiate the SSP framework with surgical policies derived from RL, IL and Control Lyapunov Functions (CLF). Validation on in both the SurRoL simulation and da Vinci Research Kit (dVRK) demonstrates that our method achieves a near-zero constraint violation rate while maintaining high task success rates compared to unconstrained baselines.

Jianshu Hu, ZhiYuan Guan, Lei Song +5

Global College, Shanghai Jiao Tong University, Shanghai, China · Department of Computer Science and Engineering, The Chinese University of Hong Kong, HongKong, China · Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Massachusetts, America

Robot Safety Surgical Robotics Control Barrier Functions

Ask PDF

Jun 22, 2026

Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Control

Multi-agent systems are widely used in safety-critical applications that require coordinated behavior under strict safety constraints. Existing approaches face a fundamental trade-off: learning-based methods achieve strong empirical performance but lack theoretical safety guarantees, while control-theoretic methods enforce safety but often lead to overly conservative and inefficient behaviors. We propose a hierarchical multi-agent reinforcement learning framework that enforces hard safety constraints under mild assumptions at low level via a constraint manifold, while enabling effective coordination through high-level policy learning. Our approach provides theoretical safety guarantees in the multi-agent setting and yields stationary learning dynamics, thereby enabling stable and efficient training. Empirically, our method achieves competitive performance while maintaining nearly perfect safety rates, and generalizes effectively to varying numbers of agents and obstacles.

Zihao Guo, Jianing Zhao, Ling Li +3

1King’s College London · University of California, Berkeley · 3The Alan Turing Institute

Robotic Control Multi-Agent Coordination Hierarchical RL Cooperative MARL Multi-Agent Reinforcement Learning

Ask PDF