SimRetract: Zero-Shot Sim-to-Real Robotic Assistance for Retractor Take-Over in Open Surgery

Xucheng Wang; Zhizhou Yang; Sung Eun Kim; Romain Hardy; Michael P. Sampson; Aakash Reddy; Ankit Pal; Xiaoman Zhang

cs.ROOct 28, 2026

SimRetract: Zero-Shot Sim-to-Real Robotic Assistance for Retractor Take-Over in Open Surgery

Authors: Xucheng Wang, Zhizhou Yang, Sung Eun Kim, Romain Hardy, Michael P. Sampson, Aakash Reddy, Ankit Pal, Xiaoman Zhang,

Organizations: Department of Biomedical Informatics, Harvard Medical School · Department of Surgery, Massachusetts General Hospital · Weill Cornell Medical College · Department of Biostatistics, Yale School of Public Health

Abstract

We demonstrate the first zero-shot sim-to-real transfer for a surgeon-robot collaboration task in open surgery: retractor take-over. A robot arm takes a retractor from the surgeon's hand and preserves retraction at handoff, using a policy trained only on simulated demonstrations, without human demonstrations or real-world fine-tuning. Imitation learning from real demonstrations is limited by collection and labeling costs, and by performance degradation when the environment changes. Our framework, SimRetract, generates simulated demonstrations designed for transfer: the policy observes masked instrument images and grasp geometry, learns to recover from off-nominal states, and acts with margin for execution drift. Deployed zero-shot, it grasps the retractor in 90% of 50 surgeon-defined phantom presentations and preserves retraction at handoff in 78%. Results on unseen retractors suggest potential for generalization to instruments with similar grasp geometry. Across ten policy configurations and 1,200 simulated and real trials, we localize the sim-to-real gap to components: in simulation, grasp rates across ablations span 80-96%, while on the real robot removing the instrument mask alone drops performance to 22% grasped and 12% held. Transfer-oriented design thus makes zero-shot sim-to-real transfer feasible for a surgical assistance task without human demonstrations. Code is available here: https://anonymous.4open.science/r/simretract-58D2.

Figures & tables

Figure 1: Overview of SimRetract . (A) A scripted expert generates complete approach and closure trajectories and retains physically validated demonstrations, including perturbation-recovery episodes. (B) Training inputs combine masked RGB, metric depth, robot joint state and grasp-plane normal, with perception augmentation. (C) Sampled observation–action pairs train $\pi_{0}$ to predict chunks of joint and gripper targets. (D) The same checkpoint is evaluated in simulation and deployed on the real robot without fine-tuning. Real outcomes at handoff are held, grasped only, or missed.

Figure 2: Real experimental setup and instruments. (A) The arm beside the draped abdominal phantom. (B) The wrist-mounted RGB-D camera and parallel gripper. (C) Left to right: Richardson and Army–Navy, both unseen during training, and the Volkmann-type training retractor.

	Simulation							Real robot
Model	Grasped (%)	Offset (mm)			Twist ( ∘ )			Grasped (%) ( $-$ pp vs. sim)	Held (%)
SimRetract (full)	94	8.0 $\pm$	6.0	[6.7]	3.8 $\pm$	6.4	[2.1]	90 ( $-$ 4)	78
w/o dropout	96	9.9 $\pm$	7.5	[8.1]	5.9 $\pm$	11.5	[2.5]	88 ( $-$ 8)	68
w/o recovery	80	15.6 $\pm$	21.0	[7.1]	6.0 $\pm$	8.8	[4.8]	78 ( $-$ 2)	34
half data (250 + 500)	82	13.1 $\pm$	13.6	[9.5]	6.8 $\pm$	8.2	[3.1]	58 ( $-$ 24)	18
w/o plane normal	84	20.0 $\pm$	21.2	[12.0]	10.9 $\pm$	18.3	[3.9]	76 ( $-$ 8)	58

Table 1: Simulation and real-robot results (50 presentations each; outcomes in § 4.2 – 4.3 ). Variants are defined in § 4.1 : w/o = without; half-data counts denote expert + recovery episodes. Offset and twist are post-release tip displacement and axial rotation-vector error over grasped simulated trials: mean

\pm

SD [median]. Parentheses give descriptive real-minus-simulation grasp-rate differences (pp). Held denotes preserved retraction at handoff; paired comparisons are in Table S6 .

Figure 3: Generalization to unseen retractors (50 presentations each): grasped and held rates at handoff for the full model and the no-plane ablation.

Figure 4: Failure types in real trials (50 per model; bar labels: missed/evaluated). Interference is pooled; contact locations are in Table S3 . Grasped-only trials are excluded.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Simulated demonstration and real rollout across five stages. Inputs include masked RGB, metric depth, plane normal and joint state. Plane overlays show the fitted plane (cyan) and normal (yellow). Simulation inputs precede augmentation. Real RGB-D matches recorded queries. Plane overlay illustrations are recomputed from recorded RGB-D. Input panels end before release; the simulated hand proxy remains visible.

Figure S2: Instrument-pose sampling domain relative to the robot. The green box bounds the proposed instrument centers; red lines show example shaft positions and orientations. The shaft can extend outside the center-sampling box.

Figure S3: TCP paths reconstructed from joint states for the same presentation and grasp. SimRetract (teal): smooth curve, then a 3 cm aligned straight entry; w/o smoothing (orange): straight reach and aligned entry; w/o standoff (red): curve directly to the grasp.

Optimizer	AdamW
Peak learning rate	$2.5\times 10^{-5}$
Schedule	1 k warm-up, cosine decay
Effective batch size	64 ( $16\times 4$ accumulation)
Optimizer updates	15 000
Sampled training pairs	960 000
Action chunk	50 targets (2 s at 25 Hz)

Appendix

Table S1: Shared policy-training hyperparameters. Sample counts include repeated draws from each model’s training pool.

Component	Parameters
Clean draw	$p=0.15$
Severity $m$	$0.4$ ( $p{=}0.40$ ), $0.8$ ( $0.35$ ), $1.4$ ( $0.10$ ), $1.0$ (rest)
Opening fill	$p=0.45$ (enclosed holes filled)
Dilate/erode	ellipse $k_{x},k_{y}=1+\lfloor\mathcal{U}\{1,2,3\}\cdot 2m\rfloor$ ; dilate $p{=}0.65$
Boundary wobble	SDF noise amplitude $\mathcal{U}(2,7)\,m$ px, feature size 10–45 px
False-positive blobs	$0..1{+}\lfloor 3m\rfloor$ blobs on the boundary, radius $\mathcal{U}(4,12)(0.5{+}m)$

Appendix

Table S2: Training-time perception augmentation. Ranges are uniform unless stated;

m

denotes severity.

Training retractor
Model	Inside	Outside	Hand	Phantom	Total
SimRetract	0	0	1	0	1
w/o dropout	3	0	0	0	3
w/o recovery	1	1	0	1	3
half data	6	3	0	0	9
w/o plane	2	1	0	0	3

Appendix

Table S3: Interference counts by contact location, with 50 trials per condition. Each interference miss receives one subtype.

Model	Time to grasp (s)	Policy queries (count)
SimRetract	10.08 $\pm$ 1.69 [10.10]	3.82 $\pm$ 0.49 [4]
SimRetract , Army–Navy	12.41 $\pm$ 2.59 [12.30]	3.90 $\pm$ 0.71 [4]
SimRetract , Richardson	8.66 $\pm$ 1.10 [9.09]	3.07 $\pm$ 0.26 [3]
w/o dropout	9.59 $\pm$ 1.62 [9.60]	3.75 $\pm$ 0.49 [4]
w/o recovery	9.00 $\pm$ 1.69 [8.22]	3.56 $\pm$ 0.55 [4]
half data	10.42 $\pm$ 1.93 [10.56]	3.79 $\pm$ 0.56 [4]

Appendix

Table S4: Real-robot timing at default playback speed on grasped trials: mean

\pm

SD [median]. Mean component latencies on an RTX 5090 are 86.6 ms for

\pi_{0}

requests, 77.6 ms for SmolVLA requests, 152 ms for SAM 3 first-frame detection, and 160 ms for asynchronous tracking (median 98 ms). ACT inference takes approximately 10 ms per 50-action chunk.

	SmolVLA	ACT
Sim. grasped	80 %	52 %
Offset (mm)	20.8 $\pm$ 17.2 [16.2]	12.3 $\pm$ 10.3 [9.8]
Twist ( ∘ )	6.2 $\pm$ 10.0 [3.2]	6.4 $\pm$ 11.5 [3.9]
Real grasped	74 %	48 %
Real held	54 %	16 %

Appendix

Table S5: Additional policy architectures, with 50 trials per domain. Offset and twist over grasped simulated trials (Appendix C.2 ) are mean

\pm

SD, with medians beneath in brackets.

Ablation	Held difference [95 % CI], pp	$p_{\rm Holm}$
w/o dropout	$+10\;[-8,28]$	0.383
w/o recovery	$+44\;[26,60]$	$<0.001$
half data	$+60\;[42,76]$	$<0.001$
w/o plane	$+20\;[2,38]$	0.128
w/o mask	$+66\;[52,80]$	$<0.001$
w/o smoothing	$+36\;[16,56]$	0.006

Appendix

Table S6: Paired held success: full model minus each ablation, in percentage points (50 presentations). Brackets show unadjusted 95 % paired bootstrap intervals; exact McNemar

p

values are Holm-adjusted across seven comparisons.

Surgical Robotics

Surgical robotics develops robotic tools and systems for minimally invasive, image-guided, and microsurgical procedures.

See all 189 papers

Imitation Learning for Robot Assistance in Open Surgery: A Multi-Policy Evaluation on Suture Following
May 27, 2026Xucheng Wang, Zhizhou Yang, Xiaoman Zhang +3Visuomotor Policy Learning Robot Imitation Learning
Department of Biomedical Informatics, Harvard Medical School, Boston, MA · Department of Surgery, Massachusetts General Hospital, Boston, MA
SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation
Feb 18, 2026Kushal Kedia, Tyler Ga Wei Lum, Jeannette Bohg +1Zero-Shot Robotic Manipulation Robot Skill Learning
Cornell University · Stanford University
MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation
Mar 17, 2026Abhay Deshpande, Maya Guru, Rose Hendrix +23Sim-to-Real Transfer Zero-Shot Robotic Manipulation
Allen Institute for AI · Technische Universität Darmstadt · University of Washington +2

May 27, 2026J/K move · Enter open · S save

Imitation Learning for Robot Assistance in Open Surgery: A Multi-Policy Evaluation on Suture Following

Xucheng Wang, Zhizhou Yang, Xiaoman Zhang, Sung Eun Kim+2

Department of Biomedical Informatics, Harvard Medical School, Boston, MA · Department of Surgery, Massachusetts General Hospital, Boston, MA

Visuomotor Policy Learning Robot Imitation Learning Surgical Robotics

Open paper AskPDF arXiv

This study presents the first evaluation of general-purpose imitation learning for surgeon-robot collaborative assistance in open surgery, targeting suture following: the grab-pull-release motion an assistant performs at every stitch. We collect 160 teleoperated demonstrations (32,374 frames) on an open-source robot arm, benchmark four architecturally diverse imitation learning policies (ACT, Diffusion Policy, SmolVLA,

π_0

) across 28 trained models evaluated in 32 configurations along three clinically motivated dimensions: dataset size, camera viewpoint, and background variation. Our results demonstrate that under ideal conditions, the four policies achieve

50

75\%

task success, with depth error as the dominant failure mode across all architectures. Among all policies,

π_0

achieves the strongest results with a pretrained vision-language backbone, demonstrating superior data efficiency, greater robustness to background variation, and smoother trajectories compatible with surgical workflow. When deployed in a surgeon-robot suturing trial,

π_0

yields a

92\%

stitch completion rate. These findings establish collaborative robotic assistance in open surgery as a feasible target for imitation learning and highlight depth perception and end-effector design as key priorities for clinical translation.

Abstract

Surgical Robotics

Explore similar work

Imitation Learning for Robot Assistance in Open Surgery: A Multi-Policy Evaluation on Suture Following

SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation

MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation