SurgVIL: Scaling Surgical Robot Imitation Learning with Open-source Surgical Videos

Xinhao Chen; JuoTung Chen; Nigel Nelson; Antony Goldenberg; Jesse Haworth; Sean D. Huver; Axel Krieger

cs.ROAug 17, 2026

SurgVIL: Scaling Surgical Robot Imitation Learning with Open-source Surgical Videos

Authors: Xinhao Chen, JuoTung Chen, Nigel Nelson, Antony Goldenberg, Jesse Haworth, Sean D. Huver, Axel Krieger

Organizations: Laboratory for Computational Sensing and Robotics (LCSR) Johns Hopkins University, USA · Nvidia, USA

Abstract

Learning-based surgical robot autonomy requires large-scale demonstrations with synchronized videos and robot actions, but such data are exceedingly rare in clinical or realistic tissue settings because robot kinematics are typically inaccessible outside controlled research systems. In contrast, phantom data collected on research platforms provide accurate action labels but lack the visual diversity of real tissue. We propose SurgVIL, a framework for scaling surgical robot imitation learning using open-source surgical videos. SurgVIL combines kinematically labeled phantom robot demonstrations with surgical videos from open-source datasets and online sources for policy learning. Since these videos lack robot motion labels, we estimate approximate kinematics as weak supervision. We evaluate SurgVIL on two da Vinci robot tasks: needle pick-up and cholecystectomy cutting. Across ACT, $π_0$ , and GR00T-H backbones, adding surgical videos substantially improves generalization to real-tissue and out-of-distribution settings, suggesting a scalable path from phantom training toward generalizable surgical robot policies.

Figures & tables

Figure 1: Overview of SurgVIL. The proposed framework combines accurately labeled in-house dataset collected in phantom settings with visually diverse open-source surgical videos. The trained policies are evaluated on real-tissue surgical tasks to assess generalization beyond the phantom.

Figure 2: Flow chart for the dataset preparation, dual-source policy training and evaluation

Figure 3: Experimental Setup. The evaluation was conducted on a dVRK-Si robot system, with an additional wrist camera mounted on the surgical instrument.

Figure 4: Needle task results on (a) Suture pad, (b) Porcine liver and gallbladder, and (c) Chicken breast with fake blood. The results are also listed as table in the Appendix.

Figure 5: Chole task results on (a) Phantom, (b) Normal ex vivo tissue, and (c) abnormal ex vivo tissue. The results are also listed as table in the Appendix.

Parameter	ACT for needle task	ACT for chole task
Model / initialization	ACT	ACT
Optimizer	AdamW	AdamW
Learning rate	$5\times 10^{-4}$	$5\times 10^{-4}$
Batch size	256	256
Adam $\beta_{1}$	0.9	0.9
Adam $\beta_{2}$	0.999	0.999

ACT hyperparameters

Parameter	$\pi_{0}$ for needle task	$\pi_{0}$ for chole task
Model / initialization	$\pi_{0}$	$\pi_{0}$
Optimizer	AdamW	AdamW
Learning rate	$1\times 10^{-4}$	$1\times 10^{-4}$
Batch size	128	128
Adam $\beta_{1}$	0.9	0.9
Adam $\beta_{2}$	0.95	0.95

\pi_{0}

fine-tuning hyperparameters

Parameter	GR00T-H for needle task	GR00T-H for chole task
Model / initialization	GR00T-H	GR00T-H
Optimizer	AdamW	AdamW
Learning rate	$2\times 10^{-5}$	$2\times 10^{-5}$
Batch size	64	64
Weight decay	$1\times 10^{-5}$	$1\times 10^{-5}$
Learning-rate schedule	Cosine decay	Cosine decay

GR00T-H fine-tuning hyperparameters

Parameter	Description	Value
$\lambda_{\mathrm{ssim}}$	Weight for SSIM image similarity loss	$0.5$
$\lambda_{\mathrm{dice}}$	Weight for Dice mask loss	$1.0$
$\lambda_{\mathrm{temp}}$	Weight for temporal consistency loss	$0.05$
$\epsilon$	Numerical stability constant in $\rho(\cdot)$	$1\times 10^{-6}$

Hyperparameters used in the video-to-action labeling loss.

Method	Position Error (mm)	Orientation Error ( ∘ )
SurgiPose [ 3 ]	$12.0$	Not reported
TUDU [ 31 ]	$5.8$	$21.48$
IGTUM [ 31 ]	5.4	$10.71$
UOL [ 31 ]	$8.3$	$17.73$
FoundPose [ 20 ]	$20.7$	$55.01$
MegaPose [ 12 ]	$13.4$	$44.09$

Comparison of pose estimation accuracy on large needle driver

Visualization of kinematic estimation results for curved scissors. (a) Original image, (b) Rendered image with estimated kinematics results from Mujoco, (c) Overlay the rendered outline on original image.

Segmentation masks of the surgical tool and needle, together with estimated needle center point and tangent vector.

Backbone	Training setting	Suture pad	Porcine liver/gallbladder	Chicken breast/fake blood
ACT	$(500+0)$	17/20 (85%)	1/20 (5%)	1/20 (5%)
ACT	$(500+500)$	17/20 (85%)	10/20 (50%)	4/20 (20%)
ACT $\_PO$	$(500+0)$	17/20 (85%)	4/20 (20%)	2/20 (10%)
ACT $\_PO$	$(500+500)$	18/20 (90%)	13/20 (65%)	8/20 (40%)
ACT $\_NA$	$(500+500)$	16/20 (80%)	2/20 (10%)	2/20 (10%)
ACT	$(1000+0)$	19/20 (95%)	0/20 (0%)	1/20 (5%)

Full needle task results. Each entry reports successful trials out of 20, with success rate in parentheses.

Backbone	Training setting	Gallbladder phantom	Normal ex vivo tissue	Abnormal ex vivo tissue
ACT	$(700+0)$	16/20 (80%)	3/20 (15%)	3/20 (15%)
ACT	$(700+700)$	17/20 (85%)	14/20 (70%)	10/20 (50%)
$\pi_{0}$	$(700+0)$	10/20 (50%)	0/20 (0%)	2/20 (10%)
$\pi_{0}$	$(700+700)$	13/20 (65%)	7/20 (35%)	4/20 (20%)
Gr00t-H	$(700+0)$	16/20 (80%)	4/20 (20%)	5/20 (25%)
Gr00t-H	$(700+700)$	14/20 (70%)	11/20 (55%)	10/20 (50%)

Full chole task results. Each entry reports successful trials out of 20, with success rate in parentheses.

Explore similar work

Imitation Learning for Robot Assistance in Open Surgery: A Multi-Policy Evaluation on Suture Following
May 27, 2026Xucheng Wang, Zhizhou Yang, Xiaoman Zhang +3Visuomotor Policy Learning Robot Imitation Learning
Department of Biomedical Informatics, Harvard Medical School, Boston, MA · Department of Surgery, Massachusetts General Hospital, Boston, MA
Surgical WAM: A World-Action Model for Data-Efficient Surgical Robot Learning
Aug 11, 2026Wenrui Bao, Tianyun Jiang, Zhiben Chen +3World Models for Robotics Bimanual Robotic Manipulation
University of Central Florida · AdventHealth
SurgVista: Long-Horizon Surgical World Modeling with Plausible Instrument-Tissue Dynamics
Jun 18, 2026Wentao Pan, Wuyang Li, Shengyuan Liu +3World Models Video Prediction
The Chinese University of Hong Kong · EPFL · Imperial College London

May 27, 2026J/K move · Enter open · S save

Imitation Learning for Robot Assistance in Open Surgery: A Multi-Policy Evaluation on Suture Following

Xucheng Wang, Zhizhou Yang, Xiaoman Zhang, Sung Eun Kim+2

Department of Biomedical Informatics, Harvard Medical School, Boston, MA · Department of Surgery, Massachusetts General Hospital, Boston, MA

Visuomotor Policy Learning Robot Imitation Learning Surgical Robotics

Open paper AskPDF arXiv

This study presents the first evaluation of general-purpose imitation learning for surgeon-robot collaborative assistance in open surgery, targeting suture following: the grab-pull-release motion an assistant performs at every stitch. We collect 160 teleoperated demonstrations (32,374 frames) on an open-source robot arm, benchmark four architecturally diverse imitation learning policies (ACT, Diffusion Policy, SmolVLA,

π_0

) across 28 trained models evaluated in 32 configurations along three clinically motivated dimensions: dataset size, camera viewpoint, and background variation. Our results demonstrate that under ideal conditions, the four policies achieve

50

75\%

task success, with depth error as the dominant failure mode across all architectures. Among all policies,

π_0

achieves the strongest results with a pretrained vision-language backbone, demonstrating superior data efficiency, greater robustness to background variation, and smoother trajectories compatible with surgical workflow. When deployed in a surgeon-robot suturing trial,

π_0

yields a

92\%

stitch completion rate. These findings establish collaborative robotic assistance in open surgery as a feasible target for imitation learning and highlight depth perception and end-effector design as key priorities for clinical translation.

Abstract

Explore similar work

Imitation Learning for Robot Assistance in Open Surgery: A Multi-Policy Evaluation on Suture Following

Surgical WAM: A World-Action Model for Data-Efficient Surgical Robot Learning

SurgVista: Long-Horizon Surgical World Modeling with Plausible Instrument-Tissue Dynamics