SurgSync: Time-Synchronized Multi-Modal Data Collection Framework and Dataset for Surgical Robotics

Haoying Zhou; Chang Liu; Yimeng Wu; Junlin Wu; Zijian Wu; Yu Chung Lee; Sara Martuscelli; Spetimiu E. Salcudean

cs.ROMar 6, 2026

SurgSync: Time-Synchronized Multi-Modal Data Collection Framework and Dataset for Surgical Robotics

Authors: Haoying Zhou, Chang Liu, Yimeng Wu, Junlin Wu, Zijian Wu, Yu Chung Lee, Sara Martuscelli, Spetimiu E. Salcudean,

Organizations: Department of Robotics Engineering, Worcester Polytechnic Institute, Worcester, MA, USA. · Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD, USA. · Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA. · Robotics and Control Laboratory, the University of British Columbia, Vancouver, Canada. ·

Abstract

Most existing robotic surgery systems adopt a human-in-the-loop paradigm, often with the surgeon directly teleoperating the robotic system. Adding intelligence to these robots would enable higher-level control, such as supervised autonomy or even full autonomy. However, artificial intelligence (AI) requires large amounts of training data, which is currently lacking. This work proposes SurgSync, a multi-modal data collection framework with offline and online synchronization to support training and real-time inference, respectively. The framework is implemented on a da Vinci Research Kit (dVRK) and introduces (1) dual-mode (online/offline-matching) synchronized recorders, (2) a modern stereo endoscope to achieve image quality on par with clinical systems, and (3) additional sensors such as a side-view camera and a novel capacitive contact sensor to provide ground truth contact data. The framework also incorporates a post-processing toolbox for tasks such as depth estimation, optical flow, and a practical kinematic reprojection method using Gaussian heatmap. User studies with participants of varying skill levels are performed with ex-vivo tissue to provide clinically realistic data, and a network for surgical skill assessment is employed to demonstrate utilization of the collected data. Through the user study experiments, we obtained a dataset of 214 validated instances across multiple canonical training tasks. All software and data are available at surgsync.github.io.

Figures & tables

Panel of Figure 1 — Fig. 1: Overview of the proposed SurgSync dataset and data collection framework. We collect 214 recordings while performing multiple canonical training tasks on ex-vivo tissues (primarily) and phantoms. We also implement post-collection processing using our toolbox. Multiple input modalities, including visual, kinematics/dynamics, tool-tissue contact, event/phase description, are provided for further algorithm training.

Fig. 2: Hardware setup and schematics. The Arduino and the proto-board are shown on the left, the schematics of the electronics on the bottom right, and the connection to the dVRK controller on the top right.

Fig. 3: Instrument wire connection. Non-polar instrument (left), monopolar (center), and bipolar (right). For non-polar instrument, the tool housing is opened and the wire insulation layer is removed for better visualization.

Fig. 4: Overall experimental setup. The endoscope is mounted on the dVRK-Si ECM using a custom holder and adapter.

System	Mean	STD
CSR Endoscope (Ours)	$529.48$	$23.77$
dVRK-Si Endoscope	$16.93$	$2.47$

TABLE I: Laplacian variance on suturing sequences.

Panel of Figure 6 — Fig. 5: Representative frames from the dVRK-Si endoscope (left) and our modern CSR endoscope (right). Brightness settings: dVRK-Si CCU at 100%, ours at 30%. Both frames are captured with similar camera-phantom distances.

Fig. 6: Kinematic Reprojection Pipeline. This approach projects the Cartesian positions of the PSM tool-yaw link to attention-weighted images.

Training Task	User Group	Number of Instances		Total
		Online	Offline
Suturing and Knot Tying	N	13	2	104
	E	36	12
	P	2	39
Peg Transfer	N	7	-	18
	E	11

TABLE II: Dataset Distribution, User Groups defined in section IV-B

Task	Fold 1	Fold 2	Fold 3	Fold 4	Mean $\pm$ Std
Suturing	0.658	0.828	0.844	0.881	0.803 $\pm$ 0.086
Knot Tying	0.870	0.875	0.618	0.699	0.765 $\pm$ 0.111

TABLE IV: Skill assessment performance on our synchronized dataset. SROCC is reported for each cross-validation fold.

Robotic Data Collection

Robotic data collection develops ways to gather, label, and validate robot and human-generated experience for learning, evaluation, and sensing.

See all 178 papers

SurgFusion-Net: Diversified Adaptive Multimodal Fusion Network for Surgical Skill Assessment
Feb 18, 2026Runlong He, Freweini M. Tesfai, Matthew W. E. Boal +13Surgical Video Understanding Multimodal Fusion
the UCL Hawkes Institute and the Department of Medical Physics & Biomedical Engineering, UCL, UK. · the Griffin Institute, Northwick Park and St Mark’s Hospital, UK. · the UCL Hawkes Institute and the Division of Informatics, Imaging & Data Sciences, The University of Manchester, UK +3
Surg $Σ$ : A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence
Mar 17, 2026Zhitao Zeng, Mengya Xu, Jian Jiang +13Multimodal Foundation Models Surgical Video Understanding
NUS · CUHK · SJTU +1
SurgVIL: Scaling Surgical Robot Imitation Learning with Open-source Surgical Videos
Aug 17, 2026Xinhao Chen, JuoTung Chen, Nigel Nelson +4Robot Learning from Video Robot Imitation Learning
Laboratory for Computational Sensing and Robotics (LCSR) Johns Hopkins University, USA · Nvidia, USA

Feb 18, 2026J/K move · Enter open · S save

SurgFusion-Net: Diversified Adaptive Multimodal Fusion Network for Surgical Skill Assessment

Runlong He, Freweini M. Tesfai, Matthew W. E. Boal, Nazir Sirajudeen+12

the UCL Hawkes Institute and the Department of Medical Physics & Biomedical Engineering, UCL, UK. · the Griffin Institute, Northwick Park and St Mark’s Hospital, UK. · the UCL Hawkes Institute and the Division of Informatics, Imaging & Data Sciences, The University of Manchester, UK+3

Surgical Video Understanding Multimodal Fusion Surgical Robotics

Open paper AskPDF arXiv

Robotic-assisted surgery (RAS) is established in clinical practice, and automated surgical skill assessment utilizing multimodal data offers transformative potential for surgical analytics and education. However, developing effective multimodal methods remains challenging due to the task complexity, limited annotated datasets and insufficient techniques for cross-modal information fusion. Existing state-of-the-art relies exclusively on RGB video and only applies on dry-lab settings, failing to address the significant domain gap between controlled simulation and real clinical cases, where the surgical environment together with camera and tissue motion introduce substantial complexities. This work introduces SurgFusion-Net and Divergence Regulated Attention (DRA), an innovative fusion strategy for multimodal surgical skill assessment. We contribute two first-of-their-kind clinical datasets: the RAH-skill dataset containing 279,691 RGB frames from 37 videos of Robot-assisted Hysterectomy (RAH), and the RARP-skill dataset containing 70,661 RGB frames from 33 videos of Robot-Assisted Radical Prostatectomy (RARP). Both datasets include M-GEARS skill annotations, corresponding optical flow and tool segmentation masks. DRA incorporates adaptive dual attention and diversity-promoting multi-head attention to fuse multimodal information, from three modalities, based on surgical context, enhancing assessment accuracy and reliability. Validated on the JIGSAWS benchmark, RAH-skill, and RARP-skill datasets, our approach outperforms recent baselines with SCC improvements of 0.02 in LOSO, 0.04 in LOUO across JIGSAWS tasks, and 0.0538 and 0.0493 gains on RAH-skill and RARP-skill, respectively.

Abstract

Robotic Data Collection

Explore similar work

SurgFusion-Net: Diversified Adaptive Multimodal Fusion Network for Surgical Skill Assessment

SurgΣΣΣ: A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence

SurgVIL: Scaling Surgical Robot Imitation Learning with Open-source Surgical Videos

Surg $Σ$ : A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence