Sensorium: An Open-Source Multimodal Data Collection and Curation Library for Embodied Surgical AI

Lorenzo Mazza; Ariel Rodriguez; Stefanie Speidel

cs.AIOct 30, 2026

Sensorium: An Open-Source Multimodal Data Collection and Curation Library for Embodied Surgical AI

Authors: Lorenzo Mazza, Ariel Rodriguez, Stefanie Speidel

Organizations: NCT/UCC Dresden, UKDD Dresden, TU Dresden, DKFZ Heidelberg. · BMFTR Research Hub 6G-Life. · Cluster of Excellence CeTI. · HZDR Dresden.

Abstract

Embodied surgical AI requires reliable multimodal data, yet surgical data collection remains costly, difficult to repeat, and strongly dependent on the hardware setup. We present Sensorium, an open-source Python library for recording, processing, annotating, and curating multimodal surgical data from configurable robot and sensor streams using the Robot Operating System (ROS) and ROS2 middleware. The library combines operator-facing data acquisition, configurable processing, semantic annotation, and episodic dataset generation. We evaluate Sensorium across 1516 recordings of phantom, ex vivo, and in vivo porcine experiments involving multiple surgical tasks and sensing configurations, and report preliminary usability feedback.

Figures & tables

Figure 1: Sensorium provides a unified workflow for recording, processing, annotating, and exporting multimodal surgical data. The mosaic shows representative endoscopic views from the tasks and experimental settings summarized in Table 1 .

Figure 2: Overview of the Sensorium workflow. The browser interfaces support (a) multimodal recording and (b) episode review and semantic annotations. Recorded streams are stored as raw episodes, processed through a ROS-independent curation pipeline, and exported as episodic CSV or LeRobot v3 datasets.

Figure 3: Action-denoising preview in the browser interface. Blue and orange bars compare raw and retained nonzero translation-action magnitudes on a $\log_{10}$ scale. The dashed line marks the selected 30th-percentile threshold. The table summarizes the samples that would be set to zero, overall and by task phase.

Dataset	Episodes	Streams	Duration	Timestamp residual (ms)
			(h:min)	Median	95th
Bowel grasping/retraction demonstrations (phantom)	180	V,R,G,T,P,F,I	1:41	8.60	17.68
Bowel grasping/retraction demonstrations (ex vivo)	123	V,R,G,T,F,I	0:59	8.13	16.65
Bowel grasping/retraction demonstrations (in vivo)	66	V,R,G,T,P,F,I	0:31	8.10	16.65
Bowel grasping/retraction policy rollouts (mixed)	426	V,R,G,T,P	4:44	8.10	16.62
Endoscope guidance (phantom)	276	V,R,T,P,G	2:10	6.41	16.32

Table 1: Deployment coverage and post-recording temporal alignment. Durations are rounded to the nearest minute. Alignment statistics summarize pooled nearest-neighbor residuals between reference-camera and robot-state timestamps over their common temporal interval.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Questionnaire item	R1	R2	R3	R4	R5	R6	R7	Mean
I could clearly determine whether all required sensors were ready	5	5	5	5	5	5	5	5.00
Starting and stopping a recording was straightforward	5	5	5	5	4	5	5	4.86
I could clearly tell when recording was active	5	4	5	5	5	5	5	4.86
The system fit into the procedural workflow without unnecessary disruption	5	5	5	5	5	5	5	5.00
I was confident that the intended data had been saved correctly	5	5	5	5	4	5	5	4.86
I could understand and respond to warnings or sensor failures	3	–	5	5	3	5	5	4.33

Appendix

Table 2: Individual operator usability ratings and item means. Questionnaire statements are reproduced as presented to respondents.

Figure 4: Sensorium during an in vivo porcine experiment. The foreground monitor shows the browser recording interface with an endoscopic view, while the surgical monitor displays the operative scene.

Robotic Data Collection

Robotic data collection develops ways to gather, label, and validate robot and human-generated experience for learning, evaluation, and sensing.

See all 182 papers

SurgSync: Time-Synchronized Multi-Modal Data Collection Framework and Dataset for Surgical Robotics
Mar 6, 2026Haoying Zhou, Chang Liu, Yimeng Wu +7Robotic Data Collection Surgical Robotics
Department of Robotics Engineering, Worcester Polytechnic Institute, Worcester, MA, USA. · Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD, USA. · Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA. +2
Surg $Σ$ : A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence
Mar 17, 2026Zhitao Zeng, Mengya Xu, Jian Jiang +13Multimodal Foundation Models Surgical Video Understanding
NUS · CUHK · SJTU +1
Open-H-Embodiment: A Large-Scale Dataset for Enabling Foundation Models in Medical Robotics
Apr 22, 2026Open-H-Embodiment Consortium, :, Nigel Nelson +213Surgical Robotics World Models for Robotics

Mar 6, 2026J/K move · Enter open · S save

SurgSync: Time-Synchronized Multi-Modal Data Collection Framework and Dataset for Surgical Robotics

Haoying Zhou, Chang Liu, Yimeng Wu, Junlin Wu+6

Department of Robotics Engineering, Worcester Polytechnic Institute, Worcester, MA, USA. · Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD, USA. · Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA.+2

Robotic Data Collection Surgical Robotics Multimodal Sensor Fusion

Open paper AskPDF arXiv

Most existing robotic surgery systems adopt a human-in-the-loop paradigm, often with the surgeon directly teleoperating the robotic system. Adding intelligence to these robots would enable higher-level control, such as supervised autonomy or even full autonomy. However, artificial intelligence (AI) requires large amounts of training data, which is currently lacking. This work proposes SurgSync, a multi-modal data collection framework with offline and online synchronization to support training and real-time inference, respectively. The framework is implemented on a da Vinci Research Kit (dVRK) and introduces (1) dual-mode (online/offline-matching) synchronized recorders, (2) a modern stereo endoscope to achieve image quality on par with clinical systems, and (3) additional sensors such as a side-view camera and a novel capacitive contact sensor to provide ground truth contact data. The framework also incorporates a post-processing toolbox for tasks such as depth estimation, optical flow, and a practical kinematic reprojection method using Gaussian heatmap. User studies with participants of varying skill levels are performed with ex-vivo tissue to provide clinically realistic data, and a network for surgical skill assessment is employed to demonstrate utilization of the collected data. Through the user study experiments, we obtained a dataset of 214 validated instances across multiple canonical training tasks. All software and data are available at surgsync.github.io.

Abstract

Robotic Data Collection

Explore similar work

SurgSync: Time-Synchronized Multi-Modal Data Collection Framework and Dataset for Surgical Robotics

SurgΣΣΣ: A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence

Open-H-Embodiment: A Large-Scale Dataset for Enabling Foundation Models in Medical Robotics

Surg $Σ$ : A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence