iMED: A Multi-Endoscope Dataset for Surgical 3D Perception

Sierra Bonilla; Fengyi Jiang; Chinedu Nwoye; Jingpei Lu; Kailey Reardon; Humphrey W. Chow; Francisco Vasconcelos; Sophia Bano

cs.CVNov 3, 2026

iMED: A Multi-Endoscope Dataset for Surgical 3D Perception

Authors: Sierra Bonilla, Fengyi Jiang, Chinedu Nwoye, Jingpei Lu, Kailey Reardon, Humphrey W. Chow, Francisco Vasconcelos, Sophia Bano,

Organizations: University College London, London WC1E 6BT, United Kingdom · Advanced Research, Intuitive, Sunnyvale, CA 94086, USA

Abstract

We introduce iMED, the first synchronized multi-endoscope dataset for robot-assisted minimally invasive surgery (RAMIS), designed to address a fundamental limitation in surgical vision benchmarking: the absence of independent held-out viewpoints. Existing surgical datasets capture single-trajectory sequences from a single endoscope, making it impossible to distinguish geometric generalization from photometric interpolation along a narrow forward-facing path. iMED provides 340 sequences ~170K synchronized timepoints with 4 views per timepoint) recorded simultaneously from two independent stereo endoscopes across ex vivo, postmortem, and live surgical settings, spanning 14 specimens with diverse anatomical regions and motion regimes. The dataset includes calibrated camera intrinsics, ArUco-based frame-wise pose estimates with uncertainty quantification, instrument segmentation masks, and rich clinical metadata. Using a train-on-one-endoscope, test-on-another evaluation protocol, we benchmark 23 state-of-the-art methods across rigid and deformable novel view synthesis, pose estimation, feature matching, and monocular depth estimation. Our experiments show that methods relying primarily on photometric supervision degrade substantially under this held-out-endoscope setting, while methods with explicit geometric regularization are more robust across views in our benchmark. Notably, large-scale foundation models transfer surprisingly well to surgical imagery, often outperforming domain-specific models, suggesting pitfalls in current fine-tuning protocols. iMED establishes a new evaluation protocol for geometric generalization in surgical vision.

Figures & tables

Figure 1: Data platform with two stereo endoscopes (top-left), common camera trajectories (top-right), and pipeline outputs including session/anatomy/instrument/motion labels, camera intrinsics, ArUco pose estimates, inferred tool masks , stereo depth (code provided), and derived 3D geometry. See Tab. 2 for provenance.

Dataset

Dataset Attributes

Scene Types

Inferred Labels

Provided Reference Signals

Seq.

Specimens

Modality

Setting

Live

Static

Dyn. Rigid

Deform. Obj.

3D Model

Tool Masks

Camera Calib.

Multi-Camera*

EndoVis 17/18 [ 4 , 2 ]

Da Vinci Xi Da Vinci X

In vivo

✓

MITI [ 22 ]

Karl Storz 3D Tipcam

In vivo

✓

STIR [ 54 ]

572

Da Vinci Xi

Ex vivo In vivo

✓

Hamlyn Centre [ 42 , 59 , 50 ]

Da Vinci Si

Phantom Ex vivo In vivo

✓

✓**

SERV-CT [ 16 ]

Da Vinci

Postmortem

✓

Table 1: Comparison of stereo-endoscopic datasets for 3D perception in minimally invasive surgery. Multi-Camera* defined as more than one stereo camera pair. **only for 2 phantom validation sequences with CT scans.

Panel of Figure 2 — Figure 2: Example geometric reconstructions and label projection enabled by iMED. Left: synchronized stereo views from both endoscopes, stereo depth estimationo [ 67 ] , aligned point clouds from provided camera calibrations, Poisson-reconstruction, and reprojected tool masks. Right: Representative anatomical regions with surface reconstructions from ex vivo specimens (chicken, porcine, bovine), human cadaver sites, and live porcine tissue. Inference code to recreate geometry and 3D tool labels is provided.

Figure 3: A: Distribution of seqs. by anatomical region and organ type including number of timepoints per category. B: Distribution of seqs. by motion regime (camera motion vs. scene motion). Calibration-only sequences are excluded. C: Distribution of seqs. by session type. D: Distribution of seqs. by cam spatial relationships, showing inter-cam separation distance and orientation difference.

Label	Level	Provenance	Source / Method	Released
Session type	Seq.	Manual	Curator	✓
Broad anatomical region	Seq.	Manual	Clinical specialist	✓
Detailed scene caption	Scene	Manual	Clinical specialist	✓
Instrument type	Seq.	Manual	Clinical specialist	✓
Motion regime/movement type	Seq.	Manual	Curator + protocol	✓
Camera intrinsics/stereo baseline	Seq.	Measured	Calibration procedure	✓

Table 2: Label provenance for iMED. We report annotation level, generation type, source method, and whether the annotation is included in the public data release. For storage reasons, stereo depth and 3D geometry are reproducible via provided code.

Panel of Figure 4 — Figure 4: Left: ArUco calibration cube design iterations leading to our final 8 mm wooden cube design. Right: ArUco-based pose estimation pipeline: (1) marker detection, (2) PnP pose estimation, (3) relative transformation via RANSAC and refinement.

Session Type	Reproj. Err. (px) $\downarrow$	Physical Err. @5cm (mm) $\downarrow$
Ex vivo	38.33	1.88
In vivo (postmortem)	17.00	0.83
In vivo (live)	11.87	0.58

Table 3: Reference trajectory accuracy after full refinement across session types. See Supplementary Sec. 6.5 for complete ablations and 3D consistency metrics.

Method	Depth	PSNR $\uparrow$	SSIM $\uparrow$	Render (FPS) $\uparrow$	Train (FPS) $\uparrow$
Rigid Novel View Synthesis
NeRF [ 41 ]	x	17.24	0.65	0.11	0.21
NeRF [ 41 ]	DA [ 71 ]	16.29	0.55	0.11	0.19
NeRF [ 41 ]	FS [ 67 ]	15.93	0.55	0.11	0.18
3DGS [ 30 ]	x	14.82	0.59	487.62	0.59
3DGS [ 30 ]	DA [ 71 ]	14.77	0.63	509.99	0.36

Table 4: NVS benchmarks. Methods trained on Endoscope 2 and evaluated on Endoscope 1 . Rigid and deformable methods perform offline optimization vs. online methods incrementally update the scene using only the most recent observations during acquisition (see Sec. 4.1 ). FPS excludes SfM [ 55 ] preprocessing.

Panel of Figure 5 — Figure 5: Left: Rendered test views show ghost-like artifacts in depth-supervised variants. Right: SSIM computed against the test held-out image during camera interpolation from a training (0%) to test (100%) viewpoint. NeRF (purple) improves monotonically, while depth-supervised variants degrade despite geometric convergence, suggesting transparent density surfaces rather than opaque geometry.

Figure 6: Qualitative comparison of the Deformable NVS methods.

Figure 7: Qualitative comparison of image matching methods showing correspondence quality. Green lines indicate inlier matches; red lines indicate outliers. The angle next to the method name is the average rotational error.

Method	Rot. Err.(°) $\downarrow$	Temporal Drift(°) $\downarrow$	Inlier Rate (%) $\uparrow$	Avg Inliers $\uparrow$
RoMa [ 15 ]	1.79±1.41	1.41	78.1	3905
MASt3R [ 32 ]	1.81±1.38	1.38	70.3	1676
GIM+LG [ 57 , 37 ]	1.87±1.48	0.41	43.1	264
ALIKE+LG [ 76 , 37 ]	1.91±1.52	0.64	68.2	211
LoFTR [ 60 ]	1.91±1.40	0.43	47.2	513
DISK+LG [ 64 , 37 ]	1.93±1.52	0.59	55.2	446

Table 5: Pose estimation and feature matching benchmarking results. Temporal consistency measures rot. drift across seqs. with tool-tissue deformation and static cameras.

Figure 8: Qualitative comparison of mono. depth estimation compared to inferred stereo depth [ 67 ] . Error maps show abs. depth error after median scaling of the predictions.

Method	Med. Scaled	Abs Rel $\downarrow$	RMSE $\downarrow$	RMSE log $\downarrow$	$\delta_{1}\uparrow$	$\delta_{2}\uparrow$	$\delta_{3}\uparrow$
UniDepth-V2 [ 48 ]	Yes	0.081	0.012	0.111	0.936	0.992	0.998
MoGe-V2 [ 66 ]	Yes	0.109	0.014	0.140	0.889	0.989	0.998
EndoDAC [ 11 ]	Yes	0.119	0.017	0.164	0.854	0.968	0.994
Endo3R [ 21 ]	Yes	0.187	0.023	0.236	0.704	0.927	0.972
UniDepth-V2 [ 48 ]	No	11.573	1.098	2.465	0.000	0.000	0.000
MoGe-V2 [ 66 ]	No	14.556	1.329	2.683	0.000	0.000	0.000

Table 6: Monocular depth prediction compared to inferred stereo depth [ 67 ] . Methods that report metric-scale predictions [ 48 , 66 ] are evaluated without median scaling.

Figure 9: Example ArUco marker removal for dynamic-camera sequences. When markers remain visible for continuous pose estimation, iMED releases ArUco masks [ 9 ] and inpainted frames generated with DiffuEraser [ 35 ] .

Tool Name	Num Sequences
Intuitive Surgical Da Vinci Tip-Up Fenestrated Grasper	61
Intuitive Surgical Da Vinci ProGrasp tm Forceps	30
Intuitive Surgical Da Vinci Long Bipolar Grasper	22
Intuitive Surgical Da Vinci Monopolar Curved Scissors	18
Ethicon Endopath XCEL Universal Sleeve Assist Port CB5ST	18
Intuitive Surgical Da Vinci Mega SutureCut tm Needle Driver	16

Table 7: Surgical instrument taxonomy and sequence frequency across the dataset (20 instrument types, 340 sequences total).

Figure 10: Three trajectory examples showing camera movements in iMED - Circular (left), Left and Right(middle), zoom in (right) camera movements. They are decomposed to their X, Y, and Z (cm) components; reprojection error optimization baseline in black and the filtered trajectories in red.

Method	SSIM $\uparrow$	PSNR $\uparrow$ (dB)	Temporal Var. $\approx$ M0 (6.69)
M0 : Baseline	$0.278$	$9.79$	$6.69$
M1 : Brightness Only	$0.283$	$12.10$	$13.88$
M2 : Frame-to-Frame	$0.479$	$15.06$	$9.55$
M3 : Frame-to-Seq	$\mathbf{0.502}$	$\mathbf{15.34}$	$\mathbf{7.64}$
M4 : Seq-to-Seq	$\mathbf{0.503}$	$\mathbf{15.34}$	$\mathbf{7.78}$

Table 8: Endoscope matching normalization strategies. SSIM and PSNR measure structural and pixel-level fidelity, while temporal variation quantifies stability across frames (

\approx

M0 baseline indicates optimal preservation of natural temporal dynamics). Methods M2-M4 evaluated with specular masking; best threshold reported. Temporal variation measures pixel stability across frames (RMS of standard deviations from 10 tracked non-specular pixels).

	2D Evaluation			3D Evaluation
Method	Reproj. Err. (pixels) $\downarrow$	Physical Err. @5cm (cm) $\downarrow$	Markers/ Image Pair $\uparrow$	RMSE(cm) $\downarrow$	Chamfer Dist. (cm) $\downarrow$
Ex Vivo Sessions (n=9)
Inlier Averaging	550.23	2.697	4.18	1.5	1.8
Optimization	362.69	1.778	4.18	1.5	1.8
Outlier Rejection	362.69	1.778	4.18	1.5	1.8
Temporally Smooth	38.33	0.188	4.18	1.4	1.7

Table 9: Trajectory estimation comparison across session types on 27 sequences (9 per type, 20 frames each). 2D reprojection error converted to physical error at 5cm working distance. 3D metrics from FoundationStereo point cloud alignment. Ex vivo baseline errors (362-550 pixels) stem from early acquisition with metallic cubes and 4×4 ArUco markers; temporal smoothing yields dramatic improvement. In vivo sessions achieve lower baseline errors (12-19 pixels) due to improved marker design (wooden cubes, " ARUCP_MIP_36h12 ") and mechanical stability.

3D Reconstruction Benchmarks

See all 27 papers

SurgCUT3R: Surgical Scene-Aware Continuous Understanding of Temporal 3D Representation
Mar 7, 2026Kaiyuan Xu, Fangzhou Hong, Daniel Elson +13D Reconstruction Endoscopic 3D Reconstruction
The Hamlyn Centre for Robotic Surgery, Imperial College London, SW7 2AZ, UK. · S-Lab, College of Computing and Data Science, Nanyang Technological University, Singapore 639798. · Department of Computer Science, University of Liverpool, L69 7ZX, UK.
SCOPE-4D: Endoscopic 4D Geometry Foundation Models
Oct 1, 2026Chaoyi Zhou, Zhongpai Gao, Anwesa Choudhuri +6Endoscopic 3D Reconstruction Endoscopy
United Imaging Intelligence, Boston, MA, USA · Clemson University, Clemson, SC, USA
Learning Surgical Robotic Manipulation with 3D Spatial Priors
Mar 4, 2026Yu Sheng, Lidian Wang, Xiaomeng Chu +6Robotic Manipulation Surgical Robotics
University of Science and Technology of China · Tuodao Medical · Yale University

Mar 7, 2026J/K move · Enter open · S save

SurgCUT3R: Surgical Scene-Aware Continuous Understanding of Temporal 3D Representation

Kaiyuan Xu, Fangzhou Hong, Daniel Elson, Baoru Huang

The Hamlyn Centre for Robotic Surgery, Imperial College London, SW7 2AZ, UK. · S-Lab, College of Computing and Data Science, Nanyang Technological University, Singapore 639798. · Department of Computer Science, University of Liverpool, L69 7ZX, UK.

3D Reconstruction Endoscopic 3D Reconstruction Monocular 3D Reconstruction Surgical Robotics Camera Pose Estimation

Open paper AskPDF arXiv

Reconstructing surgical scenes from monocular endoscopic video is critical for advancing robotic-assisted surgery. However, the application of state-of-the-art general-purpose reconstruction models is constrained by two key challenges: the lack of supervised training data and performance degradation over long video sequences. To overcome these limitations, we propose SurgCUT3R, a systematic framework that adapts unified 3D reconstruction models to the surgical domain. Our contributions are threefold. First, we develop a data generation pipeline that exploits public stereo surgical datasets to produce large-scale, metric-scale pseudo-ground-truth depth maps, effectively bridging the data gap. Second, we propose a hybrid supervision strategy that couples our pseudo-ground-truth with geometric self-correction to enhance robustness against inherent data imperfections. Third, we introduce a hierarchical inference framework that employs two specialized models to effectively mitigate accumulated pose drift over long surgical videos: one for global stability and one for local accuracy. Experiments on the SCARED and StereoMIS datasets demonstrate that our method achieves a competitive balance between accuracy and efficiency, delivering near state-of-the-art but substantially faster pose estimation and offering a practical and effective solution for robust reconstruction in surgical environments. Project page: https://chumo-xu.github.io/SurgCUT3R-ICRA26/.

Abstract

3D Reconstruction Benchmarks

Explore similar work

SurgCUT3R: Surgical Scene-Aware Continuous Understanding of Temporal 3D Representation

SCOPE-4D: Endoscopic 4D Geometry Foundation Models

Learning Surgical Robotic Manipulation with 3D Spatial Priors