The Dresden Dataset for 4D Reconstruction of Non-Rigid Abdominal Surgical Scenes

Reuben Docea; Rayan Younis; Yonghao Long; Maxime Fleury; Jinjing Xu; Chenyang Li; André Schulze; Ann Wierick

cs.CVMar 3, 2026

The Dresden Dataset for 4D Reconstruction of Non-Rigid Abdominal Surgical Scenes

Authors: Reuben Docea, Rayan Younis, Yonghao Long, Maxime Fleury, Jinjing Xu, Chenyang Li, André Schulze, Ann Wierick,

Organizations: Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT), NCT/UCC Dresden, a partnership between DKFZ, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, and Helmholtz-Zentrum Dresden-Rossendorf (HZDR), Dresden, Germany · Department of Translational Surgical Oncology, NCT/UCC Dresden, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Fetscherstraße 74, 01307 Dresden, Germany · Department for Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany · Centre for Tactile Internet with Human-in-the-Loop (CeTI), Technical University Dresden, Dresden, Germany ·

Abstract

The D4D Dataset provides paired endoscopic video and high-quality structured-light geometry for evaluating 3D reconstruction of deforming abdominal soft tissue in realistic surgical conditions. Data were acquired from six porcine cadaver sessions using a da Vinci Xi stereo endoscope and a Zivid structured-light camera, registered via optical tracking and manually curated iterative alignment methods. Three sequence types - whole deformations, incremental deformations, and moved-camera clips - probe algorithm robustness to non-rigid motion, deformation magnitude, and out-of-view updates. Each clip provides rectified stereo images, per-frame instrument masks, stereo depth, start/end structured-light point clouds, curated camera poses and camera intrinsics. In postprocessing, ICP and semi-automatic registration techniques are used to register data, and instrument masks are created. The dataset enables quantitative geometric evaluation in both visible and occluded regions, alongside photometric view-synthesis baselines. Comprising over 300,000 frames and 369 point clouds across 98 curated recordings, this resource can serve as a comprehensive benchmark for developing and evaluating non-rigid SLAM, 4D reconstruction, and depth estimation methods.

Figures & tables

Figure 1: Experimental setup showing porcine cadaver in supine position for open surgery with da Vinci . The Zivid structured light camera is visible in the foreground, with optical markers attached to the boom supporting the Zivid and the endoscope.

Figure 2: Sequence Alignment Tool used for creating clips.json files: manually identifying start and end points of clips, and associating them with specific point cloud data.

Panel of Figure 3 — Figure 3: Point cloud acquired with the SLC rendered from the perspective of the Zivid camera (left) and the endoscope camera after PnP-RANSAC (right).

Figure 4: Visualization of alignment with feature matching (LightGlue).

Panel of Figure 5 — Figure 5: Comparison between rendered, aligned, and anaglyph images for nominal (left) and PnP-RANSAC adjusted (right) poses.

Figure 6: Example of the semi-automatic region-based ICP refinement used when standard ICP failed to produce a reliable alignment.

Figure 7: Comparison of pose quality across the Nominal, Curated, and ICP sets. The box plots summarize the distribution of nearest–neighbour distances between points in the Stereo Depth point cloud and their closest corresponding points in the SLC point cloud, illustrating the relative accuracy of the poses used in each set.

Figure 8: Overview of visual data associated with Clip 1 and the subsequent four clips for sequence 2025_09_09-15_40_48 of Specimen 5 . For each clip (from top to bottom: the initial state of Clip 1 followed by the end states of the next four clips), the figure displays: the left endoscopic image, the RGB rendering produced from the SLC point cloud using the curated pose, the stereo depth map, and the SLC point-cloud depth image generated from the same curated pose. This layout illustrates the visual consistency and depth alignment achieved through the curated registration process.

Panel of Figure 9 — Figure 9: Recording 2025_09_09-15_40_48 of Specimen 5 , Clip 5 start (i.e., Clip 4 end) images with respective mask overlays: (a) Left endoscope view, (b) Zivid view

Panel of Figure 10 — Figure 10: Sample pointclouds extracted for initial (left) and end (right) states on one sequence from our dataset, obtained using the ForPlane [ 20 ] method.

	Pre-Filter	Retained
Specimen	Prior to Filter	Total	(i) Single	(ii) Incremental	(iii) Moved Camera
0	$24$	$0$	$0$	$0$	$0$
1	$37$	$16$	$9$	$7$	$0$
2	$14$	$14$	$7$	$5$	$2$
3	$19$	$15$	$8$	$6$	$1$
4	$33$	$32$	$15$	$12$	$5$

Table 1: Summary of Clip Counts by Specimen

Metric / Set	Mean	Median	Std	Min	Max
Median Dist (mm) - Nominal	5.63	4.25	3.51	1.10	15.25
Median Dist (mm) - Curated	2.37	2.15	1.20	0.52	7.26
Median Dist (mm) - ICP	2.70	2.32	1.69	0.52	14.11
Inliers $<$ 1mm (%) - Nominal	8.60	5.48	8.40	0.00	46.98
Inliers $<$ 1mm (%) - Curated	33.90	32.62	10.38	6.84	79.11
Inliers $<$ 1mm (%) - ICP	26.02	26.16	12.85	0.26	79.13

Table 2: Summary statistics of point-to-point distances from points in Stereo Depth -derived point cloud to SLC pointcloud after transformation with Nominal, Curated, and ICP poses evaluation sets (N = 540 pointclouds).

3D Reconstruction

3D reconstruction recovers a scene’s shape, appearance, and camera motion from images, video, depth, or other sensor measurements.

See all 930 papers

4D Monocular Surgical Reconstruction under Arbitrary Camera Motions
Feb 19, 2026Jiwei Shan, Zeyu Cai, Cheng-Tai Hsieh +54D Reconstruction 3D Reconstruction
Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong. · Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China · State Key Laboratory of Robotics and Intelligent Systems, Shenyang, China +3
Syn4D: A Multiview Synthetic 4D Dataset
May 6, 2026Zeren Jiang, Yushi Lan, Yihang Luo +83D Point Tracking 4D Reconstruction
VGG, University of Oxford · Nanyang Technological University · Naver Labs Europe
Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild
Jun 22, 2026Yehonathan Litman, Xiaoxuan Ma, Manan Shah +44D Reconstruction Dynamic Scene Reconstruction
Carnegie Mellon University

Feb 19, 2026J/K move · Enter open · S save

4D Monocular Surgical Reconstruction under Arbitrary Camera Motions

Jiwei Shan, Zeyu Cai, Cheng-Tai Hsieh, Yirui Li+4

Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong. · Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China · State Key Laboratory of Robotics and Intelligent Systems, Shenyang, China+3

4D Reconstruction 3D Reconstruction Endoscopic 3D Reconstruction Dynamic Scene Reconstruction 3D Gaussian Splatting

Open paper AskPDF arXiv

Reconstructing deformable surgical scenes from endoscopic videos is challenging and clinically important. Recent state-of-the-art methods based on implicit neural representations or 3D Gaussian splatting have made notable progress. However, most are designed for deformable scenes with fixed endoscope viewpoints and rely on stereo depth priors or accurate structure-from-motion for initialization and optimization, limiting their ability to handle monocular sequences with large camera motion in real clinical settings. To address this, we propose Local-EndoGS, a high-quality 4D reconstruction framework for monocular endoscopic sequences with arbitrary camera motion. Local-EndoGS introduces a progressive, window-based global representation that allocates local deformable scene models to each observed window, enabling scalability to long sequences with substantial motion. To overcome unreliable initialization without stereo depth or accurate structure-from-motion, we design a coarse-to-fine strategy integrating multi-view geometry, cross-window information, and monocular depth priors, providing a robust foundation for optimization. We further incorporate long-range 2D pixel trajectory constraints and physical motion priors to improve deformation plausibility. Experiments on three public endoscopic datasets with deformable scenes and varying camera motions show that Local-EndoGS consistently outperforms state-of-the-art methods in appearance quality and geometry. Ablation studies validate the effectiveness of our key designs. Code will be released upon acceptance at: https://github.com/IRMVLab/Local-EndoGS.

Abstract

3D Reconstruction

Explore similar work

4D Monocular Surgical Reconstruction under Arbitrary Camera Motions

Syn4D: A Multiview Synthetic 4D Dataset

Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild