Real-time reconstruction of deformable surgical scenes is vital for advancing robotic surgery, improving intraoperative guidance, and enabling automation. Recent methods achieve dense reconstructions from da Vinci robotic surgery videos, with Gaussian Splatting (GS) offering real-time performance via graphics acceleration. However, reconstruction quality in occluded regions remains limited, and depth accuracy has not been fully assessed, as benchmarks like EndoNeRF and StereoMIS lack 3D ground truth. We propose Diff2DGS, a two-stage framework for reliable 3D reconstruction of occluded surgical scenes. First, a diffusion-based video module with temporal priors inpaints tissue occluded by instruments with high spatiotemporal consistency. Second, we adapt 2D Gaussian Splatting (2DGS) with a Learnable Deformation Model (LDM) to capture dynamic tissue deformation and anatomical geometry, and introduce adaptive depth weight to improve geometric fidelity. We further extend evaluation beyond image-quality metrics by performing quantitative depth analysis on the SCARED dataset. Diff2DGS outperforms state-of-the-art methods in both appearance and geometry, reaching 38.02 dB PSNR on EndoNeRF and 34.40 dB on StereoMIS. Our experiments also show that optimizing image quality alone does not necessarily ensure accurate 3D geometry.The code is available at https://diff2dgs.github.io/.
Figures & tables
Fig. 1: Traditional endoscopic scene reconstruction methods often focus primarily on image quality from the camera viewpoint, neglecting the depth accuracy of the reconstructed geometry. Consequently, when the camera viewpoint changes, reconstruction accuracy degrades significantly. We present Diff2DGS, a framework that effectively balances depth information and image quality, achieving high-quality 3D reconstruction while better eliminating artifacts in occluded regions and providing more accurate depth estimation.
Fig. 2: Our reliable surgical scene reconstruction framework, Diff2DGS, consists of Surgical Instrument Inpainting, Point Cloud Initialization, Deformation Modeling, and 2D Gaussian Splatting.
Models
EndoNeRF-Cutting
EndoNeRF-Pulling
FPS
PSNR ↑
SSIM (%) ↑
LPIPS ↓
RMSE ↓
PSNR ↑
SSIM (%) ↑
LPIPS ↓
RMSE ↓
EndoNeRF [ 12 ]
35.84
94.25
0.085
4.299
35.43
93.89
0.077
4.746
0.03
LerPlane [ 14 ]
35.21
92.96
0.095
5.782
34.87
91.94
0.118
6.245
0.93
EndoGaussian [ 19 ]
37.19
95.29
0.052
4.032
35.98
94.23
0.047
5.341
83.13
Deform3DGS [ 20 ]
37.33
95.89
0.075
4.234
37.70
95.57
0.069
5.727
228.58
SurgicalGS [ 23 ]
37.93
96.19
0.045
1.779
37.13
95.43
0.062
2.193
154.39
TABLE I: Comparison between Diff2DGS and Existing Methods on EndoNeRF.
Models
StereoMIS-P2 7
StereoMIS-P3
FPS
PSNR ↑
SSIM (%) ↑
LPIPS ↓
RMSE ↓
PSNR ↑
SSIM (%) ↑
LPIPS ↓
RMSE ↓
EndoNeRF [ 12 ]
28.87
80.23
0.259
4.354
28.33
80.09
0.287
4.165
0.06
LerPlane [ 14 ]
29.53
78.36
0.205
8.356
29.41
78.14
0.213
7.983
0.97
EndoGaussian [ 19 ]
31.11
87.29
0.127
9.793
31.03
87.13
0.136
8.932
90.31
Deform3DGS [ 20 ]
31.83
87.75
0.129
10.036
30.91
87.73
0.131
8.482
214.38
SurgicalGS [ 23 ]
31.63
88.26
0.152
2.747
31.49
87.75
0.165
2.536
154.82
TABLE II: Comparison between Diff2DGS and Existing Methods on StereoMIS.
Fig. 3: Qualitative comparison of 3D reconstruction results on the StereoMIS and EndoNeRF datasets.
Method
Full Image
Masked Region
SSIM (%)↑
FID ↓
TCS ↓
PSNR ↑
LPIPS ↓
RMSE ↓
PSNR ↑
LPIPS ↓
RMSE ↓
Inpaint_result
31.84
0.100
6.68
27.01
0.017
12.05
90.24
29.92
0.003
Deform3DGS
25.34
0.342
13.83
19.52
0.037
27.19
72.61
74.16
0.027
EndoGaussian
25.96
0.398
12.89
23.56
0.032
17.61
68.39
81.95
0.029
SurgicalGS
24.92
0.434
14.51
21.63
0.040
21.57
65.53
108.89
0.032
Diff2DGS
27.77
0.314
10.53
30.53
0.022
8.21
76.14
61.02
0.027
TABLE III: Quantitative comparison of different methods on SCARED dataset.
Fig. 4: Depth quality visualization on the SCARED dataset. Green contours highlight visible depth inconsistency near tissue boundaries.
Fig. 5: Qualitative comparison of reconstructed instrument-occluded regions on the SCARED dataset.
Models
StereoMIS
EndoNeRF
PSNR ↑
SSIM(%) ↑
RMSE ↓
PSNR ↑
SSIM(%) ↑
RMSE ↓
Diffusion+Deform3DGS
34.32
90.13
8.13
37.63
95.76
4.78
w/o deformation
32.12
86.87
6.30
33.94
89.85
5.74
w/o inpainting
31.95
86.11
4.39
37.57
95.73
2.65
w/o Ldepth ( λ=0 )
33.76
90.14
9.80
36.95
95.41
7.37
w/o adaptive ( λ=1 )
33.99
90.21
8.58
37.86
95.72
6.74
TABLE IV: Ablation study of Diff2DGS.
Fig. 6: Ablation experiment on EndoNeRF and StereoMIS.
Method
EPE (px) ↓
3 px (%) ↓
MAE (mm) ↓
DLNR [ 37 ]
1.45
4.12
1.32
STTR [ 38 ]
6.03
9.52
11.31
RAFT [ 34 ]
1.16
4.59
1.01
TABLE V: Error analysis of RAFT on SCARED dataset.
Dynamic endoscopic reconstruction is fundamental to robotic surgery and computer-assisted interventions. While 3D Gaussian Splatting (3DGS) realises real-time rendering, its application to deformable intraoperative environments remains constrained by spurious geometry and varying illuminations. To address these limitations, we introduce EndoPrior-GS, a novel pipeline that explicitly couples frame-extracted vision heuristics and estimated depth maps. EndoPrior-GS derives a joint texture prior from a tool-filtered valid tissue mask, a non-specular photometric filter, and anatomical structural salience, yielding a probability map that guides primitive initialisation and subsequent density control. The prior is further extended to the temporal domain through a texture-aware term that dynamically weighs pairwise primitive contributions during training. We conduct extensive experiments on benchmark datasets EndoNeRF and SCARED, and the obtained results show that our method EndoPrior-GS reduces Flow Error by 27.7% and 25.8% over the representative approaches while preserving competitive rendering quality and real-time rendering speed. Our project website is available at https://jiaqi-huang-77.github.io/EndoPrior-GS/.
Jiaqi Huang, Shidong Wang, Tong Xin +1
School of Engineering, Newcastle University, Newcastle upon Tyne NE1 7RU, UK · School of Computing, Newcastle University, Newcastle upon Tyne NE4 5TG, UK
Accurate 3D reconstruction of deformable soft tissues is essential for surgical robotic perception. However, low-texture surfaces, specular highlights, and instrument occlusions often fragment geometric continuity, posing a challenge for existing fixed-topology approaches. To address this, we propose EndoVGGT, a geometry-centric framework equipped with a Deformation-aware Graph Attention (DeGAT) module. Rather than using static spatial neighborhoods, DeGAT dynamically constructs feature-space semantic graphs to capture long-range correlations among coherent tissue regions. This enables robust propagation of structural cues across occlusions, enforcing global consistency and improving non-rigid deformation recovery. Extensive experiments on SCARED show that our method significantly improves fidelity, increasing PSNR by 24.6% and SSIM by 9.1% over prior state-of-the-art. Crucially, EndoVGGT exhibits strong zero-shot cross-dataset generalization to the unseen SCARED and EndoNeRF domains, confirming that DeGAT learns domain-agnostic geometric priors. These results highlight the efficacy of dynamic feature-space modeling for consistent surgical 3D reconstruction.
Reconstructing deformable endoscopic tissues is crucial for achieving robot-assisted surgery. However, 3D Gaussian Splatting-based approaches encounter challenges in achieving consistent tissue surface reconstruction, while existing NeRF-based methods lack real-time rendering capabilities. In pursuit of both smooth deformable surfaces and real-time rendering, we introduce a novel approach based on 3D Gaussian Splatting. Specifically, we introduce surface-aware reconstruction, initially employing a Sign Distance Field-based method to construct a mesh, subsequently utilizing this mesh to constrain the Gaussian Splatting reconstruction process. Furthermore, to ensure the generation of physically plausible deformations, we incorporate local rigidity and global non-rigidity restrictions to guide Gaussian deformation, tailored for the highly deformable nature of soft endoscopic tissue. Based on 3D Gaussian Splatting, our proposed method delivers a fast rendering process and smooth surface appearances. Quantitative and qualitative analysis against alternative methodologies shows that our approach achieves solid reconstruction quality in both textures and geometries.