Instrument-Splatting++: Towards Controllable Surgical Instrument Digital Twin Using Gaussian Splatting
Authors: Shuojue Yang, Zijian Wu, Chengjiaao Liao, Qian Li, Daiyun Shen, Chang Han Low, Septimiu E. Salcudean, Yueming Jin
Organizations: Department of Biomedical Engineering, National University of Singapore (NUS), Singapore. · Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, Canada · Department of Electrical and Computer Engineering, NUS.
High-quality and controllable digital twins of surgical instruments are critical for Real2Sim in robot-assisted surgery, as they enable realistic simulation, synthetic data generation, and perception learning under novel poses. We present Instrument-Splatting++, a monocular 3D Gaussian Splatting (3DGS) framework that reconstructs surgical instruments as a fully controllable Gaussian asset with high fidelity. Our pipeline starts with part-wise geometry pretraining that injects CAD priors into Gaussian primitives and equips the representation with part-aware semantic rendering. Built on the pretrained model, we propose a semantics-aware pose estimation and tracking (SAPET) method to recover per-frame 6-DoF pose and joint angles from unposed endoscopic videos, where a gripper-tip network trained purely from synthetic semantics provides robust supervision and a loose regularization suppresses singular articulations. Finally, we introduce Robust Texture Learning (RTL), which alternates pose refinement and robust appearance optimization, mitigating pose noise during texture learning. The proposed framework can perform pose estimation and learn realistic texture from unposed videos. We validate our method on sequences extracted from EndoVis17/18, SAR-RARP, and an in-house dataset, showing superior photometric quality and improved geometric accuracy over state-of-the-art baselines. We further demonstrate a downstream keypoint detection task where unseen-pose data augmentation from our controllable instrument Gaussian improves performance.
Figures & tables
Fig. 1: Overall pipeline of our Real2Sim pipeline Instrument-Splatting++ , which yields controllable GS digital twin of surgical instrument (Textured GS) with high visual and geometric fidelity.
Fig. 2: Overview of the proposed Instrument-Splatting++. Black arrows indicate operation flow, while red arrows denote supervisory signals. The textured GS in (c) denotes the generated GS digital twin of instrument with learned textures. Fire and snowflake icons represent optimizable and frozen parameters, respectively.
Fig. 3: Forward kinematics definition of the da Vinci EndoWrist Large Needle Driver (LND).
Fig. 4: Left: Gaussian update under perturbed instrument poses. Right: The pose update is affected by the quality of Gaussian model.
Method
Intraoperative
In-house
PSNR ↑
SSIM ↑
LPIPS ↓
RMSE ↓
RMSE-M ↓
PSNR ↑
SSIM ↑
LPIPS ↓
RMSE ↓
RMSE-M ↓
WIM
22.34
92.75
0.170
25.7
16.5
23.44
92.15
0.098
19.8
16.1
SC-GS
20.42
89.48
0.146
19.8
11.3
24.41
93.28
0.070
17.4
15.6
Shape-of-motion
22.35
93.46
0.088
13.2
8.1
24.88
94.34
0.067
20.5
6.6
TTGS
22.98
93.29
0.093
18.8
16.1
25.45
94.46
0.078
23.5
14.0
EndoGaussian
22.39
93.64
0.089
27.9
19.8
25.26
93.85
0.075
25.2
14.8
TABLE I: Quantitative comparison on intraoperative and in-house datasets. Intraoperative Dataset represents sequences from both EndoVis and SAR-RARP dataset. RMSE and RMSE-M represent depth error within entire image space and instrument mask, respectively.
Fig. 5: Visualization of the texture learning results on testing sets in comparison with SOTA reconstruction methods. We visualize the RGB and depth rendering. For SAR-RARP sequence (the first row), we use the depth rendering from CAD mesh with the estimated pose as the reference depth. For the other dataset, stereo depth estimations serve as the reference depth.
Fig. 6: Visualization of the variant methods on texture learning.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
RMSE ↓
w/o pose ref.
29.24
95.69
0.051
5.6
w/o Rob
28.64
95.51
0.047
5.6
w/ joint Optim.
22.48
91.17
0.068
5.9
RTL
29.82
96.27
0.039
5.6
TABLE II: Analysis of texture learning on In-House Dataset. PSNR, SSIM, and LPIPS measure the photometric accuracy while RMSE measures depth accuracy.
Method
EndoVis17&18 (Dice ↑ )
In-House (Keypoint)
Shaft
Wrist
Gripper
RMSE ↓ (pix)
RMSE ↓ (mm)
w/o Matching
93.73
83.12
66.03
9.79
8.64
w/o Reg
95.52
78.40
63.39
6.36
5.45
w/ SVD
96.83
86.65
73.32
5.85
5.12
w/o Gripper-Tip Net
94.63
83.94
63.17
10.11
7.58
SAPET
95.92
87.54
75.27
4.75
3.69
TABLE III: Analysis on pose estimation and tracking performance. Comparison of model variants on EndoVis17&18 and the In-House dataset.
Fig. 7: Visualization of the pose estimation results on in-house dataset. We visualize the predicted and GT trajectories of the left and right gripper tip points across a 100-frame video clip. The predicted and GT trajectories are colored in green and red, respectively. With estimated poses, we render the mesh to 2D original images to visualize the reprojection errors at frame 2 and 58, respectively.
Fig. 8: Visual comparison between synthetic and reference original images. Tissue and instrument GS are both learned from the real training set. Reference images are selected from another video with different instrument poses, which are estimated by SAPET and serve as user-defined input to repose the instrument GS.
Fig. 9: Qualitative visualization of downstream keypoint detection task on the testing set. We visualize estimated keypoints with/without unseen-pose data augmentation during training.