Instrument-Splatting++: Towards Controllable Surgical Instrument Digital Twin Using Gaussian Splatting
Authors: Shuojue Yang, Zijian Wu, Chengjiaao Liao, Qian Li, Daiyun Shen, Chang Han Low, Septimiu E. Salcudean, Yueming Jin
Organizations: Department of Biomedical Engineering, National University of Singapore (NUS), Singapore. · Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, Canada · Department of Electrical and Computer Engineering, NUS.
High-quality and controllable digital twins of surgical instruments are critical for Real2Sim in robot-assisted surgery, as they enable realistic simulation, synthetic data generation, and perception learning under novel poses. We present Instrument-Splatting++, a monocular 3D Gaussian Splatting (3DGS) framework that reconstructs surgical instruments as a fully controllable Gaussian asset with high fidelity. Our pipeline starts with part-wise geometry pretraining that injects CAD priors into Gaussian primitives and equips the representation with part-aware semantic rendering. Built on the pretrained model, we propose a semantics-aware pose estimation and tracking (SAPET) method to recover per-frame 6-DoF pose and joint angles from unposed endoscopic videos, where a gripper-tip network trained purely from synthetic semantics provides robust supervision and a loose regularization suppresses singular articulations. Finally, we introduce Robust Texture Learning (RTL), which alternates pose refinement and robust appearance optimization, mitigating pose noise during texture learning. The proposed framework can perform pose estimation and learn realistic texture from unposed videos. We validate our method on sequences extracted from EndoVis17/18, SAR-RARP, and an in-house dataset, showing superior photometric quality and improved geometric accuracy over state-of-the-art baselines. We further demonstrate a downstream keypoint detection task where unseen-pose data augmentation from our controllable instrument Gaussian improves performance.
Figures & tables
Fig. 1: Overall pipeline of our Real2Sim pipeline Instrument-Splatting++ , which yields controllable GS digital twin of surgical instrument (Textured GS) with high visual and geometric fidelity.
Fig. 2: Overview of the proposed Instrument-Splatting++. Black arrows indicate operation flow, while red arrows denote supervisory signals. The textured GS in (c) denotes the generated GS digital twin of instrument with learned textures. Fire and snowflake icons represent optimizable and frozen parameters, respectively.
Fig. 3: Forward kinematics definition of the da Vinci EndoWrist Large Needle Driver (LND).
Fig. 4: Left: Gaussian update under perturbed instrument poses. Right: The pose update is affected by the quality of Gaussian model.
Method
Intraoperative
In-house
PSNR ↑
SSIM ↑
LPIPS ↓
RMSE ↓
RMSE-M ↓
PSNR ↑
SSIM ↑
LPIPS ↓
RMSE ↓
RMSE-M ↓
WIM
22.34
92.75
0.170
25.7
16.5
23.44
92.15
0.098
19.8
16.1
SC-GS
20.42
89.48
0.146
19.8
11.3
24.41
93.28
0.070
17.4
15.6
Shape-of-motion
22.35
93.46
0.088
13.2
8.1
24.88
94.34
0.067
20.5
6.6
TTGS
22.98
93.29
0.093
18.8
16.1
25.45
94.46
0.078
23.5
14.0
EndoGaussian
22.39
93.64
0.089
27.9
19.8
25.26
93.85
0.075
25.2
14.8
TABLE I: Quantitative comparison on intraoperative and in-house datasets. Intraoperative Dataset represents sequences from both EndoVis and SAR-RARP dataset. RMSE and RMSE-M represent depth error within entire image space and instrument mask, respectively.
Fig. 5: Visualization of the texture learning results on testing sets in comparison with SOTA reconstruction methods. We visualize the RGB and depth rendering. For SAR-RARP sequence (the first row), we use the depth rendering from CAD mesh with the estimated pose as the reference depth. For the other dataset, stereo depth estimations serve as the reference depth.
Fig. 6: Visualization of the variant methods on texture learning.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
RMSE ↓
w/o pose ref.
29.24
95.69
0.051
5.6
w/o Rob
28.64
95.51
0.047
5.6
w/ joint Optim.
22.48
91.17
0.068
5.9
RTL
29.82
96.27
0.039
5.6
TABLE II: Analysis of texture learning on In-House Dataset. PSNR, SSIM, and LPIPS measure the photometric accuracy while RMSE measures depth accuracy.
Method
EndoVis17&18 (Dice ↑ )
In-House (Keypoint)
Shaft
Wrist
Gripper
RMSE ↓ (pix)
RMSE ↓ (mm)
w/o Matching
93.73
83.12
66.03
9.79
8.64
w/o Reg
95.52
78.40
63.39
6.36
5.45
w/ SVD
96.83
86.65
73.32
5.85
5.12
w/o Gripper-Tip Net
94.63
83.94
63.17
10.11
7.58
SAPET
95.92
87.54
75.27
4.75
3.69
TABLE III: Analysis on pose estimation and tracking performance. Comparison of model variants on EndoVis17&18 and the In-House dataset.
Fig. 7: Visualization of the pose estimation results on in-house dataset. We visualize the predicted and GT trajectories of the left and right gripper tip points across a 100-frame video clip. The predicted and GT trajectories are colored in green and red, respectively. With estimated poses, we render the mesh to 2D original images to visualize the reprojection errors at frame 2 and 58, respectively.
Fig. 8: Visual comparison between synthetic and reference original images. Tissue and instrument GS are both learned from the real training set. Reference images are selected from another video with different instrument poses, which are estimated by SAPET and serve as user-defined input to repose the instrument GS.
Fig. 9: Qualitative visualization of downstream keypoint detection task on the testing set. We visualize estimated keypoints with/without unseen-pose data augmentation during training.
Surgical instrument pose estimation provides crucial information for promising applications, including autonomous robotic surgery, skill assessment, and standardization of surgical workflow. However, this task remains highly challenging due to high precision requirements, frequent occlusions, textureless instruments, scarcity of depth information and very limited annotated data. These constraints often lead to unsatisfactory performance when employing general object pose estimation approaches to surgical scenarios. To address these issues, we first construct a new dataset SynSurg6D, to alleviate the data shortage in this task. We further propose SurfSurg6D, a dense-correspondence framework tailored for surgical instrument pose estimation. Experimental results on the SurgRIPE, EndoVis2018 and SurgPose datasets demonstrate that the introduction of our generated dataset SynSurg6D is able to diversify the pose distributions, thus enhancing the performance of existing approaches. Furthermore, SurfSurg6D outperforms existing methods, providing a robust solution for precise and efficient RGB-only pose estimation.
Daiyun Shen, Shuojue Yang, Chang Han Low +4
National University of Singapore, Singapore · The Chinese University of Hong Kong, HKSAR, China
Purpose: Accurate detection and 6D pose estimation of surgical instruments are crucial for many computer-assisted interventions. However, supervised methods lack flexibility for new or unseen tools and require extensive annotated data. This work introduces a training-free pipeline for accurate multi-view 6D pose estimation of unseen surgical instruments, which only requires a textured CAD model as prior knowledge. Methods: Our pipeline consists of two main stages. First, for detection, we generate object mask proposals in each view and score their similarity to rendered templates using a pre-trained feature extractor. Detections are matched across views, triangulated into 3D instance candidates, and filtered using multi-view geometric consistency. Second, for pose estimation, a set of pose hypotheses is iteratively refined and scored using feature-metric scores with cross-view attention. The best hypothesis undergoes a final refinement using a novel multi-view, occlusion-aware contour registration, which minimizes reprojection errors of unoccluded contour points. Results: The proposed method was rigorously evaluated on real-world surgical data from the MVPSP dataset. The method achieves millimeter-accurate pose estimates that are on par with supervised methods under controlled conditions, while maintaining full generalization to unseen instruments. These results demonstrate the feasibility of training-free, marker-less detection and tracking in surgical scenes, and highlight the unique challenges in surgical environments. Conclusion: We present a novel and flexible pipeline that effectively combines state-of-the-art foundational models, multi-view geometry, and contour-based refinement for high-accuracy 6D pose estimation of surgical instruments without task-specific training. This approach enables robust instrument tracking and scene understanding in dynamic clinical environments.
Jonas Hein, Lilian Calvet, Matthias Seibold +3
University Hospital Balgrist, University of Zurich, Switzerland. · ETH Zurich, Switzerland.
Deploying robots in unstructured real-world environments needs accurate, interactive models of the objects. Constructing these models at scale remains a critical bottleneck for robotic system integration. We present ArtiTwinSplat, a framework that automatically constructs articulated, photo-realistic digital twins of objects directly from RGB-D videos, requiring no CAD models, simulation assets, or manual annotations. Our method is built on 3D Gaussian Splatting that preserve geometric fidelity and photometric realism, coupled with an unsupervised articulation discovery pipeline that recovers part structure and joint kinematics from observed motion alone. With tracking and optimization stages our method provides stable, queryable digital twins that support real-time rendering, viewpoint control, and interactive manipulation. Unlike prior methods confined to simulation, ArtiTwinSplat operates directly on real-world observations and produces twins that are immediately usable by downstream robot planning and learning systems. This method offers a practical, scalable pathway toward digital twin construction, lowering the integration barrier for articulated object manipulation in embodied AI and human-robot collaboration contexts.
Pranjal Mishra, René Zurbrügg, Max Wilder-Smith +4
1ETH Zürich, Zürich, Switzerland. · 2Microsoft, Zürich, Switzerland. · University of Bonn, Bonn, Germany.