Purpose: Accurate detection and 6D pose estimation of surgical instruments are crucial for many computer-assisted interventions. However, supervised methods lack flexibility for new or unseen tools and require extensive annotated data. This work introduces a training-free pipeline for accurate multi-view 6D pose estimation of unseen surgical instruments, which only requires a textured CAD model as prior knowledge. Methods: Our pipeline consists of two main stages. First, for detection, we generate object mask proposals in each view and score their similarity to rendered templates using a pre-trained feature extractor. Detections are matched across views, triangulated into 3D instance candidates, and filtered using multi-view geometric consistency. Second, for pose estimation, a set of pose hypotheses is iteratively refined and scored using feature-metric scores with cross-view attention. The best hypothesis undergoes a final refinement using a novel multi-view, occlusion-aware contour registration, which minimizes reprojection errors of unoccluded contour points. Results: The proposed method was rigorously evaluated on real-world surgical data from the MVPSP dataset. The method achieves millimeter-accurate pose estimates that are on par with supervised methods under controlled conditions, while maintaining full generalization to unseen instruments. These results demonstrate the feasibility of training-free, marker-less detection and tracking in surgical scenes, and highlight the unique challenges in surgical environments. Conclusion: We present a novel and flexible pipeline that effectively combines state-of-the-art foundational models, multi-view geometry, and contour-based refinement for high-accuracy 6D pose estimation of surgical instruments without task-specific training. This approach enables robust instrument tracking and scene understanding in dynamic clinical environments.
Figures & tables
Figure 1: Overview of our proposed pipeline. We first detect surgical instruments from multiple views using the detection stage outlined in Section 3.1 . The 6D pose of all detected instances is then estimated using the pose estimation stage presented in Section 3.2 .
Figure 2: Comparison of pose estimate error distributions from the supervised and training-free approaches. To enable a fair comparison to the supervised baseline, we use ground-truth detections in form of modal object masks as input to our pose estimation stage. All comparisons are done on 5 input views of the MVPSP wetlab test set (left) and the OR-X test set (right).
Figure 3: Comparison of pose estimate error distributions depending on the instrument detections. We compare our detection stage against the SAM2 with a scoring oracle and the ground-truth detections with 5 input views on the MVPSP OR-X test set. Dashed lines indicate results of MVFP without refinement; solid lines of the same color show the results after contour-based refinement.
Figure 4: Qualitative results from the detection and pose estimation stages. The left image shows an exemplary instrument detection of our 5-view model. The center image shows the extraction of unoccluded contour points, where the outline of the pose hypothesis is rendered in orange and the extracted contour points are highlighted in blue. Note the absence of contour points along the hand and the scrubs on the bottom of the image. An exemplary result of the final pose estimate of our pipeline with five views and on ground-truth detections is displayed in the right image.
Figure 5: Representative failure cases from the masklet generation (left), masklet classification (middle), and coarse pose estimation step (right). We superimpose the detections with their predicted masklets and their bounding boxes in blue and black. For the pose estimation, we superimpose extracted contour points in blue and the estimated instrument pose as an orange outline. The coarse pose estimate is flipped along the instruments axis and unrecoverable for the refinement stage.
Surgical instrument pose estimation provides crucial information for promising applications, including autonomous robotic surgery, skill assessment, and standardization of surgical workflow. However, this task remains highly challenging due to high precision requirements, frequent occlusions, textureless instruments, scarcity of depth information and very limited annotated data. These constraints often lead to unsatisfactory performance when employing general object pose estimation approaches to surgical scenarios. To address these issues, we first construct a new dataset SynSurg6D, to alleviate the data shortage in this task. We further propose SurfSurg6D, a dense-correspondence framework tailored for surgical instrument pose estimation. Experimental results on the SurgRIPE, EndoVis2018 and SurgPose datasets demonstrate that the introduction of our generated dataset SynSurg6D is able to diversify the pose distributions, thus enhancing the performance of existing approaches. Furthermore, SurfSurg6D outperforms existing methods, providing a robust solution for precise and efficient RGB-only pose estimation.
Daiyun Shen, Shuojue Yang, Chang Han Low +4
National University of Singapore, Singapore · The Chinese University of Hong Kong, HKSAR, China
Purpose: Accurate 6 DoF pose estimation of surgical tools is critical for automa- tion, robotic proprioception, and safe interaction with the tissue operated on. Kinematics-based approaches suffer from accumulated errors due to the cable- driven nature of robotic arms, while vision-based methods often rely on external markers or trackers. Although more recent vision-based advances have been pro- posed, these two-stage pose estimation methods often lack real-time robustness due to accumulated errors and computational overhead. Methods: We propose a novel end-to-end trainable model, PICO. Our model employs a multi-task learning architecture to predict segmentation and depth maps, alongside regression of translation and rotation parameters. We define two proxy tasks that enforce geometric consistency in both 2D and 3D spaces, improving accuracy and robustness. For this, we propose a projection loss, and a point-to-point loss. Results: We evaluate our method on the SurgRIPE dataset, benchmarking its performance against state-of-the-art approaches using standard 6DoF pose esti- mation metrics. Our results demonstrate consistently strong performance across all four subsets, specifically in rotation, ranking second even under occlusion. It also demonstrates comparable translational performance, remaining competitive, especially in occluded cases. Conclusion: PICO demonstrates the effectiveness of multi-task learning and geometry-aware proxy tasks for robust and reliable surgical tool pose estimation, especially in occluded scenarios, highlighting potential for future applications.
Lucy Fothergill, Pietro Valdastri, Dominic Jones +1
School of Computer Science, University of Leeds. · School of Electronic and Electrical Engineering, University of Leeds.
Accurate and efficient tracking of surgical instruments is fundamental for Robot-Assisted Minimally Invasive Surgery. Although vision-based robot pose estimation has enabled markerless calibration without tedious physical setups, reliable tool tracking for surgical robots still remains challenging due to partial visibility and specialized articulation design of surgical instruments. Previous works in the field are usually prone to unreliable feature detections under degraded visual quality and data scarcity, whereas rendering-based methods often struggle with computational costs and suboptimal convergence. In this work, we incorporate CMA-ES, an evolutionary optimization strategy, into a versatile tracking pipeline that jointly estimates surgical instrument pose and joint configurations. Using batch rendering to efficiently evaluate multiple pose candidates in parallel, the method significantly reduces inference time and improves convergence robustness. The proposed framework further generalizes to joint angle-free and bi-manual tracking settings, making it suitable for both vision feedback control and online surgery video calibration. Extensive experiments on synthetic and real-world datasets demonstrate that the proposed method significantly outperforms prior approaches in both accuracy and runtime. Source code and data are available at https://github.com/hanyang-hu/online_dvrk_tracking.
Hanyang Hu, Zekai Liang, Florian Richter +1
Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA 92093 USA.