Surgical navigation provides real-time guidance by estimating the pose of patient anatomy and surgical instruments to visualize relevant intraoperative information. In conventional systems, instruments are typically tracked using fiducial markers and stationary optical tracking systems (OTS). Augmented reality (AR) has further enabled intuitive visualization and motivated tracking using sensors embedded in head-mounted displays (HMDs). However, most existing approaches rely on a clear line of sight, which is difficult to maintain in dynamic operating room environments due to frequent occlusions caused by equipment, surgical tools, and personnel. This work introduces a framework for tracking surgical instruments under occlusion by fusing multiple sensing modalities within a dynamic scene graph representation. The proposed approach integrates tracking systems with different accuracy levels and motion characteristics while estimating tracking reliability in real time. Experimental results demonstrate improved robustness and enhanced consistency of AR visualization in the presence of occlusions.
Figures & tables
Figure 1: We integrate numerous trackers of diverse types into a standard surgical tracking scenario, ensuring continuity during clinical procedures and preventing track loss. (a) In cases of occlusion, our framework autonomously searches plausible paths within the Dynamic Scene Graph to reestablish tracking continuity. When direct tracking is unattainable, our framework delivers fusion uncertainty estimations, enabling the completion of the AR scene for surgical guidance assistance. (b) The architecture of our framework has a central server that communicates with other sensors through TCP/IP. Each sensor tracks surgical tools independently and updates its tracking results based on the feedback from the central server. AR HMD works as a sensor and a visualization tool to render the tracked tools with their optimized poses.
Figure 2: (a) A mapping from a mock AR-guided orthopedic surgery to DSG representation. The surgeon with an AR-HMD interacts with a bone phantom positioned on an X-ray table under a C-arm device. Both the phantom and the surgeon’s pointer are tagged with retro-reflective markers to enable tracking by OTS and AR-HMD. Within the DSG, passive nodes such as the C-arm, bone model, and pointer are depicted using npj,j∈1,2,3 (highlighted in orange), while active nodes like the OTS and AR-HMD are denoted as nai,i∈1,2 (in dark blue). The inter-layer edges bridge the nai and npj , meaning the passive nodes are being tracked by the active nodes. The intra-layer edges connect only npj , representing the relative pose between the passive nodes. (b) demonstrates that when a direct tracking from na1 to np1 is lost, we can still track it indirectly through querying a path within the DSG.
Figure 3: Example illustrating how our framework compensates for tracking loss. Top : At frame 103, all three objects are directly visible to the HoloLens, and a green indicator is displayed next to each rendered mesh to denote reliable tracking. Bottom : After the pointer is occluded by a hand at frame 115, the green indicator transitions into a yellow uncertainty ellipsoid, which represents the estimated pose uncertainty of the restored object based on the optimized dynamic scene graph. Note that the apparent misalignment between the virtual and physical objects is caused by the mixed reality capture functionality of the HoloLens, and not due to tracking errors.
Figure 4: (a) Experiment setup for a real-world quantitative experiment. Two operators were wearing HoloLens 2 for tracking and real-time rendering of the tracked markers. An NDI Polaris optical tracker and an Atracsys FusionTrack optical tracker were placed next to the operating table with their view point set to cover the table. The operators used pointers to mimic a drilling procedure on a phantom anatomy. (b) Three types of markers were used in the experiment. Each marker is equipped with four reflective spheres.
Figure 5: A segment of tracking for the ground truth, the HoloLens tracking, the NDI tracking, and our framework. Top panel shows the tracking status (tracked or lose track), and the bottom panel shows the trajectories decomposed into three axes.
Figure 6: Example measured trajectory recorded when drawing an ”IEEE VR” text with a pointer. HoloLens, although it covers a larger area of the text, has the most outliers, and its letter ”VR” drifted from the main trajectory, indicating a wrong detection of the markers. As opposed to HoloLens, NDI Polaris has better tracking consistency, yet due to its limited FoV, the letter ”I” is untrackable. Our proposed fusion can leverage the advantages of both sensors and outperforms each of the single sensors.
Augmented reality (AR) devices with head mounted displays (HMDs) facilitate the direct superimposition of 3D preoperative imaging data onto the patient during surgery. To use an HMD-AR device as a stand-alone surgical navigation system, the device should be able to locate the patient and surgical instruments, align preoperative imaging data with the patient, and visualize navigation data in real time during surgery. Whereas some of the technologies required for this are known, integration in such devices is cumbersome and requires specific knowledge and expertise, hampering scientific progress in this field. This work therefore aims to present and evaluate an integrated HMD-based AR surgical navigation framework that is adaptable to diverse surgical applications. The framework tracks 2D patterns as reference markers attached to the patient and surgical instruments. It allows for the calibration of surgical tools using pivot and reference-based calibration techniques. It enables image-to-patient registration using point-based matching and manual positioning. The integrated functionalities of the framework are evaluated on two HMD devices, the HoloLens 2 and Magic Leap 2, with two surgical use cases being evaluated in a phantom setup: AR-guided needle insertion and rib fracture localization. The framework was able to achieve a mean tooltip calibration accuracy of 1 mm, a registration accuracy of 3 mm, and a targeting accuracy below 5 mm on the two surgical use cases. The framework presents an easy-to-use configurable tool for HMD-based AR surgical navigation, which can be extended and adapted to many surgical applications. The framework is publicly available at https://github.com/abdullahthabit/SurgNavAR.
Abdullah Thabit, Mohamed Benmahdjoub, Rafiuddin Jinabade +5
Biomedical Imaging Group Rotterdam, Department of Radiology & Nuclear Medicine, Erasmus MC, 3015 GD Rotterdam, The Netherlands · Department of Oral and Maxillofacial Surgery, Erasmus MC, 3015 GD Rotterdam, The Netherlands · University of Groningen, 9712 CP Groningen, The Netherlands +3
Surgical augmented reality (AR) can provide contextual guidance by overlaying virtual annotations, tool cues, and procedural information onto the surgical workspace. However, the virtual content may obstruct task-relevant real-world information, such as surgical instruments, and interfere with users' perception during time-sensitive surgical tasks. In this paper, we investigate visual obstruction detection for surgical AR and present a latency-aware pipeline that combines vision-language model (VLM)-based surgical-object recognition with segmentation-based obstruction reasoning. To reduce inference overhead, the system adopts a cascaded small-to-large VLM architecture with segmentation-guided early exiting and attention-based visual token pruning. The small VLM handles easy frames when its key-object prediction is supported by segmentation consistency, while difficult frames are forwarded to a large VLM with pruned visual tokens. We construct a pseudo-AR surgical obstruction detection benchmark by overlaying virtual content onto surgical-tool images and labeling whether the virtual content obstructs task-relevant instruments. Evaluation results show that the proposed system achieves 87.43% obstruction detection accuracy with an average end-to-end latency of 479 ms, reducing latency by 62.90% compared with a cloud large-model baseline. These results demonstrate the feasibility of latency-aware obstruction detection for surgical AR and motivate future work on dynamic surgical videos, multi-object scenes, and clinically grounded AR guidance content.
Shih-Chin Yang, Yanming Xiu, Hanting Ye +3
Department of Computer Science, Duke University · Department of Electrical and Computer Engineering, Duke University
We present a real-time, hybrid framework for occlusion handling and depth-aware tracking in minimally invasive surgeries (MIS). Occlusions caused by surgical instruments often obscure the region of interest (ROI), affecting surgical precision. To address this, our approach integrates enhanced You Only Look Once (YOLO) based detection, Particle Filtering (PF), and monocular depth estimation using Multiple Depth Estimation Accuracy with Single Network (MiDaS). Occlusion is detected using a distance-based threshold between instruments and ROI. Experimental results demonstrate robust instrument detection with a confidence score of 0.85, effective occlusion event handling, and accurate depth prediction with an MAE of 0.34 cm and RMSE of 0.53 cm. Independent phantom-based validation against stereo depth measurements yielded an MAE of 0.65 cm and RMSE of 0.66 cm. The framework further achieved an occlusion detection accuracy of 81.95% while maintaining real-time performance at 38.7 FPS. Preliminary surgeon feedback indicates that the proposed visualization framework is clinically relevant and holds promise for future intraoperative image-guided navigation and decision support in MIS.
Anuska Sarkar, Partha Acharya, Vishnu Nandury +2
School of Medical Science and Technology, Indian Institute of Technology Kharagpur, Kharagpur 721302 India · Computer Science Department, Arizona State University, Tempe, Arizona 85287 USA · Narayana Hospital, Kolkata 700107 India