cs.HCMar 9, 2026

Extend Your Horizon: A Device-Agnostic Surgical Tool Tracking Framework with Multi-View Optimization for Augmented Reality

Authors: Jiaming Zhang, Mingxu Liu, Hongchao Shu, Ruixing Liang, Yihao Liu, Ojas Taskar, Amir Kheradmand, Mehran Armand, +1 more

Organizations: Johns Hopkins University · Johns Hopkins University, School of Medicine · University of Arkansas, Johns Hopkins University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Surgical navigation provides real-time guidance by estimating the pose of patient anatomy and surgical instruments to visualize relevant intraoperative information. In conventional systems, instruments are typically tracked using fiducial markers and stationary optical tracking systems (OTS). Augmented reality (AR) has further enabled intuitive visualization and motivated tracking using sensors embedded in head-mounted displays (HMDs). However, most existing approaches rely on a clear line of sight, which is difficult to maintain in dynamic operating room environments due to frequent occlusions caused by equipment, surgical tools, and personnel. This work introduces a framework for tracking surgical instruments under occlusion by fusing multiple sensing modalities within a dynamic scene graph representation. The proposed approach integrates tracking systems with different accuracy levels and motion characteristics while estimating tracking reliability in real time. Experimental results demonstrate improved robustness and enhanced consistency of AR visualization in the presence of occlusions.

Figures & tables

Figure 1: We integrate numerous trackers of diverse types into a standard surgical tracking scenario, ensuring continuity during clinical procedures and preventing track loss. (a) In cases of occlusion, our framework autonomously searches plausible paths within the Dynamic Scene Graph to reestablish tracking continuity. When direct tracking is unattainable, our framework delivers fusion uncertainty estimations, enabling the completion of the AR scene for surgical guidance assistance. (b) The architecture of our framework has a central server that communicates with other sensors through TCP/IP. Each sensor tracks surgical tools independently and updates its tracking results based on the feedback from the central server. AR HMD works as a sensor and a visualization tool to render the tracked tools with their optimized poses.
Figure 2: (a) A mapping from a mock AR-guided orthopedic surgery to DSG representation. The surgeon with an AR-HMD interacts with a bone phantom positioned on an X-ray table under a C-arm device. Both the phantom and the surgeon’s pointer are tagged with retro-reflective markers to enable tracking by OTS and AR-HMD. Within the DSG, passive nodes such as the C-arm, bone model, and pointer are depicted using npj,j∈1,2,3n_{pj},j\in{1,2,3} (highlighted in orange), while active nodes like the OTS and AR-HMD are denoted as nai,i∈1,2n_{ai},i\in{1,2} (in dark blue). The inter-layer edges bridge the nain_{ai} and npjn_{pj} , meaning the passive nodes are being tracked by the active nodes. The intra-layer edges connect only npjn_{pj} , representing the relative pose between the passive nodes. (b) demonstrates that when a direct tracking from na1n_{a1} to np1n_{p1} is lost, we can still track it indirectly through querying a path within the DSG.
Figure 3: Example illustrating how our framework compensates for tracking loss. Top : At frame 103, all three objects are directly visible to the HoloLens, and a green indicator is displayed next to each rendered mesh to denote reliable tracking. Bottom : After the pointer is occluded by a hand at frame 115, the green indicator transitions into a yellow uncertainty ellipsoid, which represents the estimated pose uncertainty of the restored object based on the optimized dynamic scene graph. Note that the apparent misalignment between the virtual and physical objects is caused by the mixed reality capture functionality of the HoloLens, and not due to tracking errors.
Figure 4: (a) Experiment setup for a real-world quantitative experiment. Two operators were wearing HoloLens 2 for tracking and real-time rendering of the tracked markers. An NDI Polaris optical tracker and an Atracsys FusionTrack optical tracker were placed next to the operating table with their view point set to cover the table. The operators used pointers to mimic a drilling procedure on a phantom anatomy. (b) Three types of markers were used in the experiment. Each marker is equipped with four reflective spheres.
Figure 5: A segment of tracking for the ground truth, the HoloLens tracking, the NDI tracking, and our framework. Top panel shows the tracking status (tracked or lose track), and the bottom panel shows the trajectories decomposed into three axes.
Figure 6: Example measured trajectory recorded when drawing an ”IEEE VR” text with a pointer. HoloLens, although it covers a larger area of the text, has the most outliers, and its letter ”VR” drifted from the main trajectory, indicating a wrong detection of the markers. As opposed to HoloLens, NDI Polaris has better tracking consistency, yet due to its limited FoV, the letter ”I” is untrackable. Our proposed fusion can leverage the advantages of both sensors and outperforms each of the single sensors.

Explore similar work

CardsList

  1. SurgNavAR: An Augmented Reality Surgical Navigation Framework for Optical See-Through Head Mounted Displays

    Mar 31, 2026Abdullah Thabit, Mohamed Benmahdjoub, Rafiuddin Jinabade +5

    Biomedical Imaging Group Rotterdam, Department of Radiology & Nuclear Medicine, Erasmus MC, 3015 GD Rotterdam, The Netherlands · Department of Oral and Maxillofacial Surgery, Erasmus MC, 3015 GD Rotterdam, The Netherlands · University of Groningen, 9712 CP Groningen, The Netherlands +3

  2. Real-Time Visual Obstruction Detection in Surgical Augmented Reality

    Jul 31, 2026Shih-Chin Yang, Yanming Xiu, Hanting Ye +3Efficient InferenceMedical VLMs

    Department of Computer Science, Duke University · Department of Electrical and Computer Engineering, Duke University

  3. Augmenting Real-time Visualization in Minimally Invasive Surgery with Particle Filtering and Depth-Aware Occlusion Handling

    Oct 13, 2026Anuska Sarkar, Partha Acharya, Vishnu Nandury +2Occlusion ReasoningSurgical Video Understanding

    School of Medical Science and Technology, Indian Institute of Technology Kharagpur, Kharagpur 721302 India · Computer Science Department, Arizona State University, Tempe, Arizona 85287 USA · Narayana Hospital, Kolkata 700107 India