SurgNavAR: An Augmented Reality Surgical Navigation Framework for Optical See-Through Head Mounted Displays
Authors: Abdullah Thabit, Mohamed Benmahdjoub, Rafiuddin Jinabade, Hizirwan S. Salim, Marie-Lise C. van Veelen, Mark G. van Vledder, Eppo B. Wolvius, Theo van Walsum
Organizations: Biomedical Imaging Group Rotterdam, Department of Radiology & Nuclear Medicine, Erasmus MC, 3015 GD Rotterdam, The Netherlands · Department of Oral and Maxillofacial Surgery, Erasmus MC, 3015 GD Rotterdam, The Netherlands · University of Groningen, 9712 CP Groningen, The Netherlands · Department of High Performance Compute & Visualization, SURF bv, 3511 EP Utrecht, The Netherlands · Department of Neurosurgery, Erasmus MC, 3015 GD Rotterdam, The Netherlands · Trauma Research Unit, Department of Surgery, Erasmus MC, 3015 GD Rotterdam, The Netherlands
Augmented reality (AR) devices with head mounted displays (HMDs) facilitate the direct superimposition of 3D preoperative imaging data onto the patient during surgery. To use an HMD-AR device as a stand-alone surgical navigation system, the device should be able to locate the patient and surgical instruments, align preoperative imaging data with the patient, and visualize navigation data in real time during surgery. Whereas some of the technologies required for this are known, integration in such devices is cumbersome and requires specific knowledge and expertise, hampering scientific progress in this field. This work therefore aims to present and evaluate an integrated HMD-based AR surgical navigation framework that is adaptable to diverse surgical applications. The framework tracks 2D patterns as reference markers attached to the patient and surgical instruments. It allows for the calibration of surgical tools using pivot and reference-based calibration techniques. It enables image-to-patient registration using point-based matching and manual positioning. The integrated functionalities of the framework are evaluated on two HMD devices, the HoloLens 2 and Magic Leap 2, with two surgical use cases being evaluated in a phantom setup: AR-guided needle insertion and rib fracture localization. The framework was able to achieve a mean tooltip calibration accuracy of 1 mm, a registration accuracy of 3 mm, and a targeting accuracy below 5 mm on the two surgical use cases. The framework presents an easy-to-use configurable tool for HMD-based AR surgical navigation, which can be extended and adapted to many surgical applications. The framework is publicly available at https://github.com/abdullahthabit/SurgNavAR.
Figures & tables
Fig. 1: Overview of the framework and its navigation modules (left). The modules work together to provide AR guidance for the surgeon (right).
Fig. 2: Framework architecture.
Fig. 3: Tracking of reference markers in the framework with sample ArUco/AprilTag and Vuforia markers
Fig. 4: Calibration of surgical instruments: tool-tip pose estimation (a) using pivot calibration (b), a calibration tool (c), and marker-to-marker user calibration (d)
Fig. 5: Image-to-patient alignment using point-based matching between the virtual model and patient.
Fig. 6: Tool-tip calibration experiment: (a) NDI pointer with a Vuforia marker attached. (b) The pointer tracked and calibrated in AR. (c-e) calibrating the pointer using pivot calibration, a calibration tool, and using marker-to-marker user calibration.
Fig. 7: Tool-tip calibration error: (a) pivot calibration, (b) using a calibration tool, (c-d) marker-to-marker user calibration with reference marker size of 8 cm (c) and 12 cm (d). V8-HL2 in the legend indicates the marker type (Vuforia), marker size (8 cm) and HMD device (HL2).
Device
Marker
Pivot calibration
Tool calibration
Marker-to-marker user calibration
8 cm (P)
5 cm (P)
8 cm (P) - 10 cm (T)
5 cm (P) - 10 cm (T)
8 cm (P) - 8 cm (R)
8 cm (P) - 12 cm (R)
5 cm (P) - 12 cm (R)
Tip (mm)
Tip (mm)
Tip (mm)
Angle (°)
Tip (mm)
Angle (°)
Tip (mm)
Angle (°)
Tip (mm)
Angle (°)
Tip (mm)
Angle (°)
HL2
ArUco
2.9 (0.9)
4.9 (2.1)
1.8 (0.4)
2.7 (1.9)
Vuforia
1.2 (0.4)
1.5 (0.4)
1.1 (0.3)
1.3 (0.1)
1.7 (0.2)
1.4 (0.1)
2.5 (0.5)
0.7 (0.3)
1.1 (0.5)
0.4 (0.3)
2.0 (0.7)
0.9 (0.4)
ML2
ArUco
1.4 (0.3)
1.8 (0.7)
3.2 (0.4)
2.3 (1.4)
Vuforia
1.1 (0.4)
1.5 (0.4)
1.0 (0.2)
0.9 (0.1)
1.8 (0.3)
1.3 (0.1)
2.9 (0.5)
1.3 (0.3)
2.8 (0.6)
1.3 (0.3)
3.9 (1.4)
1.8 (0.7)
TABLE I: Tool-tip calibration error [mean(std)] of the proposed calibration approaches for different HMD devices and markers, with (P) pointer marker, (T) tool marker, and (R) reference marker.
Fig. 8: Image-to-patient alignment experiment: (a) patient-board with Vuforia and NDI markers attached, (b) aligned virtual model, (c) points annotation using the pointer, (d-e) manual positioning using 6-DoF and 1-DoF manipulation.
Number of registration points
Annotation
Error metric
Vuforia
ArUco
8 cm
5 cm
8 cm
5 cm
GT-ttp
Pivot-ttp
GT-ttp
Pivot-ttp
GT-ttp
Pivot-ttp
GT-ttp
Pivot-ttp
10 points
Landmarks (mm)
NDI FRE
1.2 (0.1)
1.1 (0.0)
1.1 (0.1)
1.1 (0.1)
HMD FRE
2.2 (0.3)
2.8 (0.3)
2.2 (0.3)
2.7 (0.3)
1.5 (0.3)
1.6 (0.3)
2.1 (0.3)
2.1 (0.4)
HMD TRE (NDI-10)
2.1 (0.3)
2.6 (0.3)
2.7 (0.5)
2.4 (0.5)
3.0 (0.3)
3.2 (0.3)
3.6 (0.3)
4.1 (0.4)
Trajectories
Distance (mm)
1.6 (0.4)
2.2 (0.4)
2.0 (0.6)
2.0 (0.5)
2.6 (0.3)
2.8 (0.4)
3.4 (0.3)
3.5 (0.4)
TABLE II: HL2 registration error [mean(std)] for point-based matching for different combinations of points and markers.
Number of registration points
Annotation
Error metric
Vuforia
ArUco
8 cm
5 cm
8 cm
5 cm
GT-ttp
Pivot-ttp
GT-ttp
Pivot-ttp
GT-ttp
Pivot-ttp
GT-ttp
Pivot-ttp
10 points
Landmarks (mm)
NDI FRE
1.2 (0.0)
1.2 (0.0)
1.2 (0.0)
1.2 (0.0)
HMD FRE
2.3 (0.3)
2.4 (0.3)
2.1 (0.3)
2.7 (0.4)
1.8 (0.2)
1.9 (0.2)
2.9 (0.5)
2.2 (0.5)
HMD TRE (NDI-10)
2.1 (0.2)
2.7 (0.2)
2.0 (0.3)
2.0 (0.3)
2.4 (0.2)
2.6 (0.3)
3.0 (0.3)
2.2 (0.3)
Trajectories
Distance (mm)
1.3 (0.4)
1.9 (0.4)
1.1 (0.4)
1.4 (0.4)
1.6 (0.1)
1.8 (0.4)
2.4 (0.3)
1.5 (0.2)
TABLE III: ML2 registration error [mean(std)] for point-based matching for different combinations of points and markers.
Fig. 9: Surface-points mean registration error for the head phantom using 10, 5, and 6 registration points for both HL2 and ML2 using 8 and 5 cm marker sizes.
Fig. 10: Registration error (HMD TRE NDI-10) for different number of reintegration points: (left) order of registration points, (right) registration error as the number of points increases from 3 to 10 points.
Fig. 11: Locating rib-fractures experiment: (a) torso phantom with Vuforia and NDI markers, (b) planned incision lines, (c) aligned torso virtual model, (d) delineated incision lines by the user.
Fig. 12: Surface-points mean registration error for point-based (PB) matching and manual positioning (MP) for both HL2 and ML2 performed by the participants on the head and torso phantoms.
Fig. 13: Delineation error of the incision lines by participants in the rib-fracture localization experiment: (left) Box plots of the delineation error for both HL2 and ML2. Left(H) and Left(V) represent the left-side horizontal and vertical planned incisions. (right) incision lines marked by the participants on both sides of the torso phantom.
Fig. 14: Ventricular catheter placement experiment: (a) head phantom with entry holes, (b) two insertion planned trajectories, (c) aligned virtual head with planned trajectory, (d) insertion guidance of the pointer with virtual extensions and numerical feedback menu.
Device
Phantom
Point-based matching
Manual registration
Accuracy
Time
Accuracy
Time
HL2 (n=8)
Head
2.6 (1.1)
2:38 (0:53)
8.9 (2.8)
8:25 (3:24)
Torso
2.5 (1.0)
1:58 (1:04)
9.4 (2.5)
6:58 (3:08)
ML2 (n=8)
Head
3.0 (1.1)
4:12 (2:21)
4.6 (1.3)
4:57 (1:37)
Torso
3.0 (1.0)
3:01 (1:25)
7.1 (2.2)
7:23 (4:24)
Total (n=16)
Head
2.8 (1.1)
3:29 (1:59)
6.7 (2.0)
6:48 (3:13)
TABLE IV: Point-based matching vs Manual-positioning in aligning head and torso virtual models. Accuracy is the HMD TRE NDI-10 [mean(std)] and time is the alignment completion time [mean(std)]
Device
Tool-tip calibration error
Registration error (trajectories)
Entry point
Exit point
Entry to exit (mean)
Perception
HMD
NDI
Perception
HMD
NDI
Perception
HMD
NDI
HL2 (n=8)
Distance (mm)
1.2 (0.2)
2.6 (0.2)
0.9 (0.6)
3.2 (1.2)
3.1 (1.6)
1.4 (0.9)
2.9 (0.9)
3.3 (1.3)
2.3 (2.3)
3.5 (2.4)
3 (1.0)
Angle (°)
0.9 (0.3)
1.6 (0.2)
0.7 (0.5)
2.3 (0.7)
2.3 (0.7)
0.7 (0.3)
2.3 (0.5)
2.5 (0.7)
1.3 (1.4)
2.8 (1.3)
2.7 (0.8)
ML2 (n=8)
Distance (mm)
1.8 (0.7)
2.2 (0.1)
0.9 (0.5)
1.7 (0.7)
2.0 (1.0)
1.4 (0.6)
2.9 (0.8)
2.6 (1.0)
2.3 (1.6)
2.7 (1.6)
2.2 (0.9)
Angle (°)
0.9 (0.8)
0.7 (0.1)
0.6 (0.4)
1.6 (0.7)
1.8 (0.7)
0.8 (0.5)
1.6 (0.7)
1.7 (0.7)
1.5 (1.0)
2.2 (1.0)
2.2 (1.0)
Total (n=16)
Distance (mm)
1.5 (0.6)
2.4 (0.1)
0.9 (0.6)
2.4 (0.9)
2.5 (1.3)
1.4 (0.7)
2.9 (0.8)
3.0 (1.2)
2.3 (2.0)
3.1 (2.0)
2.6 (0.9)
TABLE V: Mean calibration, registration and insertion errors for the ventricular catheter placement user study, for both HL2 and ML2.
Surgical navigation provides real-time guidance by estimating the pose of patient anatomy and surgical instruments to visualize relevant intraoperative information. In conventional systems, instruments are typically tracked using fiducial markers and stationary optical tracking systems (OTS). Augmented reality (AR) has further enabled intuitive visualization and motivated tracking using sensors embedded in head-mounted displays (HMDs). However, most existing approaches rely on a clear line of sight, which is difficult to maintain in dynamic operating room environments due to frequent occlusions caused by equipment, surgical tools, and personnel. This work introduces a framework for tracking surgical instruments under occlusion by fusing multiple sensing modalities within a dynamic scene graph representation. The proposed approach integrates tracking systems with different accuracy levels and motion characteristics while estimating tracking reliability in real time. Experimental results demonstrate improved robustness and enhanced consistency of AR visualization in the presence of occlusions.
Jiaming Zhang, Mingxu Liu, Hongchao Shu +6
Johns Hopkins University · Johns Hopkins University, School of Medicine · University of Arkansas, Johns Hopkins University
This paper presents the computational architecture and quantitative preclinical evaluation of MIRAGE, a multimodal image-guided robotic surgery framework for real-time visualization and navigation. It integrates stereovision scene reconstruction, augmented-reality (AR) video overlay, quantified heads-up display (HUD) navigation, and intraoperative ultrasound within the da Vinci console's TilePro display (Intuitive Surgical). Additional modules include optical/electromagnetic tracker registration, stereo calibration, fiducial rigid registration with RMSE feedback, and 3D point-cloud reconstruction. Video I/O and rendering run in a C++ backend, while a Python GUI coordinates calibration, registration, reconstruction, visualization, and user interaction through an in-process zero-copy binding layer. The framework formalizes spatial transforms, module states, and synchronization of heterogeneous video and tracking streams. Preoperative calibration uses synchronized checkerboard image pairs and optimization, with estimated intrinsics, extrinsics, and rectification reused intraoperatively. MIRAGE was validated in phantom experiments across da Vinci S, Si, Xi, and SP platforms. Across clinically relevant working distances, mean 3D reprojection error was 0.6mm, with best 0.1mm RMSE at 100mm on Xi. Results support MIRAGE as a preclinical technical-validation platform for accurate, responsive, and extensible multimodal surgical guidance.
Ehsan Nasiri, Yi Zheng, Michael A. Kokko +4
Thayer School of Engineering at Dartmouth 15 Thayer Dr, Hanover, NH, USA
The handling and assembly of instruments during surgery imposes high cognitive demands on scrub nurses, particularly when instruments are unfamiliar. We present a supporting guidance system for surgical instrumentation that combines multi-camera 6D pose estimation with augmented reality in-situ visualization on a head-mounted display without the requirement for additional markers. Pose estimation and consecutive camera calibration are achieved through known objects. The 6D pose estimation network is trained purely on synthetic data, aiming for better generalizability and real-world applicability. The AR guidance displays tooltip localization cues and step-wise assembly animations. Via gaze-based selection and a foot pedal, users can switch between assembly steps in intraoperative use. In a technical evaluation, our approach outperforms state-of-art 6D pose estimation. A user study with 29 scrub nurses was conducted in a surgical simulation of knee arthroplasty, comparing the system against a paper manual. AR guidance significantly reduced the perceived workload compared. Objectively, AR guidance reduced task completion time by 21.3% (4.76 minutes). Specifically, scrub nurses less experienced with the instrument set benefited when using the system. Error frequencies were comparable between conditions. Qualitative feedback highlighted improved process clarity, reduced information overload, and perceived independence. To summarize, our marker-free multi-camera AR guidance approach for surgical instruments can, subjectively and objectively, improve intraoperative instrumentation performance, particularly for untrained scrub nurses.
Shiyu Li, Julian Kreimeier, Hannah Schieber +4
Technical University of Munich, Human-Centered-Computing and Extended Reality Lab · TUM University Hospital, Orthopedics and Sports Orthopedics · Munich Institute of Robotics and Machine Intelligence (MIRMI) +2