SurgNavAR: An Augmented Reality Surgical Navigation Framework for Optical See-Through Head Mounted Displays
Authors: Abdullah Thabit, Mohamed Benmahdjoub, Rafiuddin Jinabade, Hizirwan S. Salim, Marie-Lise C. van Veelen, Mark G. van Vledder, Eppo B. Wolvius, Theo van Walsum
Organizations: Biomedical Imaging Group Rotterdam, Department of Radiology & Nuclear Medicine, Erasmus MC, 3015 GD Rotterdam, The Netherlands · Department of Oral and Maxillofacial Surgery, Erasmus MC, 3015 GD Rotterdam, The Netherlands · University of Groningen, 9712 CP Groningen, The Netherlands · Department of High Performance Compute & Visualization, SURF bv, 3511 EP Utrecht, The Netherlands · Department of Neurosurgery, Erasmus MC, 3015 GD Rotterdam, The Netherlands · Trauma Research Unit, Department of Surgery, Erasmus MC, 3015 GD Rotterdam, The Netherlands
Augmented reality (AR) devices with head mounted displays (HMDs) facilitate the direct superimposition of 3D preoperative imaging data onto the patient during surgery. To use an HMD-AR device as a stand-alone surgical navigation system, the device should be able to locate the patient and surgical instruments, align preoperative imaging data with the patient, and visualize navigation data in real time during surgery. Whereas some of the technologies required for this are known, integration in such devices is cumbersome and requires specific knowledge and expertise, hampering scientific progress in this field. This work therefore aims to present and evaluate an integrated HMD-based AR surgical navigation framework that is adaptable to diverse surgical applications. The framework tracks 2D patterns as reference markers attached to the patient and surgical instruments. It allows for the calibration of surgical tools using pivot and reference-based calibration techniques. It enables image-to-patient registration using point-based matching and manual positioning. The integrated functionalities of the framework are evaluated on two HMD devices, the HoloLens 2 and Magic Leap 2, with two surgical use cases being evaluated in a phantom setup: AR-guided needle insertion and rib fracture localization. The framework was able to achieve a mean tooltip calibration accuracy of 1 mm, a registration accuracy of 3 mm, and a targeting accuracy below 5 mm on the two surgical use cases. The framework presents an easy-to-use configurable tool for HMD-based AR surgical navigation, which can be extended and adapted to many surgical applications. The framework is publicly available at https://github.com/abdullahthabit/SurgNavAR.
Figures & tables
Fig. 1: Overview of the framework and its navigation modules (left). The modules work together to provide AR guidance for the surgeon (right).
Fig. 2: Framework architecture.
Fig. 3: Tracking of reference markers in the framework with sample ArUco/AprilTag and Vuforia markers
Fig. 4: Calibration of surgical instruments: tool-tip pose estimation (a) using pivot calibration (b), a calibration tool (c), and marker-to-marker user calibration (d)
Fig. 5: Image-to-patient alignment using point-based matching between the virtual model and patient.
Fig. 6: Tool-tip calibration experiment: (a) NDI pointer with a Vuforia marker attached. (b) The pointer tracked and calibrated in AR. (c-e) calibrating the pointer using pivot calibration, a calibration tool, and using marker-to-marker user calibration.
Fig. 7: Tool-tip calibration error: (a) pivot calibration, (b) using a calibration tool, (c-d) marker-to-marker user calibration with reference marker size of 8 cm (c) and 12 cm (d). V8-HL2 in the legend indicates the marker type (Vuforia), marker size (8 cm) and HMD device (HL2).
Device
Marker
Pivot calibration
Tool calibration
Marker-to-marker user calibration
8 cm (P)
5 cm (P)
8 cm (P) - 10 cm (T)
5 cm (P) - 10 cm (T)
8 cm (P) - 8 cm (R)
8 cm (P) - 12 cm (R)
5 cm (P) - 12 cm (R)
Tip (mm)
Tip (mm)
Tip (mm)
Angle (°)
Tip (mm)
Angle (°)
Tip (mm)
Angle (°)
Tip (mm)
Angle (°)
Tip (mm)
Angle (°)
HL2
ArUco
2.9 (0.9)
4.9 (2.1)
1.8 (0.4)
2.7 (1.9)
Vuforia
1.2 (0.4)
1.5 (0.4)
1.1 (0.3)
1.3 (0.1)
1.7 (0.2)
1.4 (0.1)
2.5 (0.5)
0.7 (0.3)
1.1 (0.5)
0.4 (0.3)
2.0 (0.7)
0.9 (0.4)
ML2
ArUco
1.4 (0.3)
1.8 (0.7)
3.2 (0.4)
2.3 (1.4)
Vuforia
1.1 (0.4)
1.5 (0.4)
1.0 (0.2)
0.9 (0.1)
1.8 (0.3)
1.3 (0.1)
2.9 (0.5)
1.3 (0.3)
2.8 (0.6)
1.3 (0.3)
3.9 (1.4)
1.8 (0.7)
TABLE I: Tool-tip calibration error [mean(std)] of the proposed calibration approaches for different HMD devices and markers, with (P) pointer marker, (T) tool marker, and (R) reference marker.
Fig. 8: Image-to-patient alignment experiment: (a) patient-board with Vuforia and NDI markers attached, (b) aligned virtual model, (c) points annotation using the pointer, (d-e) manual positioning using 6-DoF and 1-DoF manipulation.
Number of registration points
Annotation
Error metric
Vuforia
ArUco
8 cm
5 cm
8 cm
5 cm
GT-ttp
Pivot-ttp
GT-ttp
Pivot-ttp
GT-ttp
Pivot-ttp
GT-ttp
Pivot-ttp
10 points
Landmarks (mm)
NDI FRE
1.2 (0.1)
1.1 (0.0)
1.1 (0.1)
1.1 (0.1)
HMD FRE
2.2 (0.3)
2.8 (0.3)
2.2 (0.3)
2.7 (0.3)
1.5 (0.3)
1.6 (0.3)
2.1 (0.3)
2.1 (0.4)
HMD TRE (NDI-10)
2.1 (0.3)
2.6 (0.3)
2.7 (0.5)
2.4 (0.5)
3.0 (0.3)
3.2 (0.3)
3.6 (0.3)
4.1 (0.4)
Trajectories
Distance (mm)
1.6 (0.4)
2.2 (0.4)
2.0 (0.6)
2.0 (0.5)
2.6 (0.3)
2.8 (0.4)
3.4 (0.3)
3.5 (0.4)
TABLE II: HL2 registration error [mean(std)] for point-based matching for different combinations of points and markers.
Number of registration points
Annotation
Error metric
Vuforia
ArUco
8 cm
5 cm
8 cm
5 cm
GT-ttp
Pivot-ttp
GT-ttp
Pivot-ttp
GT-ttp
Pivot-ttp
GT-ttp
Pivot-ttp
10 points
Landmarks (mm)
NDI FRE
1.2 (0.0)
1.2 (0.0)
1.2 (0.0)
1.2 (0.0)
HMD FRE
2.3 (0.3)
2.4 (0.3)
2.1 (0.3)
2.7 (0.4)
1.8 (0.2)
1.9 (0.2)
2.9 (0.5)
2.2 (0.5)
HMD TRE (NDI-10)
2.1 (0.2)
2.7 (0.2)
2.0 (0.3)
2.0 (0.3)
2.4 (0.2)
2.6 (0.3)
3.0 (0.3)
2.2 (0.3)
Trajectories
Distance (mm)
1.3 (0.4)
1.9 (0.4)
1.1 (0.4)
1.4 (0.4)
1.6 (0.1)
1.8 (0.4)
2.4 (0.3)
1.5 (0.2)
TABLE III: ML2 registration error [mean(std)] for point-based matching for different combinations of points and markers.
Fig. 9: Surface-points mean registration error for the head phantom using 10, 5, and 6 registration points for both HL2 and ML2 using 8 and 5 cm marker sizes.
Fig. 10: Registration error (HMD TRE NDI-10) for different number of reintegration points: (left) order of registration points, (right) registration error as the number of points increases from 3 to 10 points.
Fig. 11: Locating rib-fractures experiment: (a) torso phantom with Vuforia and NDI markers, (b) planned incision lines, (c) aligned torso virtual model, (d) delineated incision lines by the user.
Fig. 12: Surface-points mean registration error for point-based (PB) matching and manual positioning (MP) for both HL2 and ML2 performed by the participants on the head and torso phantoms.
Fig. 13: Delineation error of the incision lines by participants in the rib-fracture localization experiment: (left) Box plots of the delineation error for both HL2 and ML2. Left(H) and Left(V) represent the left-side horizontal and vertical planned incisions. (right) incision lines marked by the participants on both sides of the torso phantom.
Fig. 14: Ventricular catheter placement experiment: (a) head phantom with entry holes, (b) two insertion planned trajectories, (c) aligned virtual head with planned trajectory, (d) insertion guidance of the pointer with virtual extensions and numerical feedback menu.
Device
Phantom
Point-based matching
Manual registration
Accuracy
Time
Accuracy
Time
HL2 (n=8)
Head
2.6 (1.1)
2:38 (0:53)
8.9 (2.8)
8:25 (3:24)
Torso
2.5 (1.0)
1:58 (1:04)
9.4 (2.5)
6:58 (3:08)
ML2 (n=8)
Head
3.0 (1.1)
4:12 (2:21)
4.6 (1.3)
4:57 (1:37)
Torso
3.0 (1.0)
3:01 (1:25)
7.1 (2.2)
7:23 (4:24)
Total (n=16)
Head
2.8 (1.1)
3:29 (1:59)
6.7 (2.0)
6:48 (3:13)
TABLE IV: Point-based matching vs Manual-positioning in aligning head and torso virtual models. Accuracy is the HMD TRE NDI-10 [mean(std)] and time is the alignment completion time [mean(std)]
Device
Tool-tip calibration error
Registration error (trajectories)
Entry point
Exit point
Entry to exit (mean)
Perception
HMD
NDI
Perception
HMD
NDI
Perception
HMD
NDI
HL2 (n=8)
Distance (mm)
1.2 (0.2)
2.6 (0.2)
0.9 (0.6)
3.2 (1.2)
3.1 (1.6)
1.4 (0.9)
2.9 (0.9)
3.3 (1.3)
2.3 (2.3)
3.5 (2.4)
3 (1.0)
Angle (°)
0.9 (0.3)
1.6 (0.2)
0.7 (0.5)
2.3 (0.7)
2.3 (0.7)
0.7 (0.3)
2.3 (0.5)
2.5 (0.7)
1.3 (1.4)
2.8 (1.3)
2.7 (0.8)
ML2 (n=8)
Distance (mm)
1.8 (0.7)
2.2 (0.1)
0.9 (0.5)
1.7 (0.7)
2.0 (1.0)
1.4 (0.6)
2.9 (0.8)
2.6 (1.0)
2.3 (1.6)
2.7 (1.6)
2.2 (0.9)
Angle (°)
0.9 (0.8)
0.7 (0.1)
0.6 (0.4)
1.6 (0.7)
1.8 (0.7)
0.8 (0.5)
1.6 (0.7)
1.7 (0.7)
1.5 (1.0)
2.2 (1.0)
2.2 (1.0)
Total (n=16)
Distance (mm)
1.5 (0.6)
2.4 (0.1)
0.9 (0.6)
2.4 (0.9)
2.5 (1.3)
1.4 (0.7)
2.9 (0.8)
3.0 (1.2)
2.3 (2.0)
3.1 (2.0)
2.6 (0.9)
TABLE V: Mean calibration, registration and insertion errors for the ventricular catheter placement user study, for both HL2 and ML2.
Technical University of Munich, Human-Centered-Computing and Extended Reality Lab · TUM University Hospital, Orthopedics and Sports Orthopedics · Munich Institute of Robotics and Machine Intelligence (MIRMI) +2