This paper presents the computational architecture and quantitative preclinical evaluation of MIRAGE, a multimodal image-guided robotic surgery framework for real-time visualization and navigation. It integrates stereovision scene reconstruction, augmented-reality (AR) video overlay, quantified heads-up display (HUD) navigation, and intraoperative ultrasound within the da Vinci console's TilePro display (Intuitive Surgical). Additional modules include optical/electromagnetic tracker registration, stereo calibration, fiducial rigid registration with RMSE feedback, and 3D point-cloud reconstruction. Video I/O and rendering run in a C++ backend, while a Python GUI coordinates calibration, registration, reconstruction, visualization, and user interaction through an in-process zero-copy binding layer. The framework formalizes spatial transforms, module states, and synchronization of heterogeneous video and tracking streams. Preoperative calibration uses synchronized checkerboard image pairs and optimization, with estimated intrinsics, extrinsics, and rectification reused intraoperatively. MIRAGE was validated in phantom experiments across da Vinci S, Si, Xi, and SP platforms. Across clinically relevant working distances, mean 3D reprojection error was 0.6mm, with best 0.1mm RMSE at 100mm on Xi. Results support MIRAGE as a preclinical technical-validation platform for accurate, responsive, and extensible multimodal surgical guidance.
Figures & tables
Figure 1: The overview of the framework pipeline.
Framework
Primary role
Scope relevant to multimodal guidance
OpenIGTLink [ 39 ]
Communication protocol
Data exchange and interoperability across image-guided therapy components.
Plus [ 40 ]
US-guided intervention toolkit
Tracked ultrasound acquisition, calibration, reconstruction, and streaming.
3D Slicer [ 41 ]
Medical-image computing platform
Provides visualization, segmentation, registration, and an extensible environment for image-analysis applications.
ROS2 [ 42 ]
Robotics middleware
Provides communication and component composition for robotic applications; transport and quality-of-service settings depend on the application configuration.
MIRAGE
Multimodal surgical image guidance framework
Unifies multimodal calibration, stereo vision, EM/optical tracking, tracked ultrasound, AR/HUD guidance, and 3D stereo reconstruction within a unified multimodal image-guidance framework.
Table 1: Roles and integration scope of MIRAGE and related software frameworks. The comparison describes documented capabilities, not measured performance rankings.
Figure 2: System integration, calibration, registration and visualization pipeline.
Figure 3: Phantoms used in the experimental validation of this manuscript.
Figure 4: HUD-TilePro navigation-visualization on surgeon console. Note that the rod phantom (left) shows a set of embedded capillary tubes are not directly visible to the surgeon without use of the CT-segmented model (right).
Figure 5: Stereo calibration pipeline.
Figure 6: The surgeon’s console view (left) and overlay GUI (right) with the overlay toggled off (left top) and on (left bottom) for both the left and right endoscope eyes.
Figure 7: Augmented reality 3D overlay (left-right endoscope eyes) displayed on the rendered kidney phantom model.
Figure 8: Ultrasound mode: acquisition, interaction, and visualization.
Figure 9: 3D stereo reconstruction CT overlay visualization and stereo calibration 3D error.
Working distance
Pixel error
Physical error
100 mm
2 px
0.1 mm
300 mm
2.5 px
0.3 mm
500 mm
2.5 px
0.6 mm
Table 2: Pixel size and physical error at different working distances.
Figure 10: Software-hardware strategy.
Figure 11: Cross-language state and buffer binding architecture. The C++ backend handles video I/O, rendering, and hardware communication, while the Python frontend coordinates registration, HUD, stereo reconstruction, ultrasound, and GUI control. Timestamped frame buffers and tracker poses are exchanged through a thread-safe binding layer and synchronized before display.
Figure 12: Representative multimodal outputs for AR overlay accuracy, HUD tool-to-target guidance, ultrasound ROI/landmarks, and system performance logs.
Study/system
Primary capability
Console view
Live tracking
Stereo recon.
Ukimura et al. [ 26 ]
TilePro 3D prostate model
Yes
No
No
Samei et al. [ 27 ]
MRI/TRUS partial AR guidance
Yes
Yes
No
Song et al. [ 28 ]
Arc-to-line US/PA registration
No
PM/TRUS
No
Meza et al. [ 29 ]
Freehand US + structured light
No
US probe
SL surface
Stoyanov [ 30 ]
Real-time stereo reconstruction
No
No
Yes
MIRAGE
Time-synchronized multimodal navigation
Yes
Yes (EM/optical)
Yes
Table 3: System-level comparison with representative prior image-guided robotic surgery approaches.
Platform
RMSE mean ± SD (mm)
da Vinci S/Si
1.2 ± 0.20
da Vinci Xi
0.9 ± 0.20
da Vinci SP
0.9 ± 0.20
Table 4: Fiducial registration accuracy reporting structure across experiments.
Metric
Before Optimization
After Optimization
2D pixel error
4.3 px
2.5 px
3D spatial error
3.0 mm
0.6 mm
Table 5: Calibration result before and after optimization on the Xi robot.
Structure
PSMNet
RAFT-Stereo
Abdominal Aorta (AA)
2.63
2.17
Inferior Vena Cava (IVC)
2.30
1.79
Right Kidney (RK)
2.15
2.25
Right Ureter (RU)
2.43
2.13
Inference time
∼ 0.60 s
∼ 0.20 s
Table 6: 3D reconstruction error (mm) between point cloud and Kidney phantom surfaces. Inference time is per stereo pair
Platform
Working distance (mm)
Error (px)
da Vinci S
100
6.0
da Vinci Si
100
3.0
da Vinci Xi
100
2.5
da Vinci SP
100
2.5
Table 7: Mean AR overlay accuracy across phantoms.
Figure 13: Overlay accuracy analysis for Si: (a) Left-Right live overlay views in MIRAGE environment. (b) Left-eye contour checks: 0.4, 0.4, and 0.6 mm, reported as 0.47±0.10 mm. (c) Right-eye checks: 0.4, 0.2, and 0.6 mm, reported as 0.40±0.20 mm. The pixel checks use the documented MIRAGE scale of 0.20 mm/pixel.
Figure 14: Supplied live overlay runtime traces: (a) navigation frame rate and (b) navigation-processing time.
Mode
Video FPS
Tracker rate
AR overlay
60
400 Hz
HUD
60
40 Hz
Stereo reconstruction
60
400 Hz
Ultrasound
60
40 Hz
Table 8: Synchronization and latency reporting structure.
Figure 15: MIRAGE integrated modules in the surgical phantom experiment.
Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD, USA. · Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA.
Technical University of Munich, Human-Centered-Computing and Extended Reality Lab · TUM University Hospital, Orthopedics and Sports Orthopedics · Munich Institute of Robotics and Machine Intelligence (MIRMI) +2
Electrical and Computer Engineering Department, School of Engineering and Applied Science, University of Virginia, Charlottesville, VA, United States of America