This paper presents the computational architecture and quantitative preclinical evaluation of MIRAGE, a multimodal image-guided robotic surgery framework for real-time visualization and navigation. It integrates stereovision scene reconstruction, augmented-reality (AR) video overlay, quantified heads-up display (HUD) navigation, and intraoperative ultrasound within the da Vinci console's TilePro display (Intuitive Surgical). Additional modules include optical/electromagnetic tracker registration, stereo calibration, fiducial rigid registration with RMSE feedback, and 3D point-cloud reconstruction. Video I/O and rendering run in a C++ backend, while a Python GUI coordinates calibration, registration, reconstruction, visualization, and user interaction through an in-process zero-copy binding layer. The framework formalizes spatial transforms, module states, and synchronization of heterogeneous video and tracking streams. Preoperative calibration uses synchronized checkerboard image pairs and optimization, with estimated intrinsics, extrinsics, and rectification reused intraoperatively. MIRAGE was validated in phantom experiments across da Vinci S, Si, Xi, and SP platforms. Across clinically relevant working distances, mean 3D reprojection error was 0.6mm, with best 0.1mm RMSE at 100mm on Xi. Results support MIRAGE as a preclinical technical-validation platform for accurate, responsive, and extensible multimodal surgical guidance.
Figures & tables
Figure 1: The overview of the framework pipeline.
Framework
Primary role
Scope relevant to multimodal guidance
OpenIGTLink [ 39 ]
Communication protocol
Data exchange and interoperability across image-guided therapy components.
Plus [ 40 ]
US-guided intervention toolkit
Tracked ultrasound acquisition, calibration, reconstruction, and streaming.
3D Slicer [ 41 ]
Medical-image computing platform
Provides visualization, segmentation, registration, and an extensible environment for image-analysis applications.
ROS2 [ 42 ]
Robotics middleware
Provides communication and component composition for robotic applications; transport and quality-of-service settings depend on the application configuration.
MIRAGE
Multimodal surgical image guidance framework
Unifies multimodal calibration, stereo vision, EM/optical tracking, tracked ultrasound, AR/HUD guidance, and 3D stereo reconstruction within a unified multimodal image-guidance framework.
Table 1: Roles and integration scope of MIRAGE and related software frameworks. The comparison describes documented capabilities, not measured performance rankings.
Figure 2: System integration, calibration, registration and visualization pipeline.
Figure 3: Phantoms used in the experimental validation of this manuscript.
Figure 4: HUD-TilePro navigation-visualization on surgeon console. Note that the rod phantom (left) shows a set of embedded capillary tubes are not directly visible to the surgeon without use of the CT-segmented model (right).
Figure 5: Stereo calibration pipeline.
Figure 6: The surgeon’s console view (left) and overlay GUI (right) with the overlay toggled off (left top) and on (left bottom) for both the left and right endoscope eyes.
Figure 7: Augmented reality 3D overlay (left-right endoscope eyes) displayed on the rendered kidney phantom model.
Figure 8: Ultrasound mode: acquisition, interaction, and visualization.
Figure 9: 3D stereo reconstruction CT overlay visualization and stereo calibration 3D error.
Working distance
Pixel error
Physical error
100 mm
2 px
0.1 mm
300 mm
2.5 px
0.3 mm
500 mm
2.5 px
0.6 mm
Table 2: Pixel size and physical error at different working distances.
Figure 10: Software-hardware strategy.
Figure 11: Cross-language state and buffer binding architecture. The C++ backend handles video I/O, rendering, and hardware communication, while the Python frontend coordinates registration, HUD, stereo reconstruction, ultrasound, and GUI control. Timestamped frame buffers and tracker poses are exchanged through a thread-safe binding layer and synchronized before display.
Figure 12: Representative multimodal outputs for AR overlay accuracy, HUD tool-to-target guidance, ultrasound ROI/landmarks, and system performance logs.
Study/system
Primary capability
Console view
Live tracking
Stereo recon.
Ukimura et al. [ 26 ]
TilePro 3D prostate model
Yes
No
No
Samei et al. [ 27 ]
MRI/TRUS partial AR guidance
Yes
Yes
No
Song et al. [ 28 ]
Arc-to-line US/PA registration
No
PM/TRUS
No
Meza et al. [ 29 ]
Freehand US + structured light
No
US probe
SL surface
Stoyanov [ 30 ]
Real-time stereo reconstruction
No
No
Yes
MIRAGE
Time-synchronized multimodal navigation
Yes
Yes (EM/optical)
Yes
Table 3: System-level comparison with representative prior image-guided robotic surgery approaches.
Platform
RMSE mean ± SD (mm)
da Vinci S/Si
1.2 ± 0.20
da Vinci Xi
0.9 ± 0.20
da Vinci SP
0.9 ± 0.20
Table 4: Fiducial registration accuracy reporting structure across experiments.
Metric
Before Optimization
After Optimization
2D pixel error
4.3 px
2.5 px
3D spatial error
3.0 mm
0.6 mm
Table 5: Calibration result before and after optimization on the Xi robot.
Structure
PSMNet
RAFT-Stereo
Abdominal Aorta (AA)
2.63
2.17
Inferior Vena Cava (IVC)
2.30
1.79
Right Kidney (RK)
2.15
2.25
Right Ureter (RU)
2.43
2.13
Inference time
∼ 0.60 s
∼ 0.20 s
Table 6: 3D reconstruction error (mm) between point cloud and Kidney phantom surfaces. Inference time is per stereo pair
Platform
Working distance (mm)
Error (px)
da Vinci S
100
6.0
da Vinci Si
100
3.0
da Vinci Xi
100
2.5
da Vinci SP
100
2.5
Table 7: Mean AR overlay accuracy across phantoms.
Figure 13: Overlay accuracy analysis for Si: (a) Left-Right live overlay views in MIRAGE environment. (b) Left-eye contour checks: 0.4, 0.4, and 0.6 mm, reported as 0.47±0.10 mm. (c) Right-eye checks: 0.4, 0.2, and 0.6 mm, reported as 0.40±0.20 mm. The pixel checks use the documented MIRAGE scale of 0.20 mm/pixel.
Figure 14: Supplied live overlay runtime traces: (a) navigation frame rate and (b) navigation-processing time.
Mode
Video FPS
Tracker rate
AR overlay
60
400 Hz
HUD
60
40 Hz
Stereo reconstruction
60
400 Hz
Ultrasound
60
40 Hz
Table 8: Synchronization and latency reporting structure.
Figure 15: MIRAGE integrated modules in the surgical phantom experiment.
Current robot-assisted minimally-invasive surgery (RMIS) platforms provide a fixed console for the surgeon to view stereo endoscopic images and teleoperate instruments inside the patient. Several researchers have proposed the use of a head-mounted display (HMD) as a portable console, with video pass-through rendering of the endoscope images which, like the fixed console, restricts the operator to a single endoscopic viewpoint and limits depth perception. We present a digital twin-driven virtual reality (VR) teleoperation platform, where the digital twin is created from markerless perception of the surgical environment and streamed for display on the HMD. This overcomes the limitations of video pass-through by providing multi-view rendering and natural motion-parallax cues, enabling decoupling of the user's hand posture from strict instrument alignment. The system utilizes VR hand controllers to increase the teleoperation workspace and to improve the robustness and stability of instrument control compared to the hand tracking approach adopted by most prior systems. A 15-participant user study on the da Vinci Research Kit (dVRK) shows that our VR platform significantly outperforms a state-of-the-art HoloLens 2 mixed reality baseline, reducing path length by 86% and jerk by 95%, while achieving depth perception confidence comparable to or exceeding the traditional console across all conditions.
Chang Liu, Chenhao Yu, Honghao Zhao +5
Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD, USA. · Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA.
The handling and assembly of instruments during surgery imposes high cognitive demands on scrub nurses, particularly when instruments are unfamiliar. We present a supporting guidance system for surgical instrumentation that combines multi-camera 6D pose estimation with augmented reality in-situ visualization on a head-mounted display without the requirement for additional markers. Pose estimation and consecutive camera calibration are achieved through known objects. The 6D pose estimation network is trained purely on synthetic data, aiming for better generalizability and real-world applicability. The AR guidance displays tooltip localization cues and step-wise assembly animations. Via gaze-based selection and a foot pedal, users can switch between assembly steps in intraoperative use. In a technical evaluation, our approach outperforms state-of-art 6D pose estimation. A user study with 29 scrub nurses was conducted in a surgical simulation of knee arthroplasty, comparing the system against a paper manual. AR guidance significantly reduced the perceived workload compared. Objectively, AR guidance reduced task completion time by 21.3% (4.76 minutes). Specifically, scrub nurses less experienced with the instrument set benefited when using the system. Error frequencies were comparable between conditions. Qualitative feedback highlighted improved process clarity, reduced information overload, and perceived independence. To summarize, our marker-free multi-camera AR guidance approach for surgical instruments can, subjectively and objectively, improve intraoperative instrumentation performance, particularly for untrained scrub nurses.
Shiyu Li, Julian Kreimeier, Hannah Schieber +4
Technical University of Munich, Human-Centered-Computing and Extended Reality Lab · TUM University Hospital, Orthopedics and Sports Orthopedics · Munich Institute of Robotics and Machine Intelligence (MIRMI) +2
Robot-assisted minimally invasive surgery improves surgical precision but introduces complexity, making technical error detection essential for ensuring patient safety. Current executional error detection methods using video data often overlook fine-grained contextual descriptions of activities and error types within the hierarchical structure of surgical procedures. They also under-utilize complementary multimodal information. We propose a unified framework for executional error detection that leverages multimodal input, including video, kinematics, and descriptive textual prompts. Through activity prompting, we integrate descriptive language in gesture-level activities, instrument-object interactions, and error definitions. We also introduce activity-aware visual embeddings derived from vision encoders pretrained on surgical activity labels to compare the effectiveness of contrastive language-image embeddings with traditional image-based embeddings for error detection. By seamlessly integrating kinematic data with video and textual modalities, our framework significantly improves error detection performance. Achieving up to 5% and 16.6% F1 score improvements over state-of-the-art baselines on the JIGSAWS and SAR-RARP50 datasets, respectively, we demonstrate the value of combining curated textual prompts with multimodal data for accurate error detection.
Electrical and Computer Engineering Department, School of Engineering and Applied Science, University of Virginia, Charlottesville, VA, United States of America