Towards Comprehensive Real-Time Scene Understanding in Ophthalmic Surgery through Multimodal Image Fusion
Authors: Nikolo Rohrmoser, Ghazal Ghazaei, Michael Sommersperger, Nassir Navab
Organizations: Chair for Computer Aided Medical Procedures, Technical University of Munich, Garching, Germany. · Corporate Research and Technology, Carl Zeiss AG, Munich, Germany.
Purpose: The integration of multimodal imaging into operating rooms paves the way for comprehensive surgical scene understanding. In ophthalmic surgery, by now, two complementary imaging modalities are available: operating microscope (OPMI) imaging and real-time intraoperative optical coherence tomography (iOCT). This first work toward temporal OPMI and iOCT feature fusion demonstrates the potential of multimodal image processing for multi-head prediction through the example of precise instrument tracking in vitreoretinal surgery. Methods: We propose a multimodal, temporal, real-time capable network architecture to perform joint instrument detection, keypoint localization, and tool-tissue distance estimation. Our network design integrates a cross-attention fusion module to merge OPMI and iOCT image features, which are efficiently extracted via a YoloNAS and a CNN encoder, respectively. Furthermore, a region-based recurrent module leverages temporal coherence. Results: Our experiments demonstrate reliable instrument localization and keypoint detection (95.79% mAP50) and show that the incorporation of iOCT significantly improves tool-tissue distance estimation, while achieving real-time processing rates of 22.5 ms per frame. Especially for close distances to the retina (below 1 mm), the distance estimation accuracy improved from 284 μm (OPMI only) to 33 μm (multimodal). Conclusion: Feature fusion of multimodal imaging can enhance multi-task prediction accuracy compared to single-modality processing and real-time processing performance can be achieved through tailored network design. While our results demonstrate the potential of multi-modal processing for image-guided vitreoretinal surgery, they also underline key challenges that motivate future research toward more reliable, consistent, and comprehensive surgical scene understanding.
Figures & tables
Figure 1: Multimodal data in ophthalmic surgery consists of OPMI (left) and two perpendicularly aligned iOCT B-scan images (middle, right). The blue and magenta lines in the OPMI image indicate the locations of the B-scans.
Figure 2: The overall architecture. A Yolo-NAS backbone processes the OPMI stream, while a modified ResNet-18 extracts features from iOCT scans. An attention-based fusion module integrates these modalities before they are further refined by a recurrent module for temporal awareness. Finally, task-specific heads generate outputs including detection, keypoint estimation, and distance prediction.
Figure 3: Cross-attention fusion. OPMI pixel features (queries) attend to iOCT column descriptors (keys/values) enriched with positional encodings to inject spatially aware depth cues into the fused representation. The output retains the spatial resolution and channel dimensionality required by the downstream prediction heads.
Model
mAP50
kp dist
dMAE
dMAE0:1
certain dMAE
[%]
[pxl]
[ μm ]
[ μm ]
[ μm ]
SM
94.27 ± 0.41
8.35 ± 1.32
480.93 ± 54.91
284.01 ± 89.44
142.99 ± 25.4
MM
95.79 ± 0.58
7.93 ± 0.47
128.32 ± 20.03
33.05 ± 2.38
53.74 ± 13.0
RMM (GRU)
94.92 ± 0.28
9.65 ± 0.42
332.91 ± 48.88
135.44 ± 37.28
61.78 ± 10.73
RMM (LSTM)
94.60 ± 0.29
9.17 ± 0.46
351.59 ± 34.66
141.85 ± 67.60
62.88 ± 14.54
Table 1: Comparison of SM and MM models on detection, keypoint localization, and distance estimation on single frames without utilization of the recurrent module. The metrics for the RMM models are reported using a sequence of 16 frames and 4 consecutive and 4 random corrupted iOCT features using a GRU/LSTM as recurrent module.
Figure 4: Distance error distributions for a single peeling sequence in the test set stratified by certainty for the SM model utilizing only OPMI. Each boxplot summarizes the error distribution for one certainty category of certain (>90%), moderate (50-90%), and uncertain (<50%).
Figure 5: Qualitative results for distance estimation on a peeling sequence using SM (a) and MM (b) models. The prediction’s (points) certainty is encoded by color and error bar, where smaller and darker indicate higher certainty. A dark dot represents certainty >90%.
#corrupted iOCT
dMAE
masked dMAE
dMAE0:1
masked dMAE0:1
frames
[ μm ]
[ μm ]
[ μm ]
[ μm ]
GRU
0
191.80 ± 11.4
-
62.31 ± 17.1
-
4
232.19 ± 14.4
458.82 ± 59.3
89.59 ± 25.0
262.40 ± 69.3
8
314.01 ± 13.4
585.18 ± 30.0
177.71 ± 23.3
532.43 ± 32.3
LSTM
Table 2: Mean ± std over 5 seeds for the RMM using GRU/LSTM as recurrent module on a peeling sequence with an increasing number of consecutive frames with corrupted iOCT data during inference. Masked metrics are computed only for frames with corrupted iOCT data.
Surgical phase recognition is key to context-aware computer-assisted feedback in vitreoretinal procedures, yet the scarcity of synchronized multimodal intraoperative data, particularly microscope views and intraoperative OCT, limits approaches that aim to replicate the multimodal integration surgeons perform naturally. Surgical narration, by contrast, is abundantly available online and offers rich semantic supervision. Prior work has mainly explored pairwise contrastive learning (e.g., intraoperative OCT-microscope or microscope-narration), leaving the joint modeling of all three modalities largely unexplored. We introduce a framework that uses microscope views as a shared anchor to bridge surgical narrations and intraoperative OCT (iOCT) without requiring a fully synchronized tri-modal dataset, leveraging real microscope-narration videos and a synthetic dataset of synchronized microscope video and tool-aligned iOCT pairs. Contrastive alignment transfers structural priors from the synthetic domain to real videos lacking iOCT, and a dual-head MS-TCN++ integrates the resulting embeddings for joint macro- and micro-phase prediction. Evaluated on real vitreoretinal surgeries, our framework improves macro-phase recognition over a zero-shot baseline (mean F1 0.38 to 0.53) and provides an exploratory route to estimating fine-grained instrument-tissue measurements that are not directly observable in real microscope video alone; these micro-phase estimates are validated quantitatively on synthetic data and shown only qualitatively on real surgery. To our knowledge, this is the first work to unify microscope view, iOCT B-scans, and surgical narrations in a shared latent space for surgical phase recognition.
Computer Aided Medical Procedures, Technical University of Munich, Germany · SynthesEyes GmbH, Munich, Germany · Department of Ophthalmology, LMU University Hospital, Munich, Germany
Accurate diagnosis of glaucoma is challenging, as early-stage changes are subtle and often lack clear structural or appearance cues. Most existing approaches rely on a single modality, such as fundus or optical coherence tomography (OCT), capturing only partial pathological information and often missing early disease progression. In this paper, we propose an iterative multimodal optimization model (IMO) for joint segmentation and grading. IMO integrates fundus and OCT features through a mid-level fusion strategy, enhanced by a cross-modal feature alignment (CMFA) module to reduce modality discrepancies. An iterative refinement decoder progressively optimizes the multimodal features through a denoising diffusion mechanism, enabling fine-grained segmentation of the optic disc and cup while supporting accurate glaucoma grading. Extensive experiments show that our method effectively integrates multimodal features, providing a comprehensive and clinically significant approach to glaucoma assessment. Source codes are available at https://github.com/warren-wzw/IMO.git.
Zhiwei Wang, Yuxing Li, Meilu Zhu +2
Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong, China · College of Information Engineering, Zhejiang University of Technology, Hangzhou, China
We present a real-time, hybrid framework for occlusion handling and depth-aware tracking in minimally invasive surgeries (MIS). Occlusions caused by surgical instruments often obscure the region of interest (ROI), affecting surgical precision. To address this, our approach integrates enhanced You Only Look Once (YOLO) based detection, Particle Filtering (PF), and monocular depth estimation using Multiple Depth Estimation Accuracy with Single Network (MiDaS). Occlusion is detected using a distance-based threshold between instruments and ROI. Experimental results demonstrate robust instrument detection with a confidence score of 0.85, effective occlusion event handling, and accurate depth prediction with an MAE of 0.34 cm and RMSE of 0.53 cm. Independent phantom-based validation against stereo depth measurements yielded an MAE of 0.65 cm and RMSE of 0.66 cm. The framework further achieved an occlusion detection accuracy of 81.95% while maintaining real-time performance at 38.7 FPS. Preliminary surgeon feedback indicates that the proposed visualization framework is clinically relevant and holds promise for future intraoperative image-guided navigation and decision support in MIS.
Anuska Sarkar, Partha Acharya, Vishnu Nandury +2
School of Medical Science and Technology, Indian Institute of Technology Kharagpur, Kharagpur 721302 India · Computer Science Department, Arizona State University, Tempe, Arizona 85287 USA · Narayana Hospital, Kolkata 700107 India