Towards Comprehensive Real-Time Scene Understanding in Ophthalmic Surgery through Multimodal Image Fusion

Nikolo Rohrmoser; Ghazal Ghazaei; Michael Sommersperger; Nassir Navab

cs.CVMar 26, 2026

Towards Comprehensive Real-Time Scene Understanding in Ophthalmic Surgery through Multimodal Image Fusion

Authors: Nikolo Rohrmoser, Ghazal Ghazaei, Michael Sommersperger, Nassir Navab

Organizations: Chair for Computer Aided Medical Procedures, Technical University of Munich, Garching, Germany. · Corporate Research and Technology, Carl Zeiss AG, Munich, Germany.

Abstract

Purpose: The integration of multimodal imaging into operating rooms paves the way for comprehensive surgical scene understanding. In ophthalmic surgery, by now, two complementary imaging modalities are available: operating microscope (OPMI) imaging and real-time intraoperative optical coherence tomography (iOCT). This first work toward temporal OPMI and iOCT feature fusion demonstrates the potential of multimodal image processing for multi-head prediction through the example of precise instrument tracking in vitreoretinal surgery. Methods: We propose a multimodal, temporal, real-time capable network architecture to perform joint instrument detection, keypoint localization, and tool-tissue distance estimation. Our network design integrates a cross-attention fusion module to merge OPMI and iOCT image features, which are efficiently extracted via a YoloNAS and a CNN encoder, respectively. Furthermore, a region-based recurrent module leverages temporal coherence. Results: Our experiments demonstrate reliable instrument localization and keypoint detection (95.79% mAP50) and show that the incorporation of iOCT significantly improves tool-tissue distance estimation, while achieving real-time processing rates of 22.5 ms per frame. Especially for close distances to the retina (below 1 mm), the distance estimation accuracy improved from 284 $μm$ (OPMI only) to 33 $μm$ (multimodal). Conclusion: Feature fusion of multimodal imaging can enhance multi-task prediction accuracy compared to single-modality processing and real-time processing performance can be achieved through tailored network design. While our results demonstrate the potential of multi-modal processing for image-guided vitreoretinal surgery, they also underline key challenges that motivate future research toward more reliable, consistent, and comprehensive surgical scene understanding.

Figures & tables

Figure 1: Multimodal data in ophthalmic surgery consists of OPMI (left) and two perpendicularly aligned iOCT B-scan images (middle, right). The blue and magenta lines in the OPMI image indicate the locations of the B-scans.

Figure 2: The overall architecture. A Yolo-NAS backbone processes the OPMI stream, while a modified ResNet-18 extracts features from iOCT scans. An attention-based fusion module integrates these modalities before they are further refined by a recurrent module for temporal awareness. Finally, task-specific heads generate outputs including detection, keypoint estimation, and distance prediction.

Figure 3: Cross-attention fusion. OPMI pixel features (queries) attend to iOCT column descriptors (keys/values) enriched with positional encodings to inject spatially aware depth cues into the fused representation. The output retains the spatial resolution and channel dimensionality required by the downstream prediction heads.

Model	mAP50	kp dist	dMAE	dMAE0:1	certain dMAE
Model	[%]	[pxl]	[ $\mu m$ ]	[ $\mu m$ ]	[ $\mu m$ ]
SM	94.27 $\pm$ 0.41	8.35 $\pm$ 1.32	480.93 $\pm$ 54.91	284.01 $\pm$ 89.44	142.99 $\pm$ 25.4
MM	95.79 $\pm$ 0.58	7.93 $\pm$ 0.47	128.32 $\pm$ 20.03	33.05 $\pm$ 2.38	53.74 $\pm$ 13.0
RMM (GRU)	94.92 $\pm$ 0.28	9.65 $\pm$ 0.42	332.91 $\pm$ 48.88	135.44 $\pm$ 37.28	61.78 $\pm$ 10.73
RMM (LSTM)	94.60 $\pm$ 0.29	9.17 $\pm$ 0.46	351.59 $\pm$ 34.66	141.85 $\pm$ 67.60	62.88 $\pm$ 14.54

Table 1: Comparison of SM and MM models on detection, keypoint localization, and distance estimation on single frames without utilization of the recurrent module. The metrics for the RMM models are reported using a sequence of 16 frames and 4 consecutive and 4 random corrupted iOCT features using a GRU/LSTM as recurrent module.

Figure 4: Distance error distributions for a single peeling sequence in the test set stratified by certainty for the SM model utilizing only OPMI. Each boxplot summarizes the error distribution for one certainty category of certain (>90%), moderate (50-90%), and uncertain (<50%).

Panel of Figure 5 — Figure 5: Qualitative results for distance estimation on a peeling sequence using SM (a) and MM (b) models. The prediction’s (points) certainty is encoded by color and error bar, where smaller and darker indicate higher certainty. A dark dot represents certainty >90%.

GRU
#corrupted iOCT	dMAE	masked dMAE	dMAE0:1	masked dMAE0:1
frames	[ $\mu m$ ]	[ $\mu m$ ]	[ $\mu m$ ]	[ $\mu m$ ]
0	191.80 $\pm$ 11.4	-	62.31 $\pm$ 17.1	-
4	232.19 $\pm$ 14.4	458.82 $\pm$ 59.3	89.59 $\pm$ 25.0	262.40 $\pm$ 69.3
8	314.01 $\pm$ 13.4	585.18 $\pm$ 30.0	177.71 $\pm$ 23.3	532.43 $\pm$ 32.3
LSTM

Table 2: Mean

\pm

std over 5 seeds for the RMM using GRU/LSTM as recurrent module on a peeling sequence with an increasing number of consecutive frames with corrupted iOCT data during inference. Masked metrics are computed only for frames with corrupted iOCT data.

Explore similar work

Multimodal Shared Latent Representation of Narration, Microscope and iOCT Images for Phase Recognition in Vitreoretinal Surgery
Aug 31, 2026Onur Izmitlioglu, Shervin Dehghani, Tarek Ghannoum +2Surgical Video Understanding Multimodal Contrastive Learning
Computer Aided Medical Procedures, Technical University of Munich, Germany · SynthesEyes GmbH, Munich, Germany · Department of Ophthalmology, LMU University Hospital, Munich, Germany
Joint Segmentation and Grading with Iterative Optimization for Multimodal Glaucoma Diagnosis
Mar 15, 2026Zhiwei Wang, Yuxing Li, Meilu Zhu +2Medical Image Segmentation Cross-Modality Medical Image Segmentation
Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong, China · College of Information Engineering, Zhejiang University of Technology, Hangzhou, China
Augmenting Real-time Visualization in Minimally Invasive Surgery with Particle Filtering and Depth-Aware Occlusion Handling
Oct 13, 2026Anuska Sarkar, Partha Acharya, Vishnu Nandury +2Occlusion Reasoning Surgical Video Understanding
School of Medical Science and Technology, Indian Institute of Technology Kharagpur, Kharagpur 721302 India · Computer Science Department, Arizona State University, Tempe, Arizona 85287 USA · Narayana Hospital, Kolkata 700107 India

Aug 31, 2026J/K move · Enter open · S save

Multimodal Shared Latent Representation of Narration, Microscope and iOCT Images for Phase Recognition in Vitreoretinal Surgery

Onur Izmitlioglu, Shervin Dehghani, Tarek Ghannoum, Benedikt Schworm+1

Computer Aided Medical Procedures, Technical University of Munich, Germany · SynthesEyes GmbH, Munich, Germany · Department of Ophthalmology, LMU University Hospital, Munich, Germany

Surgical Video Understanding Multimodal Contrastive Learning Multimodal Fusion

Open paper AskPDF arXiv

Surgical phase recognition is key to context-aware computer-assisted feedback in vitreoretinal procedures, yet the scarcity of synchronized multimodal intraoperative data, particularly microscope views and intraoperative OCT, limits approaches that aim to replicate the multimodal integration surgeons perform naturally. Surgical narration, by contrast, is abundantly available online and offers rich semantic supervision. Prior work has mainly explored pairwise contrastive learning (e.g., intraoperative OCT-microscope or microscope-narration), leaving the joint modeling of all three modalities largely unexplored. We introduce a framework that uses microscope views as a shared anchor to bridge surgical narrations and intraoperative OCT (iOCT) without requiring a fully synchronized tri-modal dataset, leveraging real microscope-narration videos and a synthetic dataset of synchronized microscope video and tool-aligned iOCT pairs. Contrastive alignment transfers structural priors from the synthetic domain to real videos lacking iOCT, and a dual-head MS-TCN++ integrates the resulting embeddings for joint macro- and micro-phase prediction. Evaluated on real vitreoretinal surgeries, our framework improves macro-phase recognition over a zero-shot baseline (mean F1 0.38 to 0.53) and provides an exploratory route to estimating fine-grained instrument-tissue measurements that are not directly observable in real microscope video alone; these micro-phase estimates are validated quantitatively on synthetic data and shown only qualitatively on real surgery. To our knowledge, this is the first work to unify microscope view, iOCT B-scans, and surgical narrations in a shared latent space for surgical phase recognition.

Abstract

Explore similar work

Multimodal Shared Latent Representation of Narration, Microscope and iOCT Images for Phase Recognition in Vitreoretinal Surgery

Joint Segmentation and Grading with Iterative Optimization for Multimodal Glaucoma Diagnosis

Augmenting Real-time Visualization in Minimally Invasive Surgery with Particle Filtering and Depth-Aware Occlusion Handling