TopoOR: A Unified Topological Scene Representation for the Operating Room
Authors: Tony Danjun Wang, Ka Young Kim, Tolga Birdal, Nassir Navab, Lennart Bastian
Organizations: Technical University of Munich, Munich, Germany · Munich Center for Machine Learning, Germany · Kyung Hee University, South Korea · Imperial College London, United Kingdom
Surgical Scene Graphs abstract the complexity of surgical operating rooms (OR) into a structure of entities and their relations, but existing paradigms suffer from strictly dyadic structural limitations. Frameworks that predominantly rely on pairwise message passing or tokenized sequences flatten the manifold geometry inherent to relational structures and lose structure in the process. We introduce TopoOR, a new paradigm that models multimodal operating rooms as a higher-order structure, innately preserving pairwise and group relationships. By lifting interactions between entities into higher-order topological cells, TopoOR natively models complex dynamics and multimodality present in the OR. This topological representation subsumes traditional scene graphs, thereby offering strictly greater expressivity. We also propose a higher-order attention mechanism that explicitly preserves manifold structure and modality-specific features throughout hierarchical relational attention. In this way, we circumvent combining 3D geometry, audio, and robot kinematics into a single joint latent representation, preserving the precise multimodal structure required for safety-critical reasoning, unlike existing methods. Extensive experiments demonstrate that our approach outperforms traditional graph and LLM-based baselines across sterility breach detection, robot phase prediction, and next-action anticipation
Figures & tables
Figure 1: We model surgical ORs as higher-order structures. Our framework explicitly instantiates cells anchored in 3D for physical entities—such as the nurse and surgical robot—as well as data sources from diverse modalities, including audio. Combining these elements into a higher-order structure enables modeling of the unified interaction among the head surgeon, robot, saw, and patient (bottom left), while preserving the complex, multi-actor dynamics of surgical procedures.
Figure 2: Overview of TopoOR. Given multi-modal sensory inputs over a temporal window (I), 3D entities and evidence features are initialized (II) and abstracted into a CC X (III). Higher-order attention is computed across incidence neighborhoods x∈N(y) , using a learnable rank-bias to preserve structural heterogeneity (IV). Pooled representations are routed to downstream tasks enabling, e.g., simultaneous next-action anticipation and robot-phase prediction (V).
Table 3
Figure 3: Qualitative results and scene abstraction. We demonstrate improved performance in (I) robot phase prediction over baseline models. While all methods operate on the same (II) explicit 3D input entities, (III) illustrates the distinct structural formulation of each approach. Unlike the flattened relations of standard networks, our higher-order representation explicitly models the hierarchical incidence of the surgical environment through rank-0, rank-1, and rank-2 cells.
Input Modalities
F1-Score ( ↑ )
Object Nodes
Human Skel.
RGB Image
Robot Logs
Audio
Temporality
Next Action
Robot Phase
✓
25.82
21.08
✓
✓
26.26
20.25
✓
✓
✓
36.35
60.71
✓
✓
✓
✓
38.26
67.06
✓
✓
✓
✓
✓
40.57
69.63
Table 3: Ablation study of incremental input modalities on Next Action Anticipation and Robot Phase Prediction.