Sterilizable Scene Graph Generation for Operating Rooms
Authors: Nick Lemke, Ssharvien Kumar Sivakumar, Antoine P. Sanner, John Kalkhof, Henry John Krumb, Ghazal Ghazaei, Anirban Mukhopadhyay
Organizations: Technical University of Darmstadt, Darmstadt, Germany · ImFusion GmbH, Munich, Germany · Carl Zeiss AG, Munich, Germany · University Medical Center Mainz, Mainz, Germany · Inria Center at University Côte d’Azur, Sophia Antipolis, France
Scene graph generation from surgical video enables a holistic and structured understanding of surgical scenes by modeling objects and their semantic relationships. Despite recent advances, state-of-the-art approaches rely on large, parameter-heavy deep learning models that are impractical for deployment in the operating room (OR) due to hardware footprint, hygiene constraints, latency, and data privacy concerns. To the best of our knowledge, this is the first scene graph generation method built on NCAs and the first NCA framework capable of learning structured representations. We introduce SG-NCA, a lightweight scene graph generation framework based on Neural Cellular Automata (NCA), designed for inference in fanless devices critical for OR hygiene protocols. SG-NCA is the first scene graph generation combining NCA-based multiclass segmentation for efficient object detection and feature extraction with a lightweight relation predictor. We evaluate SG-NCA on videos of cataract surgery and cholecystectomy, demonstrating performance comparable to established baselines while requiring 55x fewer parameters. We showcase deployment on fanless edge devices better suited for the OR and demonstrate downstream applications such as surgical video captioning, highlighting SG-NCA's potential for affordable, privacy-preserving, and OR-ready intraoperative scene understanding.
Figures & tables
Figure 1: Lightweight scene graph generation on fanless devices such as smartphones.
Figure 2: The curriculum-based training (A), the segmentation-grounded feature extraction (B) and the relation classification (C).
Cholecystectomy
Cataracts
macro Dice
micro Dice
#Params
macro Dice
micro Dice
#Params
SG-NCA
70.9 ± 16.4
76.6 ± 21.9
27,465
75.0 ± 15.9
82.1 ± 18.0
42,505
SegFormer
71.3 ± 28.4
81.7 ± 22.1
3,717,484
70.8 ± 21.2
82.9 ± 20.5
3,719,283
UNet
44.4 ± 25.7
53.6 ± 25.2
68,331,670
43.1 ± 29.5
63.7 ± 25.9
68,332,580
SwinUNetv2
67.4 ± 27.9
78.0 ± 23.2
27,941,028
67.4 ± 20.2
79.0 ± 20.8
27,941,700
Table 1: Segmentation results and number of parameters for each segmentation model.
Figure 3: Scene graph generation evaluation of semantic relationships. SG-NCA requires 55× fewer parameters while achieving comparable performance, running on an edge device.
Cholecystectomy
Cataracts
C+
H+
macro Dice
micro Dice
#Params
macro Dice
micro Dice
#Params
8
32
73.7 ± 11.7
77.4 ± 20.8
59,625
75.9 ± 13.6
81.7 ± 17.7
148,905
16
73.9 ± 12.0
77.6 ± 21.4
40,185
76.4 ± 12.7
82.6 ± 17.6
85,625
8
74.3 ± 11.9
77.7 ± 20.7
30,465
75.2 ± 13.6
81.6 ± 18.4
53,985
4
16
74.0 ± 12.0
77.6 ± 21.0
34,785
78.0 ± 09.7
82.5 ± 17.6
55,785
8
72.6 ± 13.1
76.8 ± 21.6
27,465
75.6 ± 14.1
82.1 ± 17.8
42,505
Table 2: Ablation on the increments of channels C+ and hidden size H+ during SG-NCA training.
Figure 4: Captions generated from SG-NCA. See supplementary material for videos.
Workstation
Smartphone
Raspberry Pi
Mem.
Temp.
Energy
Temp.
Energy
Temp.
Energy
SG-NCA
44.25
4.18
223
0.66
1.6
0.62
4.5
Table 3: Maximum Memory demand (in MB), Temperature increase (in °K), and Energy consumption (in W) during inference of SG-NCA on various hardware. Due to thermal and power requirements, other models exceed the OR-limits.
Current surgical scene graph generation methods depend on dense multi-modal supervision and specialized hardware (synchronized RGB-D sensors, calibration rigs), making dataset construction expensive and restricting all existing benchmarks to simulated environments. We propose SAGE-OR, a feature-centric framework that replaces the traditional detect-then-reason paradigm with a decoupled representation-reasoning paradigm in which localization is derived from frozen foundation models, encoded implicitly in pre-computed features, and used without any localization supervision, while a lightweight graph transformer performs relational reasoning over cached features. We employ a semi-supervised formulation with general-purpose segmentation prompts to eliminate localization supervision while enabling unsupervised context augmentation through additional prompt-driven entities, such as hands, which are absent from annotations. General-purpose prompts are used to induce near-perfect recall, while precision is delegated to downstream attention-based reasoning, enabling simple adaptation to new entities via prompt-level modification. This design enables a lightweight 15M-parameter graph transformer that trains in 1.4 hours and runs relational inference at ∼1ms per frame with peak memory under 2GB, suitable for edge hardware used in the operating room; feature extraction runs offline as a separate caching stage (4.27s per frame). On the 4D-OR benchmark, the core model achieves 76% F1, matching the fully supervised 4D-OR baseline while eliminating all localization annotations, and unsupervised hand augmentation raises this to 86%, within 4 points of state-of-the-art (SOTA) methods requiring dense multi-modal supervision, providing a practical pathway for adaptation to new surgical settings without annotation other than relationship and class labels.
Brandon Leblanc, Charalambos Poullis
The Immersive and Creative Technologies Lab Concordia University Montreal, CA
Scene Graph Generation (SGG) aims to extract a detailed graph structure from an image, a representation that holds significant promise as a robust intermediate step for complex downstream tasks like reasoning for embodied agents. However, practical deployment in real-world applications - especially on resource constrained edge devices - requires speed and resource efficiency, challenges that have received limited attention in existing research. To bridge this gap, we introduce DSFlash, a low-latency model for panoptic scene graph generation designed to overcome these limitations. DSFlash can process a video stream at 56 frames per second on a standard RTX 3090 GPU, without compromising performance against existing state-of-the-art methods. Crucially, unlike prior approaches that often restrict themselves to salient relationships, DSFlash computes comprehensive scene graphs, offering richer contextual information while maintaining its superior latency. Furthermore, DSFlash is light on resources, requiring less than 24 hours to train on a single, nine-year-old GTX 1080 GPU. This accessibility makes DSFlash particularly well-suited for researchers and practitioners operating with limited computational resources, empowering them to adapt and fine-tune SGG models for specialized applications.
Julian Lorenz, Vladyslav Kovganko, Elias Kohout +3
Scene graph generation provides a compact structured representation for visual perception, but accurate and fast graph prediction from images and videos remains challenging. Recent VLM-based methods can generate scene graphs end-to-end as structured text, yet often produce long outputs with irrelevant objects and relations. We present SceneGraphVLM, a compact method for image and video scene graph generation with small visual language models. SceneGraphVLM serializes graphs in a token-efficient TOON format and trains the model in two stages: supervised fine-tuning followed by reinforcement learning with hallucination-aware rewards that balance relation coverage and precision while penalizing unsupported objects and relations. For videos, the model can optionally condition each frame on the previously generated graph, providing lightweight short-term context without tracking or post-processing. We evaluate SceneGraphVLM on PSG, PVSG, and Action Genome. With compact VLMs and vLLM-accelerated decoding, SceneGraphVLM achieves a strong quality-speed trade-off, improves precision-oriented SGG metrics while preserving reasonable recall, and generates complete scene graphs with approximately one-second latency. Code and implementation details are available at: https://github.com/markus0440/SceneGraphVLM.git.