Sterilizable Scene Graph Generation for Operating Rooms

Nick Lemke; Ssharvien Kumar Sivakumar; Antoine P. Sanner; John Kalkhof; Henry John Krumb; Ghazal Ghazaei; Anirban Mukhopadhyay

cs.CVAug 17, 2026

Sterilizable Scene Graph Generation for Operating Rooms

Authors: Nick Lemke, Ssharvien Kumar Sivakumar, Antoine P. Sanner, John Kalkhof, Henry John Krumb, Ghazal Ghazaei, Anirban Mukhopadhyay

Organizations: Technical University of Darmstadt, Darmstadt, Germany · ImFusion GmbH, Munich, Germany · Carl Zeiss AG, Munich, Germany · University Medical Center Mainz, Mainz, Germany · Inria Center at University Côte d’Azur, Sophia Antipolis, France

Abstract

Scene graph generation from surgical video enables a holistic and structured understanding of surgical scenes by modeling objects and their semantic relationships. Despite recent advances, state-of-the-art approaches rely on large, parameter-heavy deep learning models that are impractical for deployment in the operating room (OR) due to hardware footprint, hygiene constraints, latency, and data privacy concerns. To the best of our knowledge, this is the first scene graph generation method built on NCAs and the first NCA framework capable of learning structured representations. We introduce SG-NCA, a lightweight scene graph generation framework based on Neural Cellular Automata (NCA), designed for inference in fanless devices critical for OR hygiene protocols. SG-NCA is the first scene graph generation combining NCA-based multiclass segmentation for efficient object detection and feature extraction with a lightweight relation predictor. We evaluate SG-NCA on videos of cataract surgery and cholecystectomy, demonstrating performance comparable to established baselines while requiring 55x fewer parameters. We showcase deployment on fanless edge devices better suited for the OR and demonstrate downstream applications such as surgical video captioning, highlighting SG-NCA's potential for affordable, privacy-preserving, and OR-ready intraoperative scene understanding.

Figures & tables

Figure 1: Lightweight scene graph generation on fanless devices such as smartphones.

Figure 2: The curriculum-based training (A), the segmentation-grounded feature extraction (B) and the relation classification (C).

	Cholecystectomy			Cataracts
	macro Dice	micro Dice	#Params	macro Dice	micro Dice	#Params
SG-NCA	70.9 $\pm$ 16.4	76.6 $\pm$ 21.9	27,465	75.0 $\pm$ 15.9	82.1 $\pm$ 18.0	42,505
SegFormer	71.3 $\pm$ 28.4	81.7 $\pm$ 22.1	3,717,484	70.8 $\pm$ 21.2	82.9 $\pm$ 20.5	3,719,283
UNet	44.4 $\pm$ 25.7	53.6 $\pm$ 25.2	68,331,670	43.1 $\pm$ 29.5	63.7 $\pm$ 25.9	68,332,580
SwinUNetv2	67.4 $\pm$ 27.9	78.0 $\pm$ 23.2	27,941,028	67.4 $\pm$ 20.2	79.0 $\pm$ 20.8	27,941,700

Table 1: Segmentation results and number of parameters for each segmentation model.

Figure 3: Scene graph generation evaluation of semantic relationships. SG-NCA requires

55\times

fewer parameters while achieving comparable performance, running on an edge device.

		Cholecystectomy			Cataracts
$C_{+}$	$H_{+}$	macro Dice	micro Dice	#Params	macro Dice	micro Dice	#Params
8	32	73.7 $\pm$ 11.7	77.4 $\pm$ 20.8	59,625	75.9 $\pm$ 13.6	81.7 $\pm$ 17.7	148,905
	16	73.9 $\pm$ 12.0	77.6 $\pm$ 21.4	40,185	76.4 $\pm$ 12.7	82.6 $\pm$ 17.6	85,625
	8	74.3 $\pm$ 11.9	77.7 $\pm$ 20.7	30,465	75.2 $\pm$ 13.6	81.6 $\pm$ 18.4	53,985
4	16	74.0 $\pm$ 12.0	77.6 $\pm$ 21.0	34,785	78.0 $\pm$ 09.7	82.5 $\pm$ 17.6	55,785
4	8	72.6 $\pm$ 13.1	76.8 $\pm$ 21.6	27,465	75.6 $\pm$ 14.1	82.1 $\pm$ 17.8	42,505

Table 2: Ablation on the increments of channels

C_{+}

and hidden size

H_{+}

during SG-NCA training.

Figure 4: Captions generated from SG-NCA. See supplementary material for videos.

		Workstation		Smartphone		Raspberry Pi
	Mem.	Temp.	Energy	Temp.	Energy	Temp.	Energy
SG-NCA	44.25	4.18	223	0.66	1.6	0.62	4.5

Table 3: Maximum Memory demand (in MB), Temperature increase (in °K), and Energy consumption (in W) during inference of SG-NCA on various hardware. Due to thermal and power requirements, other models exceed the OR-limits.

Explore similar work

SAGE-OR: Semi-supervised Adaptive Scene Graph Generation for Operating Rooms
Aug 15, 2026Brandon Leblanc, Charalambos PoullisScene Graph Generation 3D Scene Understanding
The Immersive and Creative Technologies Lab Concordia University Montreal, CA
DSFlash: Comprehensive Panoptic Scene Graph Generation in Realtime
Mar 11, 2026Julian Lorenz, Vladyslav Kovganko, Elias Kohout +3Scene Graph Generation
University of Augsburg
SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models
May 13, 2026Vladislav Makarov, Mark Gizetdinov, Dmitry YudinScene Graph Generation Vision-Language Models
MIRAI · MIREA · MIRAI, AXXX

Aug 15, 2026J/K move · Enter open · S save

SAGE-OR: Semi-supervised Adaptive Scene Graph Generation for Operating Rooms

Brandon Leblanc, Charalambos Poullis

The Immersive and Creative Technologies Lab Concordia University Montreal, CA

Scene Graph Generation 3D Scene Understanding

Open paper AskPDF arXiv

Current surgical scene graph generation methods depend on dense multi-modal supervision and specialized hardware (synchronized RGB-D sensors, calibration rigs), making dataset construction expensive and restricting all existing benchmarks to simulated environments. We propose SAGE-OR, a feature-centric framework that replaces the traditional detect-then-reason paradigm with a decoupled representation-reasoning paradigm in which localization is derived from frozen foundation models, encoded implicitly in pre-computed features, and used without any localization supervision, while a lightweight graph transformer performs relational reasoning over cached features. We employ a semi-supervised formulation with general-purpose segmentation prompts to eliminate localization supervision while enabling unsupervised context augmentation through additional prompt-driven entities, such as hands, which are absent from annotations. General-purpose prompts are used to induce near-perfect recall, while precision is delegated to downstream attention-based reasoning, enabling simple adaptation to new entities via prompt-level modification. This design enables a lightweight 15M-parameter graph transformer that trains in 1.4 hours and runs relational inference at

\sim

1ms per frame with peak memory under 2GB, suitable for edge hardware used in the operating room; feature extraction runs offline as a separate caching stage (4.27s per frame). On the 4D-OR benchmark, the core model achieves 76% F1, matching the fully supervised 4D-OR baseline while eliminating all localization annotations, and unsupervised hand augmentation raises this to 86%, within 4 points of state-of-the-art (SOTA) methods requiring dense multi-modal supervision, providing a practical pathway for adaptation to new surgical settings without annotation other than relationship and class labels.

Abstract

Explore similar work

SAGE-OR: Semi-supervised Adaptive Scene Graph Generation for Operating Rooms

DSFlash: Comprehensive Panoptic Scene Graph Generation in Realtime

SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models