SAGE-OR: Semi-supervised Adaptive Scene Graph Generation for Operating Rooms

Brandon Leblanc; Charalambos Poullis

cs.CVAug 15, 2026

SAGE-OR: Semi-supervised Adaptive Scene Graph Generation for Operating Rooms

Authors: Brandon Leblanc, Charalambos Poullis

Organizations: The Immersive and Creative Technologies Lab Concordia University Montreal, CA

Abstract

Current surgical scene graph generation methods depend on dense multi-modal supervision and specialized hardware (synchronized RGB-D sensors, calibration rigs), making dataset construction expensive and restricting all existing benchmarks to simulated environments. We propose SAGE-OR, a feature-centric framework that replaces the traditional detect-then-reason paradigm with a decoupled representation-reasoning paradigm in which localization is derived from frozen foundation models, encoded implicitly in pre-computed features, and used without any localization supervision, while a lightweight graph transformer performs relational reasoning over cached features. We employ a semi-supervised formulation with general-purpose segmentation prompts to eliminate localization supervision while enabling unsupervised context augmentation through additional prompt-driven entities, such as hands, which are absent from annotations. General-purpose prompts are used to induce near-perfect recall, while precision is delegated to downstream attention-based reasoning, enabling simple adaptation to new entities via prompt-level modification. This design enables a lightweight 15M-parameter graph transformer that trains in 1.4 hours and runs relational inference at $\sim$ 1ms per frame with peak memory under 2GB, suitable for edge hardware used in the operating room; feature extraction runs offline as a separate caching stage (4.27s per frame). On the 4D-OR benchmark, the core model achieves 76% F1, matching the fully supervised 4D-OR baseline while eliminating all localization annotations, and unsupervised hand augmentation raises this to 86%, within 4 points of state-of-the-art (SOTA) methods requiring dense multi-modal supervision, providing a practical pathway for adaptation to new surgical settings without annotation other than relationship and class labels.

Figures & tables

Figure 1: Overview of the 3-stage decoupled feature caching pipeline: 3D reconstruction (Pi3), open-vocabulary segmentation (SAM3), and instance fusion to produce a static per-frame cache of visual features, geometric features, centroids, and category hints. All stages use frozen models; the resulting cache is reused across all training and inference without modification. Bracketed values denote tensor shapes.

Figure 2: SAGE-OR training pipeline. Cached features are projected, positionally encoded, then processed by an edge-aware graph transformer. The node head produces entity classifications, while pairwise features are constructed and refined by an edge transformer for relation prediction. Hungarian matching aligns predicted nodes to ground-truth entities without localization supervision. Bracketed values denote tensor shapes.

Method	Params	Time/s	Supervision		Assist	Cement	Clean	CloseTo	Cut	Drill	Hammer	Hold	LyingOn	Operate	Prepare	Saw	Suture	Touch	Avg
4D-OR [ 9 ]	84.8M	1.28	Full	P	42	78	53	97	49	87	71	55	100	55	62	69	60	41	68
				R	93	78	63	89	49	100	89	95	99	99	91	91	100	69	87
				F1	58	78	57	93	49	93	79	70	99	71	74	79	75	51	75
LABRAD-OR ‡ [ 18 ]	85.8M	1.46	Full	P	60	96	86	96	91	100	93	71	100	85	77	78	100	71	87
				R	86	93	72	94	68	94	95	95	100	99	91	93	100	72	90
				F1	71	94	78	95	78	97	94	81	100	91	84	85	100	71	88

Table 1: Comparison with state-of-the-art OR-SGG methods on the 4D-OR test set. All values are percentages. Best F1 per class in bold among spatially grounded methods. Time/s reports seconds per scene following the convention of [ 19 ] .

\ddagger

Temporal methods. Pix2SG (gray) is non-spatially grounded and autoregressive, reports average F1 only, and is listed for reference.

\dagger

Graph inference only; feature extraction is offline (4.27s/frame, Table 5 ).

Configuration		Assist	Cement	Clean	CloseTo	Cut	Drill	Hammer	Hold	LyingOn	Operate	Prepare	Saw	Suture	Touch	Avg
Without hands	P	44	100	84	92	93	96	70	77	99	83	78	79	74	54	81
	R	68	56	71	92	47	73	91	82	100	92	77	51	65	57	75
	F1	54	71	77	92	63	83	79	79	100	87	77	62	69	56	76
With hands (final)	P	50	96	91	93	84	94	89	77	100	85	81	87	93	61	85
	R	61	96	80	92	75	94	93	87	100	92	79	87	96	63	86
	F1	55	96	85	93	80	94	91	82	100	88	80	87	94	62	86

Table 2: Per-class impact of adding unsupervised hand nodes on the 4D-OR test set. Despite receiving no supervision, hand nodes improve average F1 by +10 points.

\Delta

denotes absolute F1 change per class.

Table 5

Table 6

Table 7

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Prompt Text	Res.	Conf.
Person	“person”	Low	0.3
Table	“medical table”	Low	0.3
Anesthesia	“anesthesia equipment”	Low	0.3
Tool	“tool in hand”	High	0.01
Hand	“hand”	High	0.3

Appendix

Table S1: SAM3 configuration and resulting segmentation recall. Left: prompt configuration with per-category resolution and confidence thresholds. Right: recall across dataset splits (over-segmentation handled by Hungarian matching; hand has no GT labels).

Target	Prompt Text	Det.	Recall
Person variants
Patient only	“patient”	14,850	100% †
Staff only	“doctor”	25,704	100% †
Staff only	“nurse”	23,502	100% †
Staff only	“surgeon”	35,637	100% †
Staff only	“assistant”	11	0%

Appendix

Table S2: Prompt ablation on the training set (4,179 frames

\times

6 cameras). Fine-grained prompts either fail to detect targets or indiscriminately label all instances.

Panel of Figure 10 — Figure S1: Qualitative segmentation examples from the train set. Each visualization shows the original camera views with instance masks overlaid. Colors indicate SAM3 categories. Examples demonstrate high recall across diverse surgical contexts.

Panel of Figure 11 — Figure S2: Qualitative comparison of Pi3 zero-shot reconstruction (left) versus 4D-OR depth sensor point clouds (right) on matched frames. Pi3 produces significantly denser point clouds with finer geometric detail, enabling robust centroid extraction for diverse entity types.

Panel of Figure 12 — Figure S3: Multi-view instance fusion results. Right: Point clouds colored by fused cluster ID after 3D IoU clustering. Left: Extracted 3D centroids (shown as spheres) for each fused instance, used as geometric features for the graph transformer. The adaptive voxelization makes fusion robust to Pi3’s scale ambiguity.

Stage	Input	Output	Trainable
Pi3 [ 23 ]	Multi-view RGB	Pointmaps, poses	×
SAM3 [ 24 ]	RGB + prompts	Instance masks	×
DINOv3 [ 25 ]	Masked crops	Visual features	×
Graph Transformer	Cached features	Scene graph	✓

Appendix

Table S3: SAGE-OR pipeline overview. Only the graph transformer is trainable; all other stages use frozen foundation models and run offline.

Category	Total Detections	Avg/Frame	F1 Impact
Hand (baseline)	31,673	4.7	+10
Knee (irrelevant)	5,916	0.9	0
Door (irrelevant)	29,905	4.4	0
Shoe (irrelevant)	32,264	4.8	0

Appendix

Table S4: Detection statistics for control study prompts across all 6,734 frames. Despite contributing comparable or more nodes than hands, irrelevant prompts provide no F1 improvement.

Explore similar work

Aug 17, 2026

Sterilizable Scene Graph Generation for Operating Rooms

Scene graph generation from surgical video enables a holistic and structured understanding of surgical scenes by modeling objects and their semantic relationships. Despite recent advances, state-of-the-art approaches rely on large, parameter-heavy deep learning models that are impractical for deployment in the operating room (OR) due to hardware footprint, hygiene constraints, latency, and data privacy concerns. To the best of our knowledge, this is the first scene graph generation method built on NCAs and the first NCA framework capable of learning structured representations. We introduce SG-NCA, a lightweight scene graph generation framework based on Neural Cellular Automata (NCA), designed for inference in fanless devices critical for OR hygiene protocols. SG-NCA is the first scene graph generation combining NCA-based multiclass segmentation for efficient object detection and feature extraction with a lightweight relation predictor. We evaluate SG-NCA on videos of cataract surgery and cholecystectomy, demonstrating performance comparable to established baselines while requiring 55x fewer parameters. We showcase deployment on fanless edge devices better suited for the OR and demonstrate downstream applications such as surgical video captioning, highlighting SG-NCA's potential for affordable, privacy-preserving, and OR-ready intraoperative scene understanding.

Nick Lemke, Ssharvien Kumar Sivakumar, Antoine P. Sanner +4

Technical University of Darmstadt, Darmstadt, Germany · ImFusion GmbH, Munich, Germany · Carl Zeiss AG, Munich, Germany +2

Scene Graph Generation Surgical Video Understanding Neural Cellular Automata

Ask PDF

Mar 10, 2026

TopoOR: A Unified Topological Scene Representation for the Operating Room

Surgical Scene Graphs abstract the complexity of surgical operating rooms (OR) into a structure of entities and their relations, but existing paradigms suffer from strictly dyadic structural limitations. Frameworks that predominantly rely on pairwise message passing or tokenized sequences flatten the manifold geometry inherent to relational structures and lose structure in the process. We introduce TopoOR, a new paradigm that models multimodal operating rooms as a higher-order structure, innately preserving pairwise and group relationships. By lifting interactions between entities into higher-order topological cells, TopoOR natively models complex dynamics and multimodality present in the OR. This topological representation subsumes traditional scene graphs, thereby offering strictly greater expressivity. We also propose a higher-order attention mechanism that explicitly preserves manifold structure and modality-specific features throughout hierarchical relational attention. In this way, we circumvent combining 3D geometry, audio, and robot kinematics into a single joint latent representation, preserving the precise multimodal structure required for safety-critical reasoning, unlike existing methods. Extensive experiments demonstrate that our approach outperforms traditional graph and LLM-based baselines across sterility breach detection, robot phase prediction, and next-action anticipation

Tony Danjun Wang, Ka Young Kim, Tolga Birdal +2

Technical University of Munich, Munich, Germany · Munich Center for Machine Learning, Germany · Kyung Hee University, South Korea +1

Topological Representation Learning Multimodal Graph Learning Surgical Video Understanding Surgical Robotics

Ask PDF

Jun 4, 2026

Visual Commonsense Driven Knowledge Refinements for Scene Graph Generation

Learning-driven Scene Graph Generation (SGG) models excel on frequent relation types but degrade sharply under annotation sparsity, failing to capture reliable visual commonsense knowledge. We propose a model-agnostic, semantically-guided knowledge refinement framework that systematically mines commonsense-grounded constraints from training data - capturing spatial, functional, and qualitative relational regularities - and uses general declarative commonsense reasoning to correct and refine ranked SGG predictions at inference time. The framework requires no manual rule authoring, no model retraining, and transfers across datasets and architectures. On three standard benchmarks, we obtain consistent improvements over strong baselines, demonstrating that structured visual commonsense reasoning over deep scene semantics is a practical and effective complement to purely learning-based scene graph generation.

Maëlic Neau, Salim Baloch, Jakob Suchan +2

Computing Science Department, Umeå University, Umeå, Sweden · School of Computer Science & Engineering, Constructor University, Bremen, Germany · CoDesign Lab EU., Örebro, Sweden +1

Scene Graph Generation Multimodal Reasoning

Ask PDF