Operating rooms (ORs) are cluttered, dynamic, highly occluded environments, where reliable spatial understanding is essential for situational awareness during complex surgical workflows. Achieving spatial understanding for panoptic segmentation from sparse multiview images poses a fundamental challenge, as limited visibility in a subset of views often leads to mispredictions across cameras. To this end, we introduce PanORama, the first panoptic segmentation for the operating room that is multiview-consistent by design. By modeling cross-view interactions at the feature level inside the backbone in a single forward pass, view consistency emerges directly rather than through post-hoc refinement. We evaluate on the MM-OR and 4D-OR datasets, achieving >70% Panoptic Quality (PQ) performance, and outperforming the previous state of the art. Importantly, PanORama is calibration-free, requiring no camera parameters, and generalizes to unseen camera viewpoints within any multiview configuration at inference time. By substantially enhancing multiview segmentation and, consequently, spatial understanding in the OR, we believe our approach opens new opportunities for surgical perception and assistance. Code will be released upon acceptance.
Figures & tables
Figure 1: Overview of PanORama. Each input view is independently encoded by a shared visual backbone fθ , producing per-view patch token embeddings. The multiview aggregator gϕ fuses these embeddings via local and global attention, enabling cross-view reasoning before mask prediction. The segmentation head hψ progressively upsamples the aggregated features through hierarchical convolutional stages with skip connections to produce panoptic masks for each input view in a single forward pass.
View
Train
Method
MM-OR PQ
4D-OR PQ
1
4
8
1
4
8
Single
MM-OR
DVIS++ [ 19 ]
64.60
59.60
59.30
–
Both
DVIS++ [ 19 ]
64.10
62.30
61.70
64.90
67.80
67.50
MM-OR
PanORama (Single)
68.91
66.28
64.75
–
Both
PanORama (Single)
69.21
66.40
64.97
62.35
59.19
56.92
Multi
MM-OR
PanORama (Multi)
71.85
68.80
66.91
–
Table 1: Main results. Panoptic Quality (PQ ↑ ) as percentage on MM-OR and 4D-OR. Train: MM-OR, Both = MM-OR+4D-OR. Columns 1/4/8 report PQ over temporal windows of 1, 4, and 8 consecutive frames.
Figure 2: Qualitative panoptic segmentation results on the MM-OR test set. Each column corresponds to one of the five synchronized camera views, with ground-truth annotations (top row) and predicted masks (bottom row) overlaid on the color image.
Figure 3: Qualitative panoptic segmentation results on the MM-OR test set for views lacking ground-truth annotations and excluded from supervision. These predictions are produced jointly with the supervised views shown in Fig. 2 in a single inference pass, yet without any view-specific training signal, demonstrating PanORama’s generalization capability within a multiview context.
#Train Views
#Eval Views
PQ
3 (supervised)
3
71.37
5 (3 supervised)
3
71.69
5 (3 supervised)
5
71.85
Table 2: Leveraging unlabeled views. Effect of the number of input views during training and inference on MM-OR. Panoptic annotations are available for 3 views only; PQ ( ↑ ) is computed on the annotated views.
Method
Cam 1
Cam 4 (Novel)
Cam 5
Overall
DVIS++ (w/o Cam 4)
54.81
32.43
53.29
48.53
PanORama (w/o Cam 4)
70.65
50.37
66.97
62.26
PanORama (as reference)
72.69
73.03
69.81
71.85
Table 3: Novel-view generalization. PQ ( ↑ ) on MM-OR when Camera 4 is withheld during training and evaluated only at test time. Bold indicates the best result under the withheld-camera setting; the final row is a reference model trained with all cameras.
Multi-view panoptic segmentation assigns a semantic class and a scene-level instance ID to every pixel of an unordered set of images, and recent feed-forward 3D models predict these labels for the input views in a single forward pass. Their predictions, however, have been evaluated with the scene-level PQ (PQ^scene) borrowed from per-scene optimization methods, typically on rendered held-out views. PQ^scene tiles all views of a scene into a single image, so that a missed appearance or a change of ID lowers the score of the matched pair only in proportion to its area. We propose View-Consistent Panoptic Quality (VC-PQ), which extends PQ from a single image to a set of input views, counts equally every view in which an instance is visible, and penalizes a prediction that is not visible in the same views as its ground truth. A decomposition of VC-PQ attributes the score a method loses to mask accuracy, view consistency, and the matching threshold. A single additional parameter recovers the area weighting of tiling for comparison. Under a fixed evaluation protocol on ScanNet++ and ScanNetv2, recent feed-forward methods are evaluated with VC-PQ and PQ^scene, and the decomposition shows where each of them loses its score. Controlled perturbations of the ground truth show that VC-PQ responds to the number of views in which an instance is missed or changes ID, whereas PQ^scene responds to their area. The aim of this work is to make view consistency part of the evaluation of multi-view panoptic segmentation, with VC-PQ reported alongside PQ^scene. The evaluation code and the input sets of the protocol are available at https://github.com/Renem24/vcpq.
Youngmin Lee, Byungha Ko, Guhnoo Yun +1
Korea University · Korea Institute of Science and Technology
In operating rooms (OR), world-scale multi-view 3D tracking supports downstream applications such as surgeon behavior recognition, where physically meaningful quantities such as distances and motion statistics must be measured in meters. However, real clinical deployments rarely satisfy the geometric prerequisites for stable multi-view fusion and tracking: camera calibration and RGB-D registration are always unreliable, leading to cross-view geometric inconsistency that produces "ghosting" during fusion and degrades 3D trajectories in a shared OR coordinate frame. To address this, we introduce Geometry OR Tracker, a two-stage pipeline that first rectifies imprecise calibration into a scaleconsistent and geometrically consistent camera setup with a single global scale via a Multi-view Metric Geometry Rectification module, and then performs Occlusion-Robust 3D Point Tracking directly in the unified OR world frame. On the MM-OR benchmark, improved geometric consistency translates into tracking gains: our rectification front-end reduces cross-view depth disagreement by more than 30× compared to raw calibration. Ablation studies further demonstrate the relationship between calibration quality and tracking accuracy, showing that improved geometric consistency yields stronger world-frame tracking.
Yihua Shao, Kang Chen, Feng Xue +6
Centre for Artificial Intelligence and Robotics Hong Kong Institute of Science and Innovation Chinese Academy of Sciences · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS) Institute of Automation Chinese Academy of Sciences (CASIA) · Peking University +4
Fine-grained understanding of operating room (OR) activity could enable workflow-aware assistance, yet remains difficult due to clutter, occlusions, and limited sensing. The prevailing approach to model this environment is scene graphs as an interpretable representation of OR interactions. Converting their frame-wise relational predictions into temporally extended, fine-grained actions however, is challenging without explicit temporal modeling. To enable a principled temporal evaluation of current OR understanding methods, we introduce the first action-centric benchmark built on a publicly available ego-exocentric OR dataset by defining a fine-grained, multi-role action taxonomy and generating dense action segments via distillation from ground-truth scene graph state changes. Experiments on this benchmark show that current scene graph prediction methods struggle to model temporal structure, even when adding explicit modeling through Graph Neural Networks. We therefore introduce a vision-only temporal model that outperforms graph-based methods significantly when using all available egocentric video as input. Building on this model we also introduce a novel multi- to single-view feature alignment strategy that improves single-view performance on multi-role action recognition, mitigating the need for extensive egocentric video capture. Benchmark and code will be released upon acceptance.
Felix Tristram, Ege Özsoy, Christian Benz +3
Technical University of Munich, Germany · Munich Center for Machine Learning (MCML) · Carl Zeiss AG, Germany