PanORama: Multiview Consistent Panoptic Segmentation in Operating Rooms

Tuna Gürbüz; Ege Özsoy; Tony Danjun Wang; Nassir Navab

cs.CVMar 20, 2026

PanORama: Multiview Consistent Panoptic Segmentation in Operating Rooms

Authors: Tuna Gürbüz, Ege Özsoy, Tony Danjun Wang, Nassir Navab

Organizations: Computer Aided Medical Procedures, Technische Universität München, Germany · MCML, Germany

Abstract

Operating rooms (ORs) are cluttered, dynamic, highly occluded environments, where reliable spatial understanding is essential for situational awareness during complex surgical workflows. Achieving spatial understanding for panoptic segmentation from sparse multiview images poses a fundamental challenge, as limited visibility in a subset of views often leads to mispredictions across cameras. To this end, we introduce PanORama, the first panoptic segmentation for the operating room that is multiview-consistent by design. By modeling cross-view interactions at the feature level inside the backbone in a single forward pass, view consistency emerges directly rather than through post-hoc refinement. We evaluate on the MM-OR and 4D-OR datasets, achieving >70% Panoptic Quality (PQ) performance, and outperforming the previous state of the art. Importantly, PanORama is calibration-free, requiring no camera parameters, and generalizes to unseen camera viewpoints within any multiview configuration at inference time. By substantially enhancing multiview segmentation and, consequently, spatial understanding in the OR, we believe our approach opens new opportunities for surgical perception and assistance. Code will be released upon acceptance.

Figures & tables

Figure 1: Overview of PanORama. Each input view is independently encoded by a shared visual backbone $f_{\theta}$ , producing per-view patch token embeddings. The multiview aggregator $g_{\phi}$ fuses these embeddings via local and global attention, enabling cross-view reasoning before mask prediction. The segmentation head $h_{\psi}$ progressively upsamples the aggregated features through hierarchical convolutional stages with skip connections to produce panoptic masks for each input view in a single forward pass.

View	Train	Method	MM-OR PQ			4D-OR PQ
View	Train	Method	1	4	8	1	4	8
Single	MM-OR	DVIS++ [ 19 ]	64.60	59.60	59.30	–
	Both	DVIS++ [ 19 ]	64.10	62.30	61.70	64.90	67.80	67.50
	MM-OR	PanORama (Single)	68.91	66.28	64.75	–
	Both	PanORama (Single)	69.21	66.40	64.97	62.35	59.19	56.92
Multi	MM-OR	PanORama (Multi)	71.85	68.80	66.91	–

Table 1: Main results. Panoptic Quality (PQ

\uparrow

) as percentage on MM-OR and 4D-OR. Train: MM-OR, Both = MM-OR+4D-OR. Columns 1/4/8 report PQ over temporal windows of 1, 4, and 8 consecutive frames.

Figure 2: Qualitative panoptic segmentation results on the MM-OR test set. Each column corresponds to one of the five synchronized camera views, with ground-truth annotations (top row) and predicted masks (bottom row) overlaid on the color image.

Figure 3: Qualitative panoptic segmentation results on the MM-OR test set for views lacking ground-truth annotations and excluded from supervision. These predictions are produced jointly with the supervised views shown in Fig. 2 in a single inference pass, yet without any view-specific training signal, demonstrating PanORama’s generalization capability within a multiview context.

#Train Views	#Eval Views	PQ
3 (supervised)	3	71.37
5 (3 supervised)	3	71.69
5 (3 supervised)	5	71.85

Table 2: Leveraging unlabeled views. Effect of the number of input views during training and inference on MM-OR. Panoptic annotations are available for 3 views only; PQ (

\uparrow

) is computed on the annotated views.

Method	Cam 1	Cam 4 (Novel)	Cam 5	Overall
DVIS++ (w/o Cam 4)	54.81	32.43	53.29	48.53
PanORama (w/o Cam 4)	70.65	50.37	66.97	62.26
PanORama (as reference)	72.69	73.03	69.81	71.85

Table 3: Novel-view generalization. PQ (

\uparrow

) on MM-OR when Camera 4 is withheld during training and evaluated only at test time. Bold indicates the best result under the withheld-camera setting; the final row is a reference model trained with all cameras.

Panoptic Segmentation

Panoptic segmentation assigns every pixel a semantic category and groups countable objects into distinct instances.

See all 38 papers

Every View Counts: View-Consistent Panoptic Quality for Multi-view Panoptic Segmentation
Oct 5, 2026Youngmin Lee, Byungha Ko, Guhnoo Yun +1Multi-View Consistency Panoptic Segmentation
Korea University · Korea Institute of Science and Technology
Geometry OR Tracker: Universal Geometric Operating Room Tracking
Feb 28, 2026Yihua Shao, Kang Chen, Feng Xue +6Point Tracking Multi-View Geometry
Centre for Artificial Intelligence and Robotics Hong Kong Institute of Science and Innovation Chinese Academy of Sciences · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS) Institute of Automation Chinese Academy of Sciences (CASIA) · Peking University +4
OR-Action: Multi-Role Video Understanding with Fine-Grained Actions
Jun 11, 2026Felix Tristram, Ege Özsoy, Christian Benz +3Temporal Action Segmentation Human Activity Recognition
Technical University of Munich, Germany · Munich Center for Machine Learning (MCML) · Carl Zeiss AG, Germany

Oct 5, 2026J/K move · Enter open · S save

Every View Counts: View-Consistent Panoptic Quality for Multi-view Panoptic Segmentation

Youngmin Lee, Byungha Ko, Guhnoo Yun, Dong Hwan Kim

Korea University · Korea Institute of Science and Technology

Multi-View Consistency Panoptic Segmentation

Open paper AskPDF arXiv

Multi-view panoptic segmentation assigns a semantic class and a scene-level instance ID to every pixel of an unordered set of images, and recent feed-forward 3D models predict these labels for the input views in a single forward pass. Their predictions, however, have been evaluated with the scene-level PQ (PQ^scene) borrowed from per-scene optimization methods, typically on rendered held-out views. PQ^scene tiles all views of a scene into a single image, so that a missed appearance or a change of ID lowers the score of the matched pair only in proportion to its area. We propose View-Consistent Panoptic Quality (VC-PQ), which extends PQ from a single image to a set of input views, counts equally every view in which an instance is visible, and penalizes a prediction that is not visible in the same views as its ground truth. A decomposition of VC-PQ attributes the score a method loses to mask accuracy, view consistency, and the matching threshold. A single additional parameter recovers the area weighting of tiling for comparison. Under a fixed evaluation protocol on ScanNet++ and ScanNetv2, recent feed-forward methods are evaluated with VC-PQ and PQ^scene, and the decomposition shows where each of them loses its score. Controlled perturbations of the ground truth show that VC-PQ responds to the number of views in which an instance is missed or changes ID, whereas PQ^scene responds to their area. The aim of this work is to make view consistency part of the evaluation of multi-view panoptic segmentation, with VC-PQ reported alongside PQ^scene. The evaluation code and the input sets of the protocol are available at https://github.com/Renem24/vcpq.

Abstract

Panoptic Segmentation

Explore similar work

Every View Counts: View-Consistent Panoptic Quality for Multi-view Panoptic Segmentation

Geometry OR Tracker: Universal Geometric Operating Room Tracking

OR-Action: Multi-Role Video Understanding with Fine-Grained Actions