PanoBench: A Panoptic Occupancy Prediction Benchmark for Autonomous Driving

Yi Feng; Zizhan Guo; Junwu E; Yu Ma; Mengtan Zhang; Jin Wu; Wei Ye; Hanli Wang

cs.CVOct 16, 2026

PanoBench: A Panoptic Occupancy Prediction Benchmark for Autonomous Driving

Authors: Yi Feng, Zizhan Guo, Junwu E, Yu Ma, Mengtan Zhang, Jin Wu, Wei Ye, Hanli Wang,

Abstract

Panoptic occupancy unifies 3D geometry, semantics, and instance identities, but its development is limited by scarce instance-aware annotations. Existing benchmarks mainly provide semantic labels reconstructed from incomplete sensor observations, while panoptic datasets have limited domain coverage or annotation completeness. We introduce PanoBench, a camera-centric panoptic occupancy prediction benchmark spanning synthetic and real-world driving scenes. It combines PanoBench-CarlaOcc, PanoBench-nuScenes, and PanoBench-KITTI360 through canonical data interfaces and unified evaluation protocols, providing multi-resolution annotations with persistent instance identities. The synthetic branch derives complete panoptic geometry from simulated meshes. For real-world data, a new pipeline reconstructs backgrounds using geometric priors and topology constraints, generates foreground mesh assets from multi-frame multi-view images, and fuses them into sparse panoptic voxels. A unified valid-mask protocol evaluates complete thing instances supported by temporal observations while scoring stuff regions at the voxel level. Analyses of observation fidelity, geometric completeness, spatial continuity, and temporal consistency demonstrate the benefits of the resulting annotations over existing occupancy benchmarks. We benchmark representative methods on semantic occupancy prediction, panoptic occupancy prediction, and 4D occupancy forecasting, and study voxel resolution, synthetic-to-real transfer, and storage efficiency. These results reveal complementary strengths of volumetric and ray-based metrics and persistent challenges in real-world instance-aware prediction. PanoBench is available at https://mias.group/PanoBench.

Figures & tables

Fig. 1: Overview of PanoBench, a unified benchmark for camera-centric panoptic occupancy prediction. By incorporating diverse driving datasets from both synthetic and real-world domains, PanoBench provides multi-resolution, high-fidelity, and temporally consistent instance annotations, establishing a unified testbed for semantic occupancy prediction, panoptic occupancy prediction, and 4D occupancy forecasting.

Dataset	Data Source	View	Sequences	Frames	Classes	Voxel Size (m)	Volume Size (m)	Instance	Valid Masks
SemanticKITTI [ 8 ]	Real	Forward	22	43,000	28	0.2	[51.2, 51.2, 6.4]	$\times$	$\times$
SSCBench-KITTI360 [ 11 ]	Real	Forward	9	12,865	19	0.2	[51.2, 51.2, 6.4]	$\times$	$\times$
Occ3D-nuScenes [ 6 ]	Real	Surround	850	34,149	17	0.4	[80.0, 80.0, 6.4]	$\times$	$\checkmark$
SurroundOcc [ 10 ]	Real	Surround	850	34,149	17	0.5	[100.0, 100.0, 8.0]	$\times$	$\times$
CarlaSC [ 29 ]	Synthetic	Surround	24	43,200	11	0.2, 0.4	[51.2, 51.2, 3.0]	$\times$	$\times$
CoHFF [ 28 ]	Synthetic	Surround	73	11,464	12	0.4	[40.0, 40.0, 3.2]	$\times$	$\times$

TABLE I: Comparison of dataset statistics between PanoBench and representative occupancy prediction benchmarks.

Fig. 2: Real-world panoptic occupancy reconstruction pipeline. Static background geometry is recovered by semantic LiDAR aggregation, HD map-guided ground lifting, topology-aware flat region recovery, and geometry-preserving processing of non-planar structures. For foreground thing instances, PanoBench mines high-quality multi-frame multi-view image crops, reconstructs complete mesh assets, selects a diverse prototype pool, and assigns a compatible mesh to each annotated instance. The aligned foreground meshes and scene-level background are voxelized and fused into sparse panoptic occupancy labels, from which temporally aware camera-centric valid masks are generated.

Fig. 3: Comparison of prior valid masks and the proposed valid masks. (a) Previous methods generate a valid mask for each single frame, in which camera rays leave large occluded regions invalid (gray). (b) The proposed temporal valid mask keeps three classes of regions under evaluation: \tiny{$\mathbf{1}$}⃝ a currently occluded but previously observed instance, recovered by aggregating visibility over a window of $T$ frames; \tiny{$\mathbf{2}$}⃝ a partially observed instance, completed as a whole once any of its voxels is visible; and \tiny{$\mathbf{3}$}⃝ stuff behind foreground instances, which remains valid to avoid occlusion-induced fragments. (c)–(e) Qualitative comparison of the raycasting-based mask, the complete ground truth, and the proposed valid mask on PanoBench-CarlaOcc. — Fig. 3: Comparison of prior valid masks and the proposed valid masks. (a) Previous methods generate a valid mask for each single frame, in which camera rays leave large occluded regions invalid (gray). (b) The proposed temporal valid mask keeps three classes of regions under evaluation: \tiny{ $\mathbf{1}$ }⃝ a currently occluded but previously observed instance, recovered by aggregating visibility over a window of $T$ frames; \tiny{ $\mathbf{2}$ }⃝ a partially observed instance, completed as a whole once any of its voxels is visible; and \tiny{ $\mathbf{3}$ }⃝ stuff behind foreground instances, which remains valid to avoid occlusion-induced fragments. (c)–(e) Qualitative comparison of the raycasting-based mask, the complete ground truth, and the proposed valid mask on PanoBench-CarlaOcc.

Domain	Dataset	Voxel Size (m)	$\mathrm{PixAcc}$ (%)
CARLA	PanoBench-CarlaOcc	0.1	91.6
	PanoBench-CarlaOcc	0.2	88.8
	PanoBench-CarlaOcc	0.4	86.4
nuScenes	OpenOccupancy [ 7 ]	0.2	70.0
	PanoBench-nuScenes	0.2	77.2
	Occ3D [ 6 ]	0.4	69.2

TABLE II: Comparison of observation fidelity across occupancy benchmarks.

Fig. 4: Illustration of observation fidelity. Occupancy is ray-cast into the camera view and compared with 2D semantic ground truth.

Fig. 5: Comparison of geometric completeness on nuScenes. (a) BEV views of two scenes. (b) Two foreground instances, each rendered from two opposite viewpoints.

Method	Geometric IoU	Semantic mIoU	Road	Sidewalk	Terrain	Building	Vegetation	Object	Vehicle	Pedestrian
Occ3D-style [ 6 ]	38.3	32.6	66.7	60.7	29.8	18.2	27.7	40.5	5.9	11.4
SurroundOcc-style [ 10 ]	37.5	30.5	68.4	56.9	28.2	18.9	27.2	26.8	7.0	10.9
PanoBench (Ours)	58.9	52.0	82.8	82.3	54.8	18.4	27.6	45.3	59.1	46.0

TABLE III: Controlled oracle evaluation of occupancy annotation pipelines on PanoBench-CarlaOcc.

Fig. 6: Comparison of occupancy annotation pipelines on PanoBench-CarlaOcc ( Town10HD ). The ground truth is voxelized from complete scene meshes, and Occ3D and SurroundOcc denote their style-faithful reimplementations.

Domain	Dataset	Voxel Size (m)	$r_{\mathrm{fp}}@0.1$	$r_{\mathrm{fp}}@0.2$
CARLA	CarlaSC [ 29 ]	0.2	9.4	15.1
	PanoBench-CarlaOcc	0.2	2.5	5.6
	CarlaSC [ 29 ]	0.4	5.3	11.8
	PanoBench-CarlaOcc	0.4	0.3	1.1
nuScenes	OpenOccupancy [ 7 ]	0.2	23.2	31.0
	PanoBench-nuScenes	0.2	11.2	17.4

TABLE IV: Comparison of spatial continuity across occupancy benchmarks. Floating-voxel ratios are in permille (lower is better), and bold denotes the better result at the same voxel size.

Domain	Dataset	Voxel Size (m)	mIoU	IoU
CARLA	PanoBench-CarlaOcc	0.1	81.9	83.3
	CarlaSC [ 29 ]	0.2	70.3	83.5
	PanoBench-CarlaOcc	0.2	82.8	89.1
	CarlaSC [ 29 ]	0.4	71.5	86.8
	PanoBench-CarlaOcc	0.4	83.6	92.3
nuScenes	OpenOccupancy [ 7 ]	0.2	60.9	68.2

TABLE V: Comparison of temporal consistency across occupancy benchmarks. Bold denotes the better result at the same voxel size.

Fig. 7: Comparison of instance temporal consistency on nuScenes: (a) Occ3D-Panoptic [ 15 ] and (b) PanoBench-nuScenes. Voxels with the same instance identity are rendered in the same color across frames, and red and blue circles highlight two representative instances.

Background Component	Configuration
Semantic LiDAR Aggregation	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Ground-Layer Lifting		$\checkmark$	$\checkmark$	$\checkmark$
Flat-Region Recovery			$\checkmark$	$\checkmark$
Non-Planar Refinement				$\checkmark$
$\mathrm{PixAcc}$ (%)	68.5	74.1	79.0	79.4

TABLE VI: Ablation study on background reconstruction components on PanoBench-nuScenes. Observation fidelity is evaluated on stuff pixels.

Method	Input Views	View Selection	IoU
LiDAR Aggregation [ 6 ]	–	–	12.4
SAM 3D [ 60 ]	Single-View	$\checkmark$	61.2
TRELLIS [ 58 ]	Single-View	$\checkmark$	57.0
TRELLIS-2 [ 59 ]	Single-View	$\checkmark$	60.5
MV-SAM3D [ 63 ]	Multi-View	$\times$	65.1
MV-SAM3D [ 63 ]	Multi-View	$\checkmark$	67.2

TABLE VII: Ablation study on foreground mesh asset acquisition on PanoBench-CarlaOcc, measured by instance IoU (%).

Fig. 8: Representative prototype meshes from the three PanoBench subsets. Rows show (a) cars, (b) two-wheelers, (c) pedestrians, and (d) other vehicles. The examples illustrate the intra-class shape diversity retained by the prototype mesh pools across synthetic and real-world domains.

Method	PanoBench-CarlaOcc					PanoBench-nuScenes					PanoBench-KITTI360
Method	IoU	mIoU	RayIoU@1	RayIoU@2	RayIoU@4	IoU	mIoU	RayIoU@1	RayIoU@2	RayIoU@4	IoU	mIoU	RayIoU@1	RayIoU@2	RayIoU@4
SparseOcc [ 15 ]	68.5	41.7	50.0	56.6	60.1	55.0	33.0	33.7	41.7	46.5	33.3	11.2	28.1	32.0	33.4
OPUS [ 39 ]	73.5	41.3	50.3	56.9	59.3	50.1	29.9	32.6	41.4	46.2	30.7	9.7	18.8	23.9	26.6
Panoptic-FlashOcc [ 16 ]	73.8	40.9	51.7	55.9	57.7	47.7	22.6	25.2	34.9	41.1	41.7	11.5	18.8	25.0	27.6
GaussianFormer [ 37 ]	82.2	46.2	56.4	61.0	62.8	66.5	32.6	32.8	41.8	46.7	65.3	15.6	21.1	25.5	27.1

TABLE VIII: Comparison of semantic occupancy prediction methods on PanoBench. Default voxel sizes are used, i.e. , 0.4 m for PanoBench-CarlaOcc and PanoBench-nuScenes and 0.2 m for PanoBench-KITTI360.

Method	PanoBench-CarlaOcc					PanoBench-nuScenes					PanoBench-KITTI360
Method	IoU	mIoU	RayPQ@1	RayPQ@2	RayPQ@4	IoU	mIoU	RayPQ@1	RayPQ@2	RayPQ@4	IoU	mIoU	RayPQ@1	RayPQ@2	RayPQ@4
SparseOcc [ 15 ]	68.5	41.7	36.5	41.7	44.1	55.0	33.0	11.2	16.5	20.1	33.3	11.2	12.7	16.0	17.4
Panoptic-FlashOcc [ 16 ]	73.8	40.9	30.6	34.8	36.8	47.7	22.6	3.2	7.0	10.8	41.7	11.5	7.2	10.8	12.6

TABLE IX: Comparison of panoptic occupancy prediction methods on PanoBench under the default voxel sizes in Table VIII .

Method	PanoBench-CarlaOcc								PanoBench-nuScenes								PanoBench-KITTI360
	Semantic mIoU				Geometric IoU				Semantic mIoU				Geometric IoU				Semantic mIoU				Geometric IoU
	1 s	2 s	3 s	Avg.	1 s	2 s	3 s	Avg.	1 s	2 s	3 s	Avg.	1 s	2 s	3 s	Avg.	1 s	2 s	3 s	Avg.	1 s	2 s	3 s	Avg.
SparseWorld [ 47 ]	33.4	28.6	24.6	28.9	68.4	63.9	59.5	63.9	25.3	23.1	21.0	23.1	49.2	47.6	45.7	47.5	6.2	4.7	3.6	4.9	24.4	20.8	17.5	20.9
SparseWorld-TC [ 48 ]	34.5	29.8	25.7	30.0	69.3	64.8	60.4	64.8	24.9	22.7	20.8	22.8	49.2	47.6	45.9	47.6	7.7	6.1	4.8	6.2	28.6	24.8	21.1	24.8

TABLE X: Comparison of 4D occupancy forecasting methods on PanoBench under the default voxel sizes in Table VIII .

Method	Voxel Size (m)	IoU	mIoU
OPUS [ 39 ]	0.1	38.5	11.7
	0.2	61.6	31.6
	0.4	73.5	41.3
Panoptic-FlashOcc [ 16 ]	0.1	38.5	11.8
	0.2	62.8	32.1
	0.4	73.8	40.9

TABLE XI: Multi-resolution occupancy prediction on PanoBench-CarlaOcc.

Target	Method	Pretraining	mIoU	RayIoU	RayPQ
nuScenes	SparseOcc [ 15 ]	–	33.0	40.6	15.9
	SparseOcc [ 15 ]	CarlaOcc	34.2	42.3	17.4
	Panoptic-FlashOcc [ 16 ]	–	22.6	33.7	7.0
	Panoptic-FlashOcc [ 16 ]	CarlaOcc	26.4	35.6	9.6
KITTI-360	SparseOcc [ 15 ]	–	11.2	31.2	15.4
	SparseOcc [ 15 ]	CarlaOcc	11.8	31.9	15.2

TABLE XII: Sim-to-real transfer results for panoptic occupancy prediction. RayIoU and RayPQ are averaged over depth tolerances of 1, 2, and 4 m.

Dataset	Storage (GiB)			Latency (ms)
Dataset	Official	Ours	$\downarrow$ (%)	Official	Ours	$\downarrow$ (%)
Occ3D-nuScenes [ 6 ]	2.6	1.3	48.7	11.1	1.6	85.8
Occ3D-Panoptic [ 15 ]	3.2	1.3	58.1	20.6	1.5	92.8
OpenOccupancy [ 7 ]	132.7	5.5	95.9	40.2	6.8	83.2
SurroundOcc [ 10 ]	40.3	1.2	97.1	13.5	1.9	85.8
SemanticKITTI [ 8 ]	90.6	1.2	98.7	36.3	2.1	94.1

TABLE XIII: Overall storage footprint and per-frame I/O latency of the official occupancy annotations versus our canonical sparse panoptic format. Latency is averaged over 1,000 randomly sampled frames per dataset on the same hardware.

3D Occupancy Prediction

3D occupancy prediction estimates the geometry and semantics of occupied and free space around an agent from camera, LiDAR, or radar data.

See all 113 papers

Mar 28, 2026

An Instance-Centric Panoptic Occupancy Prediction Benchmark for Autonomous Driving

Panoptic occupancy prediction aims to jointly infer voxel-wise semantics and instance identities within a unified 3D scene representation. Nevertheless, progress in this field remains constrained by the absence of high-quality 3D mesh resources, instance-level annotations, and physically consistent occupancy datasets. Existing benchmarks typically provide incomplete and low-resolution geometry without instance-level annotations, limiting the development of models capable of achieving precise geometric reconstruction, reliable occlusion reasoning, and holistic 3D understanding. To address these challenges, this paper presents an instance-centric benchmark for the 3D panoptic occupancy prediction task. Specifically, we introduce ADMesh, the first unified 3D mesh library tailored for autonomous driving, which integrates over 15K high-quality 3D models with diverse textures and rich semantic annotations. Building upon ADMesh, we further construct CarlaOcc, a large-scale, physically consistent panoptic occupancy dataset generated using the CARLA simulator. This dataset contains over 100K frames with fine-grained, instance-level occupancy ground truth at voxel resolutions as fine as 0.05 m. Furthermore, standardized evaluation metrics are introduced to quantify the quality of existing occupancy datasets. Finally, a systematic benchmark of representative models is established on the proposed dataset, which provides a unified platform for fair comparison and reproducible research in the field of 3D panoptic perception. Code and dataset are available at https://mias.group/CarlaOcc.

Yi Feng, Junwu E, Zizhan Guo +3

Tongji University

3D Occupancy Prediction Autonomous Driving Benchmarks Autonomous Driving Datasets

Ask PDF

Dec 17, 2024

Open-World Panoptic Segmentation

Robots need to be able to understand their surroundings in order to operate safely and robustly, and to interact with the surrounding environment. Robots deployed in unconstrained real-world scenarios must additionally be able to deal with novel situations and objects that have never been seen before. In this article, we tackle the problem of open-world panoptic segmentation, i.e., the task of discovering new semantic categories and new object instances at test time, while enforcing consistency among the categories that we incrementally discover. We present Con2MAV, a general method for open-world panoptic segmentation. Experiments across a wide range of datasets, from road scenes to underwater environments, highlight its compelling capabilities in open-world segmentation and its competitive performance on known classes. We will open-source the implementation of our approach upon acceptance. In addition, we propose PANIC (Panoptic ANomalies In Context), a benchmark for evaluating open-world segmentation tasks in autonomous driving scenarios. This dataset, recorded with a multi-modal sensor suite mounted on a car, and then manually annotated, provides high-quality, pixel-wise annotations of anomalous objects at both semantic and instance level. PANIC contains 800 images, more than 50 unknown classes, i.e., classes that do not appear in the training set, and over 4,000 object instances, providing a comprehensive benchmark for evaluating open-world segmentation methods in autonomous driving scenarios. We provide competitions for multiple open-world segmentation tasks on a hidden test set. Our dataset and competitions are available at https://www.ipb.uni-bonn.de/data/panic.

Matteo Sodano, Federico Magistri, Jens Behley +1

Center for Robotics, University of Bonn, Germany, and the Lamarr Institute for Machine Learning and Artificial Intelligence, Germany

Image Segmentation Panoptic Segmentation Autonomous Driving Perception

Ask PDF

Mar 6, 2026

FreeOcc: Training-free Panoptic Occupancy Prediction via Foundation Models

Semantic and panoptic occupancy prediction for road scene analysis provides a dense 3D representation of the ego vehicle's surroundings. Current camera-only approaches typically rely on costly dense 3D supervision or require training models on data from the target domain, limiting deployment in unseen environments. We propose FreeOcc, a training-free pipeline that leverages pretrained foundation models to recover both semantics and geometry from multi-view images. FreeOcc extracts per-view panoptic priors with a promptable foundation segmentation model and prompt-to-taxonomy rules, and reconstructs metric 3D points with a reconstruction foundation model. Depth- and confidence- aware filtering lifts reliable labels into 3D, which are fused over time and voxelized with a deterministic refinement stack. For panoptic occupancy, instances are recovered by fitting and merging robust current-view 3D box candidates, enabling instance-aware occupancy without any learned 3D model. On Occ3D-nuScenes, FreeOcc achieves 16.9 mIoU and 16.5 RayIoU train-free, on par with state-of-the-art weakly supervised methods. When employed as a pseudo-label generation pipeline for training downstream models, it achieves 21.1 RayIoU, surpassing the previous state-of-the-art weakly supervised baseline. Furthermore, FreeOcc sets new baselines for both train-free and weakly supervised panoptic occupancy prediction, achieving 3.1 RayPQ and 3.9 RayPQ, respectively. These results highlight foundation-model-driven perception as a practical route to training-free 3D scene understanding.

Andrew Caunes, Thierry Chateau, Vincent Fremont

Logiroad, Nantes, France · LS2N - Ecole Centrale de Nantes, France

3D Occupancy Prediction Autonomous Driving

Ask PDF

Abstract

3D Occupancy Prediction

Explore similar work

An Instance-Centric Panoptic Occupancy Prediction Benchmark for Autonomous Driving

Open-World Panoptic Segmentation

FreeOcc: Training-free Panoptic Occupancy Prediction via Foundation Models