An Instance-Centric Panoptic Occupancy Prediction Benchmark for Autonomous Driving

Yi Feng; Junwu E; Zizhan Guo; Yu Ma; Hanli Wang; Rui Fan

cs.CVMar 28, 2026

An Instance-Centric Panoptic Occupancy Prediction Benchmark for Autonomous Driving

Authors: Yi Feng, Junwu E, Zizhan Guo, Yu Ma, Hanli Wang, Rui Fan

Organizations: Tongji University

Abstract

Panoptic occupancy prediction aims to jointly infer voxel-wise semantics and instance identities within a unified 3D scene representation. Nevertheless, progress in this field remains constrained by the absence of high-quality 3D mesh resources, instance-level annotations, and physically consistent occupancy datasets. Existing benchmarks typically provide incomplete and low-resolution geometry without instance-level annotations, limiting the development of models capable of achieving precise geometric reconstruction, reliable occlusion reasoning, and holistic 3D understanding. To address these challenges, this paper presents an instance-centric benchmark for the 3D panoptic occupancy prediction task. Specifically, we introduce ADMesh, the first unified 3D mesh library tailored for autonomous driving, which integrates over 15K high-quality 3D models with diverse textures and rich semantic annotations. Building upon ADMesh, we further construct CarlaOcc, a large-scale, physically consistent panoptic occupancy dataset generated using the CARLA simulator. This dataset contains over 100K frames with fine-grained, instance-level occupancy ground truth at voxel resolutions as fine as 0.05 m. Furthermore, standardized evaluation metrics are introduced to quantify the quality of existing occupancy datasets. Finally, a systematic benchmark of representative models is established on the proposed dataset, which provides a unified platform for fair comparison and reproducible research in the field of 3D panoptic perception. Code and dataset are available at https://mias.group/CarlaOcc.

Figures & tables

Source	Instances	Main Categories
CARLA [ 9 ]	$\sim$ 8,000	30 Standardized Categories
BuildingNet [ 43 ]	$\sim$ 2,000	Building
MeshFleet [ 3 ]	$\sim$ 3,000	Car, Truck, Bus
ShapeNetCore [ 6 ]	$\sim$ 2,000	Car, Train, Bus, Motorcycle
Total	15,000+	30 Standardized Categories

Table 1: Overview of data sources and statistics of ADMesh.

Figure 2: Overview of the proposed ADMesh library and the CarlaOcc generation pipeline. The ADMesh library is constructed by extracting and organizing diverse 3D assets from multiple sources, which are subsequently used to reconstruct dynamic scenes with both static structures and temporally aligned non-rigid motions. The resulting unified scene meshes are then used to rectify sensor artifacts and further processed with a topology-aware mesh permutation strategy to produce non-overlapping panoptic occupancy labels.

Dataset	Synthetic	Surround-View	Classes	Frames	Volume Range (m)	Voxel Size (m)	Instance Annotations
SemanticKITTI [ 1 ]	$\times$	$\times$	28	43,000	[51.2, 51.2, 6.4]	0.2	$\times$
KITTI-360-SSCBench [ 30 ]	$\times$	$\times$	19	12,865	[51.2, 51.2, 6.4]	0.2	$\times$
Occ3D-nuScenes [ 46 ]	$\times$	$\checkmark$	17	34,149	[80, 80, 6.4]	0.4	$\times$
SurroundOcc [ 51 ]	$\times$	$\checkmark$	17	34,149	[100, 100, 8]	0.5	$\times$
CoHFF [ 44 ]	$\checkmark$	$\checkmark$	12	11,464	[40, 40, 3.2]	0.4	$\times$
CarlaSC [ 52 ]	$\checkmark$	$\checkmark$	11	43,200	[51.2, 51.2, 3]	0.4	$\times$

Table 2: Comparison between CarlaOcc and other public occupancy prediction datasets.

Figure 3: Statistics of ADMesh and CarlaOcc: (a) semantic mesh distribution in ADMesh, (b) semantic voxel distribution in CarlaOcc, and (c) instance count distribution in CarlaOcc.

Figure 4: Visualization of data modalities in CarlaOcc.

Figure 5: Visualizations of instance-guided rectification of sensor artifacts: (a) RGB images; (b) and (d) the rendered depth and semantic maps from CARLA; (c) and (e) the corresponding refined results. Regions with incorrect semantics or depth values caused by transparency are highlighted in red, while the instances that lose their categorical labels are highlighted in blue.

Dataset	Synthetic	$s_{sc}$	$s_{tc}$
SemanticKITTI [ 1 ]	$\times$	0.353	0.023
KITTI-360-SSCBench [ 30 ]	$\times$	0.658	0.402
Occ3D-nuScenes [ 46 ]	$\times$	0.721	0.431
SurroundOcc [ 51 ]	$\times$	0.878	0.589
CarlaSC [ 52 ]	$\checkmark$	0.887	0.775
CarlaOcc (Ours)	$\checkmark$	0.996	0.873

Table 3: Comparison between CarlaOcc and other public occupancy prediction datasets on occupancy quality metrics.

Method	PQ $\uparrow$	SQ $\uparrow$	RQ $\uparrow$	mIoU $\uparrow$
SparseOcc [ 33 ]	10.3	48.8	21.1	16.5
Panoptic-FlashOcc [ 57 ]	13.5	49.1	27.5	18.4

Table 4: Comparison of panoptic occupancy prediction methods on the CarlaOcc dataset.

View	Method	IoU $\uparrow$	mIoU $\uparrow$
Surround	GaussianFormer [ 22 ]	35.8	19.2
	GaussianFormer2 [ 23 ]	38.3	20.7
	OPUS [ 47 ]	37.0	19.3
Single	VoxFormer [ 29 ]	28.5	13.9
	OccFormer [ 60 ]	30.1	14.4
	COTR [ 35 ]	31.1	15.4

Table 5: Comparison of semantic occupancy prediction methods on the CarlaOcc dataset.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Top-down illustration of the sensor configuration in the CarlaOcc dataset.

Figure 7: Visualization of occupancy ground truth at different voxel resolutions: (a) RGB images, (b) panoptic occupancy labels at 0.5 m voxel size, (c) 0.1 m voxel size, and (d) 0.05 m voxel size.

Figure 8: Visualization of multi-modal data in CarlaOcc: (a) RGB images with rich annotations, (b) depth maps, (c) surface normal maps, (d) semantic segmentation maps, (e) panoptic meshes shown in wireframe, and (f) panoptic occupancy ground truth. “Stuff” classes are visualized using the official color map, while “thing” instances are displayed in randomly assigned colors.

Figure 9: Ground truth quality comparison between existing public occupancy datasets and CarlaOcc: (a) KITTI-360-SSCBench [ 30 ] , (b) Occ3D-nuScenes [ 46 ] , (c) CarlaSC [ 52 ] , and (d) our proposed CarlaOcc dataset. For each dataset, we present the (I) forward view and the (II) left-side view for better comparison.

Figure 10: Visualization of the reconstructed CARLA maps. Left: unified static scene meshes obtained through our reconstruction pipeline. Right: corresponding RGB images captured in Unreal Engine 5.

Figure 11: Parametric scene generation and editing on ADMesh. Each row shows a dynamic sequence rendered at four successive time steps. (a) Original scene generated by assigning trajectories and poses to the static mesh actors. (b) Edited scene obtained by replacing the mesh asset of each actor while keeping all trajectories, camera parameters, and illumination fixed. The example illustrates that our parametric representation can generate diverse, high-quality dynamic urban scenes with simple actor-level edits.

Method	Voxel size (m)	IoU	mIoU
Symphonies [ 24 ]	0.05	15.2	10.1
	0.10	21.5	12.1
	0.50	32.6	15.9
GaussianFormer2 [ 23 ]	0.05	18.4	10.9
	0.10	23.1	14.8
	0.50	38.3	20.7

Appendix

Table 6: Ablation study on voxel size for semantic occupancy prediction on the CarlaOcc dataset.

Pretrain	Target	Method	Scratch mIoU	mIoU
CarlaOcc	KITTI-360	Symphonies	15.9	17.4
CarlaOcc	nuScenes	SparseOcc	16.5	17.3

Appendix

Table 7: Sim-to-real evaluation results of representative baselines pretrained on CarlaOcc and finetuned on real-world datasets.

Method	IoU	mIoU	$\blacksquare$ Road	$\blacksquare$ Sidewalk	$\blacksquare$ Building	$\blacksquare$ Wall	$\blacksquare$ Fence	$\blacksquare$ Pole	$\blacksquare$ TrafficLight	$\blacksquare$ TrafficSign	$\blacksquare$ Vegetation	$\blacksquare$ Ground	$\blacksquare$ Person	$\blacksquare$ Car	$\blacksquare$ Truck	$\blacksquare$ OtherVehicle	$\blacksquare$ Other
Symphonies [ 24 ]	32.6	15.9	36.74	20.04	21.38	6.68	10.02	5.34	6.68	6.01	26.72	16.70	5.34	30.06	23.38	16.70	6.71
SparseOcc [ 33 ]	30.1	14.4	33.28	18.15	19.36	6.05	9.08	4.84	6.05	5.45	24.20	15.13	4.84	27.23	21.18	15.13	6.03
OPUS [ 47 ]	37.0	19.3	44.60	24.33	25.95	8.11	12.16	6.49	8.11	7.30	32.44	20.27	6.49	36.49	28.38	20.27	8.11
GaussianFormer2 [ 23 ]	38.3	20.7	47.84	26.09	27.83	8.70	13.05	6.96	8.70	7.83	34.79	21.74	6.96	39.14	30.44	21.74	8.69

Appendix

Table 8: Semantic occupancy prediction results on the CarlaOcc dataset.

Category	Method	AbsRel $\downarrow$	SqRel $\downarrow$	RMSE $\downarrow$	RMSElog $\downarrow$	$\delta<1.25$ $\uparrow$	$\delta<1.25^{2}$ $\uparrow$	$\delta<1.25^{3}$ $\uparrow$
Discriminative	MiDaS [ 2 ]	0.128	1.642	7.215	0.335	0.842	0.948	0.977
	LeReS [ 55 ]	0.112	1.392	6.932	0.314	0.868	0.957	0.980
	DPT [ 41 ]	0.101	1.248	6.621	0.302	0.887	0.964	0.982
	DepthAnythingV2 [ 54 ]	0.093	1.035	6.159	0.297	0.899	0.965	0.983
Generative	Marigold [ 26 ]	0.089	1.012	6.045	0.284	0.904	0.968	0.985
Generative	LOTUS [ 19 ]	0.085	0.985	5.918	0.276	0.911	0.972	0.987

Appendix

Table 9: Depth estimation results on the CarlaOcc dataset.

Figure 12: Visualizations of sim-to-real experiment. As shown in yellow circles, SparseOcc exhibits more complete instance contours after pretraining on CarlaOcc compared to official results.

Method	mAP $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$
DETR3D [ 49 ]	0.39	0.63	0.26	0.37
PETR [ 34 ]	0.45	0.57	0.27	0.39
Sparse4D [ 32 ]	0.48	0.66	0.24	0.33

Appendix

Table 10: Representative 3D object detection baselines on the CarlaOcc dataset.

Explore similar work

ProOOD: Prototype-Guided Out-of-Distribution 3D Occupancy Prediction
Apr 1, 2026Yuheng Zhang, Mengfei Duan, Kunyu Peng +53D Occupancy Prediction OOD Detection
Hunan University · Karlsruhe Institute of Technology · INSAIT, Sofia University “St. Kliment Ohridski”
MobileOcc: A Human-Aware Semantic Occupancy Dataset for Mobile Robots
Nov 21, 2025Junseo Kim, Guido Dumont, Xinyu Gao +3Mobile Robotics Semantic Occupancy Mapping
Delft University of Technology, Mekelweg 5, 2628 CD Delft, Netherlands
$M^2$ -Occ: Resilient 3D Semantic Occupancy Prediction for Autonomous Driving with Incomplete Camera Inputs
Mar 10, 2026Kaixin Lin, Kunyu Peng, Di Wen +3Multi-View Learning Autonomous Driving Perception
School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China · Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, Germany

Apr 1, 2026J/K move · Enter open · S save

ProOOD: Prototype-Guided Out-of-Distribution 3D Occupancy Prediction

Yuheng Zhang, Mengfei Duan, Kunyu Peng, Yuhang Wang+4

Hunan University · Karlsruhe Institute of Technology · INSAIT, Sofia University “St. Kliment Ohridski”

3D Occupancy Prediction OOD Detection Prototype Learning

Open paper AskPDF arXiv

3D semantic occupancy prediction is central to autonomous driving, yet current methods are vulnerable to long-tailed class bias and out-of-distribution (OOD) inputs, often overconfidently assigning anomalies to rare classes. We present ProOOD, a lightweight, plug-and-play method that couples prototype-guided refinement with training-free OOD scoring. ProOOD comprises (i) prototype-guided semantic imputation that fills occluded regions with class-consistent features, (ii) prototype-guided tail mining that strengthens rare-class representations to curb OOD absorption, and (iii) EchoOOD, which fuses local logit coherence with local and global prototype matching to produce reliable voxel-level OOD scores. Extensive experiments on five datasets demonstrate that ProOOD achieves state-of-the-art performance on both in-distribution 3D occupancy prediction and OOD detection. On SemanticKITTI, it surpasses baselines by +3.57% mIoU overall and +24.80% tail-class mIoU; on VAA-KITTI, it improves AuPRCr by +19.34 points, with consistent gains across benchmarks. These improvements yield more calibrated occupancy estimates and more reliable OOD detection in safety-critical urban driving. The source code is publicly available at https://github.com/7uHeng/ProOOD.

Abstract

Explore similar work

ProOOD: Prototype-Guided Out-of-Distribution 3D Occupancy Prediction

MobileOcc: A Human-Aware Semantic Occupancy Dataset for Mobile Robots

M2M^2M2-Occ: Resilient 3D Semantic Occupancy Prediction for Autonomous Driving with Incomplete Camera Inputs

$M^2$ -Occ: Resilient 3D Semantic Occupancy Prediction for Autonomous Driving with Incomplete Camera Inputs