PanoBench: A Panoptic Occupancy Prediction Benchmark for Autonomous Driving

Yi Feng; Zizhan Guo; Junwu E; Yu Ma; Mengtan Zhang; Jin Wu; Wei Ye; Hanli Wang

cs.CVOct 16, 2026

PanoBench: A Panoptic Occupancy Prediction Benchmark for Autonomous Driving

Authors: Yi Feng, Zizhan Guo, Junwu E, Yu Ma, Mengtan Zhang, Jin Wu, Wei Ye, Hanli Wang,

Abstract

Panoptic occupancy unifies 3D geometry, semantics, and instance identities, but its development is limited by scarce instance-aware annotations. Existing benchmarks mainly provide semantic labels reconstructed from incomplete sensor observations, while panoptic datasets have limited domain coverage or annotation completeness. We introduce PanoBench, a camera-centric panoptic occupancy prediction benchmark spanning synthetic and real-world driving scenes. It combines PanoBench-CarlaOcc, PanoBench-nuScenes, and PanoBench-KITTI360 through canonical data interfaces and unified evaluation protocols, providing multi-resolution annotations with persistent instance identities. The synthetic branch derives complete panoptic geometry from simulated meshes. For real-world data, a new pipeline reconstructs backgrounds using geometric priors and topology constraints, generates foreground mesh assets from multi-frame multi-view images, and fuses them into sparse panoptic voxels. A unified valid-mask protocol evaluates complete thing instances supported by temporal observations while scoring stuff regions at the voxel level. Analyses of observation fidelity, geometric completeness, spatial continuity, and temporal consistency demonstrate the benefits of the resulting annotations over existing occupancy benchmarks. We benchmark representative methods on semantic occupancy prediction, panoptic occupancy prediction, and 4D occupancy forecasting, and study voxel resolution, synthetic-to-real transfer, and storage efficiency. These results reveal complementary strengths of volumetric and ray-based metrics and persistent challenges in real-world instance-aware prediction. PanoBench is available at https://mias.group/PanoBench.

Figures & tables

Fig. 1: Overview of PanoBench, a unified benchmark for camera-centric panoptic occupancy prediction. By incorporating diverse driving datasets from both synthetic and real-world domains, PanoBench provides multi-resolution, high-fidelity, and temporally consistent instance annotations, establishing a unified testbed for semantic occupancy prediction, panoptic occupancy prediction, and 4D occupancy forecasting.

Dataset	Data Source	View	Sequences	Frames	Classes	Voxel Size (m)	Volume Size (m)	Instance	Valid Masks
SemanticKITTI [ 8 ]	Real	Forward	22	43,000	28	0.2	[51.2, 51.2, 6.4]	$\times$	$\times$
SSCBench-KITTI360 [ 11 ]	Real	Forward	9	12,865	19	0.2	[51.2, 51.2, 6.4]	$\times$	$\times$
Occ3D-nuScenes [ 6 ]	Real	Surround	850	34,149	17	0.4	[80.0, 80.0, 6.4]	$\times$	$\checkmark$
SurroundOcc [ 10 ]	Real	Surround	850	34,149	17	0.5	[100.0, 100.0, 8.0]	$\times$	$\times$
CarlaSC [ 29 ]	Synthetic	Surround	24	43,200	11	0.2, 0.4	[51.2, 51.2, 3.0]	$\times$	$\times$
CoHFF [ 28 ]	Synthetic	Surround	73	11,464	12	0.4	[40.0, 40.0, 3.2]	$\times$	$\times$

TABLE I: Comparison of dataset statistics between PanoBench and representative occupancy prediction benchmarks.

Fig. 2: Real-world panoptic occupancy reconstruction pipeline. Static background geometry is recovered by semantic LiDAR aggregation, HD map-guided ground lifting, topology-aware flat region recovery, and geometry-preserving processing of non-planar structures. For foreground thing instances, PanoBench mines high-quality multi-frame multi-view image crops, reconstructs complete mesh assets, selects a diverse prototype pool, and assigns a compatible mesh to each annotated instance. The aligned foreground meshes and scene-level background are voxelized and fused into sparse panoptic occupancy labels, from which temporally aware camera-centric valid masks are generated.

Fig. 3: Comparison of prior valid masks and the proposed valid masks. (a) Previous methods generate a valid mask for each single frame, in which camera rays leave large occluded regions invalid (gray). (b) The proposed temporal valid mask keeps three classes of regions under evaluation: \tiny{$\mathbf{1}$}⃝ a currently occluded but previously observed instance, recovered by aggregating visibility over a window of $T$ frames; \tiny{$\mathbf{2}$}⃝ a partially observed instance, completed as a whole once any of its voxels is visible; and \tiny{$\mathbf{3}$}⃝ stuff behind foreground instances, which remains valid to avoid occlusion-induced fragments. (c)–(e) Qualitative comparison of the raycasting-based mask, the complete ground truth, and the proposed valid mask on PanoBench-CarlaOcc. — Fig. 3: Comparison of prior valid masks and the proposed valid masks. (a) Previous methods generate a valid mask for each single frame, in which camera rays leave large occluded regions invalid (gray). (b) The proposed temporal valid mask keeps three classes of regions under evaluation: \tiny{ $\mathbf{1}$ }⃝ a currently occluded but previously observed instance, recovered by aggregating visibility over a window of $T$ frames; \tiny{ $\mathbf{2}$ }⃝ a partially observed instance, completed as a whole once any of its voxels is visible; and \tiny{ $\mathbf{3}$ }⃝ stuff behind foreground instances, which remains valid to avoid occlusion-induced fragments. (c)–(e) Qualitative comparison of the raycasting-based mask, the complete ground truth, and the proposed valid mask on PanoBench-CarlaOcc.

Domain	Dataset	Voxel Size (m)	$\mathrm{PixAcc}$ (%)
CARLA	PanoBench-CarlaOcc	0.1	91.6
	PanoBench-CarlaOcc	0.2	88.8
	PanoBench-CarlaOcc	0.4	86.4
nuScenes	OpenOccupancy [ 7 ]	0.2	70.0
	PanoBench-nuScenes	0.2	77.2
	Occ3D [ 6 ]	0.4	69.2

TABLE II: Comparison of observation fidelity across occupancy benchmarks.

Fig. 4: Illustration of observation fidelity. Occupancy is ray-cast into the camera view and compared with 2D semantic ground truth.

Fig. 5: Comparison of geometric completeness on nuScenes. (a) BEV views of two scenes. (b) Two foreground instances, each rendered from two opposite viewpoints.

Method	Geometric IoU	Semantic mIoU	Road	Sidewalk	Terrain	Building	Vegetation	Object	Vehicle	Pedestrian
Occ3D-style [ 6 ]	38.3	32.6	66.7	60.7	29.8	18.2	27.7	40.5	5.9	11.4
SurroundOcc-style [ 10 ]	37.5	30.5	68.4	56.9	28.2	18.9	27.2	26.8	7.0	10.9
PanoBench (Ours)	58.9	52.0	82.8	82.3	54.8	18.4	27.6	45.3	59.1	46.0

TABLE III: Controlled oracle evaluation of occupancy annotation pipelines on PanoBench-CarlaOcc.

Fig. 6: Comparison of occupancy annotation pipelines on PanoBench-CarlaOcc ( Town10HD ). The ground truth is voxelized from complete scene meshes, and Occ3D and SurroundOcc denote their style-faithful reimplementations.

Domain	Dataset	Voxel Size (m)	$r_{\mathrm{fp}}@0.1$	$r_{\mathrm{fp}}@0.2$
CARLA	CarlaSC [ 29 ]	0.2	9.4	15.1
	PanoBench-CarlaOcc	0.2	2.5	5.6
	CarlaSC [ 29 ]	0.4	5.3	11.8
	PanoBench-CarlaOcc	0.4	0.3	1.1
nuScenes	OpenOccupancy [ 7 ]	0.2	23.2	31.0
	PanoBench-nuScenes	0.2	11.2	17.4

TABLE IV: Comparison of spatial continuity across occupancy benchmarks. Floating-voxel ratios are in permille (lower is better), and bold denotes the better result at the same voxel size.

Domain	Dataset	Voxel Size (m)	mIoU	IoU
CARLA	PanoBench-CarlaOcc	0.1	81.9	83.3
	CarlaSC [ 29 ]	0.2	70.3	83.5
	PanoBench-CarlaOcc	0.2	82.8	89.1
	CarlaSC [ 29 ]	0.4	71.5	86.8
	PanoBench-CarlaOcc	0.4	83.6	92.3
nuScenes	OpenOccupancy [ 7 ]	0.2	60.9	68.2

TABLE V: Comparison of temporal consistency across occupancy benchmarks. Bold denotes the better result at the same voxel size.

Fig. 7: Comparison of instance temporal consistency on nuScenes: (a) Occ3D-Panoptic [ 15 ] and (b) PanoBench-nuScenes. Voxels with the same instance identity are rendered in the same color across frames, and red and blue circles highlight two representative instances.

Background Component	Configuration
Semantic LiDAR Aggregation	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Ground-Layer Lifting		$\checkmark$	$\checkmark$	$\checkmark$
Flat-Region Recovery			$\checkmark$	$\checkmark$
Non-Planar Refinement				$\checkmark$
$\mathrm{PixAcc}$ (%)	68.5	74.1	79.0	79.4

TABLE VI: Ablation study on background reconstruction components on PanoBench-nuScenes. Observation fidelity is evaluated on stuff pixels.

Method	Input Views	View Selection	IoU
LiDAR Aggregation [ 6 ]	–	–	12.4
SAM 3D [ 60 ]	Single-View	$\checkmark$	61.2
TRELLIS [ 58 ]	Single-View	$\checkmark$	57.0
TRELLIS-2 [ 59 ]	Single-View	$\checkmark$	60.5
MV-SAM3D [ 63 ]	Multi-View	$\times$	65.1
MV-SAM3D [ 63 ]	Multi-View	$\checkmark$	67.2

TABLE VII: Ablation study on foreground mesh asset acquisition on PanoBench-CarlaOcc, measured by instance IoU (%).

Fig. 8: Representative prototype meshes from the three PanoBench subsets. Rows show (a) cars, (b) two-wheelers, (c) pedestrians, and (d) other vehicles. The examples illustrate the intra-class shape diversity retained by the prototype mesh pools across synthetic and real-world domains.

Method	PanoBench-CarlaOcc					PanoBench-nuScenes					PanoBench-KITTI360
Method	IoU	mIoU	RayIoU@1	RayIoU@2	RayIoU@4	IoU	mIoU	RayIoU@1	RayIoU@2	RayIoU@4	IoU	mIoU	RayIoU@1	RayIoU@2	RayIoU@4
SparseOcc [ 15 ]	68.5	41.7	50.0	56.6	60.1	55.0	33.0	33.7	41.7	46.5	33.3	11.2	28.1	32.0	33.4
OPUS [ 39 ]	73.5	41.3	50.3	56.9	59.3	50.1	29.9	32.6	41.4	46.2	30.7	9.7	18.8	23.9	26.6
Panoptic-FlashOcc [ 16 ]	73.8	40.9	51.7	55.9	57.7	47.7	22.6	25.2	34.9	41.1	41.7	11.5	18.8	25.0	27.6
GaussianFormer [ 37 ]	82.2	46.2	56.4	61.0	62.8	66.5	32.6	32.8	41.8	46.7	65.3	15.6	21.1	25.5	27.1

TABLE VIII: Comparison of semantic occupancy prediction methods on PanoBench. Default voxel sizes are used, i.e. , 0.4 m for PanoBench-CarlaOcc and PanoBench-nuScenes and 0.2 m for PanoBench-KITTI360.

Method	PanoBench-CarlaOcc					PanoBench-nuScenes					PanoBench-KITTI360
Method	IoU	mIoU	RayPQ@1	RayPQ@2	RayPQ@4	IoU	mIoU	RayPQ@1	RayPQ@2	RayPQ@4	IoU	mIoU	RayPQ@1	RayPQ@2	RayPQ@4
SparseOcc [ 15 ]	68.5	41.7	36.5	41.7	44.1	55.0	33.0	11.2	16.5	20.1	33.3	11.2	12.7	16.0	17.4
Panoptic-FlashOcc [ 16 ]	73.8	40.9	30.6	34.8	36.8	47.7	22.6	3.2	7.0	10.8	41.7	11.5	7.2	10.8	12.6

TABLE IX: Comparison of panoptic occupancy prediction methods on PanoBench under the default voxel sizes in Table VIII .

Method	PanoBench-CarlaOcc								PanoBench-nuScenes								PanoBench-KITTI360
	Semantic mIoU				Geometric IoU				Semantic mIoU				Geometric IoU				Semantic mIoU				Geometric IoU
	1 s	2 s	3 s	Avg.	1 s	2 s	3 s	Avg.	1 s	2 s	3 s	Avg.	1 s	2 s	3 s	Avg.	1 s	2 s	3 s	Avg.	1 s	2 s	3 s	Avg.
SparseWorld [ 47 ]	33.4	28.6	24.6	28.9	68.4	63.9	59.5	63.9	25.3	23.1	21.0	23.1	49.2	47.6	45.7	47.5	6.2	4.7	3.6	4.9	24.4	20.8	17.5	20.9
SparseWorld-TC [ 48 ]	34.5	29.8	25.7	30.0	69.3	64.8	60.4	64.8	24.9	22.7	20.8	22.8	49.2	47.6	45.9	47.6	7.7	6.1	4.8	6.2	28.6	24.8	21.1	24.8

TABLE X: Comparison of 4D occupancy forecasting methods on PanoBench under the default voxel sizes in Table VIII .

Method	Voxel Size (m)	IoU	mIoU
OPUS [ 39 ]	0.1	38.5	11.7
	0.2	61.6	31.6
	0.4	73.5	41.3
Panoptic-FlashOcc [ 16 ]	0.1	38.5	11.8
	0.2	62.8	32.1
	0.4	73.8	40.9

TABLE XI: Multi-resolution occupancy prediction on PanoBench-CarlaOcc.

Target	Method	Pretraining	mIoU	RayIoU	RayPQ
nuScenes	SparseOcc [ 15 ]	–	33.0	40.6	15.9
	SparseOcc [ 15 ]	CarlaOcc	34.2	42.3	17.4
	Panoptic-FlashOcc [ 16 ]	–	22.6	33.7	7.0
	Panoptic-FlashOcc [ 16 ]	CarlaOcc	26.4	35.6	9.6
KITTI-360	SparseOcc [ 15 ]	–	11.2	31.2	15.4
	SparseOcc [ 15 ]	CarlaOcc	11.8	31.9	15.2

TABLE XII: Sim-to-real transfer results for panoptic occupancy prediction. RayIoU and RayPQ are averaged over depth tolerances of 1, 2, and 4 m.

Dataset	Storage (GiB)			Latency (ms)
Dataset	Official	Ours	$\downarrow$ (%)	Official	Ours	$\downarrow$ (%)
Occ3D-nuScenes [ 6 ]	2.6	1.3	48.7	11.1	1.6	85.8
Occ3D-Panoptic [ 15 ]	3.2	1.3	58.1	20.6	1.5	92.8
OpenOccupancy [ 7 ]	132.7	5.5	95.9	40.2	6.8	83.2
SurroundOcc [ 10 ]	40.3	1.2	97.1	13.5	1.9	85.8
SemanticKITTI [ 8 ]	90.6	1.2	98.7	36.3	2.1	94.1

TABLE XIII: Overall storage footprint and per-frame I/O latency of the official occupancy annotations versus our canonical sparse panoptic format. Latency is averaged over 1,000 randomly sampled frames per dataset on the same hardware.

3D Occupancy Prediction

3D occupancy prediction estimates the geometry and semantics of occupied and free space around an agent from camera, LiDAR, or radar data.

See all 113 papers

An Instance-Centric Panoptic Occupancy Prediction Benchmark for Autonomous Driving
Mar 28, 2026Yi Feng, Junwu E, Zizhan Guo +33D Occupancy Prediction Autonomous Driving Benchmarks
Tongji University
Open-World Panoptic Segmentation
Dec 17, 2024Matteo Sodano, Federico Magistri, Jens Behley +1Image Segmentation Panoptic Segmentation
Center for Robotics, University of Bonn, Germany, and the Lamarr Institute for Machine Learning and Artificial Intelligence, Germany
FreeOcc: Training-free Panoptic Occupancy Prediction via Foundation Models
Mar 6, 2026Andrew Caunes, Thierry Chateau, Vincent Fremont3D Occupancy Prediction Autonomous Driving
Logiroad, Nantes, France · LS2N - Ecole Centrale de Nantes, France

Mar 28, 2026J/K move · Enter open · S save

An Instance-Centric Panoptic Occupancy Prediction Benchmark for Autonomous Driving

Yi Feng, Junwu E, Zizhan Guo, Yu Ma+2

Tongji University

3D Occupancy Prediction Autonomous Driving Benchmarks Autonomous Driving Datasets

Open paper AskPDF arXiv

Panoptic occupancy prediction aims to jointly infer voxel-wise semantics and instance identities within a unified 3D scene representation. Nevertheless, progress in this field remains constrained by the absence of high-quality 3D mesh resources, instance-level annotations, and physically consistent occupancy datasets. Existing benchmarks typically provide incomplete and low-resolution geometry without instance-level annotations, limiting the development of models capable of achieving precise geometric reconstruction, reliable occlusion reasoning, and holistic 3D understanding. To address these challenges, this paper presents an instance-centric benchmark for the 3D panoptic occupancy prediction task. Specifically, we introduce ADMesh, the first unified 3D mesh library tailored for autonomous driving, which integrates over 15K high-quality 3D models with diverse textures and rich semantic annotations. Building upon ADMesh, we further construct CarlaOcc, a large-scale, physically consistent panoptic occupancy dataset generated using the CARLA simulator. This dataset contains over 100K frames with fine-grained, instance-level occupancy ground truth at voxel resolutions as fine as 0.05 m. Furthermore, standardized evaluation metrics are introduced to quantify the quality of existing occupancy datasets. Finally, a systematic benchmark of representative models is established on the proposed dataset, which provides a unified platform for fair comparison and reproducible research in the field of 3D panoptic perception. Code and dataset are available at https://mias.group/CarlaOcc.

Abstract

3D Occupancy Prediction

Explore similar work

An Instance-Centric Panoptic Occupancy Prediction Benchmark for Autonomous Driving

Open-World Panoptic Segmentation

FreeOcc: Training-free Panoptic Occupancy Prediction via Foundation Models