Physical AI Smart Spaces: A Large-Scale Benchmark for Multi-Camera 3D Perception in Smart Spaces

Yuxing Wang; Yizhou Wang; Anqi Li; Shuo Wang; Sameer Satish Pusegaonkar; Haoquan Liang; Jiajun Li; Shenxin Jiang

cs.CVOct 1, 2026

Physical AI Smart Spaces: A Large-Scale Benchmark for Multi-Camera 3D Perception in Smart Spaces

Authors: Yuxing Wang, Yizhou Wang, Anqi Li, Shuo Wang, Sameer Satish Pusegaonkar, Haoquan Liang, Jiajun Li, Shenxin Jiang,

Organizations: NVIDIA, Santa Clara, CA, USA · Santa Clara University, Santa Clara, CA, USA · NVIDIA

Abstract

Physical AI Smart Spaces is, to the best of our knowledge, the first benchmark to simultaneously provide large-scale, multi-class, and multi-camera 3D perception data for indoor smart spaces. It contains over 280 hours of synchronized 1080p footage captured by nearly 1,800 cameras in warehouses, hospitals, retail venues, and similar settings, together with automatic annotations for multi-camera identities, 2D bounding boxes, 3D bounding boxes, camera calibration, and depth where available. The benchmark spans Isaac Sim synthetic generation, Cosmos Transfer appearance augmentation, and real-world Sim2Real evaluation. For the real-world target, we include two warehouse deployments with time-synchronized streams, automatic VGGT-based calibration, and a 3D labeling interface that projects world-frame 3D boxes into each view for cross-camera verification. We describe the dataset scope, annotation and calibration schema, generation workflow, benchmark protocols, and official evaluation system, which standardizes submission format, and leaderboard reporting. A central contribution is a 3D instantiation of Higher Order Tracking Accuracy (HOTA), extending the usual 2D box-based tracking evaluation to 3D locations and 3D boxes. We further report empirical baselines from the AI City Challenge leaderboards, showing how methods evolve from person-only 3D location tracking to multi-class 3D box tracking under realistic smart-space constraints. The release is available at https://huggingface.co/datasets/nvidia/PhysicalAI-SmartSpaces.

Figures & tables

Figure 1: Example synchronized views and top-down scene context from Physical AI Smart Spaces (PAISS). The dataset provides multi-camera RGB streams together with geometry-aware annotations for 3D perception.

Dataset	Year	Frames	Cameras	IDs	Topology	Domain	3D boxes	Depth	Resolution
PETS2009 ( Ferryman and Shahrokni, 2009 )	2009	1.2K	8	30	OV	real outdoor	N/A	no	768 $\times$ 576
USC Campus ( Javed et al., 2008 )	2010	135K	3	146	NOV	real outdoor	N/A	no	852 $\times$ 480
Passageway ( Berclaz et al., 2011 )	2011	120K	4	4	OV	real outdoor	N/A	no	320 $\times$ 240
NLPR_MCT ( NLPR-MCT Challenge Organizers, 2014 )	2015	355.5K	$\leq$ 5	$\leq$ 235	NOV	real indoor/outdoor	N/A	no	320 $\times$ 240
CamNet ( Zhang et al., 2015 )	2015	360K	8	50	NOV	real indoor/outdoor	N/A	no	640 $\times$ 480
DukeMTMC ( Ristani et al., 2016 )	2016	2.45M	8	2,834	NOV	real outdoor	N/A	no	1920 $\times$ 1080

Table 1: Comparison of PAISS with representative public multi-camera benchmarks. OV/NOV indicates overlapping and non-overlapping camera views.

Component	Scenes	Hours	Cameras	Objects	Object classes	3D labels	2D boxes	Depth
PAISS 2024	90	212.0	953	2,481	P	52.0M	135.0M	no
PAISS 2025	23	42.0	504	363	P, FL, NC, TP, FG, AD	8.9M	73.0M	yes
PAISS 2026	26	28.5	342	1,379	P, FL, PT, NC, TP, FG, AD	11.1M	33.2M	yes
Full release	139	282.5	1,799	4,223	7 classes	72.0M+	241.2M	partial

Table 2: Scale of the Physical AI Smart Spaces release. The dataset provides synchronized 1080p video at 30 FPS, camera calibration, object identities, and dense 2D/3D supervision. Object class abbreviations: P = Person, FL = Forklift, PT = PalletTruck, TP = Transporter, FG = FourierGR1T2, AD = AgilityDigit, and NC = NovaCarter.

Figure 2: The PAISS pipeline. Synthetic Generation : Isaac Sim composes warehouse, hospital, and retail layouts with object assets and renders synchronized RGB/depth streams together with automatic 2D and 3D bounding boxes and track IDs. Synthetic Augmentation : Cosmos Transfer 2.5 takes the rendered RGB, depth-derived edges, and a text prompt to produce paired appearance-shifted RGB while preserving geometry, cameras, depth, and labels. Real-World Target : multi-camera streams are captured under time synchronization, calibrated through a VGGT-based pipeline that jointly estimates camera poses, 3D point clouds, and the ground plane into a standardized world coordinate system, and annotated via a multi-camera 3D labeling UI with cross-view box-projection verification.

Figure 3: Visual domains in the benchmark. Isaac Sim data provides controllable synthetic scenes with dense labels; Cosmos Transfer changes appearance while preserving labels and calibration; Real-world scenes define the hidden RGB-only Sim2Real target.

Figure 4: Top-down visualizations of PAISS camera networks across three scenes. Each colored cone depicts the field of view of one camera, with labels showing per-camera identifiers within the scene. To keep individual FOVs legible, the cameras shown here are downsampled from the full network in each scene; the released datasets contain a substantially denser deployment, with the full per-scene camera counts.

Team	Team ID	Representative method paper	HOTA	Online
Met_BEV	331	MCBLT ( Wang et al., 2025b )	81.2182	no
Yachiyo	221	overlap suppression clustering ( Yoshida et al., 2024 )	71.9446	no
SJTU-Lenovo	79	geometric consistency and state-aware ReID ( Xie et al., 2024 )	67.7258	yes
Nota	40	cluster self-refinement ( Kim et al., 2024 )	62.1666	yes
Fraunhofer IOSB	142	OCMCTrack ( Specker, 2024 )	60.8792	yes
UW-ETRI	8	online long-sequence tracking ( Yang et al., 2024 )	57.1445	yes

Table 3: AI City Challenge 2024 Track 1 leaderboard for MTMC people tracking. Scores are reported as HOTA in the challenge paper ( Wang et al., 2024 ) .

Team	Team ID	Representative method paper	HOTA	Online	Input Modalities
ZV	65	3D point clouds and ReID ( Lee et al., 2025 )	69.9118	no	RGB and Depth
Metropolis-Sparse4D	220	A unified 3D detection and tracking network ( Wang et al., 2026 )	67.3143	yes	RGB
SKKU-AutoLab	15	DepthTrack ( Tran et al., 2025 )	63.1396	no	RGB and Depth
TeamQDT	133	Robust 2D tracking with late depth aggregation ( Vu-Minh et al., 2025 )	31.6267	yes	RGB and Depth
UTE AI Lab	116	VGCRTrack ( Phan et al., 2025 )	28.0345	yes	RGB and Depth

Table 4: AI City Challenge 2025 Track 1 leaderboard for multi-class 3D multi-camera tracking. Scores are reported as raw HOTA in the challenge paper ( Tang et al., 2025 ) .

3D MOT

3D multi-object tracking (3D MOT) estimates object locations and identities over time from cameras, LiDAR, radar, and other sensors.

See all 40 papers

Geometry Beats Estimated Depth: RGB-Only Multi-Camera 3D Tracking under Sim2Real
Aug 4, 2026Abdullah Naeem, Anav Katwal, Ayon Dey +2Synthetic-to-Real Domain Adaptation 3D Point Tracking
LSU New Orleans, New Orleans, USA · PinPark, Inc.
Syn2RealTrack: Bridging the Gap Between Synthetic and Real-World Datasets for Online Multi-View Multi-Target Tracking
Aug 25, 2026Duong Nguyen-Ngoc Tran, Ngoc Doan-Minh Huynh, Cu Quoc Le +103D MOT Multi-Object Tracking
Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, South Korea
HumanPhys: A Human-Centric Physical-World Perception Benchmark for Real-World Visual Reasoning
Oct 10, 2026Yiming Zhang, Enzhi Zhou, Junan Lin +4Human Motion Understanding VLM Reasoning
Nanyang Technological University, Singapore · Ant Group, Hangzhou, China

Aug 4, 2026J/K move · Enter open · S save

Geometry Beats Estimated Depth: RGB-Only Multi-Camera 3D Tracking under Sim2Real

Abdullah Naeem, Anav Katwal, Ayon Dey, Noman Khan+1

LSU New Orleans, New Orleans, USA · PinPark, Inc.

Synthetic-to-Real Domain Adaptation 3D Point Tracking Multi-View 3D Object Detection 3D MOT

Open paper AskPDF arXiv

The AI City Challenge 2026 Track 1 evaluates multi-camera 3D perception in large indoor warehouses under a synthetic-to-real (Sim2Real) setting; depth is available only for training and validation, so inference is RGB-only. We use two RGB-only routes as a controlled test of one hypothesis: that cross-view geometric consistency, not monocular depth accuracy, governs performance under Sim2Real. The first is a geometry-first pipeline: YOLO11x detection, homography lifting to the world frame, class-level 3D size priors, multi-camera fusion, world-coordinate tracking, and offline tracklet stitching. The second is estimated-depth pseudo-LiDAR: monocular depth (D4RT, Metric3D~v2) back-projected into a fused point cloud and passed to a 3D detector (V-DETR), mirroring prior point-cloud winners that used depth. The gap is decisive: geometry-first reaches 13.0 3D HOTA (51.6 LocA), whereas pseudo-LiDAR collapses to 0.12 (9.2 LocA). We trace the collapse to cross-view inconsistency of monocular depth---scale correction is necessary but not sufficient---which domain-adaptation fine-tuning does not repair within budget. Within the geometry pipeline, offline stitching is the only intervention that helps; SAHI detection, appearance Re-ID, learned lifting, RT-DETR ensembling, test-time augmentation, and domain randomization all fail to beat the baseline detector. The bottlenecks are complementary: detection quality bounds the geometry route (DetA), localization consistency bounds pseudo-LiDAR (LocA). We release a complete, reproducible RGB-only pipeline and ablation.

Abstract

3D MOT

Explore similar work

Geometry Beats Estimated Depth: RGB-Only Multi-Camera 3D Tracking under Sim2Real

Syn2RealTrack: Bridging the Gap Between Synthetic and Real-World Datasets for Online Multi-View Multi-Target Tracking

HumanPhys: A Human-Centric Physical-World Perception Benchmark for Real-World Visual Reasoning