cs.CVAug 25, 2026

Syn2RealTrack: Bridging the Gap Between Synthetic and Real-World Datasets for Online Multi-View Multi-Target Tracking

Authors: Duong Nguyen-Ngoc Tran, Ngoc Doan-Minh Huynh, Cu Quoc Le, Hoang-Khang Nguyen, Long Hoang Pham, Huy-Hung Nguyen, Quoc Pham-Nam Ho, Trinh Le Ba Khanh, 

Organizations: Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, South Korea

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multi-camera 3D perception systems for warehouse scenes are trained largely on synthetic data and evaluated on physically captured environments. The resulting synthetic-to-real gap, which corrupts ground-plane localization and cross-camera identity association, is usually treated as one deficiency for a single domain-adaptation module to absorb; we argue instead that it enters the pipeline at three separable points: the camera calibration, the object shape prior, and the assumption that the object census is known, each admitting a different local remedy. Our online pipeline, Syn2RealTrack, follows this decomposition: lens distortion is recovered from images alone under a calibration that provides none, detections are fused across views by a visibility-weighted part-based descriptor that abstains on occluded parts rather than guessing, person height is measured in closed form from calibration instead of copied from a synthetic prior, and a closed-world cardinality prior is paired with a causal filter that removes the phantom boxes the prior manufactures. The system therefore adapts by reallocating trust between geometry and appearance without retraining a feature extractor. On the AI City Challenge 2026 Track~1 evaluation server it reaches a 3D Higher Order Tracking Accuracy (HOTA) of 52.0118%. The code will be released at https://github.com/SKKUAutoLab/aic26_mc3dp

Figures & tables

Figure 1: Representative point-cloud-guided 3D box refinement examples on (a) a synthetic and (b) a real-world scene of the AI City Challenge 2026 Track 1 dataset [ 31 ] . Red boxes denote the original multi-view tracking prior, green boxes denote the refined 3D boxes, and the colored points denote the local cloud reconstructed by Depth Anything 3 (DA3) [ 15 ] .
Figure 2: Overview of the proposed framework, read left to right. (A) Data preprocessing. AnyCalib [ 35 ] estimates per-camera distortion from 3131 sampled frames, and the Unified Camera Model (UCM) parameter ξ\xi separates reference ( ξ≤0.3\xi\leq 0.3 , blue) and fisheye-candidate (red) views. (B) Per-view perception. RF-DETR [ 25 ] , ViTPose++ [ 46 ] , and Keypoint Promptable Re-Identification (KPR) [ 28 ] provide detections, keypoints, and visibility-aware part descriptors, which an appearance–IoU Kalman tracker links into local identities. (C) Multi-view fusion. Class-specific ground anchors and visibility-weighted part similarity merge observations within per-class bird’s-eye-view (BEV) gates; local-identity carry-forward and gated Hungarian assignment then form global tracks under cap NkN_{k} . (D) 3D box construction and refinement. Trajectory-derived yaw, monocular person height, footprint and yaw refinement guided by Depth Anything 3 (DA3) [ 15 ] , and BEV visibility filtering produce the final boxes. Numbered badges identify the three synthetic-to-real gaps (calibration, shape priors, and closed-world cardinality), and dashed links show cross-stage dependencies between measured height, ground anchoring, and BEV filtering.
Figure 3: Single-view distortion estimation and camera grouping. (1) Frames are evenly sampled from every camera of a scene. (2) AnyCalib [ 35 ] predicts a dense per-pixel ray and field-of-view map and fits a pinhole baseline, a radial Brown–Conrady model, and the Unified Camera Model (UCM), whose parameter ξ\xi is used as a fisheye score; the estimates are used for distortion analysis and grouping only, and the dataset calibration is kept fixed for bird’s-eye-view (BEV) projection. (3) Cameras with ξ\xi below the threshold become reference views (blue), and the rest become fisheye candidates (red), whose reprojection displacements grow toward the image borders.
ParameterSynthetic regimeReal regime
Cameras per scene1010 – 202044 – 77
Intrinsicsideal pinhole, clonedper-camera, fx≠fyf_{x}\neq f_{y}
Person ground anchorbbox ++ height poleankle skeleton
Ankle confidence θank\theta_{\text{ank}}0.50.50.80.8
Skeleton gate nkpn_{\text{kp}}3388
Position smoothing α\alpha1.01.0 (none)0.60.6
Table 1: The synthetic-to-real configuration surface: parameter settings used in the synthetic and real regimes.
Figure 4: Multi-view RGB inputs and DA3-estimated point clouds for (a) the synthetic scene Warehouse 025 and (b) the real-world scene Warehouse 027 at frame 0.
Figure 5: Local fixed-shape object refinement pipeline for the synthetic scene. Starting from a colored DA3 .ply crop around a tracked object, we project the crop to a BEV density grid, keep density-excess cells after floor and ghost filtering, and fit a fixed-size box to the remaining support. The red dashed box is the multi-view tracking prior, and the green box is the refined result.
Figure 6: BEV use of camera grouping. (a) Floor map of reference (blue) and fisheye (red) cameras; dashed violet marks fisheye-only zones, where objects retain their fisheye BEV positions. (b) For a matched object, the reference view anchors the fused target, and each fisheye projection’s error is measured in ground-plane meters.
RankIDName3D HOTA (%)DetA (%)AssA (%)LocA (%)
1289EVA56.544755.644449.392979.5558
234SKKU-AL-T152.011845.305656.504776.2410
3130Playbox38.010540.259231.097875.1778
44QDTers34.184529.466333.112217.6841
5149Calix33.765430.874830.981650.4720
Table 2: Leaderboard of Multi-Camera 3D Perception, AI City Challenge 2026 Track 1, ranked by 3D HOTA (%). Our entry (shaded, bold) ranks second.
Table 9
Table 10
#ConfigurationHOTA (%)DetA (%)AssA (%)LocA (%)Δ\Delta HOTA
1RF-DETR detector only50.807842.411156.259974.1915–
2++ gIoU association51.114543.603155.861775.0459+0.31+0.31
3++ all association cues51.856844.792156.460876.2410+0.74+0.74
4++ cross-view ground-plane clustering51.920545.161156.460176.2053+0.06+0.06
5++ monocular person height estimation51.989845.071256.519076.2397+0.07+0.07
6++ BEV-map prediction filtering52.011845.305656.504776.2410+0.02+0.02
Table 7: Stage-wise ablation of the full pipeline on the Track 1 evaluation server. Each row adds one component to the row above it; the shaded row 6 is the submitted system of Tab. 2 .
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Visualization of the 2D bounding box filtering process. The bounding boxes removed from the training annotations are indicated by red ovals.
Figure 8: Qualitative 2D detection results on scenes Warehouse 023–027 of the AI City Challenge 2026 Track 1 test set.
Figure 9: Qualitative pose-estimation results (ViTPose++ [ 46 ] keypoint overlays) on scenes Warehouse 023–027.
Figure 10: Qualitative 2D single-view tracking results on scenes Warehouse 023–027; each tracked object carries a persistent identity.
Figure 11: Scene-level BEV visualizations constructed from DA3-estimated point clouds for (a) Warehouse 026 and (b) Warehouse 027.
More in this topic

3D MOT

3D multi-object tracking (3D MOT) estimates object locations and identities over time from cameras, LiDAR, radar, and other sensors.

See all 40 papers

Explore similar work

CardsList
Aug 4, 2026

Geometry Beats Estimated Depth: RGB-Only Multi-Camera 3D Tracking under Sim2Real

The AI City Challenge 2026 Track 1 evaluates multi-camera 3D perception in large indoor warehouses under a synthetic-to-real (Sim2Real) setting; depth is available only for training and validation, so inference is RGB-only. We use two RGB-only routes as a controlled test of one hypothesis: that cross-view geometric consistency, not monocular depth accuracy, governs performance under Sim2Real. The first is a geometry-first pipeline: YOLO11x detection, homography lifting to the world frame, class-level 3D size priors, multi-camera fusion, world-coordinate tracking, and offline tracklet stitching. The second is estimated-depth pseudo-LiDAR: monocular depth (D4RT, Metric3D~v2) back-projected into a fused point cloud and passed to a 3D detector (V-DETR), mirroring prior point-cloud winners that used depth. The gap is decisive: geometry-first reaches 13.0 3D HOTA (51.6 LocA), whereas pseudo-LiDAR collapses to 0.12 (9.2 LocA). We trace the collapse to cross-view inconsistency of monocular depth---scale correction is necessary but not sufficient---which domain-adaptation fine-tuning does not repair within budget. Within the geometry pipeline, offline stitching is the only intervention that helps; SAHI detection, appearance Re-ID, learned lifting, RT-DETR ensembling, test-time augmentation, and domain randomization all fail to beat the baseline detector. The bottlenecks are complementary: detection quality bounds the geometry route (DetA), localization consistency bounds pseudo-LiDAR (LocA). We release a complete, reproducible RGB-only pipeline and ablation.

Abdullah Naeem, Anav Katwal, Ayon Dey +2

LSU New Orleans, New Orleans, USA · PinPark, Inc.

Synthetic-to-Real Domain Adaptation3D Point TrackingMulti-View 3D Object Detection3D MOT
AskPDF
Oct 1, 2026

Physical AI Smart Spaces: A Large-Scale Benchmark for Multi-Camera 3D Perception in Smart Spaces

Physical AI Smart Spaces is, to the best of our knowledge, the first benchmark to simultaneously provide large-scale, multi-class, and multi-camera 3D perception data for indoor smart spaces. It contains over 280 hours of synchronized 1080p footage captured by nearly 1,800 cameras in warehouses, hospitals, retail venues, and similar settings, together with automatic annotations for multi-camera identities, 2D bounding boxes, 3D bounding boxes, camera calibration, and depth where available. The benchmark spans Isaac Sim synthetic generation, Cosmos Transfer appearance augmentation, and real-world Sim2Real evaluation. For the real-world target, we include two warehouse deployments with time-synchronized streams, automatic VGGT-based calibration, and a 3D labeling interface that projects world-frame 3D boxes into each view for cross-camera verification. We describe the dataset scope, annotation and calibration schema, generation workflow, benchmark protocols, and official evaluation system, which standardizes submission format, and leaderboard reporting. A central contribution is a 3D instantiation of Higher Order Tracking Accuracy (HOTA), extending the usual 2D box-based tracking evaluation to 3D locations and 3D boxes. We further report empirical baselines from the AI City Challenge leaderboards, showing how methods evolve from person-only 3D location tracking to multi-class 3D box tracking under realistic smart-space constraints. The release is available at https://huggingface.co/datasets/nvidia/PhysicalAI-SmartSpaces.

Yuxing Wang, Yizhou Wang, Anqi Li +13

NVIDIA, Santa Clara, CA, USA · Santa Clara University, Santa Clara, CA, USA · NVIDIA

3D MOT3D Object Detection
AskPDF
Jul 22, 2026

Calibration-Free 3D Multi-Camera People Tracking for Indoor Environment

Multi-Camera People Tracking (MCPT) traditionally relies on precise intrinsic and extrinsic camera calibration to project 2D detections into a unified 3D world coordinate system.However, manual calibration constitutes a major bottleneck in large-scale dataset generation from unconstrained video archives. This work proposes a unified calibration-free 3D MCPT framework that infers geometric structure directly from visual data using deep foundation models. The system integrates anchor-free detection (YOLOX), robust tracking (BoT-SORT), omni-scale appearance embedding (OsNet), pose estimation (HRNet via MMPose), and transformer-based geometric reconstruction using the Visual Geometry Grounded Transformer (VGGT). A pose-guided 3D lifting strategy projects head keypoints onto a reconstructed manifold, eliminating dependence on ground-plane homography. Global identity association is formulated as hierarchical agglomerative clustering under a joint appearance-geometry cost with strict velocity gating. Evaluation on the AI City Challenge 2024 demonstrates a HOTA score of 53.13% without access to ground-truth calibration matrices, establishing a strong baseline for purely vision-based 3D tracking.

Ponleur Veng, Dominique Vaufreydaz, Phutphalla Kong

CADT, M-PSI · Cambodia Academy of Digital Technology · Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France +2

3D MOTMulti-Object Tracking
AskPDF