Syn2RealTrack: Bridging the Gap Between Synthetic and Real-World Datasets for Online Multi-View Multi-Target Tracking

Duong Nguyen-Ngoc Tran; Ngoc Doan-Minh Huynh; Cu Quoc Le; Hoang-Khang Nguyen; Long Hoang Pham; Huy-Hung Nguyen; Quoc Pham-Nam Ho; Trinh Le Ba Khanh

cs.CVAug 25, 2026

Syn2RealTrack: Bridging the Gap Between Synthetic and Real-World Datasets for Online Multi-View Multi-Target Tracking

Authors: Duong Nguyen-Ngoc Tran, Ngoc Doan-Minh Huynh, Cu Quoc Le, Hoang-Khang Nguyen, Long Hoang Pham, Huy-Hung Nguyen, Quoc Pham-Nam Ho, Trinh Le Ba Khanh,

Organizations: Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, South Korea

Abstract

Multi-camera 3D perception systems for warehouse scenes are trained largely on synthetic data and evaluated on physically captured environments. The resulting synthetic-to-real gap, which corrupts ground-plane localization and cross-camera identity association, is usually treated as one deficiency for a single domain-adaptation module to absorb; we argue instead that it enters the pipeline at three separable points: the camera calibration, the object shape prior, and the assumption that the object census is known, each admitting a different local remedy. Our online pipeline, Syn2RealTrack, follows this decomposition: lens distortion is recovered from images alone under a calibration that provides none, detections are fused across views by a visibility-weighted part-based descriptor that abstains on occluded parts rather than guessing, person height is measured in closed form from calibration instead of copied from a synthetic prior, and a closed-world cardinality prior is paired with a causal filter that removes the phantom boxes the prior manufactures. The system therefore adapts by reallocating trust between geometry and appearance without retraining a feature extractor. On the AI City Challenge 2026 Track~1 evaluation server it reaches a 3D Higher Order Tracking Accuracy (HOTA) of 52.0118%. The code will be released at https://github.com/SKKUAutoLab/aic26_mc3dp

Figures & tables

Panel of Figure 1 — Figure 1: Representative point-cloud-guided 3D box refinement examples on (a) a synthetic and (b) a real-world scene of the AI City Challenge 2026 Track 1 dataset [ 31 ] . Red boxes denote the original multi-view tracking prior, green boxes denote the refined 3D boxes, and the colored points denote the local cloud reconstructed by Depth Anything 3 (DA3) [ 15 ] .

Figure 2: Overview of the proposed framework, read left to right. (A) Data preprocessing. AnyCalib [ 35 ] estimates per-camera distortion from $31$ sampled frames, and the Unified Camera Model (UCM) parameter $\xi$ separates reference ( $\xi\leq 0.3$ , blue) and fisheye-candidate (red) views. (B) Per-view perception. RF-DETR [ 25 ] , ViTPose++ [ 46 ] , and Keypoint Promptable Re-Identification (KPR) [ 28 ] provide detections, keypoints, and visibility-aware part descriptors, which an appearance–IoU Kalman tracker links into local identities. (C) Multi-view fusion. Class-specific ground anchors and visibility-weighted part similarity merge observations within per-class bird’s-eye-view (BEV) gates; local-identity carry-forward and gated Hungarian assignment then form global tracks under cap $N_{k}$ . (D) 3D box construction and refinement. Trajectory-derived yaw, monocular person height, footprint and yaw refinement guided by Depth Anything 3 (DA3) [ 15 ] , and BEV visibility filtering produce the final boxes. Numbered badges identify the three synthetic-to-real gaps (calibration, shape priors, and closed-world cardinality), and dashed links show cross-stage dependencies between measured height, ground anchoring, and BEV filtering.

Figure 3: Single-view distortion estimation and camera grouping. (1) Frames are evenly sampled from every camera of a scene. (2) AnyCalib [ 35 ] predicts a dense per-pixel ray and field-of-view map and fits a pinhole baseline, a radial Brown–Conrady model, and the Unified Camera Model (UCM), whose parameter $\xi$ is used as a fisheye score; the estimates are used for distortion analysis and grouping only, and the dataset calibration is kept fixed for bird’s-eye-view (BEV) projection. (3) Cameras with $\xi$ below the threshold become reference views (blue), and the rest become fisheye candidates (red), whose reprojection displacements grow toward the image borders.

Parameter	Synthetic regime	Real regime
Cameras per scene	$10$ – $20$	$4$ – $7$
Intrinsics	ideal pinhole, cloned	per-camera, $f_{x}\neq f_{y}$
Person ground anchor	bbox $+$ height pole	ankle skeleton
Ankle confidence $\theta_{\text{ank}}$	$0.5$	$0.8$
Skeleton gate $n_{\text{kp}}$	$3$	$8$
Position smoothing $\alpha$	$1.0$ (none)	$0.6$

Table 1: The synthetic-to-real configuration surface: parameter settings used in the synthetic and real regimes.

Panel of Figure 4 — Figure 4: Multi-view RGB inputs and DA3-estimated point clouds for (a) the synthetic scene Warehouse 025 and (b) the real-world scene Warehouse 027 at frame 0.

Figure 5: Local fixed-shape object refinement pipeline for the synthetic scene. Starting from a colored DA3 .ply crop around a tracked object, we project the crop to a BEV density grid, keep density-excess cells after floor and ghost filtering, and fit a fixed-size box to the remaining support. The red dashed box is the multi-view tracking prior, and the green box is the refined result.

Panel of Figure 6 — Figure 6: BEV use of camera grouping. (a) Floor map of reference (blue) and fisheye (red) cameras; dashed violet marks fisheye-only zones, where objects retain their fisheye BEV positions. (b) For a matched object, the reference view anchors the fused target, and each fisheye projection’s error is measured in ground-plane meters.

Rank	ID	Name	3D HOTA (%)	DetA (%)	AssA (%)	LocA (%)
1	289	EVA	56.5447	55.6444	49.3929	79.5558
2	34	SKKU-AL-T1	52.0118	45.3056	56.5047	76.2410
3	130	Playbox	38.0105	40.2592	31.0978	75.1778
4	4	QDTers	34.1845	29.4663	33.1122	17.6841
5	149	Calix	33.7654	30.8748	30.9816	50.4720

Table 2: Leaderboard of Multi-Camera 3D Perception, AI City Challenge 2026 Track 1, ranked by 3D HOTA (%). Our entry (shaded, bold) ranks second.

Table 9

Table 10

#	Configuration	HOTA (%)	DetA (%)	AssA (%)	LocA (%)	$\Delta$ HOTA
1	RF-DETR detector only	50.8078	42.4111	56.2599	74.1915	–
2	$+$ gIoU association	51.1145	43.6031	55.8617	75.0459	$+0.31$
3	$+$ all association cues	51.8568	44.7921	56.4608	76.2410	$+0.74$
4	$+$ cross-view ground-plane clustering	51.9205	45.1611	56.4601	76.2053	$+0.06$
5	$+$ monocular person height estimation	51.9898	45.0712	56.5190	76.2397	$+0.07$
6	$+$ BEV-map prediction filtering	52.0118	45.3056	56.5047	76.2410	$+0.02$

Table 7: Stage-wise ablation of the full pipeline on the Track 1 evaluation server. Each row adds one component to the row above it; the shaded row 6 is the submitted system of Tab. 2 .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Visualization of the 2D bounding box filtering process. The bounding boxes removed from the training annotations are indicated by red ovals.

Panel of Figure 8 — Figure 8: Qualitative 2D detection results on scenes Warehouse 023–027 of the AI City Challenge 2026 Track 1 test set.

Panel of Figure 9 — Figure 9: Qualitative pose-estimation results (ViTPose++ [ 46 ] keypoint overlays) on scenes Warehouse 023–027.

Panel of Figure 10 — Figure 10: Qualitative 2D single-view tracking results on scenes Warehouse 023–027; each tracked object carries a persistent identity.

Panel of Figure 11 — Figure 11: Scene-level BEV visualizations constructed from DA3-estimated point clouds for (a) Warehouse 026 and (b) Warehouse 027.

3D MOT

3D multi-object tracking (3D MOT) estimates object locations and identities over time from cameras, LiDAR, radar, and other sensors.

See all 40 papers

Geometry Beats Estimated Depth: RGB-Only Multi-Camera 3D Tracking under Sim2Real
Aug 4, 2026Abdullah Naeem, Anav Katwal, Ayon Dey +2Synthetic-to-Real Domain Adaptation 3D Point Tracking
LSU New Orleans, New Orleans, USA · PinPark, Inc.
Physical AI Smart Spaces: A Large-Scale Benchmark for Multi-Camera 3D Perception in Smart Spaces
Oct 1, 2026Yuxing Wang, Yizhou Wang, Anqi Li +133D MOT 3D Object Detection
NVIDIA, Santa Clara, CA, USA · Santa Clara University, Santa Clara, CA, USA · NVIDIA
Calibration-Free 3D Multi-Camera People Tracking for Indoor Environment
Jul 22, 2026Ponleur Veng, Dominique Vaufreydaz, Phutphalla Kong3D MOT Multi-Object Tracking
CADT, M-PSI · Cambodia Academy of Digital Technology · Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France +2

Aug 4, 2026J/K move · Enter open · S save

Geometry Beats Estimated Depth: RGB-Only Multi-Camera 3D Tracking under Sim2Real

Abdullah Naeem, Anav Katwal, Ayon Dey, Noman Khan+1

LSU New Orleans, New Orleans, USA · PinPark, Inc.

Synthetic-to-Real Domain Adaptation 3D Point Tracking Multi-View 3D Object Detection 3D MOT

Open paper AskPDF arXiv

The AI City Challenge 2026 Track 1 evaluates multi-camera 3D perception in large indoor warehouses under a synthetic-to-real (Sim2Real) setting; depth is available only for training and validation, so inference is RGB-only. We use two RGB-only routes as a controlled test of one hypothesis: that cross-view geometric consistency, not monocular depth accuracy, governs performance under Sim2Real. The first is a geometry-first pipeline: YOLO11x detection, homography lifting to the world frame, class-level 3D size priors, multi-camera fusion, world-coordinate tracking, and offline tracklet stitching. The second is estimated-depth pseudo-LiDAR: monocular depth (D4RT, Metric3D~v2) back-projected into a fused point cloud and passed to a 3D detector (V-DETR), mirroring prior point-cloud winners that used depth. The gap is decisive: geometry-first reaches 13.0 3D HOTA (51.6 LocA), whereas pseudo-LiDAR collapses to 0.12 (9.2 LocA). We trace the collapse to cross-view inconsistency of monocular depth---scale correction is necessary but not sufficient---which domain-adaptation fine-tuning does not repair within budget. Within the geometry pipeline, offline stitching is the only intervention that helps; SAHI detection, appearance Re-ID, learned lifting, RT-DETR ensembling, test-time augmentation, and domain randomization all fail to beat the baseline detector. The bottlenecks are complementary: detection quality bounds the geometry route (DetA), localization consistency bounds pseudo-LiDAR (LocA). We release a complete, reproducible RGB-only pipeline and ablation.

Abstract

3D MOT

Explore similar work

Geometry Beats Estimated Depth: RGB-Only Multi-Camera 3D Tracking under Sim2Real

Physical AI Smart Spaces: A Large-Scale Benchmark for Multi-Camera 3D Perception in Smart Spaces

Calibration-Free 3D Multi-Camera People Tracking for Indoor Environment