Online multi camera 3D tracking must maintain scene global identities across synchronized views, yet query-based trackers carry these identities only implicitly in the instance bank, where they fragment upon query interruption. We present an online architecture that recovers association accuracy by predicting IDs explicitly over recurrent sparse queries. An outside-in Sparse4D detector fuses calibrated views into world frame 3D detections while propagating a sparse query bank, and a causal MOTIP ID decoder associates detections against a finite trajectory memory. We adapt MOTIP's relative-ID prediction and recycled slot runtime to globally fused 3D observations, and introduce metric spatial gating and proximity based newborn recovery. On the official 2026 AI City Challenge Track 1 test set, our method raises HOTA from 29.63 with native instance bank identities to 38.01, primarily through an AssA increase from 20.83 to 31.10, and ranks third on the public leaderboard. Full-sequence validation over all 9,000 frames of each scene shows that decoupled ID training improves HOTA over native identities, whereas continuing detector training alongside the detached ID objective produces scene-dependent gains and losses.
Figures & tables
Figure 1: Difference in multi-camera tracking fusion strategies. Conventional MTMC first forms per-camera tracklets and then performs difficult post-hoc cross-camera fusion. Our decomposition instead fuses calibrated views into one world-frame 3D observation stream before online ID prediction.
Figure 2: Architecture details. The recurrent detector fuses all calibrated views with query bank Qt−1 to produce world-frame observations and Qt . Detections query the finite trajectory memory Mt−1 for relative identities. Geometric masking, discrete assignment, newborn handling, and the MOTIP least-recently-used (LRU) slot lifecycle map those labels through μt to scene-global IDs. Only Qt , Mt , and μt cross the frame boundary.
Rank
Team
HOTA
DetA
AssA
LocA
1
EVA
56.54
55.64
49.39
79.56
2
SKKU-AL-T1
52.01
45.31
56.50
76.24
3
Playbox (ours)
38.01
40.26
31.10
75.18
4
VTX_QDTers
34.18
29.47
33.11
17.68
5
TU-YMLab
25.88
26.52
30.62
69.70
–
Sparse4D [ 25 ]
29.63
39.57
20.83
75.17
Table 1: Public leaderboard of Track 1, 2026 AI City Challenge (full test set, snapshot from August 15, 2026, top five of ten shown). The last row is our baseline, the recurrent Sparse4D tracker with its native instance-bank identities [ 25 ] , evaluated through the same official server.
Warehouse_020
Warehouse_021
Warehouse_022
System
HOTA
DetA
AssA
HOTA
DetA
AssA
HOTA
DetA
AssA
Submitted frozen det. (native IDs)
41.06
49.38
37.68
5.98
10.33
4.20
13.43
28.19
6.56
+ ID pred. (decoupled)
45.90
49.48
44.50
7.15
10.21
5.48
14.38
28.05
7.54
+ ID pred. (continued det.)
31.05
33.43
30.71
15.32
16.70
15.22
14.52
26.71
8.50
Finetuned detector (native IDs)
26.11
26.09
29.30
16.16
16.45
21.81
8.02
12.97
6.17
+ ID pred. (decoupled)
27.16
26.32
30.98
16.20
16.73
18.71
9.23
14.55
6.89
Table 2: Native identities and detector–ID training strategy. Full 9,000-frame validation sequences at threshold 0.4. Native-ID rows use the corresponding detector’s instance-bank identities. “Decoupled” freezes the submitted detector while training the ID modules. “Continued det.” resumes the converged ID model and updates both modules with ID-loss gradients detached from the detector. Best per scene and metric in bold.
Token source
Encoding
W020
W021
W022
HOTA
AssA
HOTA
AssA
HOTA
AssA
Detector query
none
45.57
43.98
7.19
5.51
14.32
7.58
3D position
Fourier
45.34
43.64
6.87
5.22
14.10
7.34
3D position
linear
41.92
37.91
6.88
5.25
14.28
7.43
Query + position
Fourier
45.94
44.69
7.27
5.55
14.54
7.65
Table 3: ID-token source and position encoding. Frozen recurrent detector, geometric constraints enabled, threshold 0.3, and full 9,000-frame validation sequences. The first three rows use matched 2k-iteration decoders. The deployed combined model was trained for 6k iterations and is shown for context, not as part of the controlled comparison. DetA varies by at most 0.6 across rows on every scene and is omitted. Best among the matched 2k rows in bold.
Spatial
Newborn
W020
W021
W022
gate
recovery
HOTA
AssA
HOTA
AssA
HOTA
AssA
✗
✗
44.31
41.37
7.11
5.32
12.57
5.77
✓
✗
45.85
44.29
7.35
5.68
11.46
4.82
✗
✓
45.66
44.21
7.04
5.23
14.24
7.38
✓
✓
45.90
44.50
7.15
5.48
14.38
7.54
Table 4: Geometric constraints on the frozen-detector ID model (full 9,000-frame validation sequences, threshold 0.4). DetA varies by at most 0.09 across configurations on every scene and is omitted.
Figure 3: Short-term identity confusion. Three synchronized camera views and the bottom-right BEV show a close crossing on Warehouse_022 . One continuously observed person’s predicted identity changes from 245 (green) to 229 (orange), while the nearest other track is 0.1 m away. White denotes the corresponding ground-truth trajectory.
Multi-camera tracking with overlapping fields of view typically relies on centralized fusion, which creates computational bottlenecks that prevent deployment at scale. We present MV3DT, a fully distributed framework for real-time multi-view 3D tracking that achieves accurate identity propagation and occlusion recovery through peer-to-peer coordination, eliminating the need for central aggregation. Each camera node executes a lightweight modular pipeline comprising monocular 3D perception, distributed multi-view association, and collaborative fusion via lightweight messaging. MV3DT achieves 96.5% IDF1, 93.1% MOTA, and 94.6% MOTP on WILDTRACK, competitive with state-of-the-art centralized methods, and unprecedented 41.7% IDF1 and 50.9% MOTA on SCOUT while demonstrating superior scalability: sustaining 30 FPS on 100 cameras with <10ms inter-camera latency and only 2.2% communication overhead. MV3DT operates in a zero-shot regime given camera calibrations, requiring no scene-specific learning and making it directly deployable in new environments. These results establish MV3DT as a practical solution for real-time multi-view tracking in large-scale overlapping camera networks.
Byron Hernandez, Fangyu Li, Aotian Wu +3
University of Florida, Gainesville, FL, USA · NVIDIA Corporation, Santa Clara, CA, USA
Multi-Camera People Tracking (MCPT) traditionally relies on precise intrinsic and extrinsic camera calibration to project 2D detections into a unified 3D world coordinate system.However, manual calibration constitutes a major bottleneck in large-scale dataset generation from unconstrained video archives. This work proposes a unified calibration-free 3D MCPT framework that infers geometric structure directly from visual data using deep foundation models. The system integrates anchor-free detection (YOLOX), robust tracking (BoT-SORT), omni-scale appearance embedding (OsNet), pose estimation (HRNet via MMPose), and transformer-based geometric reconstruction using the Visual Geometry Grounded Transformer (VGGT). A pose-guided 3D lifting strategy projects head keypoints onto a reconstructed manifold, eliminating dependence on ground-plane homography. Global identity association is formulated as hierarchical agglomerative clustering under a joint appearance-geometry cost with strict velocity gating. Evaluation on the AI City Challenge 2024 demonstrates a HOTA score of 53.13% without access to ground-truth calibration matrices, establishing a strong baseline for purely vision-based 3D tracking.
Ponleur Veng, Dominique Vaufreydaz, Phutphalla Kong
CADT, M-PSI · Cambodia Academy of Digital Technology · Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France +2
Multi-view tracking extends visual coverage beyond a single camera, providing complementary observations when targets are occluded or leave the view. This raises a question: how can observations from different views be matched to the correct identities without offline scene-specific training or camera calibration? Progress toward this goal remains limited because many offline formulations assume such deployment-specific knowledge, while online adaptation can amplify association errors. We study this problem as online multi-view identity maintenance (OMIM): targets are initialized once per view, with no further identity labels provided after each initialization. To this end, we introduce OmniGlimpse, an asymmetric propose-verify framework that separates adaptive association from stable identity consolidation. At the representation level, short-term local and cross-view evidence is mapped into target-specific adaptive spaces to propose associations, whereas long-term identity distributions are maintained for verification in a canonical space defined by a frozen encoder. At the update level, short-term evidence may guide current associations, whereas, after bootstrapping, only verified observations may update persistent memory or drive adaptation. Under a unified target-specified evaluation protocol, OmniGlimpse improves over the best evaluated online baseline for each metric by 9.1 IDF1 and 22.1 CVIDF1 points on M3Track. The corresponding gains on MvMHAT are 20.3 and 26.2 points. Despite using no offline scene-specific training, it outperforms offline-trained references in IDF1 and remains competitive in cross-view association. These results support an approach to online identity maintenance that adapts to changing appearances while consolidating reliable evidence into stable identity anchors.