OmniGlimpse: Asymmetric Identity Memory for Online Multi-View Tracking

Rangya Zhang; Jiaping Xiao; Yujing Shang; Yuhang Zhang; Mingsheng Li; Mir Feroskhan

cs.CVOct 27, 2026

OmniGlimpse: Asymmetric Identity Memory for Online Multi-View Tracking

Authors: Rangya Zhang, Jiaping Xiao, Yujing Shang, Yuhang Zhang, Mingsheng Li, Mir Feroskhan

Organizations: 1Nanyang Technological University

Abstract

Multi-view tracking extends visual coverage beyond a single camera, providing complementary observations when targets are occluded or leave the view. This raises a question: how can observations from different views be matched to the correct identities without offline scene-specific training or camera calibration? Progress toward this goal remains limited because many offline formulations assume such deployment-specific knowledge, while online adaptation can amplify association errors. We study this problem as online multi-view identity maintenance (OMIM): targets are initialized once per view, with no further identity labels provided after each initialization. To this end, we introduce OmniGlimpse, an asymmetric propose-verify framework that separates adaptive association from stable identity consolidation. At the representation level, short-term local and cross-view evidence is mapped into target-specific adaptive spaces to propose associations, whereas long-term identity distributions are maintained for verification in a canonical space defined by a frozen encoder. At the update level, short-term evidence may guide current associations, whereas, after bootstrapping, only verified observations may update persistent memory or drive adaptation. Under a unified target-specified evaluation protocol, OmniGlimpse improves over the best evaluated online baseline for each metric by 9.1 IDF1 and 22.1 CVIDF1 points on M3Track. The corresponding gains on MvMHAT are 20.3 and 26.2 points. Despite using no offline scene-specific training, it outperforms offline-trained references in IDF1 and remains competitive in cross-view association. These results support an approach to online identity maintenance that adapts to changing appearances while consolidating reliable evidence into stable identity anchors.

Figures & tables

Figure 1: Overview of OmniGlimpse. Top: targets are initialized once per view and maintained through adaptive association, canonical verification, and verified updates. Bottom: OmniGlimpse supports cross-view recovery, appearance adaptation, and resistance to distractors.

Figure 2: Overview of OmniGlimpse. Short-term evidence supports association in adaptive spaces, while long-term distributions provide canonical-space verification. After initialization and bootstrapping, only admitted observations may update persistent memory and local adapters.

Method	HOTA $\uparrow$	DetA $\uparrow$	AssA $\uparrow$	IDF1 $\uparrow$	MOTA $\uparrow$	MOTP $\uparrow$	IDs $\downarrow$	MDA $\uparrow$	CVIDP $\uparrow$	CVIDR $\uparrow$	CVIDF1 $\uparrow$	CVMA $\uparrow$
Offline-trained MVMT references
MvMHAT	.5832	.4407	.7723	.6267	.4641	.9335	11	.1620	.2914	.1920	.2277	.3204
CrossMOT	.5970	.5540	.6445	.7945	.6628	.8182	116	.5635	.6150	.5499	.5786	.6335
AGW	.4748	.3928	.5773	.6347	.4630	.7936	9	.2350	.3374	.2556	.2864	.3717
Online baselines
OCLReID	.5862	.4810	.7173	.7424	.5795	.7972	18	.3290	.4480	.3552	.3918	.4586

Table 1: Selected-target results on M3Track. Offline references use target-selection information, including ground-truth-assisted trajectory selection for CrossMOT and AGW and full-sequence appearance matching for MvMHAT (Appendix B.2 ). Best and second-best in bold and underline.

Method	HOTA $\uparrow$	DetA $\uparrow$	AssA $\uparrow$	IDF1 $\uparrow$	MOTA $\uparrow$	MOTP $\uparrow$	IDs $\downarrow$	MDA $\uparrow$	CVIDP $\uparrow$	CVIDR $\uparrow$	CVIDF1 $\uparrow$	CVMA $\uparrow$
Offline-trained MVMT references
MvMHAT	.3646	.1786	.7443	.3012	.1448	.9556	1	.0500	.0522	.0505	.0513	.2354
CrossMOT	.5217	.4448	.6130	.6901	.4874	.8196	17	.5626	.6380	.5819	.6078	.7190
AGW	.5761	.4352	.7628	.5875	.3518	.9993	6	.3448	.4687	.3759	.4131	.5114
Online baselines
OCLReID	.4644	.3060	.7133	.5292	.3290	.8403	3	.2280	.2975	.2454	.2676	.4245

Table 2: Selected-target results on MvMHAT. Offline references use information for target selection: CrossMOT uses ground-truth-assisted trajectory selection; MvMHAT and AGW use full-sequence appearance matching (Appendix B.2 ). Best and second-best results are shown in bold and underline.

Variant	Rel.	Cross	Adapt.	Gauss.	V-Upd.	HOTA $\uparrow$	DetA $\uparrow$	IDF1 $\uparrow$	MOTA $\uparrow$	IDs $\downarrow$	MDA $\uparrow$	CVIDP $\uparrow$	CVIDF1 $\uparrow$	CVMA $\uparrow$
Base	–	–	–	–	–	.6240	.5548	.7999	.6801	19	.3995	.5602	.4881	.5395
+ Reliability Controller	$\surd$	–	–	–	–	.6386	.5802	.8246	.6989	46	.4769	.6271	.5608	.5805
+ Cross-view Assistance	$\surd$	$\surd$	–	–	–	.6389	.5890	.8280	.7100	45	.5360	.6460	.6000	.6250
+ Online Contrastive	$\surd$	$\surd$	$\surd$	–	–	.6528	.5964	.8453	.7322	83	.5255	.6271	.5850	.6398
Full w/o Rel.	–	$\surd$	$\surd$	$\surd$	$\surd$	.6178	.5482	.7938	.6710	32	.3924	.5507	.4796	.5333
w/o Online	$\surd$	$\surd$	–	$\surd$	$\surd$	.6250	.5650	.8050	.6900	26	.5140	.6210	.5710	.6210

Table 3: Ablation study on 19 non-dark M3Track test instances. The upper block shows incremental variants and the lower block full-model controls (Appendix B.3 ). V-Upd. denotes verification-gated updates; Adaptive Query uses adapted features. Best and second-best are in bold and underline.

Figure 3: Identity maintenance in the M3Track garden scene. Columns show synchronized observations; the first row shows the reference view. Yellow dashed boxes mark the target, and cyan boxes mark confirmed predictions.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Detector / canonical backbone	YOLOX-L / ResNet-18
Canonical checkpoint source	OCLReID-trained ResNet-18
Number of part cues $K$	4
Global and part dimensions $D_{k}$	512
Detection score / NMS IoU threshold	0.60 / 0.35
Support aggregation $L$	3

Appendix

Table 4: Association, verification, and admission settings. Capacities count observations or prototypes, not individual feature coordinates.

Setting	Value
Adapter hidden width	128
Optimizer	Adam
Learning rate / weight decay	$10^{-4}$ / $10^{-4}$
Optimization interval per target–view pair	5 newly admitted local observations
Optimizer steps per eligible batch	1
Maximum / minimum triplets per batch	16 / 4

Appendix

Table 5: Local adapter learning and classifier settings. Ridge regularization refers to the class-normalized objective in Eq. 12 .

Figure 4: Additional comparison in the M3Track wood scene. Columns correspond to dataset frames 1861, 1867, 1871, and 1875. The reference view is shown once; the other rows use identical images and crops for the three configurations. Yellow dashed boxes denote the target and cyan boxes denote confirmed outputs; tentative candidates are not drawn. Displayed View 1 and View 2 correspond to source view2 and view1, respectively.

Multi-Object Tracking

Multi-object tracking (MOT) links detections across video frames to maintain object identities through motion, occlusion, and changing viewpoints.

See all 193 papers

Online Multi-Camera 3D Tracking via ID Prediction over Recurrent Sparse Queries
Sep 16, 2026Pragyan Shrestha, Haruto Nakayama, Atom Scott3D MOT Multi-Object Tracking
Playbox Inc., Tokyo, Japan
ModTrack: Sensor-Agnostic Multi-View Tracking via Identity-Informed PHD Filtering with Covariance Propagation
Mar 16, 2026Aditya Iyer, Jack Roberts, Nora AyanianMulti-Object Tracking Uncertainty-Aware Multimodal Fusion
Department of Computer Science, Brown University, Providence, RI, USA · School of Engineering, Brown University, Providence, RI, USA
Beyond Appearance: A Multi-cue Framework and Large-scale Benchmark for Pedestrian Association and Tracking on Mobile Aerial-Ground Platforms
Jul 26, 2026Ruiqi Wu, Bingliang Jiao, Ruize Han +6Feature Fusion Multi-Object Tracking
School of Computer Science, Northwestern Polytechnical University, Xi’an, China · Ningbo Institute, Northwestern Polytechnical University, Ningbo 315000, China · National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, Xi’an, China

Sep 16, 2026J/K move · Enter open · S save

Online Multi-Camera 3D Tracking via ID Prediction over Recurrent Sparse Queries

Pragyan Shrestha, Haruto Nakayama, Atom Scott

Playbox Inc., Tokyo, Japan

3D MOT Multi-Object Tracking

Open paper AskPDF arXiv

Online multi camera 3D tracking must maintain scene global identities across synchronized views, yet query-based trackers carry these identities only implicitly in the instance bank, where they fragment upon query interruption. We present an online architecture that recovers association accuracy by predicting IDs explicitly over recurrent sparse queries. An outside-in Sparse4D detector fuses calibrated views into world frame 3D detections while propagating a sparse query bank, and a causal MOTIP ID decoder associates detections against a finite trajectory memory. We adapt MOTIP's relative-ID prediction and recycled slot runtime to globally fused 3D observations, and introduce metric spatial gating and proximity based newborn recovery. On the official 2026 AI City Challenge Track 1 test set, our method raises HOTA from 29.63 with native instance bank identities to 38.01, primarily through an AssA increase from 20.83 to 31.10, and ranks third on the public leaderboard. Full-sequence validation over all 9,000 frames of each scene shows that decoupled ID training improves HOTA over native identities, whereas continuing detector training alongside the detached ID objective produces scene-dependent gains and losses.

Abstract

Multi-Object Tracking

Explore similar work

Online Multi-Camera 3D Tracking via ID Prediction over Recurrent Sparse Queries

ModTrack: Sensor-Agnostic Multi-View Tracking via Identity-Informed PHD Filtering with Covariance Propagation

Beyond Appearance: A Multi-cue Framework and Large-scale Benchmark for Pedestrian Association and Tracking on Mobile Aerial-Ground Platforms