ModTrack: Sensor-Agnostic Multi-View Tracking via Identity-Informed PHD Filtering with Covariance Propagation

Aditya Iyer; Jack Roberts; Nora Ayanian

cs.CVMar 16, 2026

ModTrack: Sensor-Agnostic Multi-View Tracking via Identity-Informed PHD Filtering with Covariance Propagation

Authors: Aditya Iyer, Jack Roberts, Nora Ayanian

Organizations: Department of Computer Science, Brown University, Providence, RI, USA · School of Engineering, Brown University, Providence, RI, USA

Abstract

Multi-View Multi-Object Tracking (MV-MOT) aims to localize and maintain consistent identities of objects observed by multiple sensors. This task is challenging, as viewpoint changes and occlusion disrupt identity consistency across views and time. Recent end-to-end approaches address this by jointly learning 2D Bird's Eye View (BEV) representations and identity associations, achieving high tracking accuracy. However, these methods offer no principled uncertainty accounting and remain tightly coupled to their training configuration, limiting generalization across sensor layouts, modalities, or datasets without retraining. We propose ModTrack, a modular MV-MOT system that matches end-to-end performance while providing cross-modal, sensor-agnostic generalization and traceable uncertainty. ModTrack confines learning methods to just the Detection and Feature Extraction stage of the MV-MOT pipeline, performing all fusion, association, and tracking with closed-form analytical methods. Our design reduces each sensor's output to calibrated position-covariance pairs $(\mathbf{z}, R)$ ; cross-view clustering and precision-weighted fusion then yield unified estimates $(\hat{\mathbf{z}}, \hat{R})$ for identity assignment and temporal tracking. A feedback-coupled, identity-informed Gaussian Mixture Probability Hypothesis Density (GM-PHD) filter with HMM motion modes uses these fused estimates to maintain identities under missed detections and heavy occlusion. ModTrack achieves 95.5 IDF1 and 91.4 MOTA on WildTrack, surpassing all prior modular methods by over 21 points and rivaling the state-of-the-art end-to-end methods while providing deployment flexibility they cannot. Specifically, the same tracker core transfers unchanged to MultiviewX and RadarScenes, with only perception-module replacement required to extend to new domains and sensor modalities.

Figures & tables

Figure 1: ModTrack pipeline. Per-camera detections and appearance features (Stage 1) are projected to BEV with Jacobian-propagated covariance (Stage 2), associated via $\chi^{2}$ graph clustering (Stage 3), and precision-weighted into fused $(\hat{\mathbf{z}},\hat{R})$ pairs (Stage 4). An identity-informed GM-PHD filter (Stage 5) maintains persistent identities, with track predictions feeding back to guide identity assignment (dashed). Neural modules (blue) are confined to Stage 1; all downstream stages are purely analytical.

Figure 2: Predicted BEV tracks on WildTrack (colored by identity). Ground truth (left) vs. ModTrack (right).

End-to-end (fixed config.)
	WildTrack				MultiviewX
Method	IDF1	MOTA	MOTP	GOSPA $\downarrow$	IDF1	MOTA	MOTP	GOSPA $\downarrow$
EarlyBird [ 22 ]	92.3	89.5	86.6	–	82.4	88.4	86.2	–
Ali et al. [ 1 ]	92.0	87.4	–	–	–	–	–	–
MCBLT [ 27 ]	95.6	92.6	94.3	–	–	–	–	–
TrackTacular [ 23 ]	95.3	91.8	85.4	–	85.6	92.4	80.1	–

Table 1: Comparison of tracking performance on WildTrack and MultiviewX datasets.

	WildTrack				MultiviewX
Mode	IDF1	MOTA	MOTP	GOSPA	IDF1	MOTA	MOTP	GOSPA
Spatial only	86.6	89.5	82.4	1.22	84.2	84.9	83.7	1.86
Semantic only	49.6	41.8	79.6	2.60	65.1	56.7	79.6	3.00
Joint (full)	95.5	91.4	87.2	1.20	86.2	86.3	84.0	1.83

Table 2: Ablation of matching modes on WildTrack and MultiviewX. Joint mode uses spatial clustering and reserves semantic features for the tracking stage.

	Cameras available
	7	6	5	4	3	2
IDF1	95.5	91.7 $\pm$ 2.9	85.1 $\pm$ 5.3	74.5 $\pm$ 9.7	57.8 $\pm$ 15.7	31.6 $\pm$ 19.5
MOTA	91.4	87.7 $\pm$ 2.0	81.1 $\pm$ 4.7	70.2 $\pm$ 9.0	51.7 $\pm$ 16.2	25.5 $\pm$ 18.3
MOTP	87.2	85.9 $\pm$ 0.9	83.8 $\pm$ 2.9	82.4 $\pm$ 3.0	80.7 $\pm$ 4.2	73.2 $\pm$ 18.2
GOSPA	1.2	1.3 $\pm$ 0.1	1.5 $\pm$ 0.2	1.9 $\pm$ 0.3	2.3 $\pm$ 0.4	2.9 $\pm$ 0.4

Table 3: Sensor dropout robustness (WildTrack, joint mode). Mean

\pm

std over all

\binom{7}{k}

subsets. GOSPA (

p{=}2

c{=}1

\alpha{=}2

): lower is better.

Figure 3: Precision-weighted fusion on WildTrack (frame 383). Each panel shows BEV covariance ellipses (95% confidence) as cameras are incrementally added. The rightmost panel shows the 6-camera shared world-plane.

Approach	LSTQ	$S_{\text{cls}}$	$S_{\text{assoc}}$	GOSPA
MOT [ 29 ]	42.4	19.4	92.7	–
CA-Net [ 15 ]	34.8	13.0	92.7	–
Radar Tracker [ 32 ]	66.8	92.7	48.2	–
ModTrack + Doppler perception	42.1	70.0	26.9	2.50
ModTrack + Oracle perception	76.9	99.1	60.9	1.09

Table 4: Comparison on RadarScenes dataset.

Panel of Figure 4 — Figure 4: IDF1 sensitivity analysis (WildTrack, joint mode). Each parameter is swept individually; all others held at defaults. Extended results are in Supp. Material D.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Time (ms)	% Total	FPS
Neural perception (Det. + Depth + Feat.)	443.9	81.0	2.25
Analytical core (Clust. + Fusion + PHD)	103.2	18.8	9.69
Other overhead	0.7	0.1	–
ModTrack (Total)	547.8 $\pm$ 85.0	100	1.83

Appendix

Table 5: Per-frame inference time breakdown on WildTrack (7 cameras, RTX 3090). The analytical core accounts for less than 20% of total latency; perception dominates runtime.

Figure 5: MOTA sensitivity to continuous clustering hyperparameters on WildTrack.

Figure 6: MOTP sensitivity to continuous clustering hyperparameters on WildTrack.

Figure 7: IDF1 sensitivity to integer tracking hyperparameters ( $N_{\mathrm{init}}$ , $\tau_{\mathrm{tent}}$ , $\tau_{\mathrm{confirmed}}$ , $K_{\max}$ ) on WildTrack.

Figure 8: MOTA sensitivity to integer tracking hyperparameters ( $N_{\mathrm{init}}$ , $\tau_{\mathrm{tent}}$ , $\tau_{\mathrm{confirmed}}$ , $K_{\max}$ ) on WildTrack.

Figure 9: MOTP sensitivity to integer tracking hyperparameters ( $N_{\mathrm{init}}$ , $\tau_{\mathrm{tent}}$ , $\tau_{\mathrm{confirmed}}$ , $K_{\max}$ ) on WildTrack.

Figure 10: MOTA sensitivity to continuous tracking hyperparameters on WildTrack.

Figure 11: MOTP sensitivity to continuous tracking hyperparameters on WildTrack.

Figure 12: IDF1 sensitivity to calibration hyperparameters ( $\sigma^{2}_{\mathrm{pose}}$ , $\sigma^{2}_{\mathrm{min}}$ , $\lambda_{Q}$ ) on WildTrack.

Figure 13: MOTA sensitivity to calibration hyperparameters ( $\sigma^{2}_{\mathrm{pose}}$ , $\sigma^{2}_{\mathrm{min}}$ , $\lambda_{Q}$ ) on WildTrack.

Figure 14: MOTP sensitivity to calibration hyperparameters ( $\sigma^{2}_{\mathrm{pose}}$ , $\sigma^{2}_{\mathrm{min}}$ , $\lambda_{Q}$ ) on WildTrack.

Stage	Symbol	WildTrack	MultiviewX	RadarScenes	Physical Interpretation
Camera Perception
	$\tau_{\text{yolo}}$	0.1	0.63	–	Min. detector confidence to retain a detection
	$\alpha_{fp}$	0.035	0.035	–	Footpoint depth uncertainty scaling factor
	$\alpha_{bbox}$	0.05	0.05	–	Bbox depth uncertainty scaling factor
	$\eta_{\mathrm{fp}}$	3	3	–	Footpoint precision boost (trust factor)
	$\sigma^{2}_{\min\mathrm{depth}}$	0.0001	0.0001	–	Min. depth variance

Appendix

Table 6: Hyperparameters: perception, projection, and clustering stages. Camera-specific parameters (top block) do not apply to RadarScenes.

Stage	Symbol	WildTrack	MultiviewX	RadarScenes	Physical Interpretation
Tracking (GM-PHD)
	$p_{S}$	0.99	0.99	0.99	Target survival probability
	$p_{D}$	0.90	0.95	0.90	Detection probability
	$\lambda_{\mathrm{assoc}}$	2.5	2.5	2.5	Identity score boost in assignment cost
	$\mu_{\mathrm{sem}}$	2.0	2.0	0.0	Appearance similarity boost (spatial-only for radar)
	$\lambda_{\mathrm{ReID}}$	3.0	3.0	0.0	Appearance boost for LOST-track re-identification

Appendix

Table 7: Hyperparameters: tracking, lifecycle, and motion model stages. RadarScenes uses the same GM-PHD tracking loop with dataset-specific tuning.

Dataset	$N$	Mean NEES	95% CI	1 $\sigma$ Cov.	2 $\sigma$ Cov.
WildTrack	879	0.46	1.87-2.13	88.4%	98.6%
MultiviewX	1317	0.38	1.89-2.11	91.9%	99.6%

Appendix

Table 8: Calibration statistics (matched track/GT pairs only).

Method	Changed Component	IDF1	MOTA
ModTrack (baseline)	None	95.5	91.4
+ RANSAC clustering	Clustering	92.8	90.8
+ Fixed covariance ( $R=0.16I$ )	Detection covariance	95.0	90.7
+ Simple mean fusion	Fusion	93.0	89.5

Appendix

Table 9: Ablation study isolating key components of ModTrack on WildTrack (joint setting).

Dataset	Method	IDF1 $\uparrow$	IDS $\downarrow$	GOSPA $\downarrow$
WildTrack	Vanilla	79.3	30	1.31
	ModTrack	95.5	2	1.20
MultiviewX	Vanilla	46.2	145	1.95
	ModTrack	85.2	14	1.83

Appendix

Table 10: Comparison of ModTrack against standard GM-PHD filter on WildTrack.

Multi-Object Tracking

Multi-object tracking (MOT) links detections across video frames to maintain object identities through motion, occlusion, and changing viewpoints.

See all 182 papers

May 5, 2026

Unified Multimodal Visual Tracking with Dual Mixture-of-Experts

Multimodal visual object tracking can be divided into to several kinds of tasks (e.g. RGB and RGB+X tracking), based on the input modality. Existing methods often train separate models for each modality or rely on pretrained models to adapt to new modalities, which limits efficiency, scalability, and usability. Thus, we introduce OneTrackerV2, a unified multi-modal tracking framework that enables end-to-end training for any modality. We propose Meta Merger to embed multi-modal information into a unified space, allowing flexible modality fusion and robustness. We further introduce Dual Mixture-of-Experts (DMoE): T-MoE models spatio-temporal relations for tracking, while M-MoE embeds multi-modal knowledge, disentangling cross-modal dependencies and reducing feature conflicts. With a shared architecture, unified parameters, and a single end-to-end training, OneTrackerV2 achieves state-of-the-art performance across five RGB and RGB+X tracking tasks and 12 benchmarks, while maintaining high inference efficiency. Notably, even after model compression, OneTrackerV2 retains strong performance. Moreover, OneTrackerV2 demonstrates remarkable robustness under modality-missing scenarios.

Lingyi Hong, Jinglun Li, Xinyu Zhou +6

Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University

Cross-Modal Learning Multimodal Visual Object Tracking Multimodal Robustness Unified Multimodal Models Mixture of Experts Visual Object Tracking+2

Ask PDF

May 10, 2026

CalibFree: Self-Supervised View Feature Separation for Calibration-Free Multi-Camera Multi-Object Tracking

Multi-camera multi-object tracking (MCMOT) faces significant challenges in maintaining consistent object identities across varying camera perspectives, particularly when precise calibration and extensive annotations are required. In this paper, we present CalibFree, a self-supervised representation learning framework that does not need any calibration or manual labeling for the MCMOT task. By promoting feature separation between view-agnostic and view-specific representations through single-view distillation and cross-view reconstruction, our method adapts to complex, dynamic scenarios with minimal overhead. Experiments on the MMP-MvMHAT dataset show a 3% improvement in overall accuracy and a 7.5% increase in the average F1 score over state-of-the-art approaches, confirming the effectiveness of our calibration-free design. Moreover, on the more diverse MvMHAT dataset, our approach demonstrates superior over-time tracking and strong cross-view performance, highlighting its adaptability to a wide range of camera configurations. Code will be publicly available upon acceptance.

Ruiqi Xian, Deep Patel, Iain Melvin +3

University of Maryland, College Park, MD, USA · NEC Laboratories America, Princeton, NJ, USA · University of North Carolina, Greenboro, NC, USA

Multi-View Learning Invariant Representation Learning Self-Supervised Visual Representation Learning Multi-Object Tracking

Ask PDF

Aug 7, 2026

AnyTrack: Unifying Visual Object Tracking with Any Modalities

Visual object tracking aims to continuously locate specific targets within sequential frames, evolving from single-modal methods to multi-modal ones. However, existing multi-modal trackers are typically designed for fixed modality combinations, requiring separate models for different inputs. This leads to a poor adaptability to missing or imperfect modalities, and limited generalization. To address these issues, we propose a novel unified framework called AnyTrack for object tracking with any modalities. Specifically, we design a Modality-aware Interaction Module (MIM) to facilitate dynamic interaction across diverse modalities. This module bridges modality discrepancies and aggregates temporal cues to maintain spatio-temporal consistency during cross-modal interaction. Furthermore, we introduce a Context Understanding Module (CUM) to establish spatial correspondence between visual features and target locations via global-local prompts. This module employs target-aware context modeling to enhance foreground-background discrimination for precise localization. Finally, to support the training and evaluation under diverse modalities, we extend existing multi-modal object tracking benchmarks by incorporating grayscale images, language descriptions, and audio clips. Extensive experiments with both complete and missing modality settings demonstrate that our AnyTrack achieves state-of-the-art performance, validating its effectiveness and flexibility. The source code is available at https://github.com/IdolLab/AnyTrack.

Hao Li, Yunzhi Zhuge, Wenning Hao +4

Army Engineering University of PLA Nanjing, China · Dalian University of Technology Dalian, China · National University of Defense Technology Nanjing, China

Multimodal Visual Object Tracking Missing-Modality Learning Unified Multimodal Models Visual Object Tracking Multimodal Fusion

Ask PDF

Abstract

Multi-Object Tracking

Explore similar work

Unified Multimodal Visual Tracking with Dual Mixture-of-Experts

CalibFree: Self-Supervised View Feature Separation for Calibration-Free Multi-Camera Multi-Object Tracking

AnyTrack: Unifying Visual Object Tracking with Any Modalities