ModTrack: Sensor-Agnostic Multi-View Tracking via Identity-Informed PHD Filtering with Covariance Propagation

Aditya Iyer; Jack Roberts; Nora Ayanian

cs.CVMar 16, 2026

ModTrack: Sensor-Agnostic Multi-View Tracking via Identity-Informed PHD Filtering with Covariance Propagation

Authors: Aditya Iyer, Jack Roberts, Nora Ayanian

Organizations: Department of Computer Science, Brown University, Providence, RI, USA · School of Engineering, Brown University, Providence, RI, USA

Abstract

Multi-View Multi-Object Tracking (MV-MOT) aims to localize and maintain consistent identities of objects observed by multiple sensors. This task is challenging, as viewpoint changes and occlusion disrupt identity consistency across views and time. Recent end-to-end approaches address this by jointly learning 2D Bird's Eye View (BEV) representations and identity associations, achieving high tracking accuracy. However, these methods offer no principled uncertainty accounting and remain tightly coupled to their training configuration, limiting generalization across sensor layouts, modalities, or datasets without retraining. We propose ModTrack, a modular MV-MOT system that matches end-to-end performance while providing cross-modal, sensor-agnostic generalization and traceable uncertainty. ModTrack confines learning methods to just the Detection and Feature Extraction stage of the MV-MOT pipeline, performing all fusion, association, and tracking with closed-form analytical methods. Our design reduces each sensor's output to calibrated position-covariance pairs $(\mathbf{z}, R)$ ; cross-view clustering and precision-weighted fusion then yield unified estimates $(\hat{\mathbf{z}}, \hat{R})$ for identity assignment and temporal tracking. A feedback-coupled, identity-informed Gaussian Mixture Probability Hypothesis Density (GM-PHD) filter with HMM motion modes uses these fused estimates to maintain identities under missed detections and heavy occlusion. ModTrack achieves 95.5 IDF1 and 91.4 MOTA on WildTrack, surpassing all prior modular methods by over 21 points and rivaling the state-of-the-art end-to-end methods while providing deployment flexibility they cannot. Specifically, the same tracker core transfers unchanged to MultiviewX and RadarScenes, with only perception-module replacement required to extend to new domains and sensor modalities.

Figures & tables

Figure 1: ModTrack pipeline. Per-camera detections and appearance features (Stage 1) are projected to BEV with Jacobian-propagated covariance (Stage 2), associated via $\chi^{2}$ graph clustering (Stage 3), and precision-weighted into fused $(\hat{\mathbf{z}},\hat{R})$ pairs (Stage 4). An identity-informed GM-PHD filter (Stage 5) maintains persistent identities, with track predictions feeding back to guide identity assignment (dashed). Neural modules (blue) are confined to Stage 1; all downstream stages are purely analytical.

Figure 2: Predicted BEV tracks on WildTrack (colored by identity). Ground truth (left) vs. ModTrack (right).

End-to-end (fixed config.)
	WildTrack				MultiviewX
Method	IDF1	MOTA	MOTP	GOSPA $\downarrow$	IDF1	MOTA	MOTP	GOSPA $\downarrow$
EarlyBird [ 22 ]	92.3	89.5	86.6	–	82.4	88.4	86.2	–
Ali et al. [ 1 ]	92.0	87.4	–	–	–	–	–	–
MCBLT [ 27 ]	95.6	92.6	94.3	–	–	–	–	–
TrackTacular [ 23 ]	95.3	91.8	85.4	–	85.6	92.4	80.1	–

Table 1: Comparison of tracking performance on WildTrack and MultiviewX datasets.

	WildTrack				MultiviewX
Mode	IDF1	MOTA	MOTP	GOSPA	IDF1	MOTA	MOTP	GOSPA
Spatial only	86.6	89.5	82.4	1.22	84.2	84.9	83.7	1.86
Semantic only	49.6	41.8	79.6	2.60	65.1	56.7	79.6	3.00
Joint (full)	95.5	91.4	87.2	1.20	86.2	86.3	84.0	1.83

Table 2: Ablation of matching modes on WildTrack and MultiviewX. Joint mode uses spatial clustering and reserves semantic features for the tracking stage.

	Cameras available
	7	6	5	4	3	2
IDF1	95.5	91.7 $\pm$ 2.9	85.1 $\pm$ 5.3	74.5 $\pm$ 9.7	57.8 $\pm$ 15.7	31.6 $\pm$ 19.5
MOTA	91.4	87.7 $\pm$ 2.0	81.1 $\pm$ 4.7	70.2 $\pm$ 9.0	51.7 $\pm$ 16.2	25.5 $\pm$ 18.3
MOTP	87.2	85.9 $\pm$ 0.9	83.8 $\pm$ 2.9	82.4 $\pm$ 3.0	80.7 $\pm$ 4.2	73.2 $\pm$ 18.2
GOSPA	1.2	1.3 $\pm$ 0.1	1.5 $\pm$ 0.2	1.9 $\pm$ 0.3	2.3 $\pm$ 0.4	2.9 $\pm$ 0.4

Table 3: Sensor dropout robustness (WildTrack, joint mode). Mean

\pm

std over all

\binom{7}{k}

subsets. GOSPA (

p{=}2

c{=}1

\alpha{=}2

): lower is better.

Figure 3: Precision-weighted fusion on WildTrack (frame 383). Each panel shows BEV covariance ellipses (95% confidence) as cameras are incrementally added. The rightmost panel shows the 6-camera shared world-plane.

Approach	LSTQ	$S_{\text{cls}}$	$S_{\text{assoc}}$	GOSPA
MOT [ 29 ]	42.4	19.4	92.7	–
CA-Net [ 15 ]	34.8	13.0	92.7	–
Radar Tracker [ 32 ]	66.8	92.7	48.2	–
ModTrack + Doppler perception	42.1	70.0	26.9	2.50
ModTrack + Oracle perception	76.9	99.1	60.9	1.09

Table 4: Comparison on RadarScenes dataset.

Panel of Figure 4 — Figure 4: IDF1 sensitivity analysis (WildTrack, joint mode). Each parameter is swept individually; all others held at defaults. Extended results are in Supp. Material D.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Time (ms)	% Total	FPS
Neural perception (Det. + Depth + Feat.)	443.9	81.0	2.25
Analytical core (Clust. + Fusion + PHD)	103.2	18.8	9.69
Other overhead	0.7	0.1	–
ModTrack (Total)	547.8 $\pm$ 85.0	100	1.83

Appendix

Table 5: Per-frame inference time breakdown on WildTrack (7 cameras, RTX 3090). The analytical core accounts for less than 20% of total latency; perception dominates runtime.

Figure 5: MOTA sensitivity to continuous clustering hyperparameters on WildTrack.

Figure 6: MOTP sensitivity to continuous clustering hyperparameters on WildTrack.

Figure 7: IDF1 sensitivity to integer tracking hyperparameters ( $N_{\mathrm{init}}$ , $\tau_{\mathrm{tent}}$ , $\tau_{\mathrm{confirmed}}$ , $K_{\max}$ ) on WildTrack.

Figure 8: MOTA sensitivity to integer tracking hyperparameters ( $N_{\mathrm{init}}$ , $\tau_{\mathrm{tent}}$ , $\tau_{\mathrm{confirmed}}$ , $K_{\max}$ ) on WildTrack.

Figure 9: MOTP sensitivity to integer tracking hyperparameters ( $N_{\mathrm{init}}$ , $\tau_{\mathrm{tent}}$ , $\tau_{\mathrm{confirmed}}$ , $K_{\max}$ ) on WildTrack.

Figure 10: MOTA sensitivity to continuous tracking hyperparameters on WildTrack.

Figure 11: MOTP sensitivity to continuous tracking hyperparameters on WildTrack.

Figure 12: IDF1 sensitivity to calibration hyperparameters ( $\sigma^{2}_{\mathrm{pose}}$ , $\sigma^{2}_{\mathrm{min}}$ , $\lambda_{Q}$ ) on WildTrack.

Figure 13: MOTA sensitivity to calibration hyperparameters ( $\sigma^{2}_{\mathrm{pose}}$ , $\sigma^{2}_{\mathrm{min}}$ , $\lambda_{Q}$ ) on WildTrack.

Figure 14: MOTP sensitivity to calibration hyperparameters ( $\sigma^{2}_{\mathrm{pose}}$ , $\sigma^{2}_{\mathrm{min}}$ , $\lambda_{Q}$ ) on WildTrack.

Stage	Symbol	WildTrack	MultiviewX	RadarScenes	Physical Interpretation
Camera Perception
	$\tau_{\text{yolo}}$	0.1	0.63	–	Min. detector confidence to retain a detection
	$\alpha_{fp}$	0.035	0.035	–	Footpoint depth uncertainty scaling factor
	$\alpha_{bbox}$	0.05	0.05	–	Bbox depth uncertainty scaling factor
	$\eta_{\mathrm{fp}}$	3	3	–	Footpoint precision boost (trust factor)
	$\sigma^{2}_{\min\mathrm{depth}}$	0.0001	0.0001	–	Min. depth variance

Appendix

Table 6: Hyperparameters: perception, projection, and clustering stages. Camera-specific parameters (top block) do not apply to RadarScenes.

Stage	Symbol	WildTrack	MultiviewX	RadarScenes	Physical Interpretation
Tracking (GM-PHD)
	$p_{S}$	0.99	0.99	0.99	Target survival probability
	$p_{D}$	0.90	0.95	0.90	Detection probability
	$\lambda_{\mathrm{assoc}}$	2.5	2.5	2.5	Identity score boost in assignment cost
	$\mu_{\mathrm{sem}}$	2.0	2.0	0.0	Appearance similarity boost (spatial-only for radar)
	$\lambda_{\mathrm{ReID}}$	3.0	3.0	0.0	Appearance boost for LOST-track re-identification

Appendix

Table 7: Hyperparameters: tracking, lifecycle, and motion model stages. RadarScenes uses the same GM-PHD tracking loop with dataset-specific tuning.

Dataset	$N$	Mean NEES	95% CI	1 $\sigma$ Cov.	2 $\sigma$ Cov.
WildTrack	879	0.46	1.87-2.13	88.4%	98.6%
MultiviewX	1317	0.38	1.89-2.11	91.9%	99.6%

Appendix

Table 8: Calibration statistics (matched track/GT pairs only).

Method	Changed Component	IDF1	MOTA
ModTrack (baseline)	None	95.5	91.4
+ RANSAC clustering	Clustering	92.8	90.8
+ Fixed covariance ( $R=0.16I$ )	Detection covariance	95.0	90.7
+ Simple mean fusion	Fusion	93.0	89.5

Appendix

Table 9: Ablation study isolating key components of ModTrack on WildTrack (joint setting).

Dataset	Method	IDF1 $\uparrow$	IDS $\downarrow$	GOSPA $\downarrow$
WildTrack	Vanilla	79.3	30	1.31
	ModTrack	95.5	2	1.20
MultiviewX	Vanilla	46.2	145	1.95
	ModTrack	85.2	14	1.83

Appendix

Table 10: Comparison of ModTrack against standard GM-PHD filter on WildTrack.

Multi-Object Tracking

Multi-object tracking (MOT) links detections across video frames to maintain object identities through motion, occlusion, and changing viewpoints.

See all 182 papers

Unified Multimodal Visual Tracking with Dual Mixture-of-Experts
May 5, 2026Lingyi Hong, Jinglun Li, Xinyu Zhou +6Cross-Modal Learning Multimodal Visual Object Tracking
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University
CalibFree: Self-Supervised View Feature Separation for Calibration-Free Multi-Camera Multi-Object Tracking
May 10, 2026Ruiqi Xian, Deep Patel, Iain Melvin +3Multi-View Learning Invariant Representation Learning
University of Maryland, College Park, MD, USA · NEC Laboratories America, Princeton, NJ, USA · University of North Carolina, Greenboro, NC, USA
AnyTrack: Unifying Visual Object Tracking with Any Modalities
Aug 7, 2026Hao Li, Yunzhi Zhuge, Wenning Hao +4Multimodal Visual Object Tracking Missing-Modality Learning
Army Engineering University of PLA Nanjing, China · Dalian University of Technology Dalian, China · National University of Defense Technology Nanjing, China

May 5, 2026J/K move · Enter open · S save

Unified Multimodal Visual Tracking with Dual Mixture-of-Experts

Lingyi Hong, Jinglun Li, Xinyu Zhou, Kaixun Jiang+5

Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University

Cross-Modal Learning Multimodal Visual Object Tracking Multimodal Robustness Unified Multimodal Models Mixture of Experts Visual Object Tracking+2

Open paper AskPDF arXiv

Multimodal visual object tracking can be divided into to several kinds of tasks (e.g. RGB and RGB+X tracking), based on the input modality. Existing methods often train separate models for each modality or rely on pretrained models to adapt to new modalities, which limits efficiency, scalability, and usability. Thus, we introduce OneTrackerV2, a unified multi-modal tracking framework that enables end-to-end training for any modality. We propose Meta Merger to embed multi-modal information into a unified space, allowing flexible modality fusion and robustness. We further introduce Dual Mixture-of-Experts (DMoE): T-MoE models spatio-temporal relations for tracking, while M-MoE embeds multi-modal knowledge, disentangling cross-modal dependencies and reducing feature conflicts. With a shared architecture, unified parameters, and a single end-to-end training, OneTrackerV2 achieves state-of-the-art performance across five RGB and RGB+X tracking tasks and 12 benchmarks, while maintaining high inference efficiency. Notably, even after model compression, OneTrackerV2 retains strong performance. Moreover, OneTrackerV2 demonstrates remarkable robustness under modality-missing scenarios.

Abstract

Multi-Object Tracking

Explore similar work

Unified Multimodal Visual Tracking with Dual Mixture-of-Experts

CalibFree: Self-Supervised View Feature Separation for Calibration-Free Multi-Camera Multi-Object Tracking

AnyTrack: Unifying Visual Object Tracking with Any Modalities