Multi-view Optimal Transport for Multi-view Crowd Localization and Tracking
Organizations: Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), China · Visual Computing Research Center, College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China · Department of Computer Science, City University of Hong Kong, HK SAR, China
Abstract
Multi-view crowd localization and tracking aim to predict the ground-plane positions of all individuals in the scene and estimate their trajectories over time. Existing multi-view localization methods typically rely on Gaussian-smoothed density map supervision, whose inherent ambiguity in crowded regions may blur local peaks and degrade localization accuracy. In this work, we introduce a multi-view optimal transport framework that directly supervises point-level predictions on the ground plane, thereby avoiding the limitations of density map regression. Specifically, we propose a Mahalanobis distance-based multi-view optimal transport loss (M-MVOT), whose transport cost incorporates view ray direction and object-to-camera distance within a Mahalanobis formulation, with the direction guiding the ellipse orientation and the distance modulating the cost magnitude. For multi-view tracking, we first extend the proposed localization framework in a tracking-by-detection manner (MVOT-TBD). To further improve performance, we introduce a tracking-specific optimal transport loss (MVOT-Track) that explicitly supervises inter-frame association. Extensive experiments demonstrate that the proposed optimal transport-based framework consistently outperforms density map-based localization and tracking on multiple multi-view crowd datasets. Code is available at https://github.com/zqyq/MVOT-Track.
Figures & tables
| Localization Variants | ||
| Method | Variance | Cam Selection |
| E-OT | without | |
| MV-OT | without | |
| ED-OT | without | |
| M-OT | without | |
| E-MVOT | with | |
| Method | MODA | MODP | Precision | Recall | F1_score |
| MVDet Hou et al. (2020) | 14.2 | 59.3 | 85.0 | 17.3 | 28.7 |
| SHOT Song et al. (2021a) | 31.7 | 72.1 | 94.5 | 33.6 | 49.6 |
| MVDeTr Hou and Zheng (2021) | 24.9 | 79.6 | 98.1 | 25.4 | 40.4 |
| 3DROM Qiu et al. (2022) | 20.1 | 74.2 | 84.1 | 23.7 | 37.0 |
| EarlyBird Teepe et al. (2024a) | 44.8 | 77.2 | 95.5 | 47.0 | 63.0 |
| TrackTacular Teepe et al. (2024b) | 42.1 | 77.2 | 95.8 | 44.1 | 60.4 |
| Dataset | MultiviewX | Wildtrack | ||||||||
| Method | MODA | MODP | P. | R. | F1. | MODA | MODP | P. | R. | F1. |
| RCNN Xu et al. (2016) | 18.7 | 46.4 | 63.5 | 43.9 | 51.9 | 11.3 | 18.4 | 68 | 43 | 52.7 |
| POM-CNN Fleuret et al. (2007) | - | - | - | - | - | 23.2 | 30.5 | 75 | 55 | 63.5 |
| DeepMCD Chavdarova and Fleuret (2017) | 70.0 | 73.0 | 85.7 | 83.3 | 84.5 | 67.8 | 64.2 | 85 | 82 | 83.5 |
| DeepOcc. Baqué et al. (2017) | 75.2 | 54.7 | 97.8 | 80.2 | 88.1 | 74.1 | 53.8 | 95 | 80 | 86.9 |
| Volumetric Iskakov et al. (2019) | 84.2 | 80.3 | 97.5 | 86.4 | 91.6 | 88.6 | 73.8 | 95.3 | 93.2 | 94.2 |
| Loss | Dir. | Dis. | MODA | MODP | P | R | F1. |
| MSE | ✗ | ✗ | 18.6 | 74.4 | 62.5 | 46.5 | 53.5 |
| E-MVOT | ✗ | ✗ | 42.8 | 72.4 | 76.7 | 61.4 | 68.2 |
| MV-MVOT | ✓ | ✗ | 44.5 | 73.1 | 79.2 | 60.3 | 68.5 |
| ED-MVOT | ✗ | ✓ | 43.8 | 73.2 | 77.9 | 61.2 | 68.5 |
| M-MVOT | ✓ | ✓ | 45.8 | 73.2 | 82.0 | 58.7 | 68.4 |
| MODA | MODP | P. | R. | F1. | MODA | MODP | P. | R. | F1. | ||
| 0 | 96.3 | 85.2 | 98.1 | 98.1 | 98.1 | 1 | 95.2 | 90.2 | 99.1 | 96.1 | 97.6 |
| 0.05 | 96.7 | 86.1 | 98.8 | 97.9 | 98.3 | 10 | 96.1 | 86.5 | 99.2 | 96.9 | 98.0 |
| 0.2 | 96.7 | 84.3 | 98.6 | 98.1 | 98.3 | 20 | 96.7 | 86.1 | 98.8 | 97.9 | 98.3 |
| 0.6 | 96.3 | 84.5 | 98.2 | 98.1 | 98.1 | 30 | 95.2 | 83.8 | 97.9 | 97.3 | 97.6 |
| MODA | MODP | P. | R. | F1. | |
| 1:1 | 96.3 | 85.2 | 98.1 | 98.1 | 98.1 |
| 1.2:1 | 96.5 | 84.6 | 98.3 | 98.2 | 98.2 |
| 1.5:1 | 96.2 | 85.2 | 98.2 | 98.0 | 98.1 |
| 2:1 | 96.3 | 86.1 | 98.5 | 97.8 | 98.1 |
| 1:1.2 | 96.5 | 85.6 | 98.3 | 98.2 | 98.2 |
| 1:1.5 | 96.5 | 85.2 | 98.5 | 97.9 | 98.2 |
| Strategy | MODA | MODP | P. | R. | F1. |
| Average | 96.5 | 85.2 | 98.7 | 97.8 | 98.2 |
| Soft | 96.5 | 86.0 | 98.7 | 97.8 | 98.2 |
| Dist-based Sel. | 96.7 | 86.1 | 98.8 | 97.9 | 98.3 |
| Comparison | MODA | MODP | P. | R. | F1. |
| LCM | 95.1 | 81.5 | 96.7 | 98.5 | 97.6 |
| Multi-UOT | 23.9 | 72.0 | 75.2 | 35.7 | 48.4 |
| M-MVOT | 96.7 | 86.1 | 98.8 | 97.9 | 98.3 |
| Backbone | Loss | MODA | MODP | P. | R. | F1. |
| MVDeTr | MSE | 93.9 | 90.3 | 98.0 | 95.9 | 96.9 |
| Focal | 94.5 | 93.3 | 99.5 | 95.0 | 97.2 | |
| M-MVOT | 96.7 | 86.1 | 98.8 | 97.9 | 98.3 | |
| EarlyBird | MSE | 92.6 | 88.0 | 97.7 | 94.8 | 96.2 |
| Focal | 94.2 | 90.1 | 98.6 | 95.7 | 97.1 | |
| M-MVOT | 95.0 | 87.7 | 98.7 | 96.3 | 97.5 |
| Thres. | Method | MODA | MODP | P. | R. | F1. |
| 0.4m | SHOT Song et al. (2021a) | 89.0 | 78.0 | 97.1 | 91.7 | 94.3 |
| MVDeTr Hou and Zheng (2021) | 93.4 | 89.4 | 99.4 | 93.9 | 96.4 | |
| M-MVOT | 96.5 | 82.7 | 98.6 | 97.8 | 98.2 | |
| 0.5m | SHOT Song et al. (2021a) | 88.3 | 82.0 | 96.6 | 91.5 | 94.0 |
| MVDeTr Hou and Zheng (2021) | 93.7 | 91.3 | 99.5 | 94.2 | 97.8 | |
| M-MVOT | 96.7 | 86.1 | 98.8 | 97.9 | 98.3 |
| Loss | Mem.(GB) | FLOPs(G) | Train(s) | Test(s) |
| MSE | 7.446 | 1224.243 | 0.595 | 0.121 |
| Focal | 7.446 | 1224.243 | 0.583 | 0.121 |
| E-MVOT | 7.446 | 1224.243 | 0.612 | 0.121 |
| M-MVOT | 7.448 | 1224.243 | 0.642 | 0.121 |
| Method | Mem.(GB) | FLOPs(G) | Tr.(s) | Te.(s) |
| MVDet Hou et al. (2020) | 12.633 | 2032.899 | 0.309 | 0.274 |
| MVDeTr Hou and Zheng (2021) | 19.430 | 1269.489 | 0.259 | 0.225 |
| SHOT Song et al. (2021a) | 6.070 | 2040.674 | 0.412 | 0.308 |
| 3DROM Qiu et al. (2022) | 14.387 | 4017.931 | 0.609 | 0.235 |
| EarlyBird Teepe et al. (2024a) | 5.880 | 1074.787 | 0.314 | 0.064 |
| TrackTacular Teepe et al. (2024b) | 9.838 | 1221.315 | 0.298 | 0.068 |
| Tracking Method | IDF1 | MOTA | MOTP | MT | ML |
| MVDet-TBD Hou et al. (2020) | 65.2 | 69.0 | 81.2 | 44.7 | 7.9 |
| SHOT-TBD Song et al. (2021a) | 74.9 | 80.7 | 83.1 | 67.1 | 6.6 |
| MVDeTr-TBD Hou and Zheng (2021) | 74.4 | 85.3 | 85.5 | 78.9 | 3.9 |
| 3DROM-TBD Qiu et al. (2022) | 74.5 | 86.4 | 83.1 | 76.3 | 3.9 |
| MVOT-TBD (Ours) | 81.3 | 90.9 | 84.3 | 89.5 | 1.3 |
| REMP Dakic et al. (2025) | – | 81.0 | 85.8 | – | – |
| Tracking Method | IDF1 | MOTA | MOTP | MT | ML |
| MVDet-TBD Hou et al. (2020) | 85.5 | 81.5 | 83.3 | 53.7 | 17.1 |
| SHOT-TBD Song et al. (2021a) | 89.3 | 86.0 | 85.6 | 68.3 | 12.2 |
| MVDeTr-TBD Hou and Zheng (2021) | 91.1 | 89.9 | 88.3 | 75.6 | 4.9 |
| 3DROM-TBD Qiu et al. (2022) | 91.4 | 89.0 | 83.8 | 70.7 | 7.3 |
| MVOT-TBD (Ours) | 92.4 | 90.4 | 87.7 | 87.8 | 4.9 |
| KSP-DO Chavdarova et al. (2018) | 73.2 | 69.6 | 61.5 | 28.7 | 25.1 |
| IDF1 | MOTA | MOTP | MT | ML | |
| 0 | 95.1 | 90.1 | 87.1 | 85.4 | 4.9 |
| 5 | 95.0 | 90.2 | 87.8 | 87.8 | 4.9 |
| 10 | 95.4 | 91.1 | 87.6 | 87.8 | 4.9 |
| 20 | 95.0 | 90.5 | 87.3 | 85.4 | 4.9 |
| Loss | IDF1 | MOTA | MOTP | MT | ML |
| Wildtrack | |||||
| 95.1 | 89.7 | 86.9 | 87.8 | 4.9 | |
| MVOT-Track + (Eq. 18 ) | 95.4 | 91.1 | 87.6 | 87.8 | 4.9 |
| MultiviewX | |||||
| 85.2 | 92.7 | 85.3 | 89.5 | 2.6 | |
| MVOT-Track + (Eq. 18 ) | 86.4 | 93.1 | 85.1 | 93.4 | 1.3 |
| Tracking performance | |||||
| #Frames | IDF1 | MOTA | MOTP | MT | ML |
| 1 | 94.4 | 89.5 | 87.9 | 82.9 | 4.9 |
| 2 | 95.4 | 91.1 | 87.6 | 87.8 | 4.9 |
| 3 | 94.6 | 89.6 | 87.4 | 85.3 | 4.9 |
| 4 | 94.7 | 89.3 | 87.9 | 85.3 | 4.9 |
| Computational cost | |||||
| # Views | Camera IDs | IDF1 | MOTA | MOTP | MT | ML |
| 7 | 1,2,3,4,5,6,7 | 95.4 | 91.1 | 87.6 | 87.8 | 4.9 |
| 6 | 1,2,3,5,6,7 | 95.4 | 91.4 | 86.6 | 82.9 | 4.9 |
| 5 | 2,3,5,6,7 | 94.5 | 89.4 | 86.4 | 82.9 | 7.3 |
| 4 | 2,5,6,7 | 91.6 | 87.8 | 84.9 | 78.0 | 7.3 |
| 3 | 2,6,7 | 93.3 | 85.2 | 84.6 | 78.0 | 9.7 |
| Method | Mem.(GB) | FLOPs(G) | Tr.(s) | Te.(s) |
| ReST Cheng et al. (2023) | 1.908 | 134.958 | 0.064 | 0.295 |
| MVTr Yang et al. (2024) | 16.101 | 1269.498 | 0.751 | 0.142 |
| MVFlow Engilberge et al. (2023) | 8.717 | 1055.347 | 0.424 | 0.075 |
| EarlyBird Teepe et al. (2024a) | 6.737 | 1220.915 | 0.477 | 0.150 |
| TrackTacular Teepe et al. (2024b) | 10.828 | 2031.936 | 0.728 | 0.609 |
| MVOT-TBD | 22.890 | 3062.920 | 0.775 | 0.465 |
Multi-Object Tracking
Multi-object tracking (MOT) links detections across video frames to maintain object identities through motion, occlusion, and changing viewpoints.
See all 193 papers