Multi-view Optimal Transport for Multi-view Crowd Localization and Tracking

Kaiyi Zhang; Qi Zhang; Jixuan Chen; Antoni B. Chan; Hui Huang

Figure 1: The comparison of different optimal transport (OT) losses on the ground plane. The original OT uses Euclidean distance cost (E-OT) and treats all deviations from the ground-truth equally. MV-OT uses the view ray direction to the camera to change the cost function, ED-OT considers the camera distance in E-OT, while M-OT considers both the influence of the view ray direction and the distance to the camera.

Figure 2: The model architecture and the proposed M-MVOT loss for multi-view crowd localization. The model consists of feature extraction, projection, and multi-view fusion and decoding. In the proposed M-MVOT, each point’s transport cost C is calculated via the Mahalanobis distance instead of the common Euclidean distance under the closest camera , which is directed by the view ray and adjusted by the object-to-camera distance .

Figure 3: The comparison of different versions for the multi-view optimal transport, where the cost for each ground-truth point is calculated using the closest camera. E-MVOT considers each point equally regardless of the camera views, which is the same as E-OT for a single camera view. MV-MVOT replaces the Euclidean distance with the Mahalanobis distance, and the direction is guided by the view ray direction. ED-MVOT introduces the object-to-camera influence in the optimal transport cost of E-OT. M-MVOT considers both the view ray direction guidance and the object-to-camera distance adjustment in the optimal transport based on M-OT.

Figure 4: The overall architecture of MVOT-TBD for multi-view crowd tracking. MVOT-TBD integrates a ReID head into the localization framework and leverages the extracted ReID embeddings to perform two-stage association for tracking. The localization head is trained using our proposed M-MVOT loss.

Figure 5: The overall multi-view crowd tracking framework with MVOT-Track loss is as follows. The model takes two adjacent frames,

t-1

and

t

, as input. Both frames are processed by view feature extraction and ground-plane projection, following the same pipeline as multi-view localization, resulting in

\mathcal{F}_{t-1}

and

\mathcal{F}_{t}

. Next,

\mathcal{F}_{t-1}

is fed into MVDeformAttn to enable cross-view feature interaction, producing

\mathcal{F}_{t-1}^{decoded}

. This feature is then concatenated with

\mathcal{F}_{t}

and further processed by MVDeformAttn layers for temporal fusion, yielding temporally enriched features

\mathcal{F}_{t}^{decoded}

. The fused features are passed into two prediction heads: one for crowd localization at frame

t

, and the other for estimating motion offsets toward frame

t-1

. The predicted motion offsets are used to warp the localization results back to frame

t-1

. Finally, the MVOT-Track loss minimizes the optimal transport distance between the warped pseudo predictions and the ground truth at frame

t-1

, with additional regularization applied to the transport matrix

\mathbf{P}^{tra}

.

Figure 6: The illustration of temporal correspondence in tracking. The predicted result at frame

t

is warped to frame

t{-}1

using the predicted motion offsets. We then compute the correspondence between the warped frame

t{-}1

and the ground-truth annotations at frame

t{-}1

. The blue circles indicate individuals who remain in the scene across both frames. The yellow diamonds denote individuals who exit the scene at frame

t{-}1

, while the orange stars represent individuals who newly appear at frame

t

. The ground-truth correspondence matrix is a binary matrix, where an entry of ✓ indicates a valid identity match across frames, and ✗ indicates no correspondence.

Localization Variants
Method	Variance	Cam Selection
E-OT	$\sigma_{1}^{2}=\sigma_{2}^{2}=1$	without
MV-OT	$\sigma_{1}^{2}=1,\sigma_{2}^{2}=1.2$	without
ED-OT	$\sigma_{1}^{2}=\sigma_{2}^{2}=1/\exp(\alpha*\MinMaxNorm(d_{cam}))$	without
M-OT	$\sigma_{1}^{2}=1,\sigma_{2}^{2}=\exp(\alpha*\MinMaxNorm(d_{cam}))$	without
E-MVOT	$\sigma_{1}^{2}=\sigma_{2}^{2}=1$	with

Table 1: Overview of the proposed localization and tracking variants. Localization methods differ in the variance defined in the covariance matrix and whether distance-based camera selection is applied. Tracking variants extend the framework to temporal association modeling.

Method	MODA $\uparrow$	MODP $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1_score $\uparrow$
MVDet Hou et al. (2020)	14.2	59.3	85.0	17.3	28.7
SHOT Song et al. (2021a)	31.7	72.1	94.5	33.6	49.6
MVDeTr Hou and Zheng (2021)	24.9	79.6	98.1	25.4	40.4
3DROM Qiu et al. (2022)	20.1	74.2	84.1	23.7	37.0
EarlyBird Teepe et al. (2024a)	44.8	77.2	95.5	47.0	63.0
TrackTacular Teepe et al. (2024b)	42.1	77.2	95.8	44.1	60.4

Table 2: The multi-view crowd localization results on the CVCS dataset. The distance threshold is set to 0.5m in the world coordinate system for all methods. Our methods achieve the highest MODA and F1 score values.

Dataset	MultiviewX					Wildtrack
Method	MODA $\uparrow$	MODP $\uparrow$	P. $\uparrow$	R. $\uparrow$	F1. $\uparrow$	MODA $\uparrow$	MODP $\uparrow$	P. $\uparrow$	R. $\uparrow$	F1. $\uparrow$
RCNN Xu et al. (2016)	18.7	46.4	63.5	43.9	51.9	11.3	18.4	68	43	52.7
POM-CNN Fleuret et al. (2007)	-	-	-	-	-	23.2	30.5	75	55	63.5
DeepMCD Chavdarova and Fleuret (2017)	70.0	73.0	85.7	83.3	84.5	67.8	64.2	85	82	83.5
DeepOcc. Baqué et al. (2017)	75.2	54.7	97.8	80.2	88.1	74.1	53.8	95	80	86.9
Volumetric Iskakov et al. (2019)	84.2	80.3	97.5	86.4	91.6	88.6	73.8	95.3	93.2	94.2

Table 3: Comparison of the multi-view crowd localization performance on single-scene datasets Wildtrack and MultiviewX. The proposed method is the best on MultiviewX and comparable on Wildtrack. The distance threshold is 0.5m in the real world Hou et al. (2020) .

Figure 7: The predicted crowd occupancy maps of different methods on the 3 datasets CVCS, MultiviewX, and Wildtrack (zoom in for better view). The value of crowd occupancy maps indicates the probability of a person existing at each location.

Loss	Dir.	Dis.	MODA $\uparrow$	MODP $\uparrow$	P $\uparrow$	R $\uparrow$	F1. $\uparrow$
MSE	✗	✗	18.6	74.4	62.5	46.5	53.5
E-MVOT	✗	✗	42.8	72.4	76.7	61.4	68.2
MV-MVOT	✓	✗	44.5	73.1	79.2	60.3	68.5
ED-MVOT	✗	✓	43.8	73.2	77.9	61.2	68.5
M-MVOT	✓	✓	45.8	73.2	82.0	58.7	68.4

Table 4: The ablation study on the MVOT loss for multi-view crowd localization on the CVCS dataset, with other settings kept the same except for the training loss.

Figure 8: Comparison of E-MVOT, MV-MVOT, ED-MVOT, and M-MVOT. The blue triangle is the camera location. M-MVOT predicts fewer artifacts than E-MVOT, MV-MVOT, and ED-MVOT (see red boxes), demonstrating the effectiveness of incorporating distance adjustment and view ray direction.

$\alpha$	MODA $\uparrow$	MODP $\uparrow$	P. $\uparrow$	R. $\uparrow$	F1. $\uparrow$	$\tau$	MODA $\uparrow$	MODP $\uparrow$	P. $\uparrow$	R. $\uparrow$	F1. $\uparrow$
0	96.3	85.2	98.1	98.1	98.1	1	95.2	90.2	99.1	96.1	97.6
0.05	96.7	86.1	98.8	97.9	98.3	10	96.1	86.5	99.2	96.9	98.0
0.2	96.7	84.3	98.6	98.1	98.3	20	96.7	86.1	98.8	97.9	98.3
0.6	96.3	84.5	98.2	98.1	98.1	30	95.2	83.8	97.9	97.3	97.6

Table 5: The ablation study on

\alpha

and

\tau

of M-MVOT loss on the MultiviewX dataset. The best results are achieved by

\alpha=0.05

and

\tau=20

.

$\sigma_{1}^{2}:\sigma_{2}^{2}$	MODA $\uparrow$	MODP $\uparrow$	P. $\uparrow$	R. $\uparrow$	F1. $\uparrow$
1:1	96.3	85.2	98.1	98.1	98.1
1.2:1	96.5	84.6	98.3	98.2	98.2
1.5:1	96.2	85.2	98.2	98.0	98.1
2:1	96.3	86.1	98.5	97.8	98.1
1:1.2	96.5	85.6	98.3	98.2	98.2
1:1.5	96.5	85.2	98.5	97.9	98.2

Table 6: The ablation study on

\sigma_{1}^{2}:\sigma_{2}^{2}

of MV-MVOT loss on the MultiviewX dataset. The best results of fixed ratios are achieved by

\sigma_{1}^{2}:\sigma_{2}^{2}

= 1:1.2, but it is still lower than M-MVOT’s performance.

Strategy	MODA $\uparrow$	MODP $\uparrow$	P. $\uparrow$	R. $\uparrow$	F1. $\uparrow$
Average	96.5	85.2	98.7	97.8	98.2
Soft	96.5	86.0	98.7	97.8	98.2
Dist-based Sel.	96.7	86.1	98.8	97.9	98.3

Table 7: The ablation study on different strategies to fuse M-MVOT loss on the MultiviewX dataset. The best results are achieved by Dist-based Selection.

Comparison	MODA $\uparrow$	MODP $\uparrow$	P. $\uparrow$	R. $\uparrow$	F1. $\uparrow$
LCM	95.1	81.5	96.7	98.5	97.6
Multi-UOT	23.9	72.0	75.2	35.7	48.4
M-MVOT	96.7	86.1	98.8	97.9	98.3

Table 8: Comparisons with learnable covariance matrix (LCM), and multi-view fusion of single-view methods (Multi-UOT).

Backbone	Loss	MODA $\uparrow$	MODP $\uparrow$	P. $\uparrow$	R. $\uparrow$	F1. $\uparrow$
MVDeTr	MSE	93.9	90.3	98.0	95.9	96.9
	Focal	94.5	93.3	99.5	95.0	97.2
	M-MVOT	96.7	86.1	98.8	97.9	98.3
EarlyBird	MSE	92.6	88.0	97.7	94.8	96.2
	Focal	94.2	90.1	98.6	95.7	97.1
	M-MVOT	95.0	87.7	98.7	96.3	97.5

Table 9: The ablation study on the different backbones and losses on the MultiviewX dataset. The best results are achieved by MVDeTr with our M-MVOT loss.

Thres.	Method	MODA $\uparrow$	MODP $\uparrow$	P. $\uparrow$	R. $\uparrow$	F1. $\uparrow$
0.4m	SHOT Song et al. (2021a)	89.0	78.0	97.1	91.7	94.3
	MVDeTr Hou and Zheng (2021)	93.4	89.4	99.4	93.9	96.4
	M-MVOT	96.5	82.7	98.6	97.8	98.2
0.5m	SHOT Song et al. (2021a)	88.3	82.0	96.6	91.5	94.0
	MVDeTr Hou and Zheng (2021)	93.7	91.3	99.5	94.2	97.8
	M-MVOT	96.7	86.1	98.8	97.9	98.3

Table 10: Ablation study on different dist. thresholds.

Loss	Mem.(GB)	FLOPs(G)	Train(s)	Test(s)
MSE	7.446	1224.243	0.595	0.121
Focal	7.446	1224.243	0.583	0.121
E-MVOT	7.446	1224.243	0.612	0.121
M-MVOT	7.448	1224.243	0.642	0.121

Table 11: Computation cost comparison among different multi-view localization loss functions.

Method	Mem.(GB)	FLOPs(G)	Tr.(s)	Te.(s)
MVDet Hou et al. (2020)	12.633	2032.899	0.309	0.274
MVDeTr Hou and Zheng (2021)	19.430	1269.489	0.259	0.225
SHOT Song et al. (2021a)	6.070	2040.674	0.412	0.308
3DROM Qiu et al. (2022)	14.387	4017.931	0.609	0.235
EarlyBird Teepe et al. (2024a)	5.880	1074.787	0.314	0.064
TrackTacular Teepe et al. (2024b)	9.838	1221.315	0.298	0.068

Table 12: Computation cost comparison among different multi-view localization methods.

Tracking Method	IDF1 $\uparrow$	MOTA $\uparrow$	MOTP $\uparrow$	MT $\uparrow$	ML $\downarrow$
MVDet-TBD Hou et al. (2020)	65.2	69.0	81.2	44.7	7.9
SHOT-TBD Song et al. (2021a)	74.9	80.7	83.1	67.1	6.6
MVDeTr-TBD Hou and Zheng (2021)	74.4	85.3	85.5	78.9	3.9
3DROM-TBD Qiu et al. (2022)	74.5	86.4	83.1	76.3	3.9
MVOT-TBD (Ours)	81.3	90.9	84.3	89.5	1.3
REMP Dakic et al. (2025)	–	81.0	85.8	–	–

Table 13: Comparison of the multi-view crowd tracking performance on MultiviewX. MVOT-Track achieves the best performance.

Tracking Method	IDF1 $\uparrow$	MOTA $\uparrow$	MOTP $\uparrow$	MT $\uparrow$	ML $\downarrow$
MVDet-TBD Hou et al. (2020)	85.5	81.5	83.3	53.7	17.1
SHOT-TBD Song et al. (2021a)	89.3	86.0	85.6	68.3	12.2
MVDeTr-TBD Hou and Zheng (2021)	91.1	89.9	88.3	75.6	4.9
3DROM-TBD Qiu et al. (2022)	91.4	89.0	83.8	70.7	7.3
MVOT-TBD (Ours)	92.4	90.4	87.7	87.8	4.9
KSP-DO Chavdarova et al. (2018)	73.2	69.6	61.5	28.7	25.1

Table 14: Comparison of multi-view crowd tracking performance on the Wildtrack dataset. † results recomputed by EarlyBird Teepe et al. (2024a) . ∗ denotes results reproduced by us for MVTrajecter Yamane et al. (2025a) , as no official implementation is publicly available.

Figure 9: Qualitative comparison of MVOT-TBD with the tracking variants of multi-view localization methods on Wildtrack and MultiviewX (zoom in for better view). Colored trajectories indicate IDTPs, gray dashed trajectories indicate IDFPs, and red boxes highlight regions where MVOT-TBD produces fewer unmatched and fragmented trajectories.

Figure 10: Qualitative comparison of MVOT-Track with state-of-the-art multi-view tracking methods on Wildtrack and MultiviewX (zoom in for better view). Colored trajectories indicate IDTPs, gray dashed trajectories indicate IDFPs, and red boxes highlight challenging regions where MVOT-Track recovers more complete trajectories with fewer tracking errors.

Figure 11: Cross-view tracking visualizations on MultiviewX (zoom in for better view). Each example shows the synchronized camera views in which the target pedestrian is visible, together with the corresponding BEV trajectories. Predictions associated with the target GT use the same color, associations to other GTs use their corresponding colors, and false positives are shown in gray. The numbers beside the BEV nodes indicate the associated GT IDs, and the current frame is marked in the BEV view.

$\gamma$	IDF1 $\uparrow$	MOTA $\uparrow$	MOTP $\uparrow$	MT $\uparrow$	ML $\downarrow$
0	95.1	90.1	87.1	85.4	4.9
5	95.0	90.2	87.8	87.8	4.9
10	95.4	91.1	87.6	87.8	4.9
20	95.0	90.5	87.3	85.4	4.9

Table 15: Ablation study on

\gamma

and

\psi

of MVOT-Track on the Wildtrack dataset. The best results are achieved when

\gamma=10

and

\psi=1

.

Loss	IDF1 $\uparrow$	MOTA $\uparrow$	MOTP $\uparrow$	MT $\uparrow$	ML $\downarrow$
Wildtrack
$L_{1}$	95.1	89.7	86.9	87.8	4.9
MVOT-Track + $L_{1}$ (Eq. 18 )	95.4	91.1	87.6	87.8	4.9
MultiviewX
$L_{1}$	85.2	92.7	85.3	89.5	2.6
MVOT-Track + $L_{1}$ (Eq. 18 )	86.4	93.1	85.1	93.4	1.3

Table 16: Ablation study on different tracking loss settings on the Wildtrack and MultiviewX datasets. Incorporating the proposed MVOT-Track loss yields clear improvements on both datasets.

Tracking performance
#Frames	IDF1 $\uparrow$	MOTA $\uparrow$	MOTP $\uparrow$	MT $\uparrow$	ML $\downarrow$
1	94.4	89.5	87.9	82.9	4.9
2	95.4	91.1	87.6	87.8	4.9
3	94.6	89.6	87.4	85.3	4.9
4	94.7	89.3	87.9	85.3	4.9
Computational cost

Table 17: Ablation study on the number of temporal frames on Wildtrack. Two frames achieve the best trade-off between tracking performance and computational cost.

# Views	Camera IDs	IDF1 $\uparrow$	MOTA $\uparrow$	MOTP $\uparrow$	MT $\uparrow$	ML $\downarrow$
7	1,2,3,4,5,6,7	95.4	91.1	87.6	87.8	4.9
6	1,2,3,5,6,7	95.4	91.4	86.6	82.9	4.9
5	2,3,5,6,7	94.5	89.4	86.4	82.9	7.3
4	2,5,6,7	91.6	87.8	84.9	78.0	7.3
3	2,6,7	93.3	85.2	84.6	78.0	9.7

Table 18: Ablation study on the number of camera views on the Wildtrack dataset. The model is retrained using different camera subsets.

Method	Mem.(GB)	FLOPs(G)	Tr.(s)	Te.(s)
ReST Cheng et al. (2023)	1.908	134.958	0.064	0.295
MVTr Yang et al. (2024)	16.101	1269.498	0.751	0.142
MVFlow Engilberge et al. (2023)	8.717	1055.347	0.424	0.075
EarlyBird Teepe et al. (2024a)	6.737	1220.915	0.477	0.150
TrackTacular Teepe et al. (2024b)	10.828	2031.936	0.728	0.609
MVOT-TBD	22.890	3062.920	0.775	0.465

Table 19: Computation cost comparison among different multi-view tracking methods.

Multi-view Optimal Transport for Multi-view Crowd Localization and Tracking

Abstract

Figures & tables

Multi-Object Tracking

Multi-view Crowd Tracking Transformer with View-Ground Interactions Under Large Real-World Scenes

SynMVCrowd: A Large Synthetic Benchmark for Multi-view Crowd Counting and Localization

ModTrack: Sensor-Agnostic Multi-View Tracking via Identity-Informed PHD Filtering with Covariance Propagation

Abstract

Multi-Object Tracking

Explore similar work

Multi-view Crowd Tracking Transformer with View-Ground Interactions Under Large Real-World Scenes

SynMVCrowd: A Large Synthetic Benchmark for Multi-view Crowd Counting and Localization

ModTrack: Sensor-Agnostic Multi-View Tracking via Identity-Informed PHD Filtering with Covariance Propagation