Existing multi-view crowd counting and localization methods are evaluated under relatively small scenes with limited crowd numbers, camera views, and frames. This makes the evaluation and comparison of existing methods impractical, as small datasets are easily overfit by these methods. To avoid these issues, 3DROM proposes a data augmentation method. Instead, in this paper, we propose a large synthetic benchmark, SynMVCrowd, for more practical evaluation and comparison of multi-view crowd counting and localization tasks. The SynMVCrowd benchmark consists of 50 synthetic scenes with a large number of multi-view frames and camera views and a much larger crowd number (up to 1000), which is more suitable for large-scene multi-view crowd vision tasks. Besides, we propose strong multi-view crowd localization and counting baselines that outperform all comparison methods on the new SynMVCrowd benchmark. Moreover, we prove that better domain transferring multi-view and single-image counting performance could be achieved with the aid of the benchmark on novel new real scenes. As a result, the proposed benchmark could advance the research for multi-view and single-image crowd counting and localization to more practical applications. The codes and datasets are here: https://github.com/zqyq/SynMVCrowd.
Figures & tables
Figure 1: The visualization of the proposed SynMVCrowd dataset: the type of scene, time stamps, and the weather condition.
Figure 2: The whole process of the multi-view crowd image generation, including scene setup, character setup, and the scene synthesis.
Figure 3: The two pie charts respectively illustrate the distribution of time stamps and weather conditions in SynMVCrowd. On the right side, each label denotes a time of 24 hours.
Figure 4: The action catalog of SynMVCrowd, consisting of various human poses in daily life.
Figure 5: The frame numbers of crowd count distributions of SynMVCrowd.
Figure 6: The scene examples of SynMVCrowd. The orange box contains a perspective from the training set. The green and blue boxes contain a perspective from the validation and test sets.
Dataset
Type
Scenes
Size
Cameras
Frames
Average
Counting Statistics
Resolution
Total
Min
Avg
Max
PETS2009 Ferryman and Shahrokni (2009)
Real
1
-
3
1,899
576 × 768
-
20
-
40
DukeMTMC Ristani et al. (2016)
Real
1
-
4
989
1080 × 1920
-
10
-
30
Wildtrack Chavdarova et al. (2018)
Real
1
12 × 36
7
400
1080 × 1920
-
-
20
-
MultiviewX Hou et al. (2020)
Syn
1
16 × 25
6
400
1080 × 1920
-
-
40
-
CityStreet Zhang and Chan (2019)
Real
1
58 × 72
3
500
1520 × 2704
64K
70
128
150
Table 1: The statistical comparison of the proposed SynMVCrowd and six existing multi-view crowd datasets.
Figure 7: The dataset samples of scale variations.
Dataset
Type
Year
Average
Number
Counting Statistics
Resolution
of Images
Total
Min
Avg
Max
USCD Chan et al. (2008)
Real
2008
158 × 238
2,000
50K
11
25
46
Mall Chen et al. (2012)
Real
2012
240 × 320
2,000
62K
13
31
53
UCF_CC_50 Idrees et al. (2013)
Real
2013
2101 × 2888
50
64K
94
1,280
4,543
WorldExpo’10 Zhang et al. (2015)
Real
2015
576 × 720
3,980
200K
1
50
253
ShanghaiTech Part A Zhang et al. (2016)
Real
2016
589 × 868
482
242K
33
501
3,139
Table 2: The statistic comparison of the proposed SynMVCrowd and nine existing single-image crowd datasets.
Figure 8: The pipeline of multi-view localization and counting baseline model: single-image feature extraction, spatial feature selection, multi-view feature projection and fusion, and multi-view decoding with MSE or Optimal Transport (OT) loss.
Figure 9: The visualization results of multi-view crowd localization: the first row is the camera view input, and the second row is the corresponding ground truth (GT) and prediction results.
Method
MODA ↑
MODP ↑
Precision ↑
Recall ↑
F1_score ↑
MVDet Hou et al. (2020)
27.0
52.2
72.2
43.9
54.6
SHOT Song et al. (2021)
32.5
52.6
74.5
49.3
59.3
MVDeTr Hou and Zheng (2021)
35.6
69.7
95.4
37.4
53.7
3DROM Qiu et al. (2022)
24.2
59.2
86.1
28.8
43.2
SVCW Zhang et al. (2024)
35.8
55.6
75.8
51.7
61.4
MVOT Zhang et al. (2024)
45.5
66.3
83.4
56.9
67.6
Table 3: The results of multi-view crowd localization methods on the SynMVCrowd dataset. Our baseline method, Baseline (OT) outperforms all comparisons.
Dataset
Wildtrack
MultiviewX
Method
MA.
MP.
P.
R.
F1.
MA.
MP.
P.
R.
F1.
RCNN Xu et al. (2016)
11.3
18.4
68
43
52.7
18.7
46.4
63.5
43.9
51.9
POM-CNN Fleuret et al. (2007)
23.2
30.5
75
55
63.5
-
-
-
-
-
DeepMCD Chavdarova and Fleuret (2017)
67.8
64.2
85
82
83.5
70.0
73.0
85.7
83.3
84.5
DeepOcc. Baqué et al. (2017)
74.1
53.8
95
80
86.9
75.2
54.7
97.8
80.2
88.1
Volumetric Iskakov et al. (2019)
88.6
73.8
95.3
93.2
94.2
84.2
80.3
97.5
86.4
91.6
Table 4: Comparison of the multi-view people detection performance on Wildtrack and MultiviewX using 5 metrics. All comparison methods train and test on Wildtrack or MultiviewX (single scene), while ours are trained on SynMVCrowd and finetuned on Wildtrack or MultiviewX or with the domain adaptation technique.
Figure 10: The visualization results of multi-view crowd counting: the first row is the camera view input, and the second is the corresponding ground truth (GT) and prediction results.
Method
MAE ↓
NAE ↓
MSE ↓
MVMS Zhang and Chan (2020)
81.51
0.173
102.74
MVMSR Zhang and Chan (2020)
70.60
0.143
93.26
CVCS backbone Zhang et al. (2021)
72.50
0.154
93.99
CVCS Zhang et al. (2021)
67.74
0.139
89.31
3DCounting Zhang and Chan (2020)
74.63
0.171
101.30
CountFormer Mo et al. (2024)
56.30
0.117
79.30
Table 5: The results of multi-view crowd counting methods on the SynMVCrowd dataset.
Model
MAE ↓
MSE ↓
NAE ↓
MODA ↑
MODP ↑
Precision ↑
Recall ↑
F1_score ↑
VGG19
59.04
83.00
0.155
43.8
69.6
88.4
51.6
65.0
ResNet18
50.20
68.19
0.104
49.6
70.2
88.6
57.0
69.4
Transformer
66.23
92.84
0.211
39.5
62.1
80.1
48.3
60.3
Table 6: The multi-view counting and localization performance comparison of using different feature extraction backbones in the Baseline (OT) model, which is trained on the SynMVCrowd dataset with 5 views as input.
Models
test views
MAE ↓
MSE ↓
NAE ↓
CountFormer
3
47.71
65.24
0.151
5
56.30
79.30
0.117
7
71.47
95.95
0.158
9
77.82
102.96
0.164
Baseline (OT)
3
57.97
69.00
0.137
5
50.20
68.19
0.104
Table 7: Testing the baseline model on the multi-view counting task under different camera view numbers.
Loss
Module
MAE ↓
MSE ↓
NAE ↓
MODA ↑
MODP ↑
Precision ↑
Recall ↑
F1_score ↑
MSE
Without
74.33
97.52
0.236
31.9
74.8
94.1
34.0
49.9
With
68.79
92.04
0.158
34.6
74.5
92.9
37.4
53.4
OT
Without
52.94
78.21
0.105
46.7
69.7
87.9
54.1
67.0
With
50.20
68.19
0.104
49.6
70.2
88.6
57.0
69.4
Table 8: Performance Comparison of baseline model with/without spatial feature selection module using MSE or OT loss.
Pitch
MAE ↓
MSE ↓
NAE ↓
MODA ↑
MODP ↑
Precision ↑
Recall ↑
F1_score ↑
[−90,−60)
48.19
67.00
0.100
46.0
64.9
85.0
55.9
67.4
[−60,−30)
49.62
68.29
0.105
45.6
67.8
84.8
55.6
67.2
[−30,0]
55.97
76.03
0.118
42.9
66.7
83.4
53.6
65.2
Table 9: The performance comparison of the different pitch ranges.
Methods
Parameters (MB)
FLOPs (G)
Memory (MB)
MVDet Hou et al. (2020)
30.10
910.44
114.85
SHOT Song et al. (2021)
19.14
1230.24
73.04
MVDeTr Hou and Zheng (2021)
12.78
148.30
48.78
3DROM Zhang and Chan (2020)
32.46
1012.36
123.86
SVCW Zhang et al. (2024)
25.71
3413.13
98.12
MVOT Zhang et al. (2024)
18.59
349.73
70.95
Table 10: Computation statistic comparison of different methods.
Dataset
CityStreet
PETS2009
Training
Method
MAE ↓
NAE ↓
MSE ↓
MAE ↓
NAE ↓
MSE ↓
Single-scene
Dmap_wtd Ryan et al. (2014)
11.10
0.121
-
7.51
0.261
-
Dect+ReID Zhang and Chan (2019)
27.60
0.385
-
9.41
0.289
-
LateFusion Zhang and Chan (2019)
8.12
0.097
-
3.92
0.138
-
EarlyFusion Zhang and Chan (2019)
8.10
0.096
-
5.43
0.199
-
MVMS (FCN-7) Zhang and Chan (2019)
8.01
0.096
10.05
3.49
0.124
4.83
Table 11: The multi-view crowd counting domain adaptation (DA) results on the CityStreet dataset and the PETS2009 dataset.
Method
MAE ↓
MSE ↓
Precision ↑
Recall ↑
F1_score ↑
CSRNet Li et al. (2018)
21.84
36.58
50.77
69.59
58.71
DM-Count Wang et al. (2020)
20.88
34.72
65.53
67.31
66.41
NoisyCC Wan and Chan (2020)
26.19
43.23
33.65
48.28
39.66
P2PNet Song et al. (2021)
61.74
115.03
80.59
67.62
73.54
GLoss Wan et al. (2021)
21.37
40.36
79.37
70.63
74.75
CLTR Liang et al. (2022)
23.86
38.53
85.47
66.02
74.50
Table 12: The performance of single-image crowd counting and localization SOTA methods ( with code available ) on the SynMVCrowd dataset. We report all methods’ single-image counting and localization performance.
Method
Sparse (0-399)
Medium (400-699)
Congested (700-1000)
MAE ↓
MSE ↓
MAE ↓
MSE ↓
MAE ↓
MSE ↓
CSRNet Li et al. (2018)
14.92
23.69
27.06
42.75
45.17
64.85
DMCount Wang et al. (2020)
18.23
29.76
25.00
41.00
35.34
56.58
NoisyCC Wan and Chan (2020)
20.45
31.01
29.78
43.77
47.61
68.95
P2PNet Song et al. (2021)
27.77
35.15
45.26
66.01
79.96
92.04
GLoss Wan et al. (2021)
17.68
27.26
26.49
42.33
37.74
57.37
Table 13: The results on detailed evaluation granularity. We divide the scenes into 3 types regarding crowd numbers: ‘Sparse (0-399)’, ‘Medium (400-699)’, and ‘Congested (700-1000)’, and evaluate methods accordingly.
Figure 11: The visualization results of single-image crowd counting and localization.
Figure 12: Architecture of the domain adaptation network for single-image crowd counting. The model takes synthetic (source) and real (target) images as input. Features are encoded and selected, then processed through two branches: 1) A decoder producing density maps, supervised by MSE loss against ground truth; 2) A domain predictor distinguishing source from target features using cross-entropy (CE) loss. This adversarial training aligns feature distributions across domains to improve generalization.
Single-image baseline
Single-image feature extractor
Layer
Feature Extractor
Layer
UpSampling
Single-image decoder
Layer
Filter
conv 1
512×256×3×3 , relu
Table 14: The single-image baseline for domain adaptation. The single-image baseline includes a single-image feature extractor, a single-image decoder, and a discriminator. The Filter dimensions are input channels, output channels, and filter size ( w×h ).
Training
Dataset
SHA
SHB
Method
MAE ↓
MSE ↓
MAE ↓
MSE ↓
Single-scene
MCNN Zhang et al. (2016)
110.20
173.20
26.40
41.30
CSRNet Li et al. (2018)
68.20
115.00
10.60
16.00
DM-Count Wang et al. (2020)
59.70
95.70
7.40
11.80
GLoss Wan et al. (2021)
61.30
95.40
7.30
11.70
GauNet Cheng et al. (2022)
54.80
89.10
6.20
9.90
Table 15: The single-image crowd counting domain adaptation results on the ShanghaiTech A and B dataset.
Figure 13: The domain adaptation visualization results of single-image crowd counting on ShanghaiTech B and ShanghaiTech A.
Datasets
UCF-CC-50
MALL
Method
MAE ↓
NAE ↓
MSE ↓
MAE ↓
NAE ↓
MSE ↓
Crowd-CNN Zhang et al. (2015)
466.90
-
498.60
1.83
-
2.76
MCNN Zhang et al. (2016)
377.60
-
509.10
2.74
-
13.50
CSRNet Li et al. (2018)
266.10
-
397.50
-
-
-
DecideNet Liu et al. (2018)
361.80
-
493.50
1.53
-
1.89
IG-CNN Sam et al. (2018)
419.60
-
541.80
2.39
-
9.14
Table 16: The single-image crowd counting domain adaptation results on UCF-CC-50 and Mall datasets. Comparisons are all trained and tested on the same datasets, while ours is trained on SynMVCrowd and tested on UCF-CC-50 or Mall.
Multi-view crowd tracking estimates each person's tracking trajectories on the ground of the scene. Recent research works mainly rely on CNNs-based multi-view crowd tracking architectures, and most of them are evaluated and compared on relatively small datasets, such as Wildtrack and MultiviewX. Since these two datasets are collected in small scenes and only contain tens of frames in the evaluation stage, it is difficult for the current methods to be applied to real-world applications where scene size and occlusion are more complicated. In this paper, we propose a Transformer-based multi-view crowd tracking model, MVTrackTrans, which adopts interactions between camera views and the ground plane for enhanced multi-view tracking performance. Besides, for better evaluation, we collect and label two large real-world multi-view tracking datasets, MVCrowdTrack and CityTrack, which contain a much larger scene size over a longer time period. Compared with existing methods on the two large and new datasets, the proposed MVTrackTrans model achieves better performance, demonstrating the advantages of the model design in dealing with large scenes. We believe the proposed datasets and model will push the frontiers of the task to more practical scenarios, and the datasets and code are available at: https://github.com/zqyq/MVTrackTrans.
Qi Zhang, Jixuan Chen, Kaiyi Zhang +3
College of Computer Science and Software Engineering, Shenzhen University, China · Department of Computer Science, City University of Hong Kong, China
Multi-view crowd localization and tracking aim to predict the ground-plane positions of all individuals in the scene and estimate their trajectories over time. Existing multi-view localization methods typically rely on Gaussian-smoothed density map supervision, whose inherent ambiguity in crowded regions may blur local peaks and degrade localization accuracy. In this work, we introduce a multi-view optimal transport framework that directly supervises point-level predictions on the ground plane, thereby avoiding the limitations of density map regression. Specifically, we propose a Mahalanobis distance-based multi-view optimal transport loss (M-MVOT), whose transport cost incorporates view ray direction and object-to-camera distance within a Mahalanobis formulation, with the direction guiding the ellipse orientation and the distance modulating the cost magnitude. For multi-view tracking, we first extend the proposed localization framework in a tracking-by-detection manner (MVOT-TBD). To further improve performance, we introduce a tracking-specific optimal transport loss (MVOT-Track) that explicitly supervises inter-frame association. Extensive experiments demonstrate that the proposed optimal transport-based framework consistently outperforms density map-based localization and tracking on multiple multi-view crowd datasets. Code is available at https://github.com/zqyq/MVOT-Track.
Kaiyi Zhang, Qi Zhang, Jixuan Chen +2
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), China · Visual Computing Research Center, College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China · Department of Computer Science, City University of Hong Kong, HK SAR, China
This paper constructs the first benchmark on semi-supervised multi-modal crowd counting. To lay the foundation for this unexplored task, we first formulate the semi-supervised multi-modal setting and a standardized protocol that specifies the labeled-unlabeled data partition across different labeled ratios. Next, to establish solid reference points, we carefully tailor a diverse set of representative baselines, including existing fully supervised multi-modal methods and semi-supervised single-modal methods. Then, we carefully evaluate their performance under our proposed benchmark. Codes and the data partition will be released on https://github.com/HenryCilence/Semi-supervised-Multimodal-Crowd-Counting.
Haoliang Meng, Xiaopeng Hong, Yabin Wang +1
Harbin Institute of Technology · Pengcheng Laboratory