Multi-View Pedestrian Detection (MVPD) aims to detect pedestrians in the form of a bird's eye view map from multi-view images. Recent MVPD methods adopt a unified framework that projects 2D image features into a 3D world space and aggregates them into a single feature. Although they are effective, they struggle to generalize to unseen camera configurations during training due to two main issues. First, they are difficult to capture accurate visual geometry across views in unseen camera configurations. Second, they make detection models highly dependent on distortion patterns during training arising from their image feature projection. To address these, we leverage a visual geometric foundation model and propose MV2GF. This foundation model has exhibited strong generalization in capturing visual geometry across views and predicting accurate 3D attributes in diverse camera configurations. MV2GF fuses task-specific features with general-purpose geometric features extracted by the foundation model to effectively capture the visual geometry even in unseen camera configurations. Furthermore, MV2GF projects each pixel in the image features to an appropriate 3D location using 3D pointmaps predicted by the foundation model, preventing the detection model from depending on distortion patterns during training. Our experiments demonstrate the effectiveness of leveraging a visual geometric foundation model for MVPD and that MV2GF generalizes better than existing methods.
Figures & tables
Figure 1: (a) Previous MVPD methods. (b) Our MV2GF leverages a pretrained visual geometric foundation model. The red blocks are trainable, and the blue block is frozen.
Figure 2: (a) Overview of MV2GF. It consists of (b) TGF and (c) FPA. We set the red blocks to be trainable, and the gray blocks do not have trainable parameters.
Method
GMVD-D
MVPerception
Wildtrack
MODA
MODP
Prec.
Rec.
MODA
MODP
Prec.
Rec.
MODA
MODP
Prec.
Rec.
MVDet [ 17 ]
69.3
76.8
94.9
73.2
65.5
71.1
90.1
73.6
60.7
72.5
82.6
76.9
SHOT [ 39 ]
71.5
77.9
93.7
76.7
69.4
73.1
91.1
76.8
70.8
73.1
82.3
90.2
MVDeTr [ 16 ]
73.8
81.7
96.1
76.9
71.3
78.0
92.8
77.3
72.4
78.1
85.6
87.1
3DROM [ 34 ]
74.5
77.8
93.7
80.0
74.1
75.6
91.1
82.1
76.2
75.7
85.2
92.1
BoosterSHOT [ 20 ]
74.9
80.2
96.2
77.8
74.5
76.4
87.4
87.1
78.6
79.0
87.4
91.8
Table 1: Comparison with previous methods under settings where camera configurations during testing are different from those during training. All models are trained on GMVD-D training split.
Method
GMVD-S
MVPerception
Wildtrack
MODA
MODP
Prec.
Rec.
MODA
MODP
Prec.
Rec.
MODA
MODP
Prec.
Rec.
MVDet [ 17 ]
82.8
80.4
97.1
85.3
90.0
81.2
94.3
95.8
88.2
75.7
94.7
93.6
SHOT [ 39 ]
84.5
79.6
97.5
86.7
92.3
85.5
95.2
97.1
90.2
76.5
96.1
94.0
MVDeTr [ 16 ]
86.9
84.6
97.4
89.3
93.0
90.2
96.5
96.6
91.5
82.1
97.4
94.0
3DROM [ 34 ]
88.1
84.7
97.0
90.5
94.1
85.5
97.4
96.7
93.5
75.9
97.2
96.2
BoosterSHOT [ 20 ]
88.7
84.9
97.0
91.5
94.5
88.5
98.3
96.1
92.8
84.9
97.5
95.3
Table 2: Comparison with previous methods under settings where camera configurations during testing are identical to those during training. All models are trained on the training split of the same dataset used for testing.
Table 3: (a) Effect of TGF and FPA. (b) Effect of using features from ResNet and DA3 in TGF. (c) Effect of using features from multiple transformer blocks in DA3. (d) Effect of θz in FPA. (e) Effect of using features from a visual geometric foundation model. (f) Effect of the visual geometric foundation model choice. Default settings in MV2GF are marked in gray . All models are trained on GMVD-D training split and tested on its testing split.
MSMVD
Ours
N
MODA
MODP
Prec.
Rec.
MODA
MODP
Prec.
Rec.
6
82.7
81.1
95.2
87.1
85.9
82.0
96.8
88.9
5
79.9
79.0
93.0
86.3
84.1
80.5
96.8
87.0
4
70.5
78.7
88.6
80.9
80.0
80.3
95.5
84.0
3
61.4
72.6
85.0
74.5
73.8
78.0
93.7
79.1
2
34.0
63.0
76.0
49.8
57.6
73.3
86.3
67.6
Table 4: Effect of the number of input views N for MSMVD [ 50 ] and our MV2GF. Both models are trained on GMVD-D training split and tested on data comprising 6 views in GMVD-D testing split.
FPS
MVDet [ 17 ]
5.2
SHOT [ 39 ]
4.8
MVDeTr [ 16 ]
4.6
3DROM [ 34 ]
4.6
BoosterSHOT [ 20 ]
4.7
OmniOcc [ 3 ]
4.6
Table 5: Comparison of inference speed. These inference speeds are measured on data comprising 6 views in GMVD-D testing split using an Nvidia 80GB A100 GPU.
Pooling MODA MODP Prec. Rec. w/ loss weighting 84.2 83.1 96.6 87.0 w/o loss weighting 84.8 83.1 96.8 87.7
(e)
(f)
Table 6: (a) Effect of the offset prediction. (b) Effect of auxiliary loss Lviewn . (c) Effect of ResNet size in TGF. (d) Effect of the pooling choice in FPA. (e) Effect of using features from a trainable ResNet. (f) Effect of loss weighting. Default settings in MV2GF are marked in gray . All models are trained on GMVD-D training split and tested on its testing split.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
MODA
MODP
Prec.
Rec.
MSMVD [ 50 ]
91.1
86.6
98.0
93.0
Ours w/o MSMVD’s multi-scale BEV design
91.3
86.1
97.2
93.9
Ours w/ MSMVD’s multi-scale BEV design
91.8
87.2
98.1
94.0
Appendix
Table 7: Effect of MSMVD’s multi-scale BEV design on GMVD-S. All models are trained on GMVD-S training split and tested on GMVD-S testing split. Camera configurations during training are identical to those during testing. Default settings in MV2GF are marked in gray .
MODA
MODP
Prec.
Rec.
MVDet [ 17 ]
17.0
65.8
60.5
48.8
SHOT [ 39 ]
53.6
72.0
75.2
79.8
MVDeTr [ 16 ]
50.2
69.1
74.0
77.3
3DROM [ 34 ]
67.5
65.6
94.5
71.7
MVFP [ 2 ]
82.6
76.2
89.6
93.4
MSMVD [ 50 ]
82.0
78.0
88.7
94.0
Appendix
Table 8: Comparison with previous methods under the setting where models are trained on MultiviewX and tested on Wildtrack.
Figure 3: Visual comparison of detection results ( i.e . , predicted BEV maps) between our MV2GF and MSMVD [ 50 ] on GMVD-D. The green points represent detected pedestrians, the blue circles represent false positives, and the red circles represent false negatives.
Figure 4: Visual comparison of detection results ( i.e . , predicted BEV maps) between our MV2GF and MSMVD [ 50 ] on MVPerception. The green points represent detected pedestrians, the blue circles represent false positives, and the red circles represent false negatives.
Figure 5: Visual comparison of detection results ( i.e . , predicted BEV maps) between our MV2GF and MSMVD [ 50 ] on Wildtrack. The green points represent detected pedestrians, the blue circles represent false positives, and the red circles represent false negatives.
Figure 6: Visualization of 3D pointmaps predicted by DA3 [ 28 ] on GMVD.
Figure 7: Visualization of 3D pointmaps predicted by DA3 [ 28 ] on MVPerception.
Figure 8: Visualization of 3D pointmaps predicted by DA3 [ 28 ] on Wildtrack.
Figure 9: Visualization of (a) the ground truth BEV map and BEV features of (b) our MV2GF and (c) MVFP [ 2 ] on GMVD-D.
AICV Lab, EECS Department, University of Arkansas · University of Information Technology · Posts and Telecommunications Institute of Technology (PTIT), Vietnam
National Taiwan University Taipei, Taiwan · National Taiwan Normal University Taipei, Taiwan · Mitsubishi Electric Research Laboratories (MERL) Cambridge, MA, USA +3