Text-Assisted Event Stream Visual Place Recognition via Multi-Scale Spatiotemporal Optimal Transport
Organizations: School of Electronic and Information Engineering, Anhui University, Hefei, China · School of Computer Science and Technology, Anhui University, Hefei, China · School of Computer Science, Peking University, Beijing, China
Abstract
Benefiting from the high dynamic range, high temporal resolution, and low energy consumption of event cameras, low-altitude economic research leveraging event cameras has become a popular topic. Several existing attempts introduce natural-language descriptions into visual place recognition (VPR) to mitigate the sparsity of structural spatial information inherent to event cameras. Nevertheless, these methods still fail to fully address two core challenges: effectively encoding event streams and semantically aligning event streams with text. In this paper, we propose a multi-scale spatial-temporal optimal transport approach, MSTOT-EPR, to address these challenges. Specifically, given event streams and scene text information, we first adopt a visual encoder and a text encoder to extract respective feature representations. We then apply optimal-transport constraints to achieve cross-modal alignment. We then design a cross-modal bottleneck fusion module to further fuse bimodal features. Furthermore, we develop multi-scale spatial optimal transport and inter-frame temporal optimal transport modules to mine high-quality spatial-temporal event features for scene recognition. Extensive experiments on the NYC-Event-VPR and EPRBench datasets fully validate the effectiveness of our proposed framework for event-based VPR. The source code of this paper will be released upon acceptance.
Figures & tables
| Method | Source | NYC-Event-VPR-Event | NYC-Event-VPR-RGB | EPRBench | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | ||
| 01. Deep visual geo-loc Berton et al. (2022b) | CVPR’22 | 25.6 | 41.9 | 49.1 | 84.9 | 89.3 | 90.4 | 63.2 | 77.9 | 82.8 |
| 02. CosPlace Berton et al. (2022a) | CVPR’22 | 32.1 | 48.1 | 55.3 | 85.4 | 88.8 | 89.7 | 73.2 | 88.9 | 92.0 |
| 03. MixVPR Ali-Bey et al. (2023) | WACV’23 | 45.7 | 63.0 | 68.9 | 88.3 | 91.0 | 91.5 | 90.2 | 93.7 | 95.6 |
| 04. EigenPlaces Berton et al. (2023) | ICCV’23 | 28.5 | 43.4 | 51.2 | 81.4 | 91.7 | 93.9 | 81.1 | 86.0 | 89.0 |
| 05. R2Former Xu et al. (2023) | CVPR’23 | 61.2 | 75.0 | 78.5 | 86.3 | 88.9 | 89.9 | 80.0 | 94.8 | 96.7 |
| Key Modules | Evaluation Metrics | ||||
|---|---|---|---|---|---|
| Spa-OT | Temp-OT | BF | R1 (%) | R5 (%) | R10 (%) |
| 91.3 | 93.4 | 95.3 | |||
| 92.3 | 94.6 | 96.6 | |||
| 93.9 | 96.9 | 98.0 | |||
| 93.5 | 95.8 | 96.2 | |||
| 95.0 | 98.5 | 98.8 | |||
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Dimension | R@1 | R@5 | R@10 |
|---|---|---|---|
| 128 | 94.6 | 96.9 | 98.5 |
| 256 | 95.0 | 98.5 | 98.8 |
| 512 | 93.9 | 96.9 | 98.0 |
| 768 | 92.7 | 96.2 | 97.7 |
Visual Place Recognition
Visual Place Recognition matches camera observations to mapped places despite changes in viewpoint, lighting, weather, or time.
See all 90 papers