Text-Assisted Event Stream Visual Place Recognition via Multi-Scale Spatiotemporal Optimal Transport
Authors: Lan Chen, Xingxing Xiong, Xiao Wang, Ziyan Zhang, Zikang Yan, Yuanhao Zheng, Bo Jiang, Yonghong Tian
Organizations: School of Electronic and Information Engineering, Anhui University, Hefei, China · School of Computer Science and Technology, Anhui University, Hefei, China · School of Computer Science, Peking University, Beijing, China
Benefiting from the high dynamic range, high temporal resolution, and low energy consumption of event cameras, low-altitude economic research leveraging event cameras has become a popular topic. Several existing attempts introduce natural-language descriptions into visual place recognition (VPR) to mitigate the sparsity of structural spatial information inherent to event cameras. Nevertheless, these methods still fail to fully address two core challenges: effectively encoding event streams and semantically aligning event streams with text. In this paper, we propose a multi-scale spatial-temporal optimal transport approach, MSTOT-EPR, to address these challenges. Specifically, given event streams and scene text information, we first adopt a visual encoder and a text encoder to extract respective feature representations. We then apply optimal-transport constraints to achieve cross-modal alignment. We then design a cross-modal bottleneck fusion module to further fuse bimodal features. Furthermore, we develop multi-scale spatial optimal transport and inter-frame temporal optimal transport modules to mine high-quality spatial-temporal event features for scene recognition. Extensive experiments on the NYC-Event-VPR and EPRBench datasets fully validate the effectiveness of our proposed framework for event-based VPR. The source code of this paper will be released upon acceptance.
Figures & tables
Figure 1: Comparison between conventional vision-only VPR and our event-text VPR paradigm.
Figure 2: An overview of our proposed text-assisted event stream visual place recognition via multi-scale spatial-temporal optimal transport, termed MSTOT-EPR. The framework integrates OT-based cross-modal alignment with bottleneck fusion, multi-scale spatial-aware OT, and temporal inter-frame OT, followed by SALAD-based feature aggregation.
Method
Source
NYC-Event-VPR-Event
NYC-Event-VPR-RGB
EPRBench
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
01. Deep visual geo-loc Berton et al. (2022b)
CVPR’22
25.6
41.9
49.1
84.9
89.3
90.4
63.2
77.9
82.8
02. CosPlace Berton et al. (2022a)
CVPR’22
32.1
48.1
55.3
85.4
88.8
89.7
73.2
88.9
92.0
03. MixVPR Ali-Bey et al. (2023)
WACV’23
45.7
63.0
68.9
88.3
91.0
91.5
90.2
93.7
95.6
04. EigenPlaces Berton et al. (2023)
ICCV’23
28.5
43.4
51.2
81.4
91.7
93.9
81.1
86.0
89.0
05. R2Former Xu et al. (2023)
CVPR’23
61.2
75.0
78.5
86.3
88.9
89.9
80.0
94.8
96.7
Table 1: Comparison with state-of-the-art visual place recognition methods on NYC-Event-VPR-Event, NYC-Event-VPR-RGB, and EPRBench. Results are reported as Recall@ K , where K∈{1,5,10} . The best result in each column is highlighted in bold.
Key Modules
Evaluation Metrics
Spa-OT
Temp-OT
BF
R1 (%) ↑
R5 (%) ↑
R10 (%) ↑
×
×
×
91.3
93.4
95.3
✓
×
×
92.3
94.6
96.6
✓
✓
×
93.9
96.9
98.0
✓
×
✓
93.5
95.8
96.2
✓
✓
✓
95.0
98.5
98.8
Table 2: Ablation study of the proposed components on the EPRBench dataset. Spatial optimal transport (Spa-OT), temporal optimal transport (Temp-OT), and bottleneck fusion (BF) are progressively introduced to evaluate their individual and complementary contributions.
Figure 3: Ablation studies on EPRBench: (a) model components, (b) anchor-frame index, (c) bottleneck dimension, and (d) alignment-loss weights.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Dimension
R@1
R@5
R@10
128
94.6
96.9
98.5
256
95.0
98.5
98.8
512
93.9
96.9
98.0
768
92.7
96.2
97.7
Appendix
Table 3: Analysis of bottleneck fusion dimension.
Table 7
Figure 4: Qualitative top-1 retrieval comparison on EPRBench. Each row shows an event-stream query and the corresponding top-1 retrieved reference image from MSTOT-EPR (ours) and competing methods. Green and red borders denote correct and incorrect retrievals, respectively. Our method retrieves the correct places more consistently under viewpoint changes, sparse event observations, and challenging scene appearances.
Event cameras are rapidly rising in popularity for robotic and computer vision tasks because their sparse activation delivers energy-efficient, high-dynamic-range, and fast sensing. Event cameras have been used in robotic navigation and localization tasks where positioning must occur in real time with sufficient accuracy. However, current event-based localization methods suffer from poor spatial understanding and are not viewpoint tolerant. In this paper, we address the problem of viewpoint-robust place recognition directly from event streams. We present EventGeM, a global-to-local feature fusion pipeline for event-based visual place recognition that combines whole-image feature detection to shortlist top candidates for 2D homography-based re-ranking with random sample consensus (RANSAC). We also contribute a regional generalized mean pooling (GeM) layer that learns to return the most relevant spatial features using per-row exponents to pool event streams into a compact global descriptor, trained on the NYC-Event-VPR dataset. These contributions overcome shortfalls in currently available event-based localization methods that fail to recognize similar places with large changes in viewpoint. To evaluate viewpoint-robust localization, we contribute a new event-based dataset that includes repeated traverses with a severe lateral shift. EventGeM improves absolute Recall@1 by 7 to 43 percentage points over the strongest baseline in each experiment. We also deploy EventGeM on a robotic platform, demonstrating real-time performance of our hierarchical pipeline. The code for EventGeM is available at https://github.com/AdamDHines/Event-GeM.
Adam D. Hines, Gokul B. Nair, Nicolás Marticorena +2
QUT Centre for Robotics, School of Electrical Engineering and Robotics, Queensland University of Technology, Brisbane, QLD 4000, Australia.
Event cameras provide several unique advantages over standard frame-based sensors, including high temporal resolution, low latency, and robustness to extreme lighting. However, existing learning-based approaches for event processing are typically confined to narrow, task-specific silos and lack the ability to generalize across modalities. We address this gap with REALM, a cross-modal framework that learns an RGB- and Event-Aligned Latent Manifold by projecting event representations into the pretrained latent space of RGB foundation models. Instead of task-specific training, we leverage low-rank adaptation (LoRA) to bridge the modality gap, effectively unlocking the geometric and semantic priors of frozen RGB backbones for asynchronous event streams. We demonstrate that REALM effectively maps events into the ViT-based foundation latent space. Our method performs downstream tasks, such as depth estimation and semantic segmentation, by simply transferring linear heads trained on the RGB teacher. Most significantly, REALM enables the direct, zero-shot application of complex, frozen image-trained decoders, such as MASt3R, to raw event data. We demonstrate state-of-the-art performance in wide-baseline feature matching, significantly outperforming specialized architectures. Code and models are available at https://papers.starslab.ca/realm/.
Vincenzo Polizzi, David B. Lindell, Jonathan Kelly
University of Toronto, Robotics Institute · University of Toronto, Department of Computer Science
Robot localization is an ongoing challenge that demands mapping and positioning systems that are tolerant to viewpoint change. Event cameras are attracting increasing interest and adoption in robotics; however, dealing with viewpoint variance is an under-investigated problem in existing event-based localizers. In addition, event-based datasets that emphasize viewpoint variance for challenging localization situations are scarce. Here, we introduce an event-based visual place recognition (VPR) system that performs robustly under viewpoint changes. We converted five large-scale geo-tagged datasets, conventionally used to train frame-based localization systems, into synthetic event streams using Image-to-Event (I2E) conversion, and used them to fine-tune a pre-trained event-based vision transformer backbone with a multi-loss function, yielding a system we call MegaEvent that learns viewpoint-robust features for place recognition. We achieved an average Recall@1 of 82% across three existing event-based localization datasets, leading the next best event-based method by 20 recall points, and frame-based VPR models applied directly to event frames by 8 to 26 recall points. We introduce a new, challenging dataset - Springfield-Event-VPR - which features a 3.7km walking route recorded in three camera orientations for a total of 11.1km, which MegaEvent outperforms the strongest baseline by 9 recall points. The code for MegaEvent is available at https://github.com/AdamDHines/megaevent.
Adam D. Hines, Michael Milford, Tobias Fischer
QUT Centre for Robotics, Queensland University of Technology, Brisbane QLD 4001