EventGeM: Global-to-Local Feature Matching for Event-Based Visual Place Recognition
Authors: Adam D. Hines, Gokul B. Nair, Nicolás Marticorena, Michael Milford, Tobias Fischer
Organizations: QUT Centre for Robotics, School of Electrical Engineering and Robotics, Queensland University of Technology, Brisbane, QLD 4000, Australia.
Event cameras are rapidly rising in popularity for robotic and computer vision tasks because their sparse activation delivers energy-efficient, high-dynamic-range, and fast sensing. Event cameras have been used in robotic navigation and localization tasks where positioning must occur in real time with sufficient accuracy. However, current event-based localization methods suffer from poor spatial understanding and are not viewpoint tolerant. In this paper, we address the problem of viewpoint-robust place recognition directly from event streams. We present EventGeM, a global-to-local feature fusion pipeline for event-based visual place recognition that combines whole-image feature detection to shortlist top candidates for 2D homography-based re-ranking with random sample consensus (RANSAC). We also contribute a regional generalized mean pooling (GeM) layer that learns to return the most relevant spatial features using per-row exponents to pool event streams into a compact global descriptor, trained on the NYC-Event-VPR dataset. These contributions overcome shortfalls in currently available event-based localization methods that fail to recognize similar places with large changes in viewpoint. To evaluate viewpoint-robust localization, we contribute a new event-based dataset that includes repeated traverses with a severe lateral shift. EventGeM improves absolute Recall@1 by 7 to 43 percentage points over the strongest baseline in each experiment. We also deploy EventGeM on a robotic platform, demonstrating real-time performance of our hierarchical pipeline. The code for EventGeM is available at https://github.com/AdamDHines/Event-GeM.
Figures & tables
Fig. 1: Schematic overview of EventGeM. A query event stream is converted to a multi-channel time surface (MCTS) and passed through the SuperEvent [ 11 ] backbone. Global features are pooled by our multi-exponent generalized mean (GeM) layer [ 12 ] and ranked by cosine similarity to produce a top- k shortlist. Local features from the SuperEvent keypoint detector head then re-rank the shortlist by 2D homography estimation with RANSAC, returning the matched database place.
Fig. 2: Multi-patch GeM pooling. (Top) The 128×30×40 feature map is split into a 4×4 grid; each row is pooled with its own learned exponent γr (values from Sect. IV-A ), giving four 512-dimensional blocks that are ℓ2 -normalized and concatenated. (Bottom) Pooled activations for one event frame under a global γ=5 , a 4×4 grid at γ=5 , and a 4×4 grid with learned per-row exponents. Warmer colors denote stronger activation.
Sunset1 (14,481)
Morning (13,456)
Daytime (14,321)
Method (Ref: Sunset2, 12,828)
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
LENS [ 9 ]
0.09
0.16
0.20
0.05
0.10
0.14
0.04
0.08
0.13
Sparse-Event-VPR [ 6 ]
0.05
0.08
0.13
0.02
0.04
0.05
0.05
0.08
0.09
EventVLAD [ 5 ]
0.43
0.57
0.64
0.09
0.17
0.22
0.08
0.16
0.22
LENS [ 9 ] +Re-rank
0.19
0.22
0.24
0.12
0.15
0.18
0.09
0.12
0.16
Sparse-Event-VPR [ 6 ] +Re-rank
0.11
0.12
0.15
0.03
0.04
0.05
0.05
0.09
0.10
TABLE I: Recall at K (R@K) on Brisbane-Event-VPR [ 4 ] , with Sunset2 as the reference. Bold is best, underline second best, per column. EventGeM (global only) is our descriptor before re-ranking. Bracketed values are the number of places.
R0-FS0 (18,218)
R0-FN0 (17,793)
Method (Ref: R0-FA0, 20,957)
R@1
R@5
R@10
R@1
R@5
R@10
LENS [ 9 ]
0.17
0.24
0.30
0.03
0.08
0.11
Sparse-Event-VPR [ 6 ]
0.14
0.24
0.29
0.02
0.08
0.11
EventVLAD [ 5 ]
0.20
0.28
0.33
0.05
0.11
0.14
LENS [ 9 ] +Re-rank
0.27
0.32
0.35
0.04
0.08
0.12
Sparse-Event-VPR [ 6 ] +Re-rank
0.19
0.27
0.31
0.04
0.08
0.11
TABLE II: Recall at K (R@K) on NSAVP [ 20 ] , with R0-FA0 as the reference. R0-FS0 is a daytime and R0-FN0 a night-time query. Bold is best, underline second best, per column. EventGeM (global only) is our descriptor before re-ranking. Bracketed values are the number of places.
Fig. 3: Example places from the left and right traverses of the Gardens-Point-Event dataset, separated by a lateral offset of approximately 5 m. Top: RGB frames. Middle: event frames, with red and blue denoting positive and negative polarity. Bottom: green lines are mutual nearest-neighbor keypoint matches from SuperEvent [ 11 ] .
Right (9,759)
Method (Ref: Left, 10,271)
R@1
R@5
R@10
LENS [ 9 ]
0.09
0.14
0.21
Sparse-Event-VPR [ 6 ]
0.06
0.14
0.21
EventVLAD [ 5 ]
0.18
0.40
0.53
EventGeM (global only)
0.37
0.58
0.67
EventGeM (ours)
0.55
0.70
0.75
TABLE III: Recall at K (R@K) on the Gardens-Point-Event dataset, Left as reference and Right as query. Bold is best, underline second best, per column. EventGeM (global only) is our descriptor before re-ranking. Bracketed values are the number of places. Only the event-native baselines are shown to compare differences in viewpoint tolerance across established event baselines.
Fig. 4: Per-query throughput against average Recall@1 (R@1) on Brisbane-Event-VPR [ 4 ] for EventGeM and the baselines [ 9 , 6 , 5 , 29 , 25 , 17 , 11 ] ; methods further right are faster. Each point averages over the Sunset1, Morning, and Daytime queries with Sunset2 as the reference.
Fig. 5: Online deployment of EventGeM on a robotic platform. Top, left to right: the AgileX Scout robot; its iniVation DAVIS346 dynamic vision sensor (DVS) and Jetson Orin AGX; the on-board pipeline; and the floor plan, with the teleoperated path in red, traversed once for the database and once as the query. Bottom, left to right: ground-truth correspondences between 1,400 queries (Q) and 1,600 database places (DB); the descriptor distance matrix, darker being closer; Recall@K; and per-query rate distribution, with median and mean marked.
Δt (ms)
R@1
R@5
R@10
1
0.13
0.18
0.20
10
0.58
0.61
0.62
50
0.87
0.88
0.89
100
0.92
0.93
0.94
250
0.95
0.96
0.96
TABLE IV: Effect of the MCTS accumulation window Δt on Recall at K (R@K) for EventGeM on the Brisbane-Event-VPR [ 4 ] dataset. The reference traverse was Sunset2 and the query traverse was Morning.
Condition
R@1
R@5
R@10
Dim.
γ=1
0.32
0.36
0.39
128
γ=2
0.53
0.57
0.58
128
γ=3
0.59
0.63
0.65
128
γ=5
0.59
0.64
0.66
128
γ=5 + 2×2 patch
0.78
0.80
0.81
512
γ=5 + 4×4 patch
0.81
0.83
0.83
2048
TABLE V: GeM pooling variants, evaluated on Brisbane-Event-VPR [ 4 ] with Sunset2 as reference and Morning as query; all models trained on NYC-Event-VPR [ 32 ] per Sect. III-D . Bold is best, underline second best, per column. R@K is Recall at K and Dim. the descriptor dimensionality. Rows marked global are scored before re-ranking.
Johns Hopkins University, Baltimore MD 21218, USA · ShanghaiTech University, Shanghai, China · University of Pennsylvania, Philadelphia PA 19104, USA +1
Institute of Visual Computing, Graz University of Technology, 8010 Graz, Austria · Neuromorphic Automation and Intelligence Lab (NAIL), School of Artificial Intelligence and Robotics, Hunan University, Changsha 410012, China