Organizations: School of Instrument Science and Opto-Electronics Engineering, Beijing Information Science and Technology University, Beijing 100192, China · Department of Precision Instrument, Tsinghua University, Beijing 100084, China
Infrared-visible (IR-VIS) feature matching plays an essential role in cross-modality visual localization, navigation and perception. Along with the rapid development of deep learning techniques, a number of representative image matching methods have been proposed. However, crossmodal feature matching is still a challenging task due to the significant appearance difference. A significant gap for cross-modal feature matching research lies in the absence of standardized benchmarks and metrics for evaluations. In this paper, we introduce a comprehensive cross-modal feature matching benchmark, CM-Bench, which encompasses 30 feature matching algorithms across diverse cross-modal datasets. Specifically, state-of-the-art traditional and deep learning-based methods are first summarized and categorized into sparse, semidense, and dense methods. These methods are evaluated by different tasks including homography estimation, relative pose estimation, and feature-matching-based geo-localization. In addition, we introduce a classification-network-based adaptive preprocessing front-end that automatically selects suitable enhancement strategies before matching. We also present a novel infrared-satellite cross-modal dataset with manually annotated ground-truth correspondences for practical geo-localization evaluation. The dataset and resource will be available at: https://github.com/SLZ98/CM-Bench.
Figures & tables
Fig. 1: Overview of eight representative matchers across four benchmark tasks, together with representative samples from the corresponding evaluation settings.
Fig. 2: Overview of the proposed ThermoSat dataset. (A) Collection area overview with seven representative thermal-to-satellite localization regions. (B) Representative ThermoSat scenes. (C) Cross-modal ground-truth correspondences, where blue dots denote satellite ground-truth points, red crosses denote thermal ground-truth points, and green lines denote ground-truth correspondences.
Fig. 3: Overview of the proposed adaptive preprocessing front-end. A lightweight MobileNetV4-based classifier predicts the most suitable preprocessing branch for each IR-VIS image pair before feature matching.
Category
Method
Success Rate
AUC@5
AUC@10
AUC@20
Sparse
SuperPoint+MINIMA-LG
0.987
0.151
0.452
0.689
SuperPoint+SuperGlue
0.971
0.114
0.358
0.559
SuperPoint+LightGlue
0.978
0.095
0.323
0.523
ALIKED+LightGlue
0.926
0.072
0.246
0.414
SuperPoint+MNN
1.000
0.053
0.213
0.376
DeDoDe
1.000
0.065
0.210
0.336
TABLE I: Homography estimation results on MSCM. Success rate denotes the ratio of valid homography evaluations over 2000 test pairs, and failed pairs are calculated as zero when computing AUC.
Category
Method
Success Rate
AUC@5
AUC@10
AUC@20
Sparse
SuperPoint+MINIMA-LG
0.941
0.192
0.373
0.556
SuperPoint+SuperGlue
0.964
0.036
0.088
0.168
SuperPoint+LightGlue
0.941
0.016
0.043
0.090
ALIKED+LightGlue
0.807
0.008
0.021
0.049
D2Net
1.000
0.007
0.020
0.052
RIFT
1.000
0.006
0.018
0.048
TABLE II: Relative pose estimation results on METU-VisTIR. Success Rate denotes the ratio of valid pose evaluations over all 2590 image pairs, and AUC is obtained by combining the cloudy-cloudy and cloudy-sunny split scores according to the number of scenes in each split.
Category
Method
Base
Hard
MedErr m
SR@3m
SR@5m
SR@10m
MedErr m
SR@3m
SR@5m
SR@10m
Sparse
XFeat(sparse)
0.746
0.786
0.851
0.921
1.417
0.305
0.349
0.398
SuperPoint+MINIMA-LG
0.452
0.686
0.719
0.758
0.470
0.284
0.294
0.319
XFeat+LightGlue
0.663
0.660
0.714
0.779
0.764
0.243
0.254
0.278
SuperPoint+SuperGlue
1.355
0.529
0.590
0.665
2.748
0.226
0.245
0.298
SuperPoint+LightGlue
1.429
0.513
0.573
0.655
3.404
0.208
0.234
0.282
TABLE III: Geo-localization results on ThermoSat. Lower MedErr m and higher SR@3m/SR@5m/SR@10m are better.
Dataset
Setting
MINIMA(RoMa)
XoFTR
MINIMA(LoFTR)
SuperPoint+LightGlue
TopicFM
LoFTR
DKM
ALIKED+LightGlue
MSCM
Baseline
0.526
0.421
0.391
0.354
0.325
0.311
0.308
0.292
MSCM
Adaptive
0.529
0.426
0.399
0.364
0.356
0.335
0.344
0.328
MSCM
Gain (%)
+0.6%
+1.2%
+2.0%
+2.8%
+9.5%
+7.7%
+11.7%
+12.3%
METU-VisTIR
Baseline
0.557
0.300
0.230
0.061
0.088
0.068
0.121
0.041
METU-VisTIR
Adaptive
0.561
0.312
0.247
0.098
0.135
0.100
0.183
0.088
METU-VisTIR
Gain (%)
+0.7%
+4.0%
+7.4%
+60.7%
+53.4%
+47.1%
+51.2%
+114.6%
TABLE IV: Adaptive preprocessing results on MSCM, METU-VisTIR, and ThermoSat using downstream benchmark metrics. For MSCM and METU-VisTIR, we report AUC@10 on the corresponding validation splits. For ThermoSat, we report SR@10m on the sequence-disjoint hard test split. The “Gain (%)” rows denote the relative percentage change from the baseline to the adaptive result. MSCM results are computed on the 400-pair validation split sampled from the 7083-pair benchmark. METU-VisTIR results are computed on the 1208-pair cloudy-sunny validation split.
Cross-modal feature matching finds corresponding features across different data types or sensors despite differences in appearance, geometry, and viewpoint.
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University, Wuhan, P. R. China · National Key Laboratory of Space Target Awareness, Space Engineering University, Beijing, P. R. China