Why and How: Knowledge-Guided Learning for Cross-Spectral Image Patch Matching
Authors: Chuang Yu, Yunpeng Liu, Jinmiao Zhao, Ao Chen, Xiujun Shu, Bo Wang, Zelin Shi, Xiangyu Yue
Organizations: Key Laboratory of Opto-Electronic Information Processing, Chinese Academy of Sciences · Shenyang Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · MMLab, The Chinese University of Hong Kong
Recently, cross-spectral image patch matching based on feature relation learning has attracted extensive attention. However, existing methods focus on mining richer feature relations by building complex relation extraction structures. Meanwhile, performance bottlenecks have gradually emerged. To address this, we make the first attempt to explore a stable and efficient bridge between descriptor learning and metric learning, and construct a Knowledge-Guided Learning Network (KGL-Net), which achieves significant performance improvements while abandoning complex network structures. Specifically, we find that there is feature extraction consistency between metric learning based on feature difference learning and descriptor learning based on Euclidean distance. This provides the foundation for bridge building. To ensure the stability and efficiency of the constructed bridge, on the one hand, we conduct an in-depth exploration of 20 combined network architectures. On the other hand, a feature-guided loss is constructed to achieve mutual guidance of features. In addition, unlike existing methods, we consider that the feature mapping ability of the metric branch should receive more attention. Therefore, a hard negative sample mining for metric learning (HNSM-M) strategy is constructed. To the best of our knowledge, this is the first time that hard negative sample mining for metric networks has been implemented and brings significant performance gains. Extensive experimental results show that our KGL-Net achieves SOTA performance in multiple cross-spectral image patch matching datasets. Our code is available at https://github.com/YuChuang1205/KGL-Net
Figures & tables
Fig. 1: Comparison of different cross-spectral image patch matching methods on the VIS-NIR patch dataset. Our KGL-Net achieves excellent performance. Compared with the latest FIL-Net [ 1 ] , KGL-Net reduces the FPR95 by 36.5% (from 0.85 to 0.54). Compared with the lightweight EFR-Net [ 15 ] , KGL-Net reduces the parameters by 30.1% (from 9.17 to 6.41).
Fig. 2: The overall structure of our KGL-Net. Each blue block denotes a convolution operation, a Filter Response Normalization (FRN) [ 55 ] , and a Thresholded Linear Unit (TLU) [ 55 ] , where “ nb×b ” means that the number of channels of the output feature map is n and the convolution kernel size is b×b . “ ECA ” denotes the efficient channel attention module [ 56 ] .
Fig. 3: Various combined network architectures of descriptor learning and metric learning. In the subtitle, the three parts divided by short dashes denote the metric network structure, the descriptor network structure, and parameter sharing between the two. “S” denotes Siamese. “P” denotes Pseudo-Siamese. “NS” denotes no parameters are shared. “FS” denotes full parameter sharing. “BS” denotes the parameters sharing of the front network layers. “AS” denotes the parameters sharing of the later network layers. In addition, the presence of m denotes that it is a dominant branch, which makes the structure of the corresponding part of another network consistent with it.
{(dVi,dNi)}i=1Nb←φdf{(Vi,Ni)}i=1Nb
Algorithm 1 KGL-Net Training (Batch image pairs input)
Fig. 4: Comparison of the receiver operating characteristic (ROC) curve and area under the curve (AUC) of each scheme in the break-down ablation experiment. From “Mean”, compared with other schemes, “Our-w/ all (KGL-Net)” has the highest AUC. From each subset, “Our-w/ all (KGL-Net)” is also mostly at the top and has the highest AUC.
Fig. 5: Comparison of different combined network architectures on the VIS-LWIR patch dataset. The percentages denote the reduction in FPR95 achieved by applying the HNSM-M strategy compared to the baseline without HNSM-M strategy. On one hand, Series C exhibits significantly lower FPR95 compared to other series architectures. On the other hand, Using the HNSM-M strategy, the FPR95 decreased by 3.8% (from 2.35 to 2.26) - 55.9% (from 2.22 to 0.98).
Table 7
Fig. 6: Exploration of the α and β in the feature-guided loss on the VIS-NIR patch dataset. The red bar denotes the optimal value, which is 0.54.
Models
Field
Forest
Indoor
Mountain
Oldbuilding
Street
Urban
Water
Mean
Traditional methods
SIFT [ 20 ] [IJCV]
39.44
11.39
10.13
28.63
19.69
31.14
10.85
40.33
23.95
GISIFT [ 24 ] [ICIP]
34.75
16.63
10.63
19.52
12.54
21.80
7.21
25.78
18.60
LGHD [ 30 ] [ICIP]
16.52
3.78
7.91
10.66
7.91
6.55
7.21
12.76
9.16
Descriptor learning
PN-Net [ 38 ] [arXiv]
20.09
3.27
6.36
11.53
5.19
5.62
3.31
10.72
8.26
TABLE V: Comparison of FPR95 among KGL-Net and nineteen SOTA methods on the VIS-NIR patch dataset.
Fig. 7: The matching effect of KGL-Net on the VIS-NIR patch dataset (VIS-NIR), VIS-LWIR patch dataset (VIS-LWIR) and OS patch dataset (VIS-SAR). Each cross-spectral scene is from top to bottom: correctly judged as matching, correctly judged as non-matching, misjudged as matching, and misjudged as non-matching. Green denotes a correct judgment, and red denotes an incorrect judgment.
Methods
VIS-NIR patch dataset
VIS-LWIR patch dataset
Mean ± Variance ↓
95% CI ↓
p -value ↓
Mean ± Variance ↓
95% CI ↓
p -value ↓
FIL-Net
0.8600 ± 0.0495
[0.7985, 0.9215]
1.67×10−6
0.6960 ± 0.0378
[0.6490, 0.7430]
2.83×10−5
KGL-Net
0.5380 ± 0.0303
[0.5003, 0.5757]
0.5080 ± 0.0319
[0.4683, 0.5477]
Methods
LLVIP patch dataset
OS patch dataset
Mean ± Variance ↓
95% CI ↓
p -value ↓
Mean ± Variance ↓
95% CI ↓
p -value ↓
FIL-Net
1.2500 ± 0.0620
[1.1730, 1.3270]
0.49×10−6
5.3660 ± 0.1050
[5.2356, 5.4964]
0.25×10−6
TABLE VI: Stability and significance analysis of FIL-Net and KGL-Net on multiple datasets. CI denotes the confidence interval. The p -value is a significance metric, representing the probability that the difference between the two groups occurred by chance.
Methods
VIS-LWIR → VIS-NIR
VIS-NIR → VIS-LWIR
SCFDM
17.64 (-64.6%)
30.90 (-77.6%)
AFD-Net
18.26 (-65.8%)
24.21 (-71.4%)
MR_3A
19.81 (-68.5%)
28.06 (-75.3%)
MFD-Net
17.25 (-63.8%)
27.69 (-75.0%)
EFR-Net
12.06 (-48.3%)
25.63 (-73.0%)
FIL-Net
19.83 (-68.5%)
23.30 (-70.3%)
TABLE VII: Performance of various methods under different generalization experimental settings. VIS-NIR denotes the VIS-NIR patch dataset, and VIS-LWIR denotes the VIS-LWIR patch dataset. The dataset on the left side of the arrow is used for training, and the one on the right side is used for testing.
Figure 13
Methods
MFD-Net (two)
MFD-Net (four)
MFD-Net (six)
MFD-Net (eight)
SPIMNet
EFR-Net
FIL-Net
KGL-Net
Parameters (M)
34.71
81.31
116.27
151.24
130.89
9.17
33.07
6.41
FLOPs (G)
2.09
4.20
6.29
8.38
10.58
1.79
4.09
1.76
Inference time (ms)
0.28
0.58
0.87
1.52
1.44
0.42
0.47
0.23
TABLE VIII: Complexity and inference efficiency comparison of cross-spectral image patch matching methods from the past five years.
Fig. 11: The part channels of the highest layer feature map. The upper and lower rows show the outputs of the metric network, The middle two rows show the outputs of the descriptor network. Whether it is the upper two rows or the lower two rows, the corresponding feature maps have obvious similarities.
Table 16
Fig. 12: Scatter distribution of Euclidean distances from the descriptor network versus confidence scores from the metric network on positive pairs in the “Field” test subset of the VIS-NIR patch dataset.
RMSE distribution
Methods
<1 ↑
1 ∼ 5
>5 ↓
Nm ↑
Irr ↑
SCFDM
121
29
10
150
93.8%
AFD-Net
121
29
10
150
93.8%
MR_3A
118
31
11
149
93.1%
MFD-Net
125
30
5
155
96.9%
EFR-Net
124
30
6
154
96.3%
TABLE XII: Registration results of various excellent image patch matching methods on the VIS-NIR registration dataset.
Fig. 13: FPR95 comparison of various excellent image patch matching methods on the two datasets. “1” and “15” denote the total value of one round. Left: the RIRE patch dataset. Right: the LLVIP patch dataset.
Fig. 14: Partial registration visualization of KGL-Net on the VIS-NIR registration dataset. Eight samples are from eight different subsets. For each sample, from left to right are the keypoint matching image, the registered VIS image, and the registered checkerboard image.
Fig. 15: Partial fusion visualization of KGL-Net on the VIS-NIR registration dataset. For each sample, from left to right are the VIS image, the NIR image, the registered checkerboard image, and the fused image.
Key Laboratory of Opto-Electronic Information Processing, Chinese Academy of Sciences · Shenyang Institute of Automation, Chinese Academy of Sciences · Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences +3
School of Automation, Southeast University, Nanjing 210096, China · Shenzhen Research Institute of Southeast University, Shenzhen 518063, China · School of Computer Science and Engineering, Southeast University, Nanjing 210096, China +2