Organizations: Key Laboratory of Opto-Electronic Information Processing, Chinese Academy of Sciences · Shenyang Institute of Automation, Chinese Academy of Sciences · Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Xidian University · MMLab, The Chinese University of Hong Kong
Recently, feature relation learning has drawn widespread attention in cross-spectral image patch matching. However, existing related research focuses on capturing diverse feature relations between image patches and ignores sufficient intrinsic feature representations of individual image patches. To address this limitation, we propose an innovative relational representation learning that simultaneously focuses on sufficiently mining the intrinsic features of individual image patches and the feature relations between image patches. Based on this, we construct a Relational Representation Learning Network (RRL-Net). Specifically, we innovatively construct an autoencoder to effectively characterize the individual intrinsic features, and introduce a feature interaction learning (FIL) module to extract deep-level feature relations. Meanwhile, to further fully mine individual intrinsic features, a lightweight multi-dimensional global-to-local attention (MGLA) module is constructed to enhance the global feature extraction of individual image patches and capture local dependencies within global features. By combining the MGLA module, we further explore the feature extraction network and construct an attention-based lightweight feature extraction (ALFE) network. Furthermore, a multi-loss post-pruning (MLPP) optimization strategy is proposed, which can greatly facilitate network optimization while avoiding increases in parameters and inference time. Extensive experiments demonstrate that our RRL-Net achieves state-of-the-art (SOTA) performance on multiple public datasets. Our code is available at https://github.com/YuChuang1205/RRL-Net
Figures & tables
Figure 1: Performance comparison of multiple excellent cross-spectral image patch matching networks on the VIS-NIR patch dataset. The closer to the origin, the better the network performance. The number in parentheses for MFD-Net denotes its branch count. Our RRL-Net achieves SOTA performance. Compared with the lightweight EFR-Net [ 75 ] , RRL-Net has the same parameters and reduces the FPR95 by 23.2% (from 0.99 to 0.76). Compared with the latest and most competitive FIL-Net [ 74 ] , RRL-Net reduces the FPR95 by 10.6% (from 0.85 to 0.76) and reduces parameter count by 72.7% (from 0.33 to 0.09).
Figure 2: Network structure of RRL-Net. The red slash denotes the post-pruning operation.
Figure 3: Structure of the MGLA module. GAP denotes the global average pooling. ⊗ denotes the matrix multiplication. ⊙ denotes the element-wise multiplication. The red arrow denotes the Reshape operation. Green , yellow , and purple denote operations applied to the height ( H ), width ( W ), and channel ( C ) dimensions, respectively.
Figure 4: Partial sample display of multiple datasets. The red boxes denote non-matching sample pairs, and the green boxes denote matching sample pairs.
Models
Field
Forest
Indoor
Mountain
Oldbuilding
Street
Urban
Water
Mean
Traditional methods
SIFT [ 37 ] [IJCV’04]
39.44
11.39
10.13
28.63
19.69
31.14
10.85
40.33
23.95
GISIFT [ 18 ] [ICIP’11]
34.75
16.63
10.63
19.52
12.54
21.80
7.21
25.78
18.60
LGHD [ 3 ] [ICIP’15]
16.52
3.78
7.91
10.66
7.91
6.55
7.21
12.76
9.16
Descriptor learning
Q-Net [ 2 ] [Sensors’17]
17.01
2.70
6.16
9.61
4.61
3.99
2.83
8.44
6.91
Table 1: Comparison of FPR95 among RRL-Net and eighteen methods on the VIS-NIR patch dataset. Bold denotes optimal, underlined denotes suboptimal.
Figure 5: Partial matching results of RRL-Net on the VIS-NIR patch dataset. From top to bottom, the four lines of samples are misjudged as non-matching , misjudged as matching , correctly judged as non-matching , and correctly judged as matching .
Figure 7
Figure 10: Performance comparison on two VIS-SAR image patch matching datasets.
Figure 11: Partial results of RRL-Net on two VIS-SAR image patch matching datasets. The left and right of the dotted line are the OS patch dataset and the SEN1-2 patch dataset respectively. The left and right sides of the arrow denote the true labels and predicted results.
Table 2: Ablation experiments on VIS-NIR patch dataset. CL : Convolutional layers before each MGLA module, CC : Cross-layer connection. r is compression ratio. HGLA, WGLA, and CGLA are single dimension H , W , and C global-to-local attention.
Figure 12: Ablation experiments of the MGLA module on the VIS-NIR patch dataset. The dotted line is the local enlarged area. The receiver operating characteristic (ROC) curves without using the MGLA module are mostly at the lowest, and those with using the MGLA module are mostly at the highest. From the “Mean” subgraph, compared with not using the MGLA module, the AUC is significantly improved whether using a single-dimensional global-to-local module (HGLA, WGLA, CGLA) or a multi-dimensional global-to-local module (MGLA). In addition, using the final MGLA module has the best AUC .
RB Loss
Methods
SMN
RB
ML
PL
PP
FPR95 ↓
Parameters ↓
Inference time (s) ↓ (GTX 2080Ti)
Inference time (s) ↓ (GTX 3090)
Ours-w/o MLPP
✗
✗
✗
✗
✗
1.01 (-24.8%)
9,404,025
0.40×10 -3
0.30×10 -3
Variant 1
✗
✗
✗
✗
✗
0.95 (-20.0%)
10,190,175 (-7.7%)
0.51×10 -3 (-21.6%)
0.41×10 -3 (-26.8%)
Variant 2
✓
✗
✗
✗
✗
0.83 (-8.4%)
12,043,262 (-21.9%)
0.44×10 -3 (-9.1%)
0.32×10 -3 (-6.3%)
Variant 3
✓
✓
✗
✓
✗
0.79 (-3.8%)
12,829,276 (-26.7%)
0.54×10 -3 (-25.9%)
0.43×10 -3 (-30.2%)
Variant 4
✓
✓
✓
✗
✗
0.79 (-3.8%)
12,829,276 (-26.7%)
0.54×10 -3 (-25.9%)
0.43×10 -3 (-30.2%)
Table 3: Ablation experiments of MLPP optimization strategy on VIS-NIR patch dataset. SN : Sub-metric network, RB : Reconstruction branches, RB-Loss : Reconstruction branch loss, ML : MSE loss, PL : Perceptual loss, PP : Post-pruning.
Figure 13: Exploration of composition details in the MLPP optimization strategy on the VIS-NIR patch dataset. On the left are the results of different combinations of stage loss and module loss on the VIS-NIR patch dataset. The loss is added from higher levels to lower levels. On the right are the results of different combinations of α and β in overall loss. The red column with a five-pointed star on top denotes the lowest value of 0.76.
Figure 14: Visualization of the output feature maps of matching and non-matching VIS-NIR samples after passing through the MGLA modules and the FIL modules. For each pair of samples, the middle two columns denote the output of the MGLA module at each stage, and the left and right columns denote the private feature maps output by the FIL module at each stage. From top to bottom, they denote the stages from shallow to deep.
Methods
VIS-NIR patch dataset
LLVIP patch dataset
Mean ± Variance ↓
95% CI ↓
p -value ↓
Mean ± Variance ↓
95% CI ↓
p -value ↓
FIL-Net
0.860 ± 0.050
[0.799, 0.922]
0.0087
1.250 ± 0.062
[1.173, 1.327]
0.0480
RRL-Net
0.764 ± 0.038
[0.717, 0.811]
1.160 ± 0.060
[1.086, 1.235]
Methods
OS patch dataset
SEN1-2 patch dataset
Mean ± Variance ↓
95% CI ↓
p -value ↓
Mean ± Variance ↓
95% CI ↓
p -value ↓
FIL-Net
5.366 ± 0.105
[5.236, 5.496]
0.0425
1.540 ± 0.034
[1.498, 1.582]
0.0003
Table 4: Stability and significance analysis of FIL-Net and RRL-Net on multiple datasets. CI denotes the confidence interval. p -value is a significance indicator, which denotes the probability that the difference between the two groups occurs by chance.
Methods
OS → VIS-NIR
SEN1-2 → VIS-NIR
AFD-Net [ 44 , 46 ]
17.17 (-16.1%)
7.15 (-15.7%)
MR_3A [ 47 ]
20.27 (-29.0%)
8.09 (-18.0%)
MFD-Net [ 71 ]
22.33 (-35.5%)
8.39 (-21.0%)
EFR-Net [ 75 ]
15.4 (-6.5%)
7.27 (-8.8%)
FIL-Net [ 74 ]
14.5 (-0.7%))
8.88 (-25.3%)
RRL-Net (Ours)
14.4
6.63
Table 5: FPR95 comparison of various methods in the domain generalization experiment setting. OS denotes the OS patch dataset, SEN1-2 denotes the SEN1-2 patch dataset, and VIS-NIR denotes the VIS-NIR patch dataset.
Figure 15: FPR95 comparison of various excellent image patch matching methods on the RIRE patch dataset. The RRL-Net achieves the best results.
Input image size (pixel)
Field ↓
Forest ↓
Indoor ↓
Mountain ↓
Oldbuilding ↓
Street ↓
Urban ↓
Water ↓
Mean ↓
48×48
4.43
0.05
1.52
1.77
0.84
0.56
0.32
2.44
1.49
64×64
2.16
0.04
0.86
0.59
0.41
0.26
0.14
1.65
0.76
80×80
1.50
0.02
0.74
0.48
0.45
0.25
0.20
1.30
0.62
96×96
0.66
0.03
0.63
0.32
0.27
0.23
0.13
1.08
0.42
Table 6: Performance comparison of different input image sizes on the VIS-NIR patch dataset.
Figure 16: Examples of some difficult matching pairs from the VIS-NIR patch dataset being correctly discriminated. The left side of the dotted line shows dynamic object shifts due to acquisition time differences, and the right side of the dotted line shows the local displacement phenomenon caused by the difference in sensor and imaging position.
Brightness adjustment range
FIL-Net
RRL-Net
[0.5 - 2.0]
1.09
1.02
[0.5 -1.5]
0.97
0.90
[0.7 -1.5]
0.92
0.83
[0.9 -1.2]
0.86
0.79
No adjustment
0.85
0.76
Table 7: Performance comparison between RRL-Net and FIL-Net under different brightness adjustment settings.
Figure 17: Visualization of different brightness adjustments of VIS image patches. The left and right sides of the dotted line are the VIS-NIR image pair and the VIS image patch under different brightness adjustments, respectively. The brightness adjustment range is [0.5, 2.0], and is presented every 0.3.
Figure 22
RMSE distribution
Methods
<1 ↑
1 ∼ 5
>5 ↓
I\textsubscriptrr ↑
mRMSE ↓
SCFDM [ 43 ]
121
29
10
93.8%
8.48
AFD-Net [ 44 , 46 ]
121
29
10
93.8%
9.30
MR_3A [ 47 ]
118
31
11
93.1%
10.43
MFD-Net [ 71 ]
125
30
5
96.9%
5.57
EFR-Net [ 75 ]
124
30
6
96.3%
4.95
Table 8: Registration results of various excellent image patch matching methods on the VIS-NIR registration dataset. Bold denotes optimal, underlined denotes suboptimal.
Key Laboratory of Opto-Electronic Information Processing, Chinese Academy of Sciences · Shenyang Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences +1
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University, Wuhan, P. R. China · National Key Laboratory of Space Target Awareness, Space Engineering University, Beijing, P. R. China