Organizations: Key Laboratory of Opto-Electronic Information Processing, Chinese Academy of Sciences · Shenyang Institute of Automation, Chinese Academy of Sciences · Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Xidian University · MMLab, The Chinese University of Hong Kong
Recently, feature relation learning has drawn widespread attention in cross-spectral image patch matching. However, existing related research focuses on capturing diverse feature relations between image patches and ignores sufficient intrinsic feature representations of individual image patches. To address this limitation, we propose an innovative relational representation learning that simultaneously focuses on sufficiently mining the intrinsic features of individual image patches and the feature relations between image patches. Based on this, we construct a Relational Representation Learning Network (RRL-Net). Specifically, we innovatively construct an autoencoder to effectively characterize the individual intrinsic features, and introduce a feature interaction learning (FIL) module to extract deep-level feature relations. Meanwhile, to further fully mine individual intrinsic features, a lightweight multi-dimensional global-to-local attention (MGLA) module is constructed to enhance the global feature extraction of individual image patches and capture local dependencies within global features. By combining the MGLA module, we further explore the feature extraction network and construct an attention-based lightweight feature extraction (ALFE) network. Furthermore, a multi-loss post-pruning (MLPP) optimization strategy is proposed, which can greatly facilitate network optimization while avoiding increases in parameters and inference time. Extensive experiments demonstrate that our RRL-Net achieves state-of-the-art (SOTA) performance on multiple public datasets. Our code is available at https://github.com/YuChuang1205/RRL-Net
Figures & tables
Figure 1: Performance comparison of multiple excellent cross-spectral image patch matching networks on the VIS-NIR patch dataset. The closer to the origin, the better the network performance. The number in parentheses for MFD-Net denotes its branch count. Our RRL-Net achieves SOTA performance. Compared with the lightweight EFR-Net [ 75 ] , RRL-Net has the same parameters and reduces the FPR95 by 23.2% (from 0.99 to 0.76). Compared with the latest and most competitive FIL-Net [ 74 ] , RRL-Net reduces the FPR95 by 10.6% (from 0.85 to 0.76) and reduces parameter count by 72.7% (from 0.33 to 0.09).
Figure 2: Network structure of RRL-Net. The red slash denotes the post-pruning operation.
Figure 3: Structure of the MGLA module. GAP denotes the global average pooling. ⊗ denotes the matrix multiplication. ⊙ denotes the element-wise multiplication. The red arrow denotes the Reshape operation. Green , yellow , and purple denote operations applied to the height ( H ), width ( W ), and channel ( C ) dimensions, respectively.
Figure 4: Partial sample display of multiple datasets. The red boxes denote non-matching sample pairs, and the green boxes denote matching sample pairs.
Models
Field
Forest
Indoor
Mountain
Oldbuilding
Street
Urban
Water
Mean
Traditional methods
SIFT [ 37 ] [IJCV’04]
39.44
11.39
10.13
28.63
19.69
31.14
10.85
40.33
23.95
GISIFT [ 18 ] [ICIP’11]
34.75
16.63
10.63
19.52
12.54
21.80
7.21
25.78
18.60
LGHD [ 3 ] [ICIP’15]
16.52
3.78
7.91
10.66
7.91
6.55
7.21
12.76
9.16
Descriptor learning
Q-Net [ 2 ] [Sensors’17]
17.01
2.70
6.16
9.61
4.61
3.99
2.83
8.44
6.91
Table 1: Comparison of FPR95 among RRL-Net and eighteen methods on the VIS-NIR patch dataset. Bold denotes optimal, underlined denotes suboptimal.
Figure 5: Partial matching results of RRL-Net on the VIS-NIR patch dataset. From top to bottom, the four lines of samples are misjudged as non-matching , misjudged as matching , correctly judged as non-matching , and correctly judged as matching .
Figure 7
Figure 10: Performance comparison on two VIS-SAR image patch matching datasets.
Figure 11: Partial results of RRL-Net on two VIS-SAR image patch matching datasets. The left and right of the dotted line are the OS patch dataset and the SEN1-2 patch dataset respectively. The left and right sides of the arrow denote the true labels and predicted results.
Table 2: Ablation experiments on VIS-NIR patch dataset. CL : Convolutional layers before each MGLA module, CC : Cross-layer connection. r is compression ratio. HGLA, WGLA, and CGLA are single dimension H , W , and C global-to-local attention.
Figure 12: Ablation experiments of the MGLA module on the VIS-NIR patch dataset. The dotted line is the local enlarged area. The receiver operating characteristic (ROC) curves without using the MGLA module are mostly at the lowest, and those with using the MGLA module are mostly at the highest. From the “Mean” subgraph, compared with not using the MGLA module, the AUC is significantly improved whether using a single-dimensional global-to-local module (HGLA, WGLA, CGLA) or a multi-dimensional global-to-local module (MGLA). In addition, using the final MGLA module has the best AUC .
RB Loss
Methods
SMN
RB
ML
PL
PP
FPR95 ↓
Parameters ↓
Inference time (s) ↓ (GTX 2080Ti)
Inference time (s) ↓ (GTX 3090)
Ours-w/o MLPP
✗
✗
✗
✗
✗
1.01 (-24.8%)
9,404,025
0.40×10 -3
0.30×10 -3
Variant 1
✗
✗
✗
✗
✗
0.95 (-20.0%)
10,190,175 (-7.7%)
0.51×10 -3 (-21.6%)
0.41×10 -3 (-26.8%)
Variant 2
✓
✗
✗
✗
✗
0.83 (-8.4%)
12,043,262 (-21.9%)
0.44×10 -3 (-9.1%)
0.32×10 -3 (-6.3%)
Variant 3
✓
✓
✗
✓
✗
0.79 (-3.8%)
12,829,276 (-26.7%)
0.54×10 -3 (-25.9%)
0.43×10 -3 (-30.2%)
Variant 4
✓
✓
✓
✗
✗
0.79 (-3.8%)
12,829,276 (-26.7%)
0.54×10 -3 (-25.9%)
0.43×10 -3 (-30.2%)
Table 3: Ablation experiments of MLPP optimization strategy on VIS-NIR patch dataset. SN : Sub-metric network, RB : Reconstruction branches, RB-Loss : Reconstruction branch loss, ML : MSE loss, PL : Perceptual loss, PP : Post-pruning.
Figure 13: Exploration of composition details in the MLPP optimization strategy on the VIS-NIR patch dataset. On the left are the results of different combinations of stage loss and module loss on the VIS-NIR patch dataset. The loss is added from higher levels to lower levels. On the right are the results of different combinations of α and β in overall loss. The red column with a five-pointed star on top denotes the lowest value of 0.76.
Figure 14: Visualization of the output feature maps of matching and non-matching VIS-NIR samples after passing through the MGLA modules and the FIL modules. For each pair of samples, the middle two columns denote the output of the MGLA module at each stage, and the left and right columns denote the private feature maps output by the FIL module at each stage. From top to bottom, they denote the stages from shallow to deep.
Methods
VIS-NIR patch dataset
LLVIP patch dataset
Mean ± Variance ↓
95% CI ↓
p -value ↓
Mean ± Variance ↓
95% CI ↓
p -value ↓
FIL-Net
0.860 ± 0.050
[0.799, 0.922]
0.0087
1.250 ± 0.062
[1.173, 1.327]
0.0480
RRL-Net
0.764 ± 0.038
[0.717, 0.811]
1.160 ± 0.060
[1.086, 1.235]
Methods
OS patch dataset
SEN1-2 patch dataset
Mean ± Variance ↓
95% CI ↓
p -value ↓
Mean ± Variance ↓
95% CI ↓
p -value ↓
FIL-Net
5.366 ± 0.105
[5.236, 5.496]
0.0425
1.540 ± 0.034
[1.498, 1.582]
0.0003
Table 4: Stability and significance analysis of FIL-Net and RRL-Net on multiple datasets. CI denotes the confidence interval. p -value is a significance indicator, which denotes the probability that the difference between the two groups occurs by chance.
Methods
OS → VIS-NIR
SEN1-2 → VIS-NIR
AFD-Net [ 44 , 46 ]
17.17 (-16.1%)
7.15 (-15.7%)
MR_3A [ 47 ]
20.27 (-29.0%)
8.09 (-18.0%)
MFD-Net [ 71 ]
22.33 (-35.5%)
8.39 (-21.0%)
EFR-Net [ 75 ]
15.4 (-6.5%)
7.27 (-8.8%)
FIL-Net [ 74 ]
14.5 (-0.7%))
8.88 (-25.3%)
RRL-Net (Ours)
14.4
6.63
Table 5: FPR95 comparison of various methods in the domain generalization experiment setting. OS denotes the OS patch dataset, SEN1-2 denotes the SEN1-2 patch dataset, and VIS-NIR denotes the VIS-NIR patch dataset.
Figure 15: FPR95 comparison of various excellent image patch matching methods on the RIRE patch dataset. The RRL-Net achieves the best results.
Input image size (pixel)
Field ↓
Forest ↓
Indoor ↓
Mountain ↓
Oldbuilding ↓
Street ↓
Urban ↓
Water ↓
Mean ↓
48×48
4.43
0.05
1.52
1.77
0.84
0.56
0.32
2.44
1.49
64×64
2.16
0.04
0.86
0.59
0.41
0.26
0.14
1.65
0.76
80×80
1.50
0.02
0.74
0.48
0.45
0.25
0.20
1.30
0.62
96×96
0.66
0.03
0.63
0.32
0.27
0.23
0.13
1.08
0.42
Table 6: Performance comparison of different input image sizes on the VIS-NIR patch dataset.
Figure 16: Examples of some difficult matching pairs from the VIS-NIR patch dataset being correctly discriminated. The left side of the dotted line shows dynamic object shifts due to acquisition time differences, and the right side of the dotted line shows the local displacement phenomenon caused by the difference in sensor and imaging position.
Brightness adjustment range
FIL-Net
RRL-Net
[0.5 - 2.0]
1.09
1.02
[0.5 -1.5]
0.97
0.90
[0.7 -1.5]
0.92
0.83
[0.9 -1.2]
0.86
0.79
No adjustment
0.85
0.76
Table 7: Performance comparison between RRL-Net and FIL-Net under different brightness adjustment settings.
Figure 17: Visualization of different brightness adjustments of VIS image patches. The left and right sides of the dotted line are the VIS-NIR image pair and the VIS image patch under different brightness adjustments, respectively. The brightness adjustment range is [0.5, 2.0], and is presented every 0.3.
Figure 22
RMSE distribution
Methods
<1 ↑
1 ∼ 5
>5 ↓
I\textsubscriptrr ↑
mRMSE ↓
SCFDM [ 43 ]
121
29
10
93.8%
8.48
AFD-Net [ 44 , 46 ]
121
29
10
93.8%
9.30
MR_3A [ 47 ]
118
31
11
93.1%
10.43
MFD-Net [ 71 ]
125
30
5
96.9%
5.57
EFR-Net [ 75 ]
124
30
6
96.3%
4.95
Table 8: Registration results of various excellent image patch matching methods on the VIS-NIR registration dataset. Bold denotes optimal, underlined denotes suboptimal.
Recently, cross-spectral image patch matching based on feature relation learning has attracted extensive attention. However, existing methods focus on mining richer feature relations by building complex relation extraction structures. Meanwhile, performance bottlenecks have gradually emerged. To address this, we make the first attempt to explore a stable and efficient bridge between descriptor learning and metric learning, and construct a Knowledge-Guided Learning Network (KGL-Net), which achieves significant performance improvements while abandoning complex network structures. Specifically, we find that there is feature extraction consistency between metric learning based on feature difference learning and descriptor learning based on Euclidean distance. This provides the foundation for bridge building. To ensure the stability and efficiency of the constructed bridge, on the one hand, we conduct an in-depth exploration of 20 combined network architectures. On the other hand, a feature-guided loss is constructed to achieve mutual guidance of features. In addition, unlike existing methods, we consider that the feature mapping ability of the metric branch should receive more attention. Therefore, a hard negative sample mining for metric learning (HNSM-M) strategy is constructed. To the best of our knowledge, this is the first time that hard negative sample mining for metric networks has been implemented and brings significant performance gains. Extensive experimental results show that our KGL-Net achieves SOTA performance in multiple cross-spectral image patch matching datasets. Our code is available at https://github.com/YuChuang1205/KGL-Net
Chuang Yu, Yunpeng Liu, Jinmiao Zhao +5
Key Laboratory of Opto-Electronic Information Processing, Chinese Academy of Sciences · Shenyang Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences +1
Reliable correspondence estimation is a fundamental problem in image processing, underpinning applications such as Structure from Motion, visual localization, and image registration. Existing learning-based methods have significantly improved local feature representations, yet most still operate at the pixel or patch level and lack explicit modeling of regions that are jointly visible across views. We propose SAMatcher, a feature matching framework that formulates correspondence estimation through co-visibility modeling. Instead of directly matching local features, SAMatcher first predicts co-visible region masks and bounding boxes as structured priors for correspondence estimation. Built upon the Segment Anything Model (SAM), it introduces a symmetric cross-view interaction mechanism that enables bidirectional feature exchange and cross-view semantic alignment. We further develop a unified supervision scheme that jointly optimizes mask prediction and box localization through mask learning, box regression, and mask-box consistency constraints. Extensive experiments on challenging benchmarks demonstrate substantial improvements over existing matching pipelines, particularly under large viewpoint and scale variations. Our results show that foundation models originally designed for monocular segmentation can be effectively extended to multi-view correspondence reasoning through explicit co-visibility modeling, offering a new perspective on structured representation learning for image matching. Code and project page: https://xupan.top/Projects/samatcher
Xu Pan, Qiyuan Ma, Mingyue Dong +3
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University, Wuhan, P. R. China · National Key Laboratory of Space Target Awareness, Space Engineering University, Beijing, P. R. China
Cross-modal image matching establishes stable and accurate geometric correspondences across modalities for planar registration. Existing semantic representations provide cross-modal consistency, but semantic similarity does not necessarily imply geometric correspondence. Meanwhile, fine-grained CNN features provide accurate local details but lack global cross-modal semantic guidance for stable refinement. To address these issues, we propose CDPM, which first establishes geometrically consistent semantic representations and then preserves their dominant role in correspondence estimation during fine-grained localization. Specifically, we progressively adapt DINOv3 using geometrically consistent cross-modal patch pairs, enabling feature similarity to better reflect true cross-modal spatial correspondences. We then construct a DINO-Centric Feature Pyramid, where multi-scale DINO representations maintain stable cross-modal correspondences, while a lightweight CNN branch provides auxiliary structural details for precise local refinement. Extensive experiments on three cross-modal datasets demonstrate the superior performance of CDPM. On VIS-IR, compared with the dense matcher RoMa, CDPM improves AUC@3/5/10/20 by 7.36, 13.40, 13.75, and 10.42 percentage points, respectively, and reduces mACE from 5.83 to 2.78 pixels. It also outperforms RoMa v2 across all metrics while requiring 45.6% fewer FLOPs. The online demo and dataset are available, and the code will be released on our project page at https://warren-wzw.github.io/CDPM/.