cs.CVMar 18, 2024

Relational Representation Learning Network for Cross-Spectral Image Patch Matching

Authors: Chuang Yu, Yunpeng Liu, Jinmiao Zhao, Dou Quan, Zelin Shi, Xiangyu Yue

Organizations: Key Laboratory of Opto-Electronic Information Processing, Chinese Academy of Sciences · Shenyang Institute of Automation, Chinese Academy of Sciences · Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences · University of Chinese Academy of Sciences · 

Abstract

Recently, feature relation learning has drawn widespread attention in cross-spectral image patch matching. However, existing related research focuses on capturing diverse feature relations between image patches and ignores sufficient intrinsic feature representations of individual image patches. To address this limitation, we propose an innovative relational representation learning that simultaneously focuses on sufficiently mining the intrinsic features of individual image patches and the feature relations between image patches. Based on this, we construct a Relational Representation Learning Network (RRL-Net). Specifically, we innovatively construct an autoencoder to effectively characterize the individual intrinsic features, and introduce a feature interaction learning (FIL) module to extract deep-level feature relations. Meanwhile, to further fully mine individual intrinsic features, a lightweight multi-dimensional global-to-local attention (MGLA) module is constructed to enhance the global feature extraction of individual image patches and capture local dependencies within global features. By combining the MGLA module, we further explore the feature extraction network and construct an attention-based lightweight feature extraction (ALFE) network. Furthermore, a multi-loss post-pruning (MLPP) optimization strategy is proposed, which can greatly facilitate network optimization while avoiding increases in parameters and inference time. Extensive experiments demonstrate that our RRL-Net achieves state-of-the-art (SOTA) performance on multiple public datasets. Our code is available at https://github.com/YuChuang1205/RRL-Net

Figures & tables

Explore similar work

Dec 15, 2024

Why and How: Knowledge-Guided Learning for Cross-Spectral Image Patch Matching

Recently, cross-spectral image patch matching based on feature relation learning has attracted extensive attention. However, existing methods focus on mining richer feature relations by building complex relation extraction structures. Meanwhile, performance bottlenecks have gradually emerged. To address this, we make the first attempt to explore a stable and efficient bridge between descriptor learning and metric learning, and construct a Knowledge-Guided Learning Network (KGL-Net), which achieves significant performance improvements while abandoning complex network structures. Specifically, we find that there is feature extraction consistency between metric learning based on feature difference learning and descriptor learning based on Euclidean distance. This provides the foundation for bridge building. To ensure the stability and efficiency of the constructed bridge, on the one hand, we conduct an in-depth exploration of 20 combined network architectures. On the other hand, a feature-guided loss is constructed to achieve mutual guidance of features. In addition, unlike existing methods, we consider that the feature mapping ability of the metric branch should receive more attention. Therefore, a hard negative sample mining for metric learning (HNSM-M) strategy is constructed. To the best of our knowledge, this is the first time that hard negative sample mining for metric networks has been implemented and brings significant performance gains. Extensive experimental results show that our KGL-Net achieves SOTA performance in multiple cross-spectral image patch matching datasets. Our code is available at https://github.com/YuChuang1205/KGL-Net
Jun 2, 2026

SAMatcher: Co-Visibility Modeling with Segment Anything for Robust Feature Matching

Reliable correspondence estimation is a fundamental problem in image processing, underpinning applications such as Structure from Motion, visual localization, and image registration. Existing learning-based methods have significantly improved local feature representations, yet most still operate at the pixel or patch level and lack explicit modeling of regions that are jointly visible across views. We propose SAMatcher, a feature matching framework that formulates correspondence estimation through co-visibility modeling. Instead of directly matching local features, SAMatcher first predicts co-visible region masks and bounding boxes as structured priors for correspondence estimation. Built upon the Segment Anything Model (SAM), it introduces a symmetric cross-view interaction mechanism that enables bidirectional feature exchange and cross-view semantic alignment. We further develop a unified supervision scheme that jointly optimizes mask prediction and box localization through mask learning, box regression, and mask-box consistency constraints. Extensive experiments on challenging benchmarks demonstrate substantial improvements over existing matching pipelines, particularly under large viewpoint and scale variations. Our results show that foundation models originally designed for monocular segmentation can be effectively extended to multi-view correspondence reasoning through explicit co-visibility modeling, offering a new perspective on structured representation learning for image matching. Code and project page: https://xupan.top/Projects/samatcher
Oct 2, 2026

Geometry-Aligned Semantic Matching for Cross-Modal Planar Image Registration

Cross-modal image matching establishes stable and accurate geometric correspondences across modalities for planar registration. Existing semantic representations provide cross-modal consistency, but semantic similarity does not necessarily imply geometric correspondence. Meanwhile, fine-grained CNN features provide accurate local details but lack global cross-modal semantic guidance for stable refinement. To address these issues, we propose CDPM, which first establishes geometrically consistent semantic representations and then preserves their dominant role in correspondence estimation during fine-grained localization. Specifically, we progressively adapt DINOv3 using geometrically consistent cross-modal patch pairs, enabling feature similarity to better reflect true cross-modal spatial correspondences. We then construct a DINO-Centric Feature Pyramid, where multi-scale DINO representations maintain stable cross-modal correspondences, while a lightweight CNN branch provides auxiliary structural details for precise local refinement. Extensive experiments on three cross-modal datasets demonstrate the superior performance of CDPM. On VIS-IR, compared with the dense matcher RoMa, CDPM improves AUC@3/5/10/20 by 7.36, 13.40, 13.75, and 10.42 percentage points, respectively, and reduces mACE from 5.83 to 2.78 pixels. It also outperforms RoMa v2 across all metrics while requiring 45.6% fewer FLOPs. The online demo and dataset are available, and the code will be released on our project page at https://warren-wzw.github.io/CDPM/.