Geometry-Aligned Semantic Matching for Cross-Modal Planar Image Registration

Zhiwei Wang; Defeng He; Yuxing Li; Meilu Zhu; Edmund Y. Lam

cs.CVOct 2, 2026

Geometry-Aligned Semantic Matching for Cross-Modal Planar Image Registration

Authors: Zhiwei Wang, Defeng He, Yuxing Li, Meilu Zhu, Edmund Y. Lam

Abstract

Cross-modal image matching establishes stable and accurate geometric correspondences across modalities for planar registration. Existing semantic representations provide cross-modal consistency, but semantic similarity does not necessarily imply geometric correspondence. Meanwhile, fine-grained CNN features provide accurate local details but lack global cross-modal semantic guidance for stable refinement. To address these issues, we propose CDPM, which first establishes geometrically consistent semantic representations and then preserves their dominant role in correspondence estimation during fine-grained localization. Specifically, we progressively adapt DINOv3 using geometrically consistent cross-modal patch pairs, enabling feature similarity to better reflect true cross-modal spatial correspondences. We then construct a DINO-Centric Feature Pyramid, where multi-scale DINO representations maintain stable cross-modal correspondences, while a lightweight CNN branch provides auxiliary structural details for precise local refinement. Extensive experiments on three cross-modal datasets demonstrate the superior performance of CDPM. On VIS-IR, compared with the dense matcher RoMa, CDPM improves AUC@3/5/10/20 by 7.36, 13.40, 13.75, and 10.42 percentage points, respectively, and reduces mACE from 5.83 to 2.78 pixels. It also outperforms RoMa v2 across all metrics while requiring 45.6% fewer FLOPs. The online demo and dataset are available, and the code will be released on our project page at https://warren-wzw.github.io/CDPM/.

Figures & tables

Panel of Figure 1 — Fig. 1: (a) Raw DINOv3 yields ambiguous cross-modal responses, while CNN refinement may sharpen but shift the peak; CDPM aligns semantic correspondence and stabilizes fine-scale refinement. (b) Green and red correspondences indicate reprojection errors of $\leq 3$ and $>3$ pixels, respectively. Yellow dashed boundaries denote the predicted alignments.

Fig. 2: Overview of the proposed CDPM, including data collection, patch-level cross-modal DINOv3 adaptation, and DINO-centric feature pyramid construction that preserves DINO correspondence representations throughout coarse-to-fine matching.

Fig. 3: Structure of the proposed Spatial Feature Block (SFB) for progressive multi-level DINO feature aggregation.

Fig. 4: Data acquisition setup and representative PCB-Layout pairs. The imaging system is shown on the left, while representative design layouts and their corresponding real PCB images are shown on the right.

Method	VIS-IR					GoogleMap					PCB-Layout
	@3 $px$	@5 $px$	@10 $px$	@20 $px$	$mACE$	@3 $px$	@5 $px$	@10 $px$	@20 $px$	$mACE$	@3 $px$	@5 $px$	@10 $px$	@20 $px$	$mACE$
SP+LG [ 35 , 36 ]	4.48	13.53	34.16	54.91	17.22	2.71	7.98	19.72	32.27	36.39	2.86	9.31	32.59	60.63	8.11
LoMa [ 37 ]	2.95	12.18	32.97	54.28	16.29	0.50	2.64	10.85	25.37	32.83	5.27	12.69	34.26	61.53	8.03
RHWF [ 25 ]	19.15	33.79	56.86	74.95	6.23	5.46	17.73	42.42	63.35	10.72	0.00	0.00	0.81	13.56	20.47
MCNet [ 26 ]	20.90	34.99	56.36	73.97	6.71	18.77	34.59	54.12	68.35	10.27	0.11	0.77	14.64	41.41	14.80
LoFTR [ 9 ]	7.92	19.63	40.73	59.62	13.99	6.54	18.28	41.43	61.27	11.59	4.23	16.28	43.56	67.19	8.26

TABLE I: Quantitative comparison with state-of-the-art methods on VIS-IR, GoogleMap, and PCB-Layout. AUC@

\{3,5,10,20\}

(

\uparrow

) uses pixel-error thresholds, and mACE (

\downarrow

) is in pixels. SP+LG denotes SuperPoint with LightGlue, and MatchAny denotes MatchAnything. The best and second-best results are highlighted in bold and underlined , respectively.

Panel of Figure 6 — Fig. 5: Qualitative comparison on VIS-IR, GoogleMap, and PCB-Layout. (a) Alignment results. Green solid and yellow dashed boundaries denote the ground truth and prediction, respectively. (b) Correspondence results. Green and red/yellow lines indicate reprojection errors of $\leq 3$ and $>3$ pixels.

Fig. 6: Qualitative results of the transferability experiment on RGB-T, RGB-Event, RGB-Depth, and Retina datasets.

Method	Input Size	Params (M) $\downarrow$	FLOPs (G) $\downarrow$	Time (ms) $\downarrow$
SP+LG [ 35 , 36 ]	$448\times 448$	13.15	317.50	$23.34\pm 0.20$
LoFTR [ 9 ]	$448\times 448$	11.56	467.97	$19.36\pm 0.11$
XoFTR [ 28 ]	$448\times 448$	11.09	281.15	$19.17\pm 0.11$
MCNet [ 26 ]	$448\times 448$	0.85	112.68	$25.52\pm 0.94$
RoMa [ 10 ]	$448\times 448$	415.67	2100.00	$245.00\pm 0.28$
GFNet [ 15 ]	$448\times 448$	308.24	1514.30	$81.81\pm 2.33$

TABLE II: Comparison of model complexity and inference efficiency at an input resolution of

448\times 448

Fig. 7: Qualitative ablation of DINO-P16 with CNN features at P8–P1, multi-scale DINO without CNN features, and the full CDPM. Green crosses and colored dots indicate the ground-truth and predicted correspondences, respectively, with localization errors reported in pixels.

Configuration	R@1 $\uparrow$	R@5 $\uparrow$	R@20 $\uparrow$	MedRank $\downarrow$
Raw DINOv3	0.109	0.338	0.656	11
Joint Adaptation	0.581	0.941	0.989	1
Staged Adaptation	0.608	0.951	0.991	1

TABLE III: We evaluate cross-modal patch retrieval using Recall@K (R@K) and median rank (MedRank), where R@K measures top-

K

retrieval accuracy, while MedRank denotes the median rank of the ground-truth correspondence.

DINO Adaptation
Configuration	AUC@3 $\uparrow$	@5 $\uparrow$	@10 $\uparrow$	@20 $\uparrow$	mACE $\downarrow$
Raw DINOv3	35.03	52.33	71.56	84.19	3.62
E2E Adaptation	36.43	53.85	73.22	85.35	3.19
Joint Adaptation	37.19	54.55	73.65	85.55	3.15
Staged Adaptation	41.43	58.82	76.49	87.17	2.78
Feature Construction

TABLE IV: Ablation study of DINO adaptation, feature construction, and refinement strategies on the VIS-IR dataset.

Explore similar work

Apr 7, 2026

CRFT: Consistent-Recurrent Feature Flow Transformer for Cross-Modal Image Registration

We present Consistent-Recurrent Feature Flow Transformer (CRFT), a unified coarse-to-fine framework based on feature flow learning for robust cross-modal image registration. CRFT learns a modality-independent feature flow representation within a transformer-based architecture that jointly performs feature alignment and flow estimation. The coarse stage establishes global correspondences through multi-scale feature correlation, while the fine stage refines local details via hierarchical feature fusion and adaptive spatial reasoning. To enhance geometric adaptability, an iterative discrepancy-guided attention mechanism with a Spatial Geometric Transform (SGT) recurrently refines the flow field, progressively capturing subtle spatial inconsistencies and enforcing feature-level consistency. This design enables accurate alignment under large affine and scale variations while maintaining structural coherence across modalities. Extensive experiments on diverse cross-modal datasets demonstrate that CRFT consistently outperforms state-of-the-art registration methods in both accuracy and robustness. Beyond registration, CRFT provides a generalizable paradigm for multimodal spatial correspondence, offering broad applicability to remote sensing, autonomous navigation, and medical imaging. Code and datasets are publicly available at https://github.com/NEU-Liuxuecong/CRFT.

Xuecong Liu, Mengzhu Ding, Zixuan Sun +2

Northeastern University, China · National University of Defense Technology, China

Cross-Modal Image Registration Cross-Modal Feature Matching

Ask PDF

Jun 2, 2026

SAMatcher: Co-Visibility Modeling with Segment Anything for Robust Feature Matching

Reliable correspondence estimation is a fundamental problem in image processing, underpinning applications such as Structure from Motion, visual localization, and image registration. Existing learning-based methods have significantly improved local feature representations, yet most still operate at the pixel or patch level and lack explicit modeling of regions that are jointly visible across views. We propose SAMatcher, a feature matching framework that formulates correspondence estimation through co-visibility modeling. Instead of directly matching local features, SAMatcher first predicts co-visible region masks and bounding boxes as structured priors for correspondence estimation. Built upon the Segment Anything Model (SAM), it introduces a symmetric cross-view interaction mechanism that enables bidirectional feature exchange and cross-view semantic alignment. We further develop a unified supervision scheme that jointly optimizes mask prediction and box localization through mask learning, box regression, and mask-box consistency constraints. Extensive experiments on challenging benchmarks demonstrate substantial improvements over existing matching pipelines, particularly under large viewpoint and scale variations. Our results show that foundation models originally designed for monocular segmentation can be effectively extended to multi-view correspondence reasoning through explicit co-visibility modeling, offering a new perspective on structured representation learning for image matching. Code and project page: https://xupan.top/Projects/samatcher

Xu Pan, Qiyuan Ma, Mingyue Dong +3

State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University, Wuhan, P. R. China · National Key Laboratory of Space Target Awareness, Space Engineering University, Beijing, P. R. China

Multi-View Geometry Image Matching Local Feature Matching

Ask PDF

May 5, 2026

UniCorrn: Unified Correspondence Transformer Across 2D and 3D

Visual correspondence across image-to-image (2D-2D), image-to-point cloud (2D-3D), and point cloud-to-point cloud (3D-3D) geometric matching forms the foundation for numerous 3D vision tasks. Despite sharing a similar problem structure, current methods use task-specific designs with separate models for each modality combination. We present UniCorrn, the first correspondence model with shared weights that unifies geometric matching across all three tasks. Our key insight is that Transformer attention naturally captures cross-modal feature similarity. We propose a dual-stream decoder that maintains separate appearance and positional feature streams. This design enables end-to-end learning through stack-able layers while supporting flexible query-based correspondence estimation across heterogeneous modalities. Our architecture employs modality-specific backbones followed by shared encoder and decoder components, trained jointly on diverse data combining pseudo point clouds from depth maps with real 3D correspondence annotations. UniCorrn achieves competitive performance on 2D-2D matching and surpasses prior state-of-the-art by 8% on 7Scenes (2D-3D) and 10% on 3DLoMatch (3D-3D) in registration recall. Project website: https://neu-vi.github.io/UniCorrn

Prajnan Goswami, Tianye Ding, Feng Liu +1

Northeastern University · Adobe Research

Image Matching Point Cloud Registration Cross-Modal Feature Matching

Ask PDF