RegNC-Mamba3: Shared Structural Feature Learning Using Non-Causal State Space Principles for Multi-Modal Image Registration
Authors: Zhikang Li, Yan Wu, Xin Hu, Yi Dai, Ming Li
Organizations: Remote Sensing Image Processing and Fusion Group, School of Electronic Engineering, Xidian University, Xi’an, 710071, China · National Key Laboratory of Radar Signal Processing, Xidian University, Xi’an, 710071, China
The main challenge of multi-modal image registration is shared structural feature extraction with higher accuracy and lower computational complexity. Compared with a State Space Model (SSM), the State Space Principles (SSP) formulation of Mamba-3 uses a more accurate discretization with lower truncation error while maintaining linear complexity. However, vanilla SSP is designed for natural language processing (NLP) and is not directly suitable for images. To address this issue, we propose a novel non-causal SSP (NC-SSP) with shared structure awareness and develop a coarse-to-fine registration method named RegNC-Mamba3. First, NC-SSP is designed to eliminate the dependence of the integral computation on the scanning path. By combining relative-weight normalization with aggregation over all circular scan paths, NC-SSP constructs a globally hidden state without long-range decay, which provides a wider receptive field and higher efficiency than the original Mamba-3. Second, NC-SSP is modified to strengthen the extraction of cross-modal structural features. When computing hidden states and feature interactions, we construct complex-valued sequences in NC-SSP by applying learnable Log-Gabor kernels, with different orientations to each head for frequency-domain modulation. This improves the perception of shared structural information across modalities and increases both registration accuracy and quality. Experiments on VIS-SAR and VIS-IR demonstrate that RegNC-Mamba3 outperforms the compared detector-based and semi-dense multi-modal registration methods.
Figures & tables
Figure 1: Schematic diagram of integral approximation: (a) generalized trapezoidal discretization in SSP; (b) Euler’s rule in SSMs.
Figure 2: Architecture of the shared structural-feature-aware NC-SSP layer.
Figure 3: The architecture of our NC-SSP block.
Figure 4: Architecture of RegNC-Mamba3. The Cross-Modal Feature Interaction (CMI) module extracts shared structural features at scales 1/2, 1/4, and 1/8 in Stages 1–3. Stage 3(a) contains NC-SSP blocks and Stage 3(b) contains MHA blocks. The Multi-Scale Feature Fusion (MSF) module fuses features at scales [1/8,1/4] in Stage 4 and [1/4,1/2] in Stage 5. Coarse correspondences are computed at 1/8 resolution; 5×5 and 7×7 window attention is then used at 1/4 and 1/2 resolution to obtain intermediate and fine correspondences.
Dataset
Method
aRMSE ↓
aNCM ↑
SMR(%) ↑
FLOPs (G)
time(ms)
memory(MB)
VIS-SAR
VSSD
2.657
325
84.4
588.2
92.05
745
VSSD+SFA
2.266
689.2
89.4
588.3
92.53
750
Ori-SSP
2.712
464.8
71.9
601.3
669.09
1724
Ori-SSP + SFA
2.442
541.1
89.2
601.4
687.53
1743
NC-SSP
2.595
258.4
82.3
627.0
93.05
761
NC-SSP+SFA
1.901
724.3
97.6
627.1
93.54
765
Table 1: Ablation of the backbone and structural feature awareness on VIS-SAR and VIS-IR.
Dataset
Metrics
RIFT
SDNet
ReDFeat
LoFTR
XoFTR
JamMa
RegNC-Mamba3
VIS-SAR
aRMSE ↓
3.104
2.984
2.400
2.584
3.075
3.033
1.901
aNCM ↑
93.3
78.3
115.6
281.4
292.4
93.0
724.3
SMR(%) ↑
14.4
55.0
75.7
74.8
60.1
60.4
97.6
VIS-IR
aRMSE ↓
2.009
1.822
1.168
0.983
0.942
1.119
0.911
aNCM ↑
513.5
116.0
357.1
1148.3
1222.8
732.4
1625.0
SMR(%) ↑
66.6
75.0
93.8
94.6
96.9
98.1
98.7
Table 2: Average RMSE, NCM, and SMR of RegNC-Mamba3 and six comparison methods on VIS-SAR and VIS-IR.
Dataset
Threshold
RIFT
SDNet
ReDFeat
LoFTR
XoFTR
JamMa
RegNC-Mamba3
VIS-SAR
@ 1px
0.5
1.9
6.1
2.8
0
0.9
17.7
@ 3px
7.3
28.3
54.0
51.9
29.2
29.5
84.2
@ 5px
14.4
55.0
75.7
74.8
60.1
60.4
97.6
VIS-IR
@ 1px
11.3
22.5
52.6
64.6
65.7
53.1
68.7
@ 3px
58.7
76.5
92.5
91.3
94.5
96.2
96.8
@ 5px
71.8
91.0
96.2
94.6
96.9
98.1
98.7
Table 3: Percentage of successfully registered pairs below 1-, 3-, and 5-pixel RMSE thresholds on VIS-SAR and VIS-IR.
Dataset
Top-K
Metric
LoFTR
XoFTR
JamMa
RegNC-Mamba3
VIS-SAR
K=50
SMR(%)
64.2
15.3
34.7
93.4
aRMSE
2.707
3.603
3.420
2.208
NCM
31.1
20.7
19.4
20.2
K=500
SMR(%)
74.1
53.3
60.8
97.6
aRMSE
2.557
3.314
3.049
1.926
NCM
180.9
148.8
91.1
185.3
Table 4: Registration results of semi-dense methods under different match budgets on VIS-SAR and VIS-IR.
Pairs
Metrics
RIFT
SDNet
ReDFeat
LoFTR
XoFTR
JamMa
RegNC-Mamba3
pair 1
RMSE ↓
10
10
10
1.88
10
10
1.26
NCM ↑
8
5
21
214
35
34
360
pair 2
RMSE ↓
10
2.21
1.57
2.11
2.13
2.47
1.29
NCM ↑
35
191
282
331
260
100
392
pair 3
RMSE ↓
4.22
3.04
1.76
1.69
1.73
1.79
1.40
NCM ↑
107
409
461
871
1363
1019
1503
Table 5: RMSE and NCM of RegNC-Mamba3 and six comparison methods on four image pairs.
Figure 5: Visualization of matching performance. Only the inlier matches after MAGSAC are shown.
Figure 6: Checkerboard images of Pair 1 to Pair 4 registration results.
Figure 7: Edge images of Pair 1 to Pair 4 registration results. Red and green rectangles circle the areas not aligned or aligned.
Table 6: Registration accuracy and computational cost of the compared methods.
In multi-modal image registration, the primary challenge lies in shared structural information extraction. Compared to Transformers, Structured State Space Duality (SSD) offers greater global structural feature extraction with higher efficiency during training and inference. Inspired by these advantages, we propose a novel algorithm for multi-modal image registration, named RegNetMamba-2. Our algorithm incorporates SSD into coarse-to-fine matching process to extract local and global structural features effectively. Firstly, SSD is applied in three different scales for multi-modal feature extraction in our network. To strengthen local representation, we pay more attention on foreground edge and structural information by feature scaling function of SSD. Secondly, for shared feature extraction of input images and multi-modal feature fusion in all scales, we propose cross-modality feature fusion model based on SSD, consisting of Cross-Modality feature Interaction (CMI) module and Multi-Scale feature Fusion (MSF) module. CMI module is designed for cross-modality feature extraction of each scale by SSD in cross form. MSF module is designed to employ a progressive upward fusion in feature-level to obtain fine features, consisting of multi-modal features in all scales. Following coarse-to-fine, the features in 1/8 scale from CMI and 1/2 scale from MSF are collected to calculate matching probability scores. Then we respectively establish matching process by correspondences of pixel-wise. Extensive experiments demonstrate that comparing with state-of-the-art deep-learning based algorithms, RegNetMamba-2 has achieved good effects in both performance and efficiency for multi-modal image registration on the following datasets: VIS-SAR (OSDataset), VIS-IR (LGHD/RoadSence) and VIS-NIR (RGB-NIR sense).
Zhikang Li, Yan Wu, Xin Hu +2
Remote Sensing Image Processing and Fusion Group, School of Electronic Engineering, Xidian University, Xi’an 710071, China · National Key Laboratory of Radar Signal Processing, Xidian University, Xi’an 710071, China
Multimodal image registration is a fundamental task and a prerequisite for downstream cross-modal analysis. Despite recent progress in shared feature extraction and multi-scale architectures, two key limitations remain. First, some methods use disentanglement to learn shared features but mainly regularize the shared part, allowing modality-private cues to leak into the shared space. Second, most multi-scale frameworks support only a single transformation type, limiting their applicability when global misalignment and local deformation coexist. To address these issues, we formulate hybrid multimodal registration as jointly learning a stable shared feature space and a unified hybrid transformation. Based on this view, we propose HRNet, a Hybrid Registration Network that couples representation disentanglement with hybrid parameter prediction. A shared backbone with Modality-Specific Batch Normalization (MSBN) extracts multi-scale features, while a Cross-scale Disentanglement and Adaptive Projection (CDAP) module suppresses modality-private cues and projects shared features into a stable subspace for matching. Built on this shared space, a Hybrid Parameter Prediction Module (HPPM) performs non-iterative coarse-to-fine estimation of global rigid parameters and deformation fields, which are fused into a coherent deformation field. Extensive experiments on four multimodal datasets demonstrate state-of-the-art performance on rigid and non-rigid registration tasks. The code is available at the project website.
Chunlei Zhang, Jiahao Xia, Yun Xiao +2
Faculty of Engineering and IT, University of Technology Sydney, Australia · School of Artificial Intelligence, Anhui University, Hefei, China · School of Computer Science and Technology, Anhui University, Hefei, China
Multi-modal image registration plays a critical role in precision medicine but faces challenges from non-linear intensity relationships and local optima. While deep learning models enable rapid inference, they often suffer from generalization collapse on unseen modalities. To address this, we propose Search-MIND, a training-free, iterative optimization framework for instance-specific registration. Our pipeline utilizes a coarse-to-fine strategy: a hierarchical coarse alignment stage followed by deformable refinement. We introduce two novel loss functions: Variance-Weighted Mutual Information (VWMI), which prioritizes informative tissue regions to shield global alignment from background noise and uniform regions, and Search-MIND (S-MIND), which broadens the convergence basin of structural descriptors by considering larger local search range. Evaluations on CARE Liver 2025 and CHAOS Challenge datasets show that Search-MIND consistently outperforms classical baselines like ANTs and foundation model-based approaches like DINO-reg, offering superior stability across diverse modalities.
Boya Wang, Ruizhe Li, Chao Chen +1
Intelligent Modelling & Analysis Group (IMA), School of Computer Science, University of Nottingham, UK · Lab for Uncertainty in Data and Decision Making (LUCID), Nottingham Biomedical Research Centre (BRC), School of Medicine, University of Nottingham, UK