Disentangle-then-Align: Non-Iterative Hybrid Multimodal Image Registration via Cross-Scale Feature Disentanglement

Chunlei Zhang; Jiahao Xia; Yun Xiao; Bo Jiang; Jian Zhang

cs.CVMar 20, 2026

Disentangle-then-Align: Non-Iterative Hybrid Multimodal Image Registration via Cross-Scale Feature Disentanglement

Authors: Chunlei Zhang, Jiahao Xia, Yun Xiao, Bo Jiang, Jian Zhang

Organizations: Faculty of Engineering and IT, University of Technology Sydney, Australia · School of Artificial Intelligence, Anhui University, Hefei, China · School of Computer Science and Technology, Anhui University, Hefei, China

Abstract

Multimodal image registration is a fundamental task and a prerequisite for downstream cross-modal analysis. Despite recent progress in shared feature extraction and multi-scale architectures, two key limitations remain. First, some methods use disentanglement to learn shared features but mainly regularize the shared part, allowing modality-private cues to leak into the shared space. Second, most multi-scale frameworks support only a single transformation type, limiting their applicability when global misalignment and local deformation coexist. To address these issues, we formulate hybrid multimodal registration as jointly learning a stable shared feature space and a unified hybrid transformation. Based on this view, we propose HRNet, a Hybrid Registration Network that couples representation disentanglement with hybrid parameter prediction. A shared backbone with Modality-Specific Batch Normalization (MSBN) extracts multi-scale features, while a Cross-scale Disentanglement and Adaptive Projection (CDAP) module suppresses modality-private cues and projects shared features into a stable subspace for matching. Built on this shared space, a Hybrid Parameter Prediction Module (HPPM) performs non-iterative coarse-to-fine estimation of global rigid parameters and deformation fields, which are fused into a coherent deformation field. Extensive experiments on four multimodal datasets demonstrate state-of-the-art performance on rigid and non-rigid registration tasks. The code is available at the project website.

Figures & tables

Figure 1: Comparison of model structures. (a) non-rigid only, (b) rigid only; (c) Our hybrid structure.

Figure 2: (a) Rigid registration fails to handle local deformations (first row). Non-rigid registration may distort structural integrity under large global offsets (second row). (b) Serial hybrid registration.

Figure 3: The schematic diagram and detailed architectures of the H ybrid R egistration N etwork, namely HRNet, which consists of three main components: a shared backbone with Modality-Specific Batch Normalization (MSBN), a Cross-scale Disentanglement and Adaptive Projection (CDAP) module for feature disentanglement, and a Hybrid Parameter Prediction Module (HPPM) for transformation parameter estimation.

stage	$\alpha_{r}$	$\alpha_{n}$	$\alpha_{s}$	$\alpha_{tri}$	$\alpha_{cs}$	$\alpha_{ccd}$	$\alpha_{bo}$
warmup (10%)	7	6	0.5	0.5	0.05	0	0.1
mid (50%)	5	10	0.5	1	0.1	0.05	0.2
late (40%)	3	12	0.7	1	0.1	0.05	0.2

Table 1: Loss weight configuration across three-phase curriculum training.

Methods	RGB-NIR		RGB-TIR		RGB-IR		RGB-SAR
Methods	RE $\downarrow$	NCC $\uparrow$	RE $\downarrow$	NCC $\uparrow$	RE $\downarrow$	NCC $\uparrow$	RE $\downarrow$	NCC $\uparrow$
IHN	$3.887_{0.009}$	$0.656_{0.000}$	$3.006_{0.004}$	$0.695_{0.000}$	$5.684_{0.004}$	$0.747_{0.000}$	$7.087_{0.030}$	$0.741_{0.003}$
InMIRNet	$7.312_{0.018}$	$0.509_{0.001}$	$5.952_{0.053}$	$0.558_{0.003}$	$17.958_{0.158}$	$0.268_{0.009}$	$15.749_{0.223}$	$0.440_{0.012}$
RHWF	$5.081_{0.011}$	$0.616_{0.001}$	$3.994_{0.024}$	$0.654_{0.001}$	$7.289_{0.033}$	$0.692_{0.001}$	$8.446_{0.006}$	$0.697_{0.000}$
SCPNet	$3.056_{0.037}$	$0.692_{0.002}$	$3.187_{0.279}$	$0.674_{0.019}$	$6.156_{0.378}$	$0.722_{0.017}$	$16.974_{0.396}$	$0.376_{0.012}$
MCNet	$3.582_{0.012}$	$0.666_{0.001}$	$3.221_{0.023}$	$0.678_{0.001}$	$12.281_{0.275}$	$0.483_{0.011}$	$8.843_{0.147}$	$0.682_{0.005}$

Table 2: Registration results (

mean_{std}

) for the rigid registration task. The best results are marked in bold and second bests are underlined .

Methods	RGB-NIR		RGB-TIR		RGB-IR		RGB-SAR
Methods	RE $\downarrow$	NCC $\uparrow$	RE $\downarrow$	NCC $\uparrow$	RE $\downarrow$	NCC $\uparrow$	RE $\downarrow$	NCC $\uparrow$
SuperFusion	$4.034_{0.021}$	$0.639_{0.001}$	$4.325_{0.042}$	$0.571_{0.002}$	$36.184_{0.019}$	$0.100_{0.000}$	$39.341_{0.011}$	$0.096_{0.000}$
InMIRNet	$16.536_{0.048}$	$0.212_{0.001}$	$14.372_{0.055}$	$0.211_{0.000}$	$26.301_{0.004}$	$0.120_{0.000}$	$28.886_{0.046}$	$0.110_{0.002}$
NBRNet	$10.426_{0.110}$	$0.351_{0.004}$	$9.661_{0.107}$	$0.341_{0.003}$	$17.609_{0.146}$	$0.275_{0.009}$	$24.787_{0.242}$	$0.185_{0.003}$
ADRNet	$4.215_{0.029}$	$0.646_{0.002}$	$6.031_{0.382}$	$0.490_{0.023}$	$4.577_{0.073}$	$0.804_{0.003}$	$9.351_{0.054}$	$0.645_{0.001}$
MMRNet	$3.772_{0.043}$	$0.675_{0.002}$	$2.954_{0.002}$	$0.668_{0.000}$	$5.802_{0.031}$	$0.738_{0.001}$	$8.636_{0.096}$	$0.675_{0.003}$

Table 3: Registration results (

mean_{std}

) for the non-rigid registration task. The best results are marked in bold and second bests are underlined .

Figure 4: Qualitative comparison of rigid registration. F: fixed image, M: moving image. From top to bottom: RGB-SAR, RGB-IR, RGB-NIR, and RGB-TIR.

Figure 5: Qualitative comparison of non-rigid registration. F: fixed image, M: moving image. From top to bottom: RGB-SAR, RGB-IR, RGB-NIR, and RGB-TIR.

	Method	RGB-TIR		RGB-SAR
	Method	RE $\downarrow$	NCC $\uparrow$	RE $\downarrow$	NCC $\uparrow$
(1)	w/o MSBN	0.924	0.790	3.475	0.855
(2)	w/o CDAP	0.822	0.793	3.407	0.857
(3)	Ours	$\mathbf{0.744}$	$\mathbf{0.796}$	$\mathbf{3.161}$	$\mathbf{0.866}$

Table 4: Ablation of MSBN and CDAP on rigid registration for RGB–TIR and RGB–SAR datasets.

Figure 6: Linear CKA comparison with/without MSBN across scales

Figure 7: Visualization of disentangled feature distributions on RGB–TIR and RGB–SAR datasets.

	$L_{ccd}$	$L_{bo}$	$L_{cs}$	$L_{tri}$	RGB-TIR		RGB-SAR
	$L_{ccd}$	$L_{bo}$	$L_{cs}$	$L_{tri}$	RE $\downarrow$	NCC $\uparrow$	RE $\downarrow$	NCC $\uparrow$
(1)					2.403	0.6905	7.445	0.7125
(2)	✓				2.356	0.6916	7.344	0.7126
(3)	✓	✓			2.309	0.6970	7.306	0.7153
(4)	✓	✓	✓		2.297	0.6982	7.242	0.7164
(5)	✓	✓	✓	✓	2.264	0.7008	7.172	0.7198

Table 5: Loss ablation on non-rigid registration for RGB–TIR and RGB–SAR datasets.

Task	Steps in HPPM		RGB-TIR		RGB-SAR
Task	$N_{\text{rigid}}$	$N_{\text{nonrigid}}$	RE $\downarrow$	NCC $\uparrow$	RE $\downarrow$	NCC $\uparrow$
Rigid task	5	0	6.378	0.5284	15.498	0.4473
	4	1	1.872	0.7277	9.712	0.6320
	3	2	1.344	0.7689	7.795	0.7712
	2	3	0.801	0.7931	4.572	0.8074
	1	4	0.744	0.7960	3.161	0.8664

Table 6: Ablation on the number of rigid vs. non-rigid steps in HPPM on non-rigid registration for RGB–TIR and RGB–SAR datasets. (total steps fixed to 5).

Explore similar work

Cross-Modality Feature Fusion Based on Structured State Space Duality for Multimodal Image Registration Network
Jun 2, 2026Zhikang Li, Yan Wu, Xin Hu +2Multi-Scale Feature Fusion Cross-Modal Feature Matching
Remote Sensing Image Processing and Fusion Group, School of Electronic Engineering, Xidian University, Xi’an 710071, China · National Key Laboratory of Radar Signal Processing, Xidian University, Xi’an 710071, China
CRFT: Consistent-Recurrent Feature Flow Transformer for Cross-Modal Image Registration
Apr 7, 2026Xuecong Liu, Mengzhu Ding, Zixuan Sun +2Cross-Modal Image Registration Cross-Modal Feature Matching
Northeastern University, China · National University of Defense Technology, China
Adapting Frozen Mono-modal Backbones for Multi-modal Registration via Contrast-Agnostic Instance Optimization
Mar 27, 2026Yi Zhang, Yidong Zhao, Qian TaoDeformable Image Registration Cross-Modal Image Registration
Department of Imaging Physics, Delft University of Technology, The Netherlands

Jun 2, 2026J/K move · Enter open · S save

Cross-Modality Feature Fusion Based on Structured State Space Duality for Multimodal Image Registration Network

Zhikang Li, Yan Wu, Xin Hu, Yi Dai+1

Remote Sensing Image Processing and Fusion Group, School of Electronic Engineering, Xidian University, Xi’an 710071, China · National Key Laboratory of Radar Signal Processing, Xidian University, Xi’an 710071, China

Multi-Scale Feature Fusion Cross-Modal Feature Matching Cross-Modal Image Registration Multimodal Fusion Feature Fusion

Open paper AskPDF arXiv

In multi-modal image registration, the primary challenge lies in shared structural information extraction. Compared to Transformers, Structured State Space Duality (SSD) offers greater global structural feature extraction with higher efficiency during training and inference. Inspired by these advantages, we propose a novel algorithm for multi-modal image registration, named RegNetMamba-2. Our algorithm incorporates SSD into coarse-to-fine matching process to extract local and global structural features effectively. Firstly, SSD is applied in three different scales for multi-modal feature extraction in our network. To strengthen local representation, we pay more attention on foreground edge and structural information by feature scaling function of SSD. Secondly, for shared feature extraction of input images and multi-modal feature fusion in all scales, we propose cross-modality feature fusion model based on SSD, consisting of Cross-Modality feature Interaction (CMI) module and Multi-Scale feature Fusion (MSF) module. CMI module is designed for cross-modality feature extraction of each scale by SSD in cross form. MSF module is designed to employ a progressive upward fusion in feature-level to obtain fine features, consisting of multi-modal features in all scales. Following coarse-to-fine, the features in 1/8 scale from CMI and 1/2 scale from MSF are collected to calculate matching probability scores. Then we respectively establish matching process by correspondences of pixel-wise. Extensive experiments demonstrate that comparing with state-of-the-art deep-learning based algorithms, RegNetMamba-2 has achieved good effects in both performance and efficiency for multi-modal image registration on the following datasets: VIS-SAR (OSDataset), VIS-IR (LGHD/RoadSence) and VIS-NIR (RGB-NIR sense).

Abstract

Explore similar work

Cross-Modality Feature Fusion Based on Structured State Space Duality for Multimodal Image Registration Network

CRFT: Consistent-Recurrent Feature Flow Transformer for Cross-Modal Image Registration

Adapting Frozen Mono-modal Backbones for Multi-modal Registration via Contrast-Agnostic Instance Optimization