DMM-Align: Closed-Loop Optimization for 2D-3D Registration with Dual-Role Diffusion

Chongjian Wang; Junjie Gao

cs.CVAug 17, 2026

DMM-Align: Closed-Loop Optimization for 2D-3D Registration with Dual-Role Diffusion

Authors: Chongjian Wang, Junjie Gao

Organizations: Shandong Women’s University Jinan, China · Shandong University of Science and Technology Qingdao, China

Abstract

2D-3D registration remains brittle in challenging scenarios such as low overlap, occlusion, repetitive structures, and severe cross-modal ambiguity. A key reason is that existing methods improve representation learning, correspondence estimation, or pose computation in isolation, while the dominant failure mode is inherently cross-level, where errors propagate between features, correspondences, and pose. To address this limitation, we propose DMM-Align: Diffusion-based Matching Matrix Alignment, a closed-loop framework that couples correspondence refinement, pose estimation, and representation learning through a shared differentiable geometric state. Our method leverages diffusion in two coordinated roles: a geometry-aware diffusion process refines the soft matching matrix for robust correspondence estimation, while a geometry-conditioned diffusion teacher injects pose-induced supervision back into feature learning. These processes are connected via a differentiable geometric hinge that converts correspondences into a global pose and exposes geometric inconsistency to upstream modules. Extensive experiments on 7-Scenes and RGB-D Scenes V2 demonstrate that DMM-Align consistently outperforms strong baselines, especially under low-overlap and heavy-occlusion conditions, highlighting the effectiveness of closed-loop geometric feedback for robust 2D-3D registration.

Figures & tables

Figure 1. Comparison of correspondence confidence maps for a query 3D point. We select a point in a structurally repetitive region and visualize its matching confidence over the image as predicted by Diff2I2P and our method. Our method produces a response that is more concentrated near the ground-truth correspondence.

Figure 2. Overview of DMM-Align. Given a query image and a scene point cloud, a detection-free coarse-to-fine backbone extracts multi-scale features and constructs an initial soft 2D–3D matching matrix. A geometry-aware diffusion module refines the correspondence structure within local candidate subspaces. The refined matches are converted into a global pose via a differentiable geometric hinge, which also produces pose-induced cues such as depth, visibility, and reprojection consistency. These cues are fed into a geometry-conditioned diffusion teacher to distill pose-level supervision back to the feature space, forming a closed-loop optimization across features, correspondences, and pose. During inference, the distillation branch is removed, and the pipeline reduces to feature extraction, matching refinement, and pose estimation.

Model	Chess	Fire	Heads	Office	Pump	Kitchen	Stairs	Mean
Inlier Ratio $\uparrow$
FCGF ( Choy2019FullyCG )	34.2	32.8	14.8	26.0	23.3	22.5	6.0	22.8
P2Net ( Wang2021P2NetJD )	55.2	46.7	13.0	36.2	32.0	32.8	5.8	31.7
Predator ( Huang2020PREDATORRO )	34.7	33.8	16.6	25.9	23.1	22.2	7.5	23.4
MATR ( Li20232D3DMATR2M )	72.1	66.0	31.3	60.7	50.2	52.5	18.1	50.1
Diff 2 I2P ( mu2025diff2i2p )	74.1	68.8	39.2	65.6	52.1	54.2	18.1	53.2

Table 1. Registration results of DMM-Align and baselines on 7-Scenes. The best result for each metric is shown in bold .

Figure 3. Qualitative results on the 7-Scenes dataset. The red lines indicate erroneous correspondences (3D distance greater than 5 cm), while the green lines represent correct correspondences.

Method	PIR (%) $\uparrow$	IR (%) $\uparrow$	FMR (%) $\uparrow$	RR (%) $\uparrow$
FCGF ( Choy2019FullyCG )	20.1	10.3	29.2	32.5
P2-Net ( Wang2021P2NetJD )	30.4	14.5	63.7	41.7
Predator ( Huang2020PREDATORRO )	32.6	15.8	68.1	33.6
MATR ( Li20232D3DMATR2M )	57.6	36.3	76.0	56.9
Diff 2 I2P ( mu2025diff2i2p )	60.8	36.9	77.1	60.5
DMM-Align	62.4	39.8	78.4	63.7

Table 2. Registration results of DMM-Align and baselines on RGB-D Scenes V2. The best result for each metric is shown in bold .

Model	IR $\uparrow$	FMR $\uparrow$	RR $\uparrow$
Matrix diffusion only	52.7	92.1	81.6
Distillation only	53.3	92.3	83.3
Coexistence without coupling	53.9	92.4	84.7
DMM-Align (full)	54.5	92.5	86.2

Table 3. Overall effect of DMM-Align on 7-Scenes. DMM-Align consistently outperforms reduced variants, showing the benefit of explicit closed-loop coupling.

Matching formulation	IR $\uparrow$	FMR $\uparrow$	RR $\uparrow$
Strict doubly stochastic	52.6	92.0	82.4
Partial doubly stochastic	53.1	92.1	83.6
Partial DS + dustbin	53.8	92.3	84.7
Partial DS + dustbin + geometry-aware diffusion	54.5	92.5	86.2

Table 4. Effect of matching matrix modeling on 7-Scenes. More flexible correspondence modeling and geometry-aware diffusion consistently improve performance.

Figure 4. Robustness under local point cloud degradation. From left to right, the occlusion ratio increases from $0$ to $0.6$ . (a) Input image and degraded point cloud. (b) Diff 2 I2P. (c) DMM-Align. Green and red lines denote correct and incorrect matches, respectively, and IR is reported for each case. DMM-Align consistently achieves higher IR under severe degradation.

Hinge variant	IR $\uparrow$	FMR $\uparrow$	RR $\uparrow$
Without hinge	53.2	92.1	83.8
+ 3D offset only	53.7	92.2	84.7
+ 3D offset + 2D offset	54.1	92.4	85.5
+ 3D offset + 2D offset + uncertainty weighting	54.5	92.5	86.2

Table 5. Effect of the differentiable geometric hinge on 7-Scenes. Residual correction and uncertainty-aware weighting consistently improve performance.

Distillation variant	IR $\uparrow$	FMR $\uparrow$	RR $\uparrow$
Depth only + binary mask	52.5	92.0	82.4
Depth + reprojection + soft mask	53.3	92.2	83.9
Full geometry bundle + soft mask	54.0	92.4	85.2
Full geometry bundle + uncertainty-aware soft mask	54.5	92.5	86.2

Table 6. Effect of geometry-conditioned distillation on 7-Scenes. Richer geometric cues and uncertainty-aware masking consistently improve performance.

2D-3D Image Registration

See all 21 papers

Aug 6, 2026

Ordered Diffusion for 3D Human Registration

3D human registration has historically been treated as a regression task, assuming a unique ground-truth alignment exists between the template and an input point cloud. In reality, acquisition noise, occlusions, and unknown soft tissue dynamics introduce inherent ambiguity into human scans. Regression-based methods consequently converge to an average prediction, often failing to represent a plausible geometry. In our work, we embrace such uncertainty by modeling the registration as a distribution of alignments. We propose ODin, which formulates registration as a 3D diffusion process that generates a point cloud aligned with the target geometry while preserving template semantics through consistent point ordering. To achieve this, ODin relies on global, local, and positional conditioning, guiding each point to its correct location. Our experiments demonstrate that such a generative formulation not only outperforms its regression-based baseline, but also establishes a new state of the art, surpassing highly engineered methods while reducing the registration time by two-thirds. Pre-trained models and code are available at https://riccardomarin.github.io/odin/.

Mattia Masiero, Ilya A. Petrov, Daniel Cremers +2

University of Tübingen, Germany · Tübingen AI Center, Germany · Technical University of Munich, Germany +2

Point Cloud Registration 3D Diffusion Models

Ask PDF

Sep 16, 2026

Mask 2D-3D: Adaptive Dual-Masked Autoencoder Network for Image-to-Point Cloud Registration

Detection-free methods for image-to-point cloud registration are prone to erroneous correspondences caused by domain and modality discrepancies, limited sensitivity of feature extractors, and the presence of non-overlapping regions. The Masked Autoencoder (MAE) has shown strong performance in visual representation for images and point clouds. It may be helpful to apply this approach to image-to-point cloud registration, a task that requires unified feature extraction and accurate cross-modal correspondences. Standard MAE's random masking may overlook key regions due to limited camera views, reducing registration effectiveness. To address this, we propose the Intermodal Dual-MAE Framework (ID-MAE) with a Similarity-based RL Masking Strategy (SRLM), which adaptively masks informative positions by leveraging cross-modal similarity and reinforcement learning, thus narrowing the modality gap. Our method enhances cross-modal representation learning by enforcing representation consistency during feature extraction, thereby enabling more reliable 2D-3D correspondence estimation. Experiments on RGB-D Scenes v2 and 7-Scenes benchmarks show that our method achieves state-of-the-art performance in image-to-point cloud registration.

Zhixin Cheng, Jiacheng Deng, Xiaotian Yin +3

School of Computer Science and Technology, Hefei University of Technology, Hefei 230601, China · School of Information Science and Technology, University of Science and Technology of China, Hefei 230027, China · Institute of Advanced Technology, University of Science and Technology of China, Hefei 230027, China

Masked Autoencoders Point Cloud Registration Cross-Modal Feature Matching Cross-Modal Image Registration 2D-3D Image Registration

Ask PDF

May 5, 2026

UniCorrn: Unified Correspondence Transformer Across 2D and 3D

Visual correspondence across image-to-image (2D-2D), image-to-point cloud (2D-3D), and point cloud-to-point cloud (3D-3D) geometric matching forms the foundation for numerous 3D vision tasks. Despite sharing a similar problem structure, current methods use task-specific designs with separate models for each modality combination. We present UniCorrn, the first correspondence model with shared weights that unifies geometric matching across all three tasks. Our key insight is that Transformer attention naturally captures cross-modal feature similarity. We propose a dual-stream decoder that maintains separate appearance and positional feature streams. This design enables end-to-end learning through stack-able layers while supporting flexible query-based correspondence estimation across heterogeneous modalities. Our architecture employs modality-specific backbones followed by shared encoder and decoder components, trained jointly on diverse data combining pseudo point clouds from depth maps with real 3D correspondence annotations. UniCorrn achieves competitive performance on 2D-2D matching and surpasses prior state-of-the-art by 8% on 7Scenes (2D-3D) and 10% on 3DLoMatch (3D-3D) in registration recall. Project website: https://neu-vi.github.io/UniCorrn

Prajnan Goswami, Tianye Ding, Feng Liu +1

Northeastern University · Adobe Research

Image Matching Point Cloud Registration Cross-Modal Feature Matching 2D-3D Image Registration

Ask PDF

Abstract

2D-3D Image Registration

Explore similar work

Ordered Diffusion for 3D Human Registration

Mask 2D-3D: Adaptive Dual-Masked Autoencoder Network for Image-to-Point Cloud Registration

UniCorrn: Unified Correspondence Transformer Across 2D and 3D