SAGE: Semantic-Augmented Geometric Experts for Point Cloud Registration

Tomoyasu Shimada; Kazuhiko Murasaki; Shoichiro Takeda; Shogo Sato; Riku Inoue; Toshihiko Nishimura; Ryuichi Tanida

cs.CVOct 27, 2026

SAGE: Semantic-Augmented Geometric Experts for Point Cloud Registration

Authors: Tomoyasu Shimada, Kazuhiko Murasaki, Shoichiro Takeda, Shogo Sato, Riku Inoue, Toshihiko Nishimura, Ryuichi Tanida

Organizations: NTT, Inc.

Abstract

Point cloud registration is an essential task for scene reconstruction, requiring the alignment of partially overlapping point clouds. Under low-overlap conditions, this task remains challenging because existing geometry-only methods cannot distinguish geometrically similar but semantically distinct structures (e.g., TV corners vs. desk corners and chair legs vs. table legs), resulting in incorrect correspondences. To address this geometric ambiguity, we present Semantic-Augmented Geometric Experts (SAGE), which integrates frozen pre-trained point cloud semantic representations into the registration pipeline without requiring additional sensors or fine-tuning the semantic encoder. To effectively leverage these semantic distinctions, SAGE employs a Semantic-Guided Mixture of Experts (SG-MoE) architecture. Through semantic-aware routing, SG-MoE enables each expert to specialize in distinct semantic-geometric patterns, reducing mismatches that confound geometry-only methods. With a structure-aware estimator, SAGE achieves the highest registration recall on 3DLoMatch while remaining competitive on 3DMatch. When evaluated with the same number of refinement iterations, SAGE further achieves the highest inlier ratio and registration recall on both benchmarks. These results indicate that frozen semantic representations provide effective cues for resolving geometric ambiguity in low-overlap registration.

Figures & tables

Figure 1: Limitation of geometric features in low-overlap point cloud registration. Point clouds are pseudo-colored by semantic regions for visualization. Top: GeoTransformer [ 26 ] , relying solely on geometric features, is overwhelmed by dominant incorrect correspondences (red lines) due to similar geometric structures, failing to establish true inliers (no green lines visible) . Bottom: SAGE (Ours) successfully disambiguates geometrically similar but semantically distinct structures. Green and red lines indicate inlier and outlier correspondences (threshold: 0.1 m).

Figure 2: Overall pipeline of SAGE. Given two input point clouds, the Two-Stage Feature Extraction (Sec. 3.2 ) first applies a frozen Sonata encoder to extract point-level semantic features from each point cloud, and then feeds these semantic features together with the raw point coordinates into the KPConv encoder to produce semantic-geometric features. The Spatial Feature Aggregation (Sec. 3.3 ) then maps both the semantic and semantic-geometric features to the superpoint level via nearest-neighbor search and average pooling. The aggregated features are refined through $N$ alternating iterations of Geometric Transformer layers (red box, self-attention) and Semantic-Guided Mixture of Experts (Sec. 3.4 ) layers (blue box, cross-attention), where semantic features guide expert routing to enable semantic-aware specialization. Finally, the framework adopts GeoTransformer’s coarse-to-fine matching for correspondence estimation and registration.

Panel of Figure 3 — Figure 3: Comparison of MoE architectures . (a) Standard MoE feeds the fused semantic-geometric features into both the router and the experts. (b) Our SG-MoE decouples the inputs, feeding pure semantic features to the router and semantic-geometric features to the experts. These processes apply symmetrically with $\mathcal{P}$ and $\mathcal{Q}$ swapped.

	3DMatch					3DLoMatch
# Samples	5000	2500	1000	500	250	5000	2500	1000	500	250
Feature Matching Recall (%) $\uparrow$
Predator [ 17 ]	96.6	96.6	96.5	96.3	96.5	78.6	77.4	76.3	75.7	75.3
CoFiNet [ 49 ]	98.1	98.3	98.1	98.2	98.3	83.1	83.5	83.3	83.1	82.6
GeoTrans. [ 26 ]	97.9	97.9	97.9	97.9	97.6	88.3	88.6	88.8	88.6	88.3
RoITr [ 50 ]	98.0	98.0	97.9	98.0	97.9	89.6	89.6	89.5	89.4	89.3

Table 1: Evaluation results on 3DMatch and 3DLoMatch using RANSAC-50K. Pink rows use iterative refinement. Boldface indicates the best result and underline indicates the second-best result within each group. Iterative PEAL and PSReg are reproduced under our setting and fall below their reported numbers.

	3DMatch			3DLoMatch
	FMR $\uparrow$	IR $\uparrow$	RR $\uparrow$	FMR $\uparrow$	IR $\uparrow$	RR $\uparrow$
GeoTrans. [ 26 ]	98.2	70.9	92.5	87.1	43.5	74.0
RoITr [ 50 ]	98.1	82.0	90.9	89.4	54.4	73.4
SAGE ( ours )	98.2	71.7	91.0	89.4	44.3	76.3
PEAL [ 51 ]	97.8	73.5	91.5	85.8	49.5	76.9
PSReg [ 18 ]	97.1	73.4	92.1	85.4	49.5	77.4

Table 2: Evaluation on 3DMatch and 3DLoMatch with LGR [ 26 ] .

Configuration	FMR $\uparrow$ [%]	IR $\uparrow$ [%]	RR $\uparrow$ [%]
baseline	87.1	43.5	74.0
Two-Stage Feature Extraction only	88.5	40.4	74.7
Standard MoE only	87.5	42.6	74.7
SG-MoE only	87.5	43.8	75.3
Full SAGE	89.4	44.3	76.3

Table 3: Ablation study on 3DLoMatch with LGR.

Figure 4: Qualitative comparison of correspondences on 3DLoMatch scenes. Each column shows a different point cloud pair. Top: GeoTransformer’s registration with correspondences (red for outliers, green for inliers). Bottom: SAGE’s registration and correspondences. SAGE effectively leverages semantic guidance to correctly identify inliers, even in low-overlap scenarios.

Figure 5: Expert activation patterns. Top: Standard MoE produces fragmented expert assignments primarily driven by local geometry. Bottom: SG-MoE consistently assigns semantically similar regions, such as the furniture highlighted by red circles, to the same expert, demonstrating semantic-guided specialization.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	GeoTransformer	SAGE
1	KPConv( $1\rightarrow 64$ )	KPConv( 1088 $\rightarrow 64$ )
	ResBlock( $64\rightarrow 128$ )	ResBlock( $64\rightarrow 128$ )
2	ResBlock( $128\rightarrow 128$ , strided)	ResBlock( $128\rightarrow 128$ , strided)
	ResBlock( $128\rightarrow 256$ )	ResBlock( $128\rightarrow 256$ )
	ResBlock( $256\rightarrow 256$ )	ResBlock( $256\rightarrow 256$ )
3	ResBlock( $256\rightarrow 256$ , strided)	ResBlock( $256\rightarrow 256$ , strided)

Appendix

Table A : Geometric Feature Extractor (KPConv) architecture. SAGE modifies only the input dimension of the first KPConv layer (red) to accommodate 1024-dimensional Sonata semantic features concatenated with 64-dimensional positional encodings.

Downsampling
Stage	GeoTransformer	SAGE
1	KPConv( $1\rightarrow 64$ )	KPConv( 1088 $\rightarrow 64$ )
	ResBlock( $64\rightarrow 128$ )	ResBlock( $64\rightarrow 128$ )
2	ResBlock( $128\rightarrow 128$ , strided)	ResBlock( $128\rightarrow 128$ , strided)
	ResBlock( $128\rightarrow 256$ )	ResBlock( $128\rightarrow 256$ )
	ResBlock( $256\rightarrow 256$ )	ResBlock( $256\rightarrow 256$ )

Appendix

Table B : KPConv-FPN architecture for KITTI. SAGE uses a 5-stage backbone with an additional downsampling stage (Stage 5) compared to 3DMatch to handle larger outdoor scenes. The input feature dimension of the first KPConv layer is modified from 1 to 1088 (highlighted in red) to incorporate semantic features.

	RTE[cm] $\downarrow$	RRE[ ∘ ] $\downarrow$	RR[%] $\uparrow$
FCGF	9.5	0.3	96.6
D3Feat	7.2	0.3	99.8
FMR	$\sim$ 66	1.49	90.6
DGR	$\sim$ 32	0.37	98.7
SpinNet	9.9	0.47	99.1
Predator	6.8	0.27	99.8

Appendix

Table C : Registration performance on KITTI.

Scenes	Kitchen	Home 1	Home 2	Hotel 1	Hotel 2	Hotel 3	Study	MIT Lab	All
#Pairs	1031	445	438	444	262	103	532	149	3404
GeoTrans.	91.7	84.9	85.4	94.1	80.2	86.4	75.2	83.9	86.3
RoITr	94.0	81.6	85.4	93.9	80.9	85.4	74.3	81.9	86.4
SAGE (Ours)	94.2	82.3	86.1	95.7	82.8	85.4	80.3	81.2	87.9

Appendix

Table D : Registration Recall (%) on the 3DMatch and 3DLoMatch benchmark across different scene categories. #Pairs denotes the total number of point cloud pairs evaluated in each scene.

Overlap Ratio	0-10%	10-20%	20-30%	30-40%	40-50%	50-60%	60-70%	70-80%	80-90%	90-100%
#Pairs	3	1040	832	387	417	304	224	131	58	8
GeoTrans.	66.7	70.6	86.4	94.8	95.2	98.7	99.6	100.0	100.0	100.0
RoITr	66.7	72.2	85.5	94.6	94.5	98.0	98.7	100.0	100.0	100.0
SAGE (Ours)	66.7	73.4	88.7	95.3	96.2	98.4	99.6	100.0	100.0	100.0

Appendix

Table E : Registration Recall (%) comparison across different overlap ratios. #Pairs denotes the number of point cloud pairs in each overlap bin.

Method	Params (M)	Mem. (GB)*	Time (ms)**	RR (%)
GeoTransformer	9.83	7.02	75	74.0
RoITr	10.10	9.28	140	73.4
SAGE (Ours)	121.71	16.69	179	76.3

Appendix

Table F : Computational Efficiency Analysis

Sonata Adaptation
Configuration	FMR	IR	RR
w/o adapter	89.4	44.3	76.3
w/ adapter	86.4	42.6	74.8
Routing Strategy
SG-MoE (ours)	89.4	44.3	76.3
SCAR	88.1	44.4	75.0

Appendix

Table G : Additional ablation studies on 3DLoMatch with LGR. All metrics in %. We evaluate: (1) adding adapters to Sonata, (2) alternative routing strategy using cross-attention (SCAR), and (3) varying the number of experts.

Figure A : The Overview of SCAR. Semantic Cross-Attention Routing (SCAR) first computes cross-attention between source and target semantic features, then uses the attended features for routing. While SCAR incorporates inter-point-cloud information earlier, this premature soft matching introduces noise in low-overlap scenarios, leading to inferior performance (76.3% vs. 75.0% RR on 3DLoMatch).

Perturbation	10%	20%	30%	40%	50%
Gaussian noise	76.4	75.7	75.6	76.3	76.2
Feature dropout	76.4	76.1	75.0	75.8	76.2
Random replacement	75.8	75.1	74.8	74.8	73.5

Appendix

Table H : Robustness to semantic feature degradation. Registration Recall (%).

Figure B : Qualitative comparison on 3DLoMatch. We compare the registration quality between GeoTransformer (baseline) and SAGE (ours) on challenging low-overlap pairs. From left to right: (a) Input source (yellow) and target (blue) point clouds. (b) Correspondences generated by GeoTransformer with IR. (c) Correspondences generated by SAGE with IR. (d) Registration result of GeoTransformer with RMSE. (e) Registration result of SAGE with RMSE. (f) Ground truth alignment with overlap ratio. SAGE consistently achieves accurate registration (low RMSE) even in cases where GeoTransformer completely fails (e.g., 3rd and 4th rows).

Figure C : Additional failure cases on 3DLoMatch. Each row shows a challenging scenario where SAGE fails to achieve accurate registration. From left to right: (a) Source point cloud (cyan), (b) Target point cloud (yellow), (c) SAGE correspondences with Inlier Ratio (red lines: outliers, green lines: inliers), (d) SAGE registration result, (e) Ground truth alignment. These failures occur due to: (1) severe occlusion reducing the overlapping area, (2) geometrically indiscriminative regions (uniform walls, floors, ceilings), and (3) severe truncation of semantic regions, preventing the semantic encoder from extracting reliable semantic representations. When semantic structures are incomplete (e.g., only partial views of furniture), the frozen Sonata encoder cannot provide discriminative semantic features, as it was trained on complete object instances.

Figure D : Expert routing patterns of SG-MoE across six scenes from 3DMatch and 3DLoMatch. Each scene is shown as a source-target pair, where superpoints are color-coded by their assigned expert (red, blue, green, yellow) and bounding boxes highlight semantically salient regions. The same semantic regions in the source and target point clouds are routed to the same expert, ensuring compatible feature representations for correspondence matching. This consistent functional specialization emerges across diverse scenes without explicit supervision, confirming that the frozen semantic features enable scene-agnostic and view-invariant expert routing.

Point Cloud Registration

Point cloud registration aligns 3D point sets by estimating the transformations and correspondences that make their geometry consistent.

See all 111 papers

GeGS-PCR: Effective and Robust 3D Point Cloud Registration with Two-Stage Color-Enhanced Geometric-3DGS Fusion
Apr 20, 2026Jiayi Tian, Haiduo Huang, Tian Xia +2Point Cloud Registration
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center of Visual Information and Applications, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, Shaanxi, China
CMHANet: A Cross-Modal Hybrid Attention Network for Point Cloud Registration
Mar 13, 2026Dongxu Zhang, Yingsen Wang, Yiding Sun +33D Shape Matching Point Cloud Registration
School of Software, Xi’an Jiaotong University, No. 28 Xianning West Road, China · CEPRI, China Electric Power Research Institute No. 15 Xiaoying East Road, Qinghe, Haidian District, Beijing, China
Unsupervised Point Cloud Registration via Training-Time Semantic Guidance
Sep 14, 2026Kezheng Xiong, Shiyun Xu, Sheng Ao +3Point Cloud Registration
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, China · Fujian Key Laboratory of Urban Intelligent Sensing and Computing, Xiamen University, China

Apr 20, 2026J/K move · Enter open · S save

GeGS-PCR: Effective and Robust 3D Point Cloud Registration with Two-Stage Color-Enhanced Geometric-3DGS Fusion

Jiayi Tian, Haiduo Huang, Tian Xia, Wenzhe Zhao+1

National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center of Visual Information and Applications, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, Shaanxi, China

Point Cloud Registration

Open paper AskPDF arXiv

We address the challenge of point cloud registration using color information, where traditional methods relying solely on geometric features often struggle in low-overlap and incomplete scenarios. To overcome these limitations, we propose GeGS-PCR, a novel two-stage method that combines geometric, color, and Gaussian information for robust registration. Our approach incorporates a dedicated color encoder that enhances color features by extracting multi-level geometric and color data from the original point cloud. We introduce the Geometric-3DGS module, which encodes the local neighborhood information of colored superpoints to ensure a globally invariant geometric-color context. Leveraging LORA optimization, we maintain high performance while preserving the expressiveness of 3DGS. Additionally, fast differentiable rendering is utilized to refine the registration process, leading to improved convergence. To further enhance performance, we propose a joint photometric loss that exploits both geometric and color features. This enables strong performance in challenging conditions with extremely low point cloud overlap. We validate our method by colorizing the Kitti dataset as ColorKitti and testing on both Color3DMatch and Color3DLoMatch datasets. Our method achieves state-of-the-art performance with Registration Recall at 99.9%, Relative Rotation Error as low as 0.013, and Relative Translation Error as low as 0.024, improving precision by at least a factor of 2.

Abstract

Point Cloud Registration

Explore similar work

GeGS-PCR: Effective and Robust 3D Point Cloud Registration with Two-Stage Color-Enhanced Geometric-3DGS Fusion

CMHANet: A Cross-Modal Hybrid Attention Network for Point Cloud Registration

Unsupervised Point Cloud Registration via Training-Time Semantic Guidance