Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · 3Beihang University
Camera localization methods are mostly multi-stage pipelines relying on dedicated and complex maps that are costly to construct, store, and maintain over time. Alternatively, well-structured maps that organize scenes with compact yet expressive primitives are often readily available, providing rich structural and semantic cues that humans can interpret. However, their potential for direct camera localization remains largely unexplored due to the significant modality gap between map primitives and visual observations. This motivates a new concept for localization, which estimates camera poses by grounding query observations to map primitives. In this paper, we instantiate the concept in the context of planar primitives with PGGT, the Primitive-Grounded Geometry Transformer network. Given a single query image and a set of planar map primitives, the network is trained to predict multiple localization targets for feed-forward 6-DoF camera localization that generalizes to unseen environments. With this straightforward and scalable architecture, our method pushes the state-of-the-art localization performance on both dense surface maps and minimal layout models. The code and models are available at https://github.com/3dv-casia/PGGT-Loc
Figures & tables
Figure 1: Overview. (a) PlanaReLoc [ 90 ] localizes cameras against dense surface maps by explicitly matching 2D plane proposals with planar map primitives. (b) SPVLoc [ 25 ] localizes cameras against minimal layout models by retrieving the best-matching panoramic rendering. (c) Our Pggt -Loc operates directly on query patches and map primitives, enabling feed-forward localization across both dense surface maps and minimal layout models with a unified base architecture.
Figure 2: Model Architecture . (a) Two embedders first tokenize the inputs into patch and primitive tokens, respectively. (b) The two-branch transformer backbone refines the two sets of tokens, together with the prepended camera and map tokens, and enables interactions between them. (c) Downstream prediction heads produce different localization targets that enable camera pose inference via three pathways, which we refer to as the (1) Param.-based, (2) Ray-based, and (3) Token-based approaches.
Figure 3: Scene architectural layout. Left : 3D layout model with ceilings culled. Right : corresponding 2D floorplan with semantic labels.
Δq ( \SIUnitSymbolDegree ) ↓
Δt ( m ) ↓
Pose Recall ( % ) ↑
Time ( s−1 ) ↓
Methods
Mean
Med.
Mean
Med.
( 0.2m , 10\SIUnitSymbolDegree )
( 0.5m , 15\SIUnitSymbolDegree )
( 1.0m , 30\SIUnitSymbolDegree )
MeshLoc
MASt3R [ 41 ]
46.0
12.2
1.02
0.43
35.4
49.5
57.6
∼0.683168
MatchAnything [ 30 ]
35.7
19.9
1.23
0.74
20.0
35.7
52.1
∼0.925184
NOPE-SAC [ 73 ]
28.7
15.9
0.90
0.77
3.3
21.2
54.6
∼0.368222
Plana3R [ 44 ]
26.8
12.9
0.92
0.52
17.9
37.6
57.1
∼0.423516
PlaneRecTR++ [ 68 ]
20.7
9.9
0.66
0.54
12.2
39.2
75.1
∼0.412271995
Table 1: Quantitative results on ScanNet. We report rotation and translation errors, along with pose recalls across three thresholds. We also report the average inference time per query, using a single NVIDIA RTX 3090 GPU. Top-3 results are highlighted as the first , second , and third .
Med. Err. ↓
Pose Recall ( % ) ↑
Primitive Matching ( % ) ↑
Methods
Δq ( \SIUnitSymbolDegree )
Δt ( m )
( 0.2m , 10\SIUnitSymbolDegree )
( 0.5m , 15\SIUnitSymbolDegree )
( 1.0m , 30\SIUnitSymbolDegree )
Prec.
Rec.
F 1
AP
MASt3R [ 41 ]
12.0
0.30
45.2
51.9
59.0
59.8
42.9
50.0
81.6
MatchAnything [ 30 ]
7.9
0.20
46.9
63.4
77.6
51.2
56.1
53.5
77.2
NOPE-SAC [ 73 ]
17.7
0.54
2.9
25.9
67.2
43.8
22.0
29.3
71.4
Plana3R [ 44 ]
15.9
0.50
7.9
30.1
69.3
-
-
-
-
PlaneRecTR++ [ 68 ]
9.8
0.39
11.1
53.7
83.5
55.0
24.9
34.3
75.4
Table 2: Quantitative results on 12Scenes. We present the cross-dataset performance in terms of both localization ( i.e . , median errors and pose recall) and primitive matching.
Figure 4: Qualitative comparisons with PlanaReLoc. Our method robustly produces accurate pose estimation, with (b) superior plane recovery and primitive correspondences as additional outputs. In contrast, (c) PlanaReLoc performs suboptimally and even fails due to poor plane recovery (marked by the boxes ) and matching outliers (bottom row, colored in red ). (d) Comparison of localization results. Legend: the ground truth , Pggt (Ours) , and PlanaReLoc .
Map Type
Use Semantics
<1m Med. Error ↓ (2D)
Pose Recall ( % , 2D) ↑
Paradigm
Methods
Δtxy ( cm )
Δqyaw ( \SIUnitSymbolDegree )
10cm
50cm
1m
1m & 30\SIUnitSymbolDegree
Render & Compare
LASER [ 52 ]
2D
-
-
7.0
6.4
10.4
8.7
F 3 Loc [ 16 ]
2D
-
-
1.5
14.6
22.4
21.3
3DP [ 14 ]
2D
-
-
5.6
27.4
55.5
24.0
SemRayLoc [ 27 ]
2D
✓
-
-
5.7
45.5
58.8
57.5
RSK [ 15 ]
2D
-
-
6.4
28.6
56.9
25.2
Table 3: Quantitative results on S3D- official . Different methods employ either 2D floorplans or 3D layout models as the reference map, with some further leveraging map semantics. We report median 2D localization errors over <1m inliers, along with pose recall at different thresholds. Top-3 results are highlighted as the first , second , and third . † method not requiring known intrinsics ( i.e . , FoV).
Med. Err. ↓ (3D)
<1m Med. Error ↓ (2D)
Pose Recall ( % , 2D) ↑
Methods
Δq ( \SIUnitSymbolDegree )
Δt ( cm )
Δqyaw ( \SIUnitSymbolDegree )
Δtxy ( cm )
10cm
50cm
1m
1m & 30\SIUnitSymbolDegree
90\SIUnitSymbolDegree FoV
LASER [ 52 ]
-
-
0.67
14.6
23.5
72.8
76.8
76.2
SPVLoc [ 25 ]
-
-
0.83
9.2
50.1
89.7
92.1
91.9
Ours (Ray)
0.81
11.1
0.38
8.4
56.0
94.5
95.6
95.5
↪ Token
0.87
11.4
0.41
8.5
55.5
94.4
95.6
95.4
↪ Param.
0.84
11.5
0.36
7.6
55.6
87.8
90.6
90.5
Table 4: Quantitative results on S3D- resampled . For each FoV subset, we evaluate both 2D and 3D localization errors, along with 2D pose recall under different thresholds. The best is highlighted.
Figure 5: Qualitative results on Structured3D. (b) We present the predicted layout segmentation, the 3D layout recovered from the query image, and the grounded map primitives, with the ceilings removed for clarity. (c) We show our localization result together with the ground truth .
Table 10Figure 11
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Map Type
Domain
Camera Type
#Frames
#Scenes (regions)
Avg.#Frames per Scene
Dataset
ScanNet [ 20 ]
Surface
Real-world
Perspective
98533
1210
81
ScanNet++ [ 92 ]
Real-world
Perspective
216645
856
253
Matterport3D [ 12 ]
Real-world
Perspective
54578
2195
25
MP3D-Habitat [ 65 ]
Synthetic
Perspective
24568
2195
11
S3D- official [ 97 ]
Layout
Synthetic
Perspective
148341
3250
46
Appendix
Table 8: Statistics of training datasets. Two groups of datasets are used to train the model for localization on surface maps and layout models, respectively.
Med. Err. ↓
Pose Recall ( % ) ↑
Pathway
Supervision
Δq ( \SIUnitSymbolDegree )
Δt ( m )
( 0.2m , 10\SIUnitSymbolDegree )
( 0.5m , 15\SIUnitSymbolDegree )
( 1.0m , 30\SIUnitSymbolDegree )
Param.-based
Lassign+Lparam
3.1
0.23
44.2
70.6
79.5
Token-based
Lpose
4.4
0.28
39.9
83.7
87.2
Ray-based
Lray
3.4
0.20
51.7
87.6
90.9
Full (Ray)
Lassign+Lparam+Lpose+Lray
3.0
0.16
62.0
92.8
95.2
Appendix
Table 9: Pathway-specific supervision on ScanNet. Each pathway is trained with its required supervision only; Full (Ray) uses all supervision terms.
Figure 7: Cumulative accuracy curves on surface maps. We plot the proportion of successfully localized frames as a function of varying translation and rotation error thresholds.
Figure 8: Analysis of the impact of primitive richness on ScanNet. (a) Left: Distribution of all test samples and successfully localized samples (by our method) at ( 0.2m , 10\SIUnitSymbolDegree ) thresholds w.r.t. the number of actually observed primitives. Right: Pose recall at ( 0.2m , 10\SIUnitSymbolDegree ) versus the number of actually observed primitives. (b) Average pose error curves, where pose errors are averaged over all samples within each bin. (c) Left: Distribution of actually observed map primitives and those successfully matched (by our method). Right: Curves of the matching F 1 score.
Segmentation
Depth Estimation
Methods
RI ↑
VOI ↓
SC ↑
RMSE ↓
Log 10 ↓
Rel ↓
σ1↑
σ2↑
σ3↑
PlaneRecTR [ 68 ]
0.91
0.93
0.78
0.24
0.05
0.09
90.8
97.7
99.2
ZeroPlane [ 47 ]
0.92
0.90
0.80
0.34
0.09
0.18
65.9
92.1
97.7
MoGe-2 [ 81 ] +RANSAC
0.89
1.28
0.71
0.65
0.58
0.20
82.3
86.0
86.5
Ours
0.94
0.78
0.86
0.16
0.03
0.07
96.0
99.1
99.7
Appendix
Table 10: Comparison of plane recovery results on our ScanNet test set. Although existing monocular methods are not direct competitors, we quantitatively compare their plane recovery quality with ours using standard segmentation and depth estimation metrics. The best and second are highlighted.
Figure 9: Per-pixel and per-plane recalls on our ScanNet test set, which are comprehensive metrics for evaluating the plane recovery quality.
Med. Err. ↓ (3D)
Depth Estimation
Segmentation ↑
Dataset
Δq ( \SIUnitSymbolDegree )
Δt ( cm )
RMSE ↓
Log 10 ↓
Rel ↓
σ1↑
σ2↑
σ3↑
mIoU
mACC
official
0.75
13.0
0.18
0.03
0.08
96.1
98.6
99.1
81.6
84.9
resampled- 90\SIUnitSymbolDegree FoV
0.81
11.1
0.11
0.02
0.05
98.3
99.6
99.8
78.3
80.1
Appendix
Table 11: Additional Evaluation for Layout Localization.
Figure 10: Performance trends of depth and pose estimation on two datasets. We plot the depth RMSE distribution as histograms (left y-axis) and corresponding pose accuracy curves (right y-axis).
Figure 11: Qualitative results on surface maps. The first two examples are from ScanNet and the last two examples are from 12Scenes. Our method robustly produces accurate pose estimation, with (b) superior plane recovery and primitive correspondences as by-products, while (c) PlanaReLoc performs suboptimally or even fails due to poor plane recovery, matching outliers and inadequate pose estimation constraints. (d) Comparison of localization results. Legend: the ground truth , Pggt (Ours) , and PlanaReLoc .
Figure 12: Qualitative results on layout models. (b) We present the predicted layout segmentation, the layout geometry recovered from the query image, and the grounded map primitives, with ceilings removed for clarity. (c) We show the localization result as well as the ground truth .
Figure 13: Representative failure cases on surface maps. (a–b) Cases where the Param.-based pose is more accurate than the Ray-based pose . (c) Pose shift under blurry observations. (d–f) Failure cases occurring in scenes with limited structural detail or highly repetitive patterns, or caused by uninformative query observations.
Figure 14: Representative failure cases on layout models. (a–b) Cases where the Param.-based pose is more accurate than the Ray-based pose . (c–d) Failure cases due to structural ambiguity.
Visual localization -- estimating a camera pose within a pre-existing map -- is a fundamental problem in computer vision. Floorplans are an attractive map representation: they are readily available for most buildings, compact, and inherently invariant to visual appearance changes. However, bridging the severe domain gap between camera observations and floorplan geometry remains challenging. Existing methods address this gap through data-driven learning, yet they require large-scale training data and environment-specific retraining, limiting their practical deployment. We propose a zero-shot floorplan localization method that generalizes to novel environments without any retraining. Our key insight is that dominant geometric primitives -- lines and circles -- are ubiquitous in human-made environments and provide appearance-invariant structural constraints. We extract these primitives from a bird's-eye-view (BEV) projection of monocular 3D reconstructions and match them to the floorplan via dedicated minimal solvers within a robust estimation framework. Experiments on both simulated and real-world datasets show that our approach outperforms state-of-the-art learning-based methods on unseen environments, while using a single fixed set of hyperparameters across all experiments. The source code will be made publicly available.
Ayumi Umemura, Toshinori Kuwahara, Marc Pollefeys +1
While structure-based relocalizers have long strived for point correspondences when establishing or regressing query-map associations, in this paper, we pioneer the use of planar primitives and 3D planar maps for lightweight 6-DoF camera relocalization in structured environments. Planar primitives, beyond being fundamental entities in projective geometry, also serve as region-based representations that encapsulate both structural and semantic richness. This motivates us to introduce PlanaReLoc, a streamlined plane-centric paradigm where a deep matcher associates planar primitives across the query image and the map within a learned unified embedding space, after which the 6-DoF pose is solved and refined under a robust framework. Through comprehensive experiments on the ScanNet and 12Scenes datasets across hundreds of scenes, our method demonstrates the superiority of planar primitives in facilitating reliable cross-modal structural correspondences and achieving effective camera relocalization without requiring realistically textured/colored maps, pose priors, or per-scene training. The code and data are available at https://github.com/3dv-casia/PlanaReLoc .
Hanqiao Ye, Yuzhou Liu, Yangdong Liu +1
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences
Embodied systems need geometric perception that exploits available observations beyond images alone. Recent feed-forward 3D models incorporate geometric priors, including camera poses, intrinsics, and depth. However, handling noisy poses, preserving accurate priors, and recovering physical scale require more than simply accepting these inputs. We introduce Vision-Prior Geometry Grounded Transformer (VPGGT), a VGGT-based framework that extends OmniVGGT for prior-aware embodied perception. We formulate sensor-motivated pose corruptions from ground-truth trajectories for training and introduce a parameter-free prior residual connection (PRC) to mitigate prior dilution, where predictions are less accurate than their supplied pose priors. Our noise formulation targets camera poses; supplied intrinsics and depth receive no additional corruption. We further introduce Metric Global Attention, which conditions a global scale token on available pose and depth scales and predicts a shared metric scaling factor for the geometric outputs. Experiments across four datasets show that PRC improves translation-direction accuracy and joint pose AUC over a matched training baseline when camera priors are provided for all views, under both exact and corrupted poses. These results support explicit prior access during refinement as a useful addition to feature-level conditioning.