Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · 3Beihang University
Camera localization methods are mostly multi-stage pipelines relying on dedicated and complex maps that are costly to construct, store, and maintain over time. Alternatively, well-structured maps that organize scenes with compact yet expressive primitives are often readily available, providing rich structural and semantic cues that humans can interpret. However, their potential for direct camera localization remains largely unexplored due to the significant modality gap between map primitives and visual observations. This motivates a new concept for localization, which estimates camera poses by grounding query observations to map primitives. In this paper, we instantiate the concept in the context of planar primitives with PGGT, the Primitive-Grounded Geometry Transformer network. Given a single query image and a set of planar map primitives, the network is trained to predict multiple localization targets for feed-forward 6-DoF camera localization that generalizes to unseen environments. With this straightforward and scalable architecture, our method pushes the state-of-the-art localization performance on both dense surface maps and minimal layout models. The code and models are available at https://github.com/3dv-casia/PGGT-Loc
Figures & tables
Figure 1: Overview. (a) PlanaReLoc [ 90 ] localizes cameras against dense surface maps by explicitly matching 2D plane proposals with planar map primitives. (b) SPVLoc [ 25 ] localizes cameras against minimal layout models by retrieving the best-matching panoramic rendering. (c) Our Pggt -Loc operates directly on query patches and map primitives, enabling feed-forward localization across both dense surface maps and minimal layout models with a unified base architecture.
Figure 2: Model Architecture . (a) Two embedders first tokenize the inputs into patch and primitive tokens, respectively. (b) The two-branch transformer backbone refines the two sets of tokens, together with the prepended camera and map tokens, and enables interactions between them. (c) Downstream prediction heads produce different localization targets that enable camera pose inference via three pathways, which we refer to as the (1) Param.-based, (2) Ray-based, and (3) Token-based approaches.
Figure 3: Scene architectural layout. Left : 3D layout model with ceilings culled. Right : corresponding 2D floorplan with semantic labels.
Δq ( \SIUnitSymbolDegree ) ↓
Δt ( m ) ↓
Pose Recall ( % ) ↑
Time ( s−1 ) ↓
Methods
Mean
Med.
Mean
Med.
( 0.2m , 10\SIUnitSymbolDegree )
( 0.5m , 15\SIUnitSymbolDegree )
( 1.0m , 30\SIUnitSymbolDegree )
MeshLoc
MASt3R [ 41 ]
46.0
12.2
1.02
0.43
35.4
49.5
57.6
∼0.683168
MatchAnything [ 30 ]
35.7
19.9
1.23
0.74
20.0
35.7
52.1
∼0.925184
NOPE-SAC [ 73 ]
28.7
15.9
0.90
0.77
3.3
21.2
54.6
∼0.368222
Plana3R [ 44 ]
26.8
12.9
0.92
0.52
17.9
37.6
57.1
∼0.423516
PlaneRecTR++ [ 68 ]
20.7
9.9
0.66
0.54
12.2
39.2
75.1
∼0.412271995
Table 1: Quantitative results on ScanNet. We report rotation and translation errors, along with pose recalls across three thresholds. We also report the average inference time per query, using a single NVIDIA RTX 3090 GPU. Top-3 results are highlighted as the first , second , and third .
Med. Err. ↓
Pose Recall ( % ) ↑
Primitive Matching ( % ) ↑
Methods
Δq ( \SIUnitSymbolDegree )
Δt ( m )
( 0.2m , 10\SIUnitSymbolDegree )
( 0.5m , 15\SIUnitSymbolDegree )
( 1.0m , 30\SIUnitSymbolDegree )
Prec.
Rec.
F 1
AP
MASt3R [ 41 ]
12.0
0.30
45.2
51.9
59.0
59.8
42.9
50.0
81.6
MatchAnything [ 30 ]
7.9
0.20
46.9
63.4
77.6
51.2
56.1
53.5
77.2
NOPE-SAC [ 73 ]
17.7
0.54
2.9
25.9
67.2
43.8
22.0
29.3
71.4
Plana3R [ 44 ]
15.9
0.50
7.9
30.1
69.3
-
-
-
-
PlaneRecTR++ [ 68 ]
9.8
0.39
11.1
53.7
83.5
55.0
24.9
34.3
75.4
Table 2: Quantitative results on 12Scenes. We present the cross-dataset performance in terms of both localization ( i.e . , median errors and pose recall) and primitive matching.
Figure 4: Qualitative comparisons with PlanaReLoc. Our method robustly produces accurate pose estimation, with (b) superior plane recovery and primitive correspondences as additional outputs. In contrast, (c) PlanaReLoc performs suboptimally and even fails due to poor plane recovery (marked by the boxes ) and matching outliers (bottom row, colored in red ). (d) Comparison of localization results. Legend: the ground truth , Pggt (Ours) , and PlanaReLoc .
Map Type
Use Semantics
<1m Med. Error ↓ (2D)
Pose Recall ( % , 2D) ↑
Paradigm
Methods
Δtxy ( cm )
Δqyaw ( \SIUnitSymbolDegree )
10cm
50cm
1m
1m & 30\SIUnitSymbolDegree
Render & Compare
LASER [ 52 ]
2D
-
-
7.0
6.4
10.4
8.7
F 3 Loc [ 16 ]
2D
-
-
1.5
14.6
22.4
21.3
3DP [ 14 ]
2D
-
-
5.6
27.4
55.5
24.0
SemRayLoc [ 27 ]
2D
✓
-
-
5.7
45.5
58.8
57.5
RSK [ 15 ]
2D
-
-
6.4
28.6
56.9
25.2
Table 3: Quantitative results on S3D- official . Different methods employ either 2D floorplans or 3D layout models as the reference map, with some further leveraging map semantics. We report median 2D localization errors over <1m inliers, along with pose recall at different thresholds. Top-3 results are highlighted as the first , second , and third . † method not requiring known intrinsics ( i.e . , FoV).
Med. Err. ↓ (3D)
<1m Med. Error ↓ (2D)
Pose Recall ( % , 2D) ↑
Methods
Δq ( \SIUnitSymbolDegree )
Δt ( cm )
Δqyaw ( \SIUnitSymbolDegree )
Δtxy ( cm )
10cm
50cm
1m
1m & 30\SIUnitSymbolDegree
90\SIUnitSymbolDegree FoV
LASER [ 52 ]
-
-
0.67
14.6
23.5
72.8
76.8
76.2
SPVLoc [ 25 ]
-
-
0.83
9.2
50.1
89.7
92.1
91.9
Ours (Ray)
0.81
11.1
0.38
8.4
56.0
94.5
95.6
95.5
↪ Token
0.87
11.4
0.41
8.5
55.5
94.4
95.6
95.4
↪ Param.
0.84
11.5
0.36
7.6
55.6
87.8
90.6
90.5
Table 4: Quantitative results on S3D- resampled . For each FoV subset, we evaluate both 2D and 3D localization errors, along with 2D pose recall under different thresholds. The best is highlighted.
Figure 5: Qualitative results on Structured3D. (b) We present the predicted layout segmentation, the 3D layout recovered from the query image, and the grounded map primitives, with the ceilings removed for clarity. (c) We show our localization result together with the ground truth .
Table 10Figure 11
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Map Type
Domain
Camera Type
#Frames
#Scenes (regions)
Avg.#Frames per Scene
Dataset
ScanNet [ 20 ]
Surface
Real-world
Perspective
98533
1210
81
ScanNet++ [ 92 ]
Real-world
Perspective
216645
856
253
Matterport3D [ 12 ]
Real-world
Perspective
54578
2195
25
MP3D-Habitat [ 65 ]
Synthetic
Perspective
24568
2195
11
S3D- official [ 97 ]
Layout
Synthetic
Perspective
148341
3250
46
Appendix
Table 8: Statistics of training datasets. Two groups of datasets are used to train the model for localization on surface maps and layout models, respectively.
Med. Err. ↓
Pose Recall ( % ) ↑
Pathway
Supervision
Δq ( \SIUnitSymbolDegree )
Δt ( m )
( 0.2m , 10\SIUnitSymbolDegree )
( 0.5m , 15\SIUnitSymbolDegree )
( 1.0m , 30\SIUnitSymbolDegree )
Param.-based
Lassign+Lparam
3.1
0.23
44.2
70.6
79.5
Token-based
Lpose
4.4
0.28
39.9
83.7
87.2
Ray-based
Lray
3.4
0.20
51.7
87.6
90.9
Full (Ray)
Lassign+Lparam+Lpose+Lray
3.0
0.16
62.0
92.8
95.2
Appendix
Table 9: Pathway-specific supervision on ScanNet. Each pathway is trained with its required supervision only; Full (Ray) uses all supervision terms.
Figure 7: Cumulative accuracy curves on surface maps. We plot the proportion of successfully localized frames as a function of varying translation and rotation error thresholds.
Figure 8: Analysis of the impact of primitive richness on ScanNet. (a) Left: Distribution of all test samples and successfully localized samples (by our method) at ( 0.2m , 10\SIUnitSymbolDegree ) thresholds w.r.t. the number of actually observed primitives. Right: Pose recall at ( 0.2m , 10\SIUnitSymbolDegree ) versus the number of actually observed primitives. (b) Average pose error curves, where pose errors are averaged over all samples within each bin. (c) Left: Distribution of actually observed map primitives and those successfully matched (by our method). Right: Curves of the matching F 1 score.
Segmentation
Depth Estimation
Methods
RI ↑
VOI ↓
SC ↑
RMSE ↓
Log 10 ↓
Rel ↓
σ1↑
σ2↑
σ3↑
PlaneRecTR [ 68 ]
0.91
0.93
0.78
0.24
0.05
0.09
90.8
97.7
99.2
ZeroPlane [ 47 ]
0.92
0.90
0.80
0.34
0.09
0.18
65.9
92.1
97.7
MoGe-2 [ 81 ] +RANSAC
0.89
1.28
0.71
0.65
0.58
0.20
82.3
86.0
86.5
Ours
0.94
0.78
0.86
0.16
0.03
0.07
96.0
99.1
99.7
Appendix
Table 10: Comparison of plane recovery results on our ScanNet test set. Although existing monocular methods are not direct competitors, we quantitatively compare their plane recovery quality with ours using standard segmentation and depth estimation metrics. The best and second are highlighted.
Figure 9: Per-pixel and per-plane recalls on our ScanNet test set, which are comprehensive metrics for evaluating the plane recovery quality.
Med. Err. ↓ (3D)
Depth Estimation
Segmentation ↑
Dataset
Δq ( \SIUnitSymbolDegree )
Δt ( cm )
RMSE ↓
Log 10 ↓
Rel ↓
σ1↑
σ2↑
σ3↑
mIoU
mACC
official
0.75
13.0
0.18
0.03
0.08
96.1
98.6
99.1
81.6
84.9
resampled- 90\SIUnitSymbolDegree FoV
0.81
11.1
0.11
0.02
0.05
98.3
99.6
99.8
78.3
80.1
Appendix
Table 11: Additional Evaluation for Layout Localization.
Figure 10: Performance trends of depth and pose estimation on two datasets. We plot the depth RMSE distribution as histograms (left y-axis) and corresponding pose accuracy curves (right y-axis).
Figure 11: Qualitative results on surface maps. The first two examples are from ScanNet and the last two examples are from 12Scenes. Our method robustly produces accurate pose estimation, with (b) superior plane recovery and primitive correspondences as by-products, while (c) PlanaReLoc performs suboptimally or even fails due to poor plane recovery, matching outliers and inadequate pose estimation constraints. (d) Comparison of localization results. Legend: the ground truth , Pggt (Ours) , and PlanaReLoc .
Figure 12: Qualitative results on layout models. (b) We present the predicted layout segmentation, the layout geometry recovered from the query image, and the grounded map primitives, with ceilings removed for clarity. (c) We show the localization result as well as the ground truth .
Figure 13: Representative failure cases on surface maps. (a–b) Cases where the Param.-based pose is more accurate than the Ray-based pose . (c) Pose shift under blurry observations. (d–f) Failure cases occurring in scenes with limited structural detail or highly repetitive patterns, or caused by uninformative query observations.
Figure 14: Representative failure cases on layout models. (a–b) Cases where the Param.-based pose is more accurate than the Ray-based pose . (c–d) Failure cases due to structural ambiguity.