Accurate camera localization is crucial for robotics and Extended Reality (XR), enabling reliable navigation and alignment of virtual and real content. Existing visual methods often suffer from drift, scale ambiguity, and depend on fiducials or loop closure. This work introduces a real-time method for localizing a camera within a pre-captured, highly accurate colored LiDAR point cloud. By rendering synthetic views from this cloud, 2D-3D correspondences are established between live frames and the point cloud. A neural rendering technique narrows the domain gap between synthetic and real images, reducing occlusion and background artifacts to improve feature matching. The result is drift-free camera tracking with correct metric scale in the global LiDAR coordinate system. Two real-time variants are presented: Online Render and Match (R&M), and Prebuild and Localize (P&L). We demonstrate improved results on the ScanNet++ dataset and outperform existing SLAM pipelines.
Figures & tables
[ 4 ]
[ 8 ]
Ours
Real-time processing
–
✓
✓
Monocular data only
✓
–
✓
Tracking module
–
✓
✓
No rough trajectory/initialization required
✓
–
✓
Neural rendering without scene-specific training
–
–
✓
Integration in existing SLAM pipelines
–
–
✓
TABLE I: Comparison to the closest related works [ 8 , 4 ] .
Fig. 1: The steps within the adapted neural renderer. (a) First, the input point cloud and camera parameters are used to (b) compute the per-pixel minimum depth via z-buffering. In the second pass, (c) colors of points projecting to the same pixel within a small depth threshold are accumulated and then (d) averaged. Next, (e) background leakage is removed using a hierarchical depth filter, and finally, (f) any remaining holes are filled with a U-Net.
Fig. 2: Overview of the proposed methods. Preprocessing: synthetic cubemap images and depth maps are rendered from uniformly sampled poses within the LiDAR point cloud, using the neural rendering method described in Section III-A . Prebuild&Localize utilizes these virtual cubemaps to compute a localization map offline, which can be used by the localization module of traditional SLAM backends. Render&Match creates a feature database that links 2D descriptors to ground-truth 3D landmarks. Relocalization: when no prior pose is available, k candidate keyframes selected with the global descriptor of the query image, are matched to the query to obtain a coarse 6DoF pose via PnP, which is refined by re-rendering the point cloud at the estimated pose. Tracking: during tracking, the previous pose is used to render a synthetic view, which is matched to the live camera frame to estimate the current 6DoF pose. Both stages operate without unbounded drift since all correspondences are derived from the LiDAR-based geometry.
Scene
#Scans
#Seq.
#Frames
Length
Trans. speed
Ang. speed
(m)
(m/s)
( ∘ /s)
Lab
4
3
4335±102.3
45.6±12.9
0.32±0.08
23.7±7.5
Synthetic
29
4
1968±265.2
134.2±24.1
2.04±0.12
10.4±1.4
TABLE II: Custom dataset characterization (mean ± std over sequences). #Scans reports the number of concatenated scans.
Method
Type
Data Required
-GPU
Met.
Rel.
ScanNet++ [ 28 ]
Off
Vis.+RGB LiDAR
✓
✓
NA
StellaVSLAM [ 39 ]
On
Vis. (Monocular)
✓
–
✓
ORB-SLAM3 [ 40 ]
On
Vis. (Monocular)
✓
–
✓
COLMAP [ 29 ]
Off
Vis. (Monocular)
✓
–
NA
VILENS [ 23 ]
On
Vis.+IMU+LiDAR
✓
✓
✓
CMRNext [ 12 ]
On
Vis.+LiDAR
–
✓
–
TABLE III: Comparison of baseline methods. Columns indicate operation mode (Type: Online [On] / Offline [Off]), required sensor modalities, GPU independence (-GPU), metric trajectory recovery (Met.), and relocalization support (Rel.).
Method
Pos.
Ang.
%Loc.
<5cm
SSIM
(m) ↓
( ∘ ) ↓
↑
<2 ∘↑
↑
ScanNet++ [ 28 ]
-
-
86.1
-
0.663
StellaVSLAM [ 39 ]
0.13
1.79
66.3
49.74
0.607
ORB-SLAM3 [ 40 ]
0.03
6.99
70.9
64.04
0.609
Ours (P&L-S)
0.02
0.44
69.1
89.23
0.691
Ours (P&L-O)
0.02
0.50
31.3
89.50
0.715
TABLE IV: Quantitative evaluation on ScanNet++. The best and second-best values are highlighted.
Fig. 3: Results for ScanNet++ and our custom datasets. Each image displays an overlay of the target camera frame and a neural rendering of the point cloud generated from the estimated camera pose. We show results for the ground-truth, R&M, and the P&L methods. Misalignment, visible as ghosting or blur, suggests pose estimation errors. Note that ScanNet++’s ground-truth is only an approximate reference.
Method
Pos.
Ang.
%Loc.
<1cm
<5cm
<20cm
(cm) ↓
( ∘ ) ↓
↑
<1 ∘↑
<2 ∘↑
<3 ∘↑
StellaVSLAM [ 39 ]
7.11
0.82
84.8
31.9
44.4
82.6
ORB-SLAM3 [ 40 ]
29.70
5.00
82.3
34.2
50.4
75.2
Ours (P&L-S)
0.59
0.18
100.0
75.4
98.6
99.9
Ours (P&L-O)
1.53
0.50
39.2
49.3
82.2
92.5
Ours (R&M)
1.21
0.21
97.4
45.0
92.4
99.7
TABLE V: Quantitative evaluation on the custom datasets. The best and second-best values are highlighted.
Neural
Depth-filtered
Point-based
Method
Dec.
Pos. ↓
%Loc. ↑
<5cm,<2 ∘↑
Pos. ↓
%Loc. ↑
<5cm,<2 ∘↑
Pos. ↓
%Loc. ↑
<5cm,<2 ∘↑
R&M
100%
0.77
94.0
98.6
1.90
86.1
88.0
1.02
82.6
96.9
50%
0.88
88.9
98.6
3.15
73.0
71.7
1.89
75.2
87.8
20%
0.98
83.6
98.3
4.98
35.9
50.6
3.67
29.6
63.3
10%
1.08
83.7
97.9
5.72
7.1
32.7
11.20
2.4
30.6
5%
1.27
83.5
97.0
-
-
-
-
-
-
TABLE VI: Impact of scene decimation on localization accuracy for the ‘lab’ dataset, comparing Neural, Depth-filtered and Point-based rendering. Results are shown by method, with the superior performance highlighted per decimation level.
Fig. 4: Results on the Oxford Spires dataset. Each image displays an overlay of the target camera frame and a neural rendering of the point cloud generated from the estimated camera pose. We show results for the ground-truth, VILENS [ 23 ] , CMRNext [ 12 ] , R&M, and the P&L methods. Misalignment, visible as ghosting or blur, suggests pose estimation errors.
Method
Pos.
Ang.
%Loc.
Pos. ↑
Ang. ↑
(m) ↓
( ∘ ) ↓
↑
<5c
<20c
<1m
<1 ∘
<2 ∘
<3 ∘
StellaVSLAM [ 39 ]
2.02
1.61
85.2
15.9
36.4
55.4
56.3
81.7
84.4
ORB-SLAM3 [ 40 ]
0.52
16.71
69.5
31.3
55.3
70.9
84.5
86.8
88.7
COLMAP [ 29 ]
0.09
-
5.2 †
31.2
90.1
100.0
-
-
-
VILENS [ 23 ]
0.13
0.40
5.3 †
35.0
73.4
95.1
87.1
98.1
99.9
CMRNext [ 12 ]
1.62
14.39
29.4
1.3
31.5
83.5
13.6
32.4
43.5
TABLE VII: Quantitative evaluation on the Oxford Spires dataset. The best and second-best values are highlighted.
Fig. 5: Error-vs-distance plot for every baseline and our methods, pooled across all sequences of each dataset and binned into 40 intervals, each curve shows the per-bin mean. Every estimated trajectory is aligned with the ground truth as described in section IV-C .
Method
Oxford Spires
ScanNet++
Custom
Succ. ↑
t2p ↓
Succ. ↑
t2p ↓
Succ. ↑
t2p ↓
StellaVSLAM
88
0.08
78
0.03
81
0.05
ORB-SLAM3
76
0.08
65
0.03
85
0.06
Ours (R&M)
69
0.51
92
1.17
93
0.06
Ours (P&L-S)
23
8.86
82
0.81
80
0.17
Ours (P&L-O)
7
11.3
50
4.56
36
0.71
TABLE VIII: Global relocalization: success rate (%, ≤ 50 cm within 10 s) and median time-to-first-pose (s).
Method
FPS ↑
Reloc. (ms) ↓
Map (s) ↓
StellaVSLAM [ 39 ]
45.5
41
-
ORB-SLAM3 [ 40 ]
32.5
23
-
CMRNext [ 12 ]
1.6
-
-
Ours (P&L-Stella)
33.2
72
449
Ours (P&L-ORB-SLAM3)
31.9
30
460
Ours (R&M)
13.5
47
83
TABLE IX: Average tracking FPS, relocalization time, and map construction time across all datasets.
Method
%Loc. ↑
Reloc. (ms) ↓
Map (s) ↓
R&M XFeat
83.8
47.2
83.0
R&M XFeat †
89.0
659.8
69.0
R&M ORB
27.5
45.9
52.8
† Without global image descriptors for relocalization.
TABLE X: Comparison of R&M feature backend variants. Mean metrics are reported across all datasets.
School of Automation and Intelligent Sensing, Institute of Medical Robotics, Shanghai Jiao Tong University · School of Mathematical Sciences, Shanghai Jiao Tong University · School of Electronic and Information Engineering, Beijing Jiaotong University