Accurate camera localization is crucial for robotics and Extended Reality (XR), enabling reliable navigation and alignment of virtual and real content. Existing visual methods often suffer from drift, scale ambiguity, and depend on fiducials or loop closure. This work introduces a real-time method for localizing a camera within a pre-captured, highly accurate colored LiDAR point cloud. By rendering synthetic views from this cloud, 2D-3D correspondences are established between live frames and the point cloud. A neural rendering technique narrows the domain gap between synthetic and real images, reducing occlusion and background artifacts to improve feature matching. The result is drift-free camera tracking with correct metric scale in the global LiDAR coordinate system. Two real-time variants are presented: Online Render and Match (R&M), and Prebuild and Localize (P&L). We demonstrate improved results on the ScanNet++ dataset and outperform existing SLAM pipelines.
Figures & tables
[ 4 ]
[ 8 ]
Ours
Real-time processing
–
✓
✓
Monocular data only
✓
–
✓
Tracking module
–
✓
✓
No rough trajectory/initialization required
✓
–
✓
Neural rendering without scene-specific training
–
–
✓
Integration in existing SLAM pipelines
–
–
✓
TABLE I: Comparison to the closest related works [ 8 , 4 ] .
Fig. 1: The steps within the adapted neural renderer. (a) First, the input point cloud and camera parameters are used to (b) compute the per-pixel minimum depth via z-buffering. In the second pass, (c) colors of points projecting to the same pixel within a small depth threshold are accumulated and then (d) averaged. Next, (e) background leakage is removed using a hierarchical depth filter, and finally, (f) any remaining holes are filled with a U-Net.
Fig. 2: Overview of the proposed methods. Preprocessing: synthetic cubemap images and depth maps are rendered from uniformly sampled poses within the LiDAR point cloud, using the neural rendering method described in Section III-A . Prebuild&Localize utilizes these virtual cubemaps to compute a localization map offline, which can be used by the localization module of traditional SLAM backends. Render&Match creates a feature database that links 2D descriptors to ground-truth 3D landmarks. Relocalization: when no prior pose is available, k candidate keyframes selected with the global descriptor of the query image, are matched to the query to obtain a coarse 6DoF pose via PnP, which is refined by re-rendering the point cloud at the estimated pose. Tracking: during tracking, the previous pose is used to render a synthetic view, which is matched to the live camera frame to estimate the current 6DoF pose. Both stages operate without unbounded drift since all correspondences are derived from the LiDAR-based geometry.
Scene
#Scans
#Seq.
#Frames
Length
Trans. speed
Ang. speed
(m)
(m/s)
( ∘ /s)
Lab
4
3
4335±102.3
45.6±12.9
0.32±0.08
23.7±7.5
Synthetic
29
4
1968±265.2
134.2±24.1
2.04±0.12
10.4±1.4
TABLE II: Custom dataset characterization (mean ± std over sequences). #Scans reports the number of concatenated scans.
Method
Type
Data Required
-GPU
Met.
Rel.
ScanNet++ [ 28 ]
Off
Vis.+RGB LiDAR
✓
✓
NA
StellaVSLAM [ 39 ]
On
Vis. (Monocular)
✓
–
✓
ORB-SLAM3 [ 40 ]
On
Vis. (Monocular)
✓
–
✓
COLMAP [ 29 ]
Off
Vis. (Monocular)
✓
–
NA
VILENS [ 23 ]
On
Vis.+IMU+LiDAR
✓
✓
✓
CMRNext [ 12 ]
On
Vis.+LiDAR
–
✓
–
TABLE III: Comparison of baseline methods. Columns indicate operation mode (Type: Online [On] / Offline [Off]), required sensor modalities, GPU independence (-GPU), metric trajectory recovery (Met.), and relocalization support (Rel.).
Method
Pos.
Ang.
%Loc.
<5cm
SSIM
(m) ↓
( ∘ ) ↓
↑
<2 ∘↑
↑
ScanNet++ [ 28 ]
-
-
86.1
-
0.663
StellaVSLAM [ 39 ]
0.13
1.79
66.3
49.74
0.607
ORB-SLAM3 [ 40 ]
0.03
6.99
70.9
64.04
0.609
Ours (P&L-S)
0.02
0.44
69.1
89.23
0.691
Ours (P&L-O)
0.02
0.50
31.3
89.50
0.715
TABLE IV: Quantitative evaluation on ScanNet++. The best and second-best values are highlighted.
Fig. 3: Results for ScanNet++ and our custom datasets. Each image displays an overlay of the target camera frame and a neural rendering of the point cloud generated from the estimated camera pose. We show results for the ground-truth, R&M, and the P&L methods. Misalignment, visible as ghosting or blur, suggests pose estimation errors. Note that ScanNet++’s ground-truth is only an approximate reference.
Method
Pos.
Ang.
%Loc.
<1cm
<5cm
<20cm
(cm) ↓
( ∘ ) ↓
↑
<1 ∘↑
<2 ∘↑
<3 ∘↑
StellaVSLAM [ 39 ]
7.11
0.82
84.8
31.9
44.4
82.6
ORB-SLAM3 [ 40 ]
29.70
5.00
82.3
34.2
50.4
75.2
Ours (P&L-S)
0.59
0.18
100.0
75.4
98.6
99.9
Ours (P&L-O)
1.53
0.50
39.2
49.3
82.2
92.5
Ours (R&M)
1.21
0.21
97.4
45.0
92.4
99.7
TABLE V: Quantitative evaluation on the custom datasets. The best and second-best values are highlighted.
Neural
Depth-filtered
Point-based
Method
Dec.
Pos. ↓
%Loc. ↑
<5cm,<2 ∘↑
Pos. ↓
%Loc. ↑
<5cm,<2 ∘↑
Pos. ↓
%Loc. ↑
<5cm,<2 ∘↑
R&M
100%
0.77
94.0
98.6
1.90
86.1
88.0
1.02
82.6
96.9
50%
0.88
88.9
98.6
3.15
73.0
71.7
1.89
75.2
87.8
20%
0.98
83.6
98.3
4.98
35.9
50.6
3.67
29.6
63.3
10%
1.08
83.7
97.9
5.72
7.1
32.7
11.20
2.4
30.6
5%
1.27
83.5
97.0
-
-
-
-
-
-
TABLE VI: Impact of scene decimation on localization accuracy for the ‘lab’ dataset, comparing Neural, Depth-filtered and Point-based rendering. Results are shown by method, with the superior performance highlighted per decimation level.
Fig. 4: Results on the Oxford Spires dataset. Each image displays an overlay of the target camera frame and a neural rendering of the point cloud generated from the estimated camera pose. We show results for the ground-truth, VILENS [ 23 ] , CMRNext [ 12 ] , R&M, and the P&L methods. Misalignment, visible as ghosting or blur, suggests pose estimation errors.
Method
Pos.
Ang.
%Loc.
Pos. ↑
Ang. ↑
(m) ↓
( ∘ ) ↓
↑
<5c
<20c
<1m
<1 ∘
<2 ∘
<3 ∘
StellaVSLAM [ 39 ]
2.02
1.61
85.2
15.9
36.4
55.4
56.3
81.7
84.4
ORB-SLAM3 [ 40 ]
0.52
16.71
69.5
31.3
55.3
70.9
84.5
86.8
88.7
COLMAP [ 29 ]
0.09
-
5.2 †
31.2
90.1
100.0
-
-
-
VILENS [ 23 ]
0.13
0.40
5.3 †
35.0
73.4
95.1
87.1
98.1
99.9
CMRNext [ 12 ]
1.62
14.39
29.4
1.3
31.5
83.5
13.6
32.4
43.5
TABLE VII: Quantitative evaluation on the Oxford Spires dataset. The best and second-best values are highlighted.
Fig. 5: Error-vs-distance plot for every baseline and our methods, pooled across all sequences of each dataset and binned into 40 intervals, each curve shows the per-bin mean. Every estimated trajectory is aligned with the ground truth as described in section IV-C .
Method
Oxford Spires
ScanNet++
Custom
Succ. ↑
t2p ↓
Succ. ↑
t2p ↓
Succ. ↑
t2p ↓
StellaVSLAM
88
0.08
78
0.03
81
0.05
ORB-SLAM3
76
0.08
65
0.03
85
0.06
Ours (R&M)
69
0.51
92
1.17
93
0.06
Ours (P&L-S)
23
8.86
82
0.81
80
0.17
Ours (P&L-O)
7
11.3
50
4.56
36
0.71
TABLE VIII: Global relocalization: success rate (%, ≤ 50 cm within 10 s) and median time-to-first-pose (s).
Method
FPS ↑
Reloc. (ms) ↓
Map (s) ↓
StellaVSLAM [ 39 ]
45.5
41
-
ORB-SLAM3 [ 40 ]
32.5
23
-
CMRNext [ 12 ]
1.6
-
-
Ours (P&L-Stella)
33.2
72
449
Ours (P&L-ORB-SLAM3)
31.9
30
460
Ours (R&M)
13.5
47
83
TABLE IX: Average tracking FPS, relocalization time, and map construction time across all datasets.
Method
%Loc. ↑
Reloc. (ms) ↓
Map (s) ↓
R&M XFeat
83.8
47.2
83.0
R&M XFeat †
89.0
659.8
69.0
R&M ORB
27.5
45.9
52.8
† Without global image descriptors for relocalization.
TABLE X: Comparison of R&M feature backend variants. Mean metrics are reported across all datasets.
Camera localization in a prior LiDAR map provides a persistent geometric reference for long-term robotic navigation, yet remains challenging because of the substantial modality gap between camera images and point-cloud maps. We present a unified localization framework that makes the LiDAR map visually addressable rather than relying on a dedicated image-LiDAR correspondence model. Map geometry and reflectivity are rendered into LiDAR-derived quasi-images with explicit 2D-3D provenance, enabling camera observations and rendered map views to share mature visual features and matchers for both global localization and continuous pose tracking. Point and line correspondences are established through this common visual interface, while the retained provenance recovers metric LiDAR geometry and line-supported planar constraints for pose estimation. To improve robustness under ambiguous associations and weak geometry, we further introduce a distribution-aware, observability-complementary optimization strategy. Instead of reducing matching ambiguity to a scalar confidence, candidate association distributions are propagated into directional pose-information uncertainty, and reliable structural factors are selectively reinforced according to their ability to complement the currently weak pose directions. Experiments on the EuRoC MAV benchmark and self-collected real-world sequences demonstrate accurate global localization and robust continuous 6-DoF tracking using only a pre-built LiDAR map as the persistent prior, including under severe illumination variations and dynamic occlusions.
Wentao Zhao, Zikun Chen, Yihe Niu +2
School of Automation and Intelligent Sensing, Institute of Medical Robotics, Shanghai Jiao Tong University · School of Mathematical Sciences, Shanghai Jiao Tong University · School of Electronic and Information Engineering, Beijing Jiaotong University
Visual localization (VL) is a fundamental technology for vision applications such as virtual reality. Recently, a novel VL paradigm, Camera Ray Regression (CRR), has emerged, which maps 2D image patches to 3D camera rays, but its accuracy is limited. To improve CRR accuracy, we notice a compelling duality: the inverse of this mapping is inherently performed by the novel view synthesis model, \ie, Neural Radiance Fields (NeRF). While NeRF renders image patches from camera rays via differentiable ray marching, CRR predicts the rays from image patches. Motivated by this complementary relationship, we propose NeRF-aided Camera Ray Regression (NaCR), a unified framework that seamlessly bridges NeRF and CRR at the ray level. First, NaCR incorporates three simple yet effective enhancements into the CRR baseline. Second, leveraging a pre-trained NeRF, NaCR augments the training data by synthesizing novel views tailored for efficient, patch-level consumption. Finally, exploiting the differentiability of NeRF, NaCR forms a closed-loop supervision pipeline where photometric rendering errors are back-propagated to optimize the predicted camera rays. To ensure stable convergence within the highly non-convex image space, we introduce a two-stage training curriculum. Extensive experiments across indoor and outdoor benchmarks demonstrate that NaCR achieves competitive accuracy. Comprehensive ablation studies validate the efficacy of each proposed component.
Geometric light detection and ranging (LiDAR)--inertial simultaneous localization and mapping (SLAM) provides efficient state estimation but lacks the continuous map representation and global refinement capabilities of neural implicit SLAM, while neural implicit methods remain computationally expensive. We propose Lightweight LiDAR--Inertial Neural SLAM (LIN-SLAM), a hybrid method that couples high-rate geometric LiDAR--inertial estimation with neural implicit tracking, continual mapping, and global refinement for accurate real-time onboard SLAM. Experiments on twelve representative sequences from three datasets spanning 32-, 64-, and 128-channel LiDAR configurations show that LIN-SLAM achieves the lowest overall mean absolute trajectory error (ATE) among the evaluated state-of-the-art geometric and neural implicit methods, while substantially reducing runtime compared with existing neural implicit SLAM methods. On a Unitree Go2 equipped with an NVIDIA Jetson Orin NX and a 16-channel LiDAR, LIN-SLAM sustains real-time localization and neural mapping over a 1.47-km indoor trajectory through challenging repetitive corridor environments, with an endpoint closure error of only 0.141~m. To the best of our knowledge, LIN-SLAM is the first LiDAR--inertial SLAM system with a neural implicit backend to achieve real-time online operation on a mobile robotic platform while maintaining state-of-the-art localization accuracy. The code will be released upon acceptance.
Shiqi Che, Yijie Liu, Shuo Wang +1
School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China