NaCR: Visual Localization via NeRF-aided Camera Ray Regression

Yesheng Zhang; Xiang Dai; Xu Zhao; Chongyang Zhang

cs.CVSep 22, 2026

NaCR: Visual Localization via NeRF-aided Camera Ray Regression

Authors: Yesheng Zhang, Xiang Dai, Xu Zhao, Chongyang Zhang

Organizations: Shanghai Jiao Tong University

Abstract

Visual localization (VL) is a fundamental technology for vision applications such as virtual reality. Recently, a novel VL paradigm, Camera Ray Regression (CRR), has emerged, which maps 2D image patches to 3D camera rays, but its accuracy is limited. To improve CRR accuracy, we notice a compelling duality: the inverse of this mapping is inherently performed by the novel view synthesis model, \ie, Neural Radiance Fields (NeRF). While NeRF renders image patches from camera rays via differentiable ray marching, CRR predicts the rays from image patches. Motivated by this complementary relationship, we propose NeRF-aided Camera Ray Regression (NaCR), a unified framework that seamlessly bridges NeRF and CRR at the ray level. First, NaCR incorporates three simple yet effective enhancements into the CRR baseline. Second, leveraging a pre-trained NeRF, NaCR augments the training data by synthesizing novel views tailored for efficient, patch-level consumption. Finally, exploiting the differentiability of NeRF, NaCR forms a closed-loop supervision pipeline where photometric rendering errors are back-propagated to optimize the predicted camera rays. To ensure stable convergence within the highly non-convex image space, we introduce a two-stage training curriculum. Extensive experiments across indoor and outdoor benchmarks demonstrate that NaCR achieves competitive accuracy. Comprehensive ablation studies validate the efficacy of each proposed component.

Figures & tables

Figure 1: Camera Ray Regression (CRR) and Neural Radiance Fields (NeRF) Represent Two Complementary Processes: CRR learns to map an image patch to a camera ray , while NeRF renders an image patch from a camera ray . Recognizing this synergy, we introduce NeRF into the CRR training pipeline, creating an end-to-end differentiable loop. In this loop, photometric rendering error provides a direct guidance for CRR optimization. Additionally, NeRF allows for generating synthetic training image patches from arbitrary rays, serving as a powerful patch-level data augmentation technique.

Figure 2: The Improved DIMM. Our work introduces three simple yet effective enhancements to DIMM. ❶ We substitute the original DINOv2 trained on 2D tasks with the DA3-DINOv2 backbone pre-trained for 3D reconstruction, which provides the model with richer geometric priors. ❷ We transform the challenging absolute camera ray regression into a more tractable task by conditioning the model on tokens from fixed reference views. ❸ We augment the DIMM output head to predict confidence for each ray. This effectively leverages the over-parameterized nature of camera rays by down-weighting uncertain predictions.

Panel of Figure 3 — Figure 3: The two-stage NeRF-aided CRR training pipeline. (a) Stage 1 trains CRR with patch-level Gradient Decorrelation Training (GDT) on real and NeRF-synthesized patches, while adversarial feature alignment bridges the render-to-real gap. (b) Stage 2 adds a differentiable rendering loss, using the photometric error between rendered and original patches as a closed-loop signal for ray refinement.

Figure 4: Qualitative results in 7-Scenes. The top row compares the ground truth image with a view rendered from our estimated camera pose, which validates the precision of our method. The bottom row visualizes the predicted and GT camera rays within the 3D scene.

Category	Method	Chess	Fire	Heads	Office	Pumpkin	Kitchen	Stairs	Average
Category	Method	( $cm$ / ∘ )	( $cm$ / ∘ )	( $cm$ / ∘ )	( $cm$ / ∘ )	( $cm$ / ∘ )	( $cm$ / ∘ )	( $cm$ / ∘ )	( $cm$ / ∘ )
SCR	DSAC* [ 4 ]	0.5/0.2	0.8/0.3	0.5/0.3	1.2/0.3	1.2/0.3	0.7/0.2	2.7/0.8	1.1/0.3
	ACE [ 3 ]	0.5/0.2	0.8/0.3	0.6/0.3	1.0/0.3	1.0/0.2	0.8/0.2	2.9/0.8	1.1/0.3
	GLACE [ 37 ]	0.6/0.2	0.9/0.3	0.5/0.3	1.1/0.3	0.9/0.2	0.8/0.2	3.2/0.9	1.2/0.3
RPR	ExReNet [ 39 ]	5.0/1.6	7.0/2.5	3.0/2.7	6.0/1.8	7.0/2.0	7.0/2.1	19.0/4.9	8.0/2.5
RPR	Reloc3r [ 8 ]	3.0/0.9	3.0/0.8	1.0/1.0	4.0/0.9	6.0/1.1	4.0/1.3	7.0/1.3	4.0/1.0

Table 1: Localization accuracy on 7-Scenes [ 35 ] using SfM poses as ground truth. We report median position errors in

cm

and orientation errors in degrees ( ∘ ). The best results within the CRR and NPR groups, together with NVS-aided methods , are highlighted. “Method NeRF/GS ” indicates that “Method” uses NeRF/GS-based novel view synthesis for data augmentation.

Scene	SCR		APR				CRR
	DSAC*	ACE	PN	MST	$marepo$	$marepo_{S}$	DIMM	NaCR	NaCR+NPR
	[ 4 ]	[ 3 ]	[ 12 , 13 , 14 ]	[ 34 ]	[ 6 ]	[ 6 ]	[ 43 ]	Ours	Ours
Throughput (fps)	17.9	17.9	166.7	28.4	55.6	55.6	20.1	33.4	16.5
Bears	82.6%/91.6%	80.7%/92.6%	12.9%/35.7%	0.5%/12.8%	80.7%/99.3%	80.7%/99.5%	82.4%/97.1%	95.5%/99.3%	96.4 %/ 99.7 %
Cubes	83.8%/98.1%	97.0 %/ 98.1 %	0.0%/0.4%	0.00%/9.9%	72.4%/96.9%	71.8%/96.9%	71.5%/89.8%	72.5%/85.7%	73.8%/87.9%
Inscription	54.1%/69.7%	49.0%/69.6%	1.1%/6.3%	1.3%/9.7%	37.8%/74.2%	37.1%/74.1%	21.3%/72.6%	57.8 %/75.1%	56.1%/ 77.3 %

Table 2: Pose accuracy comparison on the outdoor Wayspots [ 1 ] dataset. Results are reported as the percentage of frames below

10cm/5^{\circ}

and

0.5m/5^{\circ}

error.

Component	Model Params (M)	Time	Role
ACE Mapping	2.10	5 min	SCR mapping
GLACE Mapping	9.45	33 min	SCR mapping
DIMM Mapping	18.31	1.32 h	baseline mapping
NaCR Mapping Cost	39.53	3.63 h	mapping
$\hookrightarrow$ Scene NeRF training	16.79	0.42 h	mapping
$\hookrightarrow$ Stage-1 training	39.53	2.08 h	mapping

Table 3: Detailed efficiency report. Mapping time is measured per scene, and inference time is measured per query image.

Method	Arch.	NVS aug.	$\mathcal{L}_{R}$	TST	7-Scenes $\downarrow$	STD	Wayspots $\uparrow$	STD
DIMM	$\times$	$\times$	$\times$	$\checkmark$	3.6/1.0	0.034	42.3/64.0	0.215
$\hookrightarrow$ DA3 only	DA3	$\times$	$\times$	$\checkmark$	3.3/0.9	0.037	45.9/63.8	0.052
DIMM ∗	Full	$\times$	$\times$	$\checkmark$	3.2/0.9	0.019	45.8/63.7	0.555
NaCR ( $\mu=0$ )	Full	$\checkmark$	$\times$	$\checkmark$	2.4/0.5	0.022	52.7/65.5	0.040
DIMM ∗ + $\mathcal{L}_{R}$	Full	$\times$	$\checkmark$	$\checkmark$	2.9/0.7	0.023	51.0/64.3	0.117
NaCR-SST	Full	$\checkmark$	$\checkmark$	$\times$	6.7/3.3	0.027	39.1/60.0	0.099

Table 4: Complete component ablation. Except for the two single-stage variants, all configurations use a 60K-step two-stage schedule. Results are averaged over seeds 42, 23, and 24 and across all scenes in 7-Scenes and Wayspots. “Arch.” denotes the architectural upgrades. Each entry reports translation/rotation error on 7-Scenes and recall at

10\text{cm}/5^{\circ}

and

0.5\text{m}/5^{\circ}

on Wayspots.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Qualitative results in Wayspots.The top row compares the ground truth image with a view rendered from our estimated camera pose. The bottom row visualizes the predicted camera rays within the 3D scene.

Scene	NaCR	NaCR w/o Adversarial Loss
chess	0.9/0.2	1.0/0.3
fire	1.0/0.2	1.2/0.5
heads	1.3/0.4	1.7/0.6
office	4.7/0.6	4.9/0.9

Appendix

Table 5: Ablation Study on the adversarial loss. The median rotation and translation errors are reported.

Figure 6: Training loss curves on the 7-Scenes Chess dataset, comparing our two-stage training strategy against a single-stage baseline. Our two-stage training demonstrates that while the rendering loss introduces slight instability to the training error (left), it yields superior final translation accuracy (middle). Conversely, the single-stage baseline (right), which applies the rendering loss from the start, struggles to converge with optimization instability, due to the unreliable supervision signal from the renderer when ray predictions are still coarse.

Proportion of Synthesized Images	Recall ( $\%$ )	Time (h)	Training Steps
DIMM (Baseline)	82.4	0.83	20K
0% (Improved DIMM)	84.1	1.15	20K
20%	84.6	1.87	30K
50%	87.1	2.25	40K
70%	92.3	2.67	50K
100%	95.5	3.22	60K

Appendix

Table 6: Mapping time versus performance. The

10\text{cm}/5^{\circ}

recall

\uparrow

and training steps are reported on Wayspots-Bears, varying the proportion of augmented synthesized images.

Scene	NaCR	NaCR	NaCR	RAP [ 17 ]	GRLoc [ 16 ]
NVS model	Nerfacto [ 36 ]	Deblur-NeRF [ 24 ]	3DGS [ 41 ] (w/o rendering loss)	3DGS	3DGS
chess	0.9 / 0.2 /23.67	1.0/ 0.2 /22.91	1.1/0.5/ 23.95	1.0/0.8/-	1.0/0.3/-
fire	1.0 / 0.2 /22.75	1.1/0.5/21.33	1.4/0.3/ 23.86	6.0/3.4/-	2.9/1.0/-
heads	1.3 / 0.4 /18.89	1.5/1.0/18.09	1.7/1.3/ 19.07	4.0/5.5/-	1.4/0.9/-
office	4.7/ 0.6 /21.98	3.2 /0.8/ 22.85	5.3/1.3/22.69	5.1/1.1/-	5.0/1.9/-

Appendix

Table 7: Ablation study on NeRF augmentation. The median translation/rotation errors

\downarrow

, along with rendering PSNR

\uparrow

, are reported on four 7-Scenes scenes.

NVS reference views (%)	NaCR $\downarrow$	Nerfacto PSNR $\uparrow$	RAP $\downarrow$	3DGS PSNR $\uparrow$
0	2.9/0.7	N/A	7.1/2.9	N/A
25	3.2/1.1	13.7	8.3/3.6	15.3
50	2.6/0.6	17.4	7.5/3.2	19.4
75	2.3/0.4	19.8	6.8/2.8	22.3
100	2.2/0.4	21.0	5.0/2.5	23.4

Appendix

Table 8: NVS reference-view sensitivity on 7-Scenes. Nerfacto and 3DGS use matched reference-view subsets. We report average median translation/rotation error (cm/ ∘ ) and test-view PSNR over all scenes.

Model	Bears	Cubes	Lawn	The Rock
I: DIMM	82.4	71.5	22.4	95.4
II: DIMM + Nerfacto	91.7	72.1	29.6	96.0
III: II + DA3 DINOv2	92.5	72.3	30.2	96.5
IV: III + Ref. Token Cond. (random)	92.1	72.1	29.8	96.9
V: III + Ref. Token Cond. (centered)	94.6	72.3	32.5	98.1
VI: V + Ray-Confidence Head	95.5	72.5	34.2	97.9

Appendix

Table 9: Ablation Study on Improved DIMM components. The recall

\uparrow

within

10\text{cm}/5^{\circ}

is reported on the Wayspots dataset.

Figure 7: The performance of image-level fine-tuning of DA3 DPT [ 30 ] decoder in 7-Scenes Chess. This is because the small VL training set generates highly correlated ray gradients within each image, which corrupts the powerful geometric priors learned during DA3’s large-scale pre-training.

Figure 8: Poor Rendering Image Quality of Nerfacto in 6-Scenes [ 7 ] .The severe illumination changes in the 6-Scenes dataset degrade the rendering quality of the NVS model, causing it to produce significant visual artifacts. This, in turn, negatively impacts NaCR’s localization performance by introducing noise into both of its NeRF-dependent components: (1) the synthetic images used for data augmentation are corrupted, and (2) the supervisory signal from the end-to-end rendering loss becomes unreliable.

Camera Pose Estimation

Camera pose estimation recovers a camera’s position and orientation from images, visual features, scene geometry, and sometimes sensor measurements.

See all 237 papers

Geometry Regularization for Vanilla NeRF via Self-Supervised Cross-Ray Consistency
Oct 20, 2026Zongji WangNeural Radiance Fields 3D Reconstruction
CLEAR-NeRF: Collinearity and Local-region Enhanced Accurate 3D Reconstruction in Unbounded Scenes
May 27, 2026Vladislav Polianskii, Elijs Dima, Isabel Salmerón Marazuela +33D Reconstruction Multi-View 3D Reconstruction
Ericsson Research, Stockholm, Sweden
DriftNeRF: Cross-Ray Density Drifting for Sparse-View Neural Radiance Fields
Oct 21, 2026Juntang Wang, Yihan Wang, Sheng'en Li +3Novel View Synthesis Neural Radiance Fields
Harvard University, Cambridge, USA · Duke University, Durham, USA · Duke Kunshan University, Kunshan, China

Oct 20, 2026J/K move · Enter open · S save

Geometry Regularization for Vanilla NeRF via Self-Supervised Cross-Ray Consistency

Zongji Wang

Neural Radiance Fields 3D Reconstruction Neural Representation Geometry

Open paper AskPDF arXiv

Neural Radiance Fields (NeRF) achieve high-quality novel-view synthesis by representing a scene as a continuous volumetric radiance field. However, the geometry encoded by the density field of vanilla NeRF can be noisy and inaccurate, since different combinations of shape and radiance may explain the same input images. To reduce this ambiguity, we augment vanilla NeRF with a geometry-optimization branch based on self-supervised cross-ray consistency. Specifically, this branch samples candidate surface points from the density field, generates virtual rays that intersect at each point from different directions, and renders their appearance using the NeRF network. Based on the visibility-weighted color consistency across these rays, we constrain the implicit geometry. We further regularize the implicit geometry by penalizing opacity at candidates with inconsistent appearance predictions and encouraging local normal smoothness. These geometric constraints are derived from the NeRF predictions and require no ground-truth depth or surface-normal supervision. Experiments on eight Replica scenes show that the proposed method reduces the mean Chamfer distance by 30.8% compared with vanilla NeRF.

Abstract

Camera Pose Estimation

Explore similar work

Geometry Regularization for Vanilla NeRF via Self-Supervised Cross-Ray Consistency

CLEAR-NeRF: Collinearity and Local-region Enhanced Accurate 3D Reconstruction in Unbounded Scenes

DriftNeRF: Cross-Ray Density Drifting for Sparse-View Neural Radiance Fields