Among visual localization strategies, scene coordinate regression (SCR) has emerged as an attractive approach for its compactness, accuracy, and speed. Despite these advantages, SCR models generalize poorly across substantially different camera models, such as omnidirectional or strongly rectified imagery. This limitation arises because, changes in field of view, lens distortion, or rectification-induced resampling can make the same scene content appear differently depending on its pixel position, degrading localization accuracy. To cope with this limitation, we introduce Ray-SCR, a lightweight extension to existing SCR architectures that conditions local visual features on their calibrated viewing rays and refines neighboring ray-conditioned features with a shallow local Transformer. We further formulate supervision directly in ray space and use sparse SfM points as metric anchors that link observations across views and camera models. Taken together, these components enable a single scene representation built from a panoramic mapping sequence to accurately localize query images captured with very different cameras, including pinhole, fisheye, and omnidirectional cameras. We evaluate Ray-SCR across multiple SCR architectures and on datasets with pinhole, fisheye, and omnidirectional cameras. The proposed ray conditioning substantially improves generalization across cameras and leads to state-of-the-art localization performance in these challenging cross-camera settings. The project page is available at https://jaeminch0.github.io/ray-scr-generalizing-scene-coordinate-regression-across-camera-models/.
Figures & tables
Figure 1: Cross-camera localization with Ray-SCR. A scene mapped from panoramic images is queried using pinhole, fisheye, and 360∘ cameras with substantially different fields of view and image distortions. Ray-SCR uses calibrated viewing rays associated with local image features to enable localization across these heterogeneous cameras.
Figure 2: Ray-SCR scene-training pipeline. We first extract encoder features from the training images, optionally augmented with sampled virtual views. SfM points are projected into each view, and features whose angular distance θi from the corresponding viewing ray ri is below a predefined threshold are retained in the training buffer. The Ray Adapter then injects camera-geometry information by encoding the associated viewing rays into the features and locally processing groups of neighboring features with a lightweight Transformer. The adapted features are subsequently passed through the standard SCR pipeline. Here, ACE-G is shown as an example, using a DINO encoder together with trainable scene-specific map codes.
Figure 3: Ray Adapter Architecture. Feature tiles sampled from the training buffer are paired viewing rays. Each ray is encoded and used to condition the visual feature through FiLM modulation and an additive ray embedding. The resulting features are then jointly refined within each local tile using a local Transformer before being passed to the SCR prediction head.
Scene
Atrium Day
Concourse Day
Camera
Pinhole
Fisheye1
Fisheye2
Fisheye3
360
Average
Pinhole
Fisheye1
Fisheye2
Fisheye3
360
Average
APR
PN [ 17 ]
15.5/85.5
11.8/79.9
11.3/80.4
10.1/79.8
4.5/82.6
10.6/81.6
18.6/95.0
10.7/88.7
10.1/87.7
7.7/85.9
2.6/39.0
9.9/79.3
MS-T [ 36 ]
20.1/77.7
14.5/73.7
14.0/71.7
12.5/65.6
13.2/54.7
14.9/68.7
17.0/69.1
11.0/60.8
10.5/59.6
9.9/63.4
4.9/34.6
10.7/57.5
PN-VC2 [ 14 ]
11.5/31.2
9.1/19.5
8.9/19.3
8.4/18.1
7.7/17.1
9.1/21.0
7.0/18.9
4.7/11.5
4.4/11.2
3.9/9.9
3.2/6.9
4.6/11.7
MS-T-VC2 [ 14 ]
10.0/46.7
6.3/36.3
6.2/36.5
4.9/28.1
5.2/47.6
6.5/39.0
3.5/19.6
1.9/12.7
1.8/12.8
1.7/12.7
1.6/13.5
2.1/14.3
CDG-Loc [ 40 ]
5.2 / 21.8
2.2 / 12.7
2.1 / 12.3
1.8 / 11.5
1.6 / 14.1
2.6 / 14.5
2.7 / 9.7
1.5 / 5.7
1.3 / 5.5
1.1 / 5.4
1.0 / 5.7
1.5 / 6.4
Table 1: Cross-camera localization on 360Loc Day. Median translation/rotation errors (m/ ∘ ). Best results are shown in bold and second-best results are underlined .
Scene
Atrium Night
Concourse Night
Camera
Pinhole
Fisheye1
Fisheye2
Fisheye3
360
Average
Pinhole
Fisheye1
Fisheye2
Fisheye3
360
Average
APR
PN [ 17 ]
19.8/90.8
16.4/87.3
16.1/86.7
15.7/83.5
8.7/22.8
15.3/74.2
19.8/96.2
13.3/85.5
13.0/88.4
11.1/87.5
5.5/45.6
12.5/80.6
MS-T [ 36 ]
24.8/84.7
17.8/80.5
17.0/80.1
15.1/80.0
5.0/46.3
15.9/74.3
22.1/74.7
17.7/69.5
17.0/68.3
17.2/72.7
8.4/57.3
16.5/68.5
PN-VC2 [ 14 ]
13.6/50.2
11.5/38.1
11.6/38.3
11.4/37.8
9.7/45.7
11.6/42.0
10.7/25.4
8.3/16.4
8.4/15.7
8.1/14.7
5.5/13.5
8.2/17.1
MS-T-VC2 [ 14 ]
13.5/70.6
9.1/64.1
9.0/63.8
8.6/64.4
7.3/61.5
9.5/64.9
5.5/28.7
4.0/20.1
3.9/20.3
3.5/20.9
3.4/23.7
4.1/22.7
CDG-Loc [ 40 ]
4.6 / 25.3
1.8 / 11.0
1.7 / 10.1
1.6 / 10.1
1.4 / 7.3
2.2 / 12.8
3.3 / 11.4
1.7 / 6.6
1.6 / 6.3
1.4 / 6.9
1.1 / 6.1
1.8 / 7.5
Table 2: Cross-camera localization on 360Loc Night. Median translation/rotation errors (m/ ∘ ). Best results are shown in bold and second-best results are underlined .
Atrium Day
Camera
Pinhole
Fisheye1
Fisheye2
Fisheye3
360
Avg
Ray-DSAC ∗ Net
14.2/28.1
5.0/10.8
4.5/9.8
5.8/13.1
7.2/12.1
7.3/14.8
Ray-ACE
13.5/23.3
5.2/11.5
4.9/11.8
6.0/13.5
7.9/8.1
7.5/13.6
Ray-GLACE
7.5/18.1
2.8/6.8
2.7/7.0
3.7/11.6
4.3/8.0
4.2/10.3
Ray-ACEG
1.9 / 4.3
1.2 / 2.9
1.2 / 3.0
1.1 / 2.7
0.9 / 2.3
1.3 / 3.0
Concourse Day
Table 3: Compatibility across SCR models on 360Loc Day. In addition to the Ray-ACEG results, we instantiate Ray-SCR with DSAC*, ACE, and GLACE. All four models are evaluated on the five query camera types. Values are median translation/rotation errors (m/ ∘ ), with the best results shown in bold .
Figure 4: Cross-camera consistency of predicted scene coordinates on Piatrium-Day. Ray-SCR produces substantially overlapping 3D scene points from all five camera types.
Figure 5: Map size and localization accuracy on OmniScenes. Explicit-map methods (circles) use a pre-captured 3D point-cloud map at query time, whereas Ray-Poker-SCR (squares) represents each scene implicitly in network weights. Ray-Poker-ACE achieves the highest accuracy with 77× less storage than FGPL [ 21 ] .
Augmented views
Query camera
360
Fish.
Pin.
Pinhole
Fisheye1
Fisheye2
Fisheye3
360
Avg
ACE-G [ 7 ]
✓
✓
✓
4.7/9.9
4.1/11.2
4.1/12.2
4.5/11.9
8.1/33.5
5.1/15.7
✓
×
✓
3.6/8.8
4.9/13.7
4.9/14.6
5.9/18.6
8.0/34.1
5.5/18.0
✓
✓
×
9.8/17.2
4.6/12.5
4.5/12.7
4.6/13.0
7.7/32.9
6.2/17.7
✓
×
×
42.9/70.9
42.5/76.0
40.7/75.5
47.0/68.0
7.6/31.9
36.1/64.5
Ray-ACEG
✓
✓
✓
1.4 / 6.0
0.9 / 4.4
1.0 / 4.5
0.8 / 4.1
0.7 / 3.5
1.0 / 4.5
Table 5: Virtual-view augmentation on 360Loc Concourse-Day. ACE-G and Ray-ACEG are trained with augmented mapping views from the rendered cross cameras and evaluated on all five query cameras. Values are median translation/rotation errors (m/ ∘ ).
Query camera
Pinhole
Fisheye1
Fisheye2
Fisheye3
360
Avg
Cubemap-ACE [ 4 ]
18.8/36.1
10.1/18.7
8.7/13.1
6.8/11.2
3.1/4.4
9.5/16.7
Ray-ACE
6.9/26.8
2.6/12.6
2.3/12.1
3.4/15.8
5.0/14.5
4.0/16.4
Cubemap-ACEG [ 7 ]
14.5/14.4
9.6/13.8
9.7/13.1
10.0/13.1
7.2/8.9
10.2/12.7
Ray-ACEG
1.4 / 6.0
0.9 / 4.4
1.0 / 4.5
0.8 / 4.1
0.7 / 3.5
1.0 / 4.5
Table 6: Cubemap-based SCR on 360Loc Concourse-Day. Standard ACE and ACE-G are trained and evaluated on perspective cubemap faces, whereas their Ray-SCR counterparts operate directly on native calibrated queries. Values are median translation/rotation errors (m/ ∘ ).
w/o Attention
w/o Lray
w/o Lsfm
Full
Day Avg.
1.32/5.89
1.74/9.16
1.53/8.39
0.96 / 4.54
Night Avg.
1.54/6.95
2.37/11.83
1.75/9.28
1.28 / 5.98
Table 7: Component analysis on 360Loc Concourse. Values are camera-averaged median translation/rotation errors (m/ ∘ ) for Day and Night queries.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
w=1
w=2
w=4
w=8 (main)
w=16
Day Avg.
1.26/5.72
1.25/6.00
1.11/5.60
0.96 / 4.54
0.98/4.77
Night Avg.
1.52/6.68
1.48/6.82
1.39/6.49
1.28/5.98
1.20 / 5.60
Appendix
Table 8: Transformer window size ablation on 360Loc Concourse. Values are camera-averaged median translation/rotation errors (m/ ∘ ) for Day and Night queries.
Figure 6: OmniScenes mapping and query examples. The mapping panoramas are captured by a robot-mounted camera, while the query panoramas are captured by a handheld camera after rearrangement of the scene.