A camera becomes an absolute pose sensor when its images can be relocalized against a prebuilt map of the scene. On mobile robots and augmented-reality devices, that map must be small and quick to build. Scene coordinate regression (SCR) stores the scene in a compact network that predicts a 3D coordinate for every pixel, and mixture-of-experts (MoE) variants extend its capacity to larger outdoor scenes. In existing MoE-based SCR, however, the router assigns each pixel to an expert from the same local descriptor that the experts regress from, so repeated facades and windows at different locations receive similar routing inputs. We present DAC-MACE, which conditions soft expert routing on recalibrated and contextualized features. A residual context adapter (RCA) reweights the channels and locations of frozen encoder features, a global context encoder (GCE) mixes them with multi-scale self-attention, and an image-level pooled descriptor is added to the router input. A dual-threshold RANSAC--Levenberg--Marquardt estimator recovers the pose. On Cambridge Landmarks, DAC-MACE attains a mean per-scene median translation error of 13.0 cm over three seeds, the lowest among the RGB-only SCR methods compared (GLACE 13.6 cm, MACE 14.4 cm), and reduces the error on the largest scene from 24 to 19 cm relative to MACE. In a cumulative study, the MoE head, GCE, RCA, and the dual-threshold estimator lower the mean error from 17.0 cm to 14.4, 14.0, 13.6, and 13.0 cm. On 7-Scenes, adding GCE to a single ACE head raises Recall@(5 cm, 5∘) from 86.56% to 87.60%.
Figures & tables
Fig. 1: DAC-MACE pipeline. The frozen ACE encoder produces Fraw . RCA recalibrates channels and locations ( Frca ), and GCE adds multi-scale self-attention context ( Fg ). The router receives Fg plus an image-level pooled residual and predicts per-pixel soft weights over K position-aware expert heads E1,…,EK , whose fused scene coordinates are passed to the dual-threshold RANSAC–LM pose stage. Training uses the reprojection loss Lsc and the load-balancing term Lbal .
TABLE II: Cambridge Landmarks: Median Translation (cm) / Rotation ( ∘ ) Errors. Avg. Is the Mean of the Five Per-Scene Medians, Computed From the Values Listed. Bold Marks the Best RGB-Only SCR Translation Error Per Column. Mapping Times Come From Different Implementations and Hardware and Are Indicative Only. † Excludes Pre-Clustering Time. ‡ Routing Stage, Excluding the ACE Pretraining of the Cluster Experts. § Expert Heads; for DAC-MACE, K=4 (5 MB for K=2 ), With the Context Modules Listed in Table X . The DAC-MACE Row Is the Mean Over Three Seeds.
Fig. 2: Memory–accuracy trade-off on Cambridge Landmarks: map size versus the mean of per-scene median translation errors (values of Table II ). DAC-MACE has the lowest error among the RGB-only SCR methods. Plotted method/value pairs: AS (200 MB, 13.8 cm), hLoc (800, 10.8), PixLoc (600, 15.0), DSAC* with depth (28, 20.6), SRC with depth (40, 41.6), SANet with depth (260, 83.6), RGB-only DSAC* (28, 18.6), DSAC* Tiny (4, 45.0), ACE (4, 25.0), Poker (16, 17.0), GLACE (13, 13.6), MACE (5, 14.4), and DAC-MACE (10, 13.0).
Variant
Great Court
King’s College
Old Hospital
Shop Facade
St Mary’s
Avg.
Δ Avg.
ACE Poker [ 8 ]
28/0.1
18/0.3
25/0.5
5/0.3
9/0.3
17.0/0.30
–
+ MoE (MACE configuration) [ 10 ]
24/0.2
15/0.3
19/0.4
5/0.2
9/0.3
14.4/0.28
−2.6
+ GCE
21/0.1
16/0.3
19/0.4
5/0.2
9/0.3
14.0/0.26
−0.4
+ RCA
22/0.1
15/0.3
18/0.4
5/0.2
8/0.3
13.6/0.26
−0.4
+ Dual-threshold pose stage
19/0.1
15/0.3
18/0.4
5/0.2
8/0.4
13.0/0.28
−0.6
DAC-MACE, mean of three seeds
19/0.1
15/0.3
18/0.4
5/0.2
8/0.3
13.0/0.26
TABLE III: Cumulative Component Analysis on Cambridge Landmarks: Median Translation (cm) / Rotation ( ∘ ). Each Row Adds One Component to the Previous Row. The First Two Rows Are the Poker and MACE Configurations With the Values Reported in [ 8 , 10 ] ; the GCE, RCA, and Dual-Threshold Rows Are the Reference Run (Seed 2089); the Last Row Is the Three-Seed Mean of the Full Model. Δ Avg. Is the Change in Mean Translation Error With Respect to the Previous Row.
TABLE IV: Training and Pose-Estimation Settings, Shared by All Scenes
Method
Chess
Fire
Heads
Office
Pumpkin
Red Kitchen
Stairs
Recall (%)
FM
DenseVLAD [ 11 ]
21/12.5
33/13.8
15/14.9
28/11.2
31/11.3
30/12.3
25/15.8
N/A
Oracle [ 18 ]
16/12.3
26/13.6
12/14.7
20/11.5
19/14.0
18/15.0
17/18.1
0.17
AS [ 3 ]
3/0.87
2/1.01
1/0.82
4/1.15
7/1.69
5/1.72
4/1.01
68.7
InLoc [ 16 ]
3/1.05
3/1.07
2/1.16
3/1.05
5/1.55
4/1.31
9/2.47
66.3
hLoc [ 4 , 14 ]
2/0.85
2/0.94
1/0.75
3/0.92
5/1.30
4/1.40
5/1.47
73.1
PixLoc [ 18 ]
2/0.80
2/0.73
1/0.82
3/0.82
4/1.21
3/1.20
5/1.30
75.7
TABLE V: 7-Scenes With D-SLAM Reference Poses: Median Translation (cm) / Rotation ( ∘ ) and Recall@(5 cm, 5 ∘ ) in %. “SCR+D” Denotes SCR Methods Trained With Depth or Geometry. ACE and ACE+GCE Were Trained by Us With Identical Settings; ACE+GCE Has a Single Head and No MoE. The Oracle Recall Is Reproduced From [ 18 ] . Bold: Best Rotation Error and Recall per Column, and Best Translation Error Where It Is Unique at 1-cm Resolution.
Fig. 3: Trajectory visualization on Great Court. (a) Mapping-camera centers (blue squares) and all 760 estimated query camera centers (pink) over the sparse SfM point cloud. (b) The same query estimates colored by translation error; values of 20 cm and above saturate the color scale. The point cloud is thinned for display, while every query is shown. Camera centers are drawn without frustums, so the visualization does not depend on a camera-forward convention.
Fig. 4: Trajectory visualization on St Mary’s Church. Left: mapping and estimated query camera positions over the SfM point cloud. Right: estimated query positions colored by translation error.
Scene
MoE
GCE
RCA
Dual-threshold
Great Court
−4
−3
+1
−3
King’s College
−3
+1
−1
0
Old Hospital
−6
0
−1
0
Shop Facade
0
0
0
0
St Mary’s Church
0
0
−1
0
Avg.
−2.6
−0.4
−0.4
−0.6
TABLE VI: Per-Scene Change in Median Translation Error (cm) Contributed by Each Component, Derived From Consecutive Rows of Table III . Negative Values Denote Improvement.
Scene
Single-threshold PnP+LM
Dual-threshold RANSAC+LM
Great Court
20/0.1
19/0.1
King’s College
16/0.3
15/0.3
Old Hospital
18/0.4
18/0.4
Shop Facade
5/0.2
5/0.2
St Mary’s Church
8/0.3
8/0.3
Avg.
13.4/0.26
13.0/0.26
TABLE VII: Pose-Stage Ablation on the Fixed Trained DAC-MACE Models (Mean of Three Seeds): Median Translation (cm) / Rotation ( ∘ ). Only the Inference-Time Pose Stage Is Changed.
Scene
Mapping
Strict, 6 px
Loose, 10 px
Query
Great Court
1,531
617 (81.2%)
143 (18.8%)
760
King’s College
1,220
343 (100.0%)
0 (0.0%)
343
Old Hospital
895
179 (98.4%)
3 (1.6%)
182
Shop Facade
231
3 (2.9%)
100 (97.1%)
103
St Mary’s Church
1,487
448 (84.5%)
82 (15.5%)
530
All scenes
5,364
1,590 (82.9%)
328 (17.1%)
1,918
TABLE VIII: Cambridge Data Split and Dual-Threshold Branch Selection. Strict and Loose Counts Sum to Each Scene’s Number of Query Images.
Fig. 5: Qualitative feature-response diagnostic on a repetitive Old Hospital facade. Left: query image; the red cross marks the query location. Middle: cosine similarity between the query feature and every feature of the image before GCE. Right: the same after GCE. Each map is min–max normalized to [0,1] independently and bilinearly upsampled, so colors indicate spatial concentration rather than calibrated confidence.
Fig. 6: Hard expert-selection ratios (%) of DAC-MACE on Cambridge Landmarks, without (left) and with (right) load balancing, i.e., the standard-deviation penalty in ( 15 ) and the router-bias update. Crossed cells mark experts that do not exist in the K=2 models.
Hnorm
Neff
Max (%)
Scene
K
w/o
w/
w/o
w/
w/o
w/
Shop Facade
2
0.000
0.986
1.00
1.98
100.0
56.9
Old Hospital
2
0.000
0.997
1.00
2.00
100.0
53.4
King’s College
4
0.494
0.977
1.98
3.88
56.3
32.9
St Mary’s Church
4
0.532
0.943
2.09
3.70
51.8
41.7
Great Court
4
0.787
0.990
2.98
3.95
37.2
30.1
TABLE IX: Expert-Usage Statistics Derived From Fig. 6 , Without/With Load Balancing (LB). With Hard Selection Ratios pk , Hnorm=−∑kpklogpk/logK and Neff=exp(−∑kpklogpk) ; Max: Largest Expert Share (%).
Component
Params (M)
GFLOPs
Scene (M)
ACE encoder (frozen)
5.57
78.1
–
RCA
0.03
<0.1
0.03
GCE, scale-1 branch
1.05
97.9
1.05
GCE, scale-2 branch
1.05
8.5
1.05
GCE, fusion
0.53
6.7
0.53
Router
0.66
8.4
0.66
TABLE X: Parameters and Floating-Point Operations of DAC-MACE ( K=4 ) for a 480×853 Cambridge Query ( 60×107 Features), Computed Analytically From the Network Definition. Last Column: Scene-Specific Parameters (M).
Among visual localization strategies, scene coordinate regression (SCR) has emerged as an attractive approach for its compactness, accuracy, and speed. Despite these advantages, SCR models generalize poorly across substantially different camera models, such as omnidirectional or strongly rectified imagery. This limitation arises because, changes in field of view, lens distortion, or rectification-induced resampling can make the same scene content appear differently depending on its pixel position, degrading localization accuracy. To cope with this limitation, we introduce Ray-SCR, a lightweight extension to existing SCR architectures that conditions local visual features on their calibrated viewing rays and refines neighboring ray-conditioned features with a shallow local Transformer. We further formulate supervision directly in ray space and use sparse SfM points as metric anchors that link observations across views and camera models. Taken together, these components enable a single scene representation built from a panoramic mapping sequence to accurately localize query images captured with very different cameras, including pinhole, fisheye, and omnidirectional cameras. We evaluate Ray-SCR across multiple SCR architectures and on datasets with pinhole, fisheye, and omnidirectional cameras. The proposed ray conditioning substantially improves generalization across cameras and leads to state-of-the-art localization performance in these challenging cross-camera settings. The project page is available at https://jaeminch0.github.io/ray-scr-generalizing-scene-coordinate-regression-across-camera-models/.
Autonomous flight in GPS-denied indoor spaces requires trajectories that keep visual-localization error tightly bounded across varied missions. Map-based visual localization methods such as feature matching require computationally intensive map reconstruction and have feature-storage scalability issues, especially for large environments. Scene coordinate regression (SCR) provides an efficient learning-based alternative that directly predicts3D coordinates for every pixel, enabling absolute pose estimation with significant potential for onboard roboticsapplications. We present a perception-aware trajectory planner that couples an evidential learning-based SCR poseestimator with a receding-horizon trajectory optimizer. The optimizer steers the onboard camera toward reliablescene coordinates with low uncertainty, while a fixed-lag smoother fuses the low-rate SCR pose estimates with high-rate IMU data to provide a high-quality, high-rate pose estimate. In simulation, our planner reduces translationand rotation RMSE by at least 4.9% and 30.8% relative to baselines, respectively. Hardware-in-the-loop experiments validate the feasibility of our proposed trajectory planner under close-to-real deployment conditions.
Juyeop Han, Lukas Lao Beyer, Guilherme V. Cavalheiro +1
Laboratory for Information and Decision Systems (LIDS), Massachusetts Institute of Technology, Cambridge, MA 02139, USA
Scene Coordinate Regression (SCR) has recently emerged as a promising approach for LiDAR-based localization, achieving accurate localization without requiring an explicit 3D map. Despite their effectiveness, existing SCR methods rely on scene classification-based global embedding that struggles to provide fine-grained discrimination among nearby locations. Moreover, their reliance on uniform sampling of local features during training assigns equal importance to all points, thereby inadvertently propagating features from dynamic objects or unstable regions and potentially degrading training stability. In this paper, we present ReLoc, a revamped SCR architecture that can effectively address these limitations. First, we redesign the global embedding module by combining learnable context tokens with a feature aggregator to capture richer and more discriminative scene context. Second, we introduce an attention-based local feature enhancement module to mitigate the impact of noisy local features while encouraging context-consistent structures, yielding more robust local feature representations. Experimental results on two large-scale outdoor datasets demonstrate that our approach achieves state-of-the-art accuracy over previous SCR-based methods while maintaining real-time inference performance.
Heejoon Moon, Yurim Cho, Je Hyeong Hong
Department of Artificial Intelligence, Hanyang University · Department of Electronic Engineering, Hanyang University