What Do Learned 3D Models Add to Localization from Posed References?
Authors: Jaehyun Kim, Wonseok Kang, Jeongmin Lee, Tae-Wan Kim
Organizations: Seoul National University, Department of Naval Architecture and Ocean Engineering, Seoul, South Korea · DeepFine Co., Ltd., Seoul, South Korea
Localizing a query from a few posed reference images avoids building a 3D point map. We ask what the camera predictions of learned 3D models add to geometric estimation from the same references, and where failures arise. Across eight learned models and variants and five test sets, we compare reference budgets, swap pose components, and intervene on the references a model sees. At fine pose-error thresholds, geometric estimation with one additional reference is significantly more accurate than every model on two references on Cambridge and NAVER B1 and never significantly worse, alone or as a fallback behind a shared acceptance gate; with the same three references, the more accurate approach depends on the scene. The gain from the learned translation magnitude depends on reference selection: it improves seven of the eight models on 12-Scenes with the spread-out pairs we select, but none of the five we test with the two top-ranked retrieved references. Where retrieval also returns references from other locations, at NAVER Gangnam Station, models fail at different stages. A robust alignment brings Pi3X and MapAnything within 3.3 pp of adding references as needed. In controlled tests, aligning on the nearby references largely restores VGGT, Pi3X and MapAnything but not Depth Anything 3, and pose-conditioned Pi3X recovers only if the far poses are masked and left out of its input-pose normalization.
Figures & tables
Figure 1: Orange: learned, blue: geometric (subscripts ℓ , g ); differences are learned minus geometric; filled marks and bars: 95% interval excludes zero; diamond: VGGT, square: pose-conditioned Pi3X, circles: other models (line: range); bars take the color of the side ahead. (a) pp over all queries (only those the shared gate rejects differ): each model on two references minus geometric estimation on three (Tab. 3 b). (b) Recall change vs. PoseLib (pp) when the learned magnitude λ , or rotation R and direction d (magnitude refitted), replace the essential-matrix estimate’s (Tab. 2 ). (c) Left: pose-conditioned Pi3X on the same three references, no gate (Tab. 3 a). Right: NAVER B2/B1, three references within 10 m of the query plus two at least 50 m away, minus the near three alone, aligned on the near three (Tab. 5 ).
12-Sc.
7-Sc.
Camb.
B2
B1
SuperPoint+LightGlue, two references
alone
+25.4
+7.3
− 26.6
+15.8
+33.3
gate, PnP where rejected
+25.2
+8.6
− 1.9
+17.5
+32.8
gate, PoseLib where rejected
+12.7
+3.4
− 2.3
+5.2
+14.0
LoMa, two references
alone
+9.0
− 1.5
− 27.3
− 5.8
− 0.7
Table 1: Sensitivity to the comparison protocol: gain in recall (pp) of VGGT’s aligned pose over geometric estimation (positive: VGGT ahead), changing one choice per block; read down a column. Alone : against PnP, every query, no gate. Gate : both keep the PnP pose the shared gate accepts; on rejected queries, VGGT against PnP or PoseLib. B2, B1: NAVER floors; the 886 and 2,544 queries with stored SuperPoint+LightGlue matches. Pose-cond. Pi3X: its own pose instead of VGGT’s, same queries. Gray: 95% interval includes zero.
Δ recall vs. PoseLib (pp)
12-Sc.
7-Sc.
Camb.
B2
B1
learned translation magnitude, geometric rotation and direction
VGGT
+3.6
+0.4
− 0.8
+1.5
+1.1
Depth Anything 3
+4.7
+0.4
− 0.9
+2.1
+2.8
Pi3X
+4.0
+0.4
− 0.3
+1.0
+2.7
Pi3X, pose-cond.
+5.0
+0.5
+0.3
+1.3
+3.2
MapAnything
+3.0
− 0.1
− 0.6
+1.0
+2.8
Table 2: Pose components with two references, for queries rejected by the shared acceptance gate: change in recall (pp over all paired queries; positive: learned part ahead) over PoseLib’s pose when the model supplies only the translation magnitude, or only the rotation and translation direction, of our essential-matrix decomposition. Decomposition alone : no learned part, also against PoseLib. The learned magnitude helps most on 12-Scenes; learned rotation and direction lower recall on NAVER. Gray: 95% interval includes zero.
12-Sc.
7-Sc.
Camb.
B2
B1
(a) the same references, no gate (recall, %)
geometric estimation, three
95.9
96.9
61.7
50.1
58.3
pose-cond. Pi3X, the same three
97.4
97.8
52.6
43.6
56.9
pose-cond. Pi3X, two
95.1
96.1
48.5
48.6
56.1
Pi3X, the same three, robust
96.2
97.1
52.1
52.2
60.2
MapAnything, the same three, robust
95.1
91.3
46.5
49.2
60.6
Table 3: Additional references. (a) Recall without a gate, the same references for both: geometric estimation on three; pose-conditioned Pi3X on the same three or the first two, and Pi3X and MapAnything on the same three with the robust alignment of Tab. 4 (gray: not significantly different from geometric estimation). (b) Where the shared gate rejects, each model on two references minus geometric estimation on three, both keeping the accepted PnP pose (pp over all queries; gray: 95% interval includes zero); last rows: the accepted queries and the rejected share. Model estimate: VGGT’s learned-scale hybrid, pose-conditioned MapAnything’s own pose, otherwise the better of the two per test set (a choice that favors the models). (c) VGGT’s hybrid estimate as soon as the gate rejects, against add refs.
12-Sc.
7-Sc.
Camb.
B2
B1
add refs (recall, %)
97.4
97.3
60.9
59.0
65.6
geometric estimation, all five
+1.0
+0.8
+7.1
+3.4
+1.8
VGGT
− 1.7
+0.2
− 13.3
− 16.7
− 18.9
robust alignment, after
− 0.4
+0.6
− 13.4
− 6.9
− 4.8
Depth Anything 3
− 2.0
+0.7
− 10.6
− 15.9
− 18.2
robust alignment, after
0.0
+1.1
− 10.8
− 6.3
− 8.6
Table 4: The same five references for every model: its own pose where the stricter initial gate rejects, minus adding references as needed (add refs; pp over all queries; negative: add refs ahead; a missing estimate counts as incorrect). Model rows: its own pose aligned over all five (mean rotation and center offset); pose-conditioned MapAnything and Reloc3r return map poses themselves. Indented: inconsistent references excluded after inference (a robust alignment of the same predictions) or, for pose-conditioned Pi3X, before it (its most compact three of the five). Geometric estimation, all five: triangulation and PnP on all five for every query. Gray: 95% interval includes zero.
Figure 2: NAVER queries rejected by the stricter initial gate, by thirds of the spread of the five reference cameras around their centroid; (5): all five references (MapAnything, Reloc3r, MASt3R, not shown: at most 1% in the widest third). Queries: 478 (B2) and 1,815 (B1).
far
norm.
B2 (636)
B1 (1802)
poses
over
near
+far
near
+far
geometric estimation
–
–
85.7
84.7
78.6
78.5
Pi3X
–
–
88.1
86.2
80.9
76.7
VGGT
–
–
73.1
72.3
73.6
72.1
Depth Anything 3
–
–
83.2
55.5
74.4
39.3
MapAnything
–
–
83.8
82.5
79.7
79.3
Table 5: Failure stages at NAVER (recall, %). Near: three references within 10 m of the query; +far: two more at least 50 m away (chosen by true position only). Models are aligned on the near three after inference (pose-conditioned MapAnything: its own map poses); geometric estimation uses three or five references. Norm. over: views whose mean distance normalizes input poses. Aligned over all five, models without pose input reach 0%.
We present RIC-Loc (Reference-Induced Consensus localization), a scene-training-free posed-reference localizer that is SfM-point-map-free in its main estimator: it uses known reference poses, but not precomputed SfM 3D map points, query-to-map 2D-3D matches, or query-to-map PnP. A frozen VGGT pass predicts local camera poses, depth, and query-reference tracks for a query and selected references. Each reference induces one map-frame SE(3) query-pose hypothesis, robust consensus estimates the pose, and the preserved hypothesis structure yields two reliability scores: spatial dispersion and a track-conditioned covariance score. On the covariance-eligible set, the two scores are complementary for held-out, ground-truth-free failure detection across indoor, outdoor, and large-scale low-texture benchmarks: the joint policy is strongest in textured scenes and the covariance score in the low-texture regime, and the hypothesis-derived scores consistently outperform the standard retrieval-score gap and random rankings. Without per-scene training the consensus estimator remains accurate -- competitive with structure-based localization indoors and improving over a comparable feed-forward baseline -- giving an effective selective operating regime for posed-reference localization. Code is available at https://github.com/SNU-DLLAB/ric_loc.
Wonseok Kang, Jaehyun Kim, Jeongmin Lee +1
DeepFine Co., Ltd., Seoul, South Korea · Seoul National University, Department of Naval Architecture and Ocean Engineering, Seoul, South Korea
Localizing objects and parts from natural language in 3D space is essential for robotics, AR, and embodied AI, yet existing methods face a trade-off between the accuracy and geometric consistency of per-scene optimization and the efficiency of feed-forward inference. We present TrianguLang, a feed-forward framework for 3D localization that requires no camera calibration at inference. Unlike prior methods that treat views independently, we introduce Geometry-Aware Semantic Attention (GASA), which utilizes predicted geometry to gate cross-view feature correspondence, suppressing semantically-plausible but geometrically-inconsistent matches without requiring ground-truth poses. Validated on five benchmarks including ScanNet++ and uCO3D, TrianguLang achieves state-of-the-art feed-forward text-guided segmentation and localization, reducing user effort from O(N) clicks to a single text query. The model processes each frame at 1008x1008 resolution in ∼57ms (∼18 FPS) without optimization, enabling practical deployment for interactive robotics and AR applications. Code and checkpoints are available at https://cwru-aism.github.io/triangulang/.
Bryce Grant, Aryeh Rothenberg, Atri Banerjee +1
Case Western Reserve University, Cleveland, OH, USA
Spatial perception aims to estimate camera motion and scene structure from visual observations, a problem traditionally addressed through geometric modeling and physical consistency constraints. Recent learning-based methods have demonstrated strong representational capacity for geometric perception and are increasingly used to augment classical geometry-centric systems in practice. However, whether learning components should directly replace geometric estimation or instead serve as intermediate modules within such pipelines remains an open question. In this work, we address this gap and investigate an end-to-end modular framework for effective spatial reasoning, where learning proposes geometric hypotheses, while geometric algorithms dispose estimation decisions. In particular, we study this principle in the context of relative camera pose estimation on RGB-D sequences. Using VGGT as a representative learning model, we evaluate learning-based pose and depth proposals under varying motion magnitudes and scene dynamics, followed by a classical point-to-plane RGB-D ICP as the geometric backend. Our experiments on the TUM RGB-D benchmark reveal three consistent findings: (1) learning-based pose proposals alone are unreliable; (2) learning-proposed geometry, when improperly aligned with camera intrinsics, can degrade performance; and (3) when learning-proposed depth is geometrically aligned and followed by a geometric disposal stage, consistent improvements emerge in moderately challenging rigid settings. These results demonstrate that geometry is not merely a refinement component, but an essential arbiter that validates and absorbs learning-based geometric observations. Our study highlights the importance of modular, geometry-aware system design for robust spatial perception.
Haichao Zhu, Zhaorui Yang, Qian Zhang
Reality Vision · Department of Computer Science and Engineering, University of California, Riverside