Target localization is a prerequisite for embodied tasks such as navigation and manipulation. Conventional approaches rely on constructing explicit 3D scene representations to enable target localization, such as point clouds, voxel grids, or scene graphs. While effective, these pipelines incur substantial mapping time, storage overhead, and scalability limitations. Recent advances in vision-language models suggest that rich semantic reasoning can be performed directly on 2D observations, raising a fundamental question: is a complete 3D scene reconstruction necessary for object localization? In this work, we revisit object localization and propose a map-free pipeline that stores only posed RGB-D keyframes as a lightweight visual memory--without constructing any global 3D representation of the scene. At query time, our method retrieves candidate views, re-ranks them with a vision-language model, and constructs a sparse, on-demand 3D estimate of the queried target through depth backprojection and multi-view fusion. Compared to reconstruction-based pipelines, this design drastically reduces preprocessing cost, enabling scene indexing that is over two orders of magnitude faster to build while using substantially less storage. We further validate the localized targets on downstream object-goal navigation tasks. Despite requiring no task-specific training, our approach achieves strong performance across multiple benchmarks, demonstrating that direct reasoning over image-based scene memory can effectively replace dense 3D reconstruction for object-centric robot navigation. Project page: https://ruizhou-cn.github.io/memory-over-maps/
Figures & tables
Fig. 1: Image-based target localization without dense 3D reconstruction. Using only posed RGB-D keyframes as a lightweight visual memory, our method directly infers the 3D target location and enables a robot to navigate to the queried object without building a global scene map.
Fig. 2: Method overview. Given a query and posed RGB-D keyframes: (1) Retrieval: SigLIP2 embeddings indexed with FAISS retrieve top- K candidates. (2) VLM Re-Rank: a VLM filters false positives (red) and promotes true matches (green). (3) 3D Localization: SAM 3 segments the target; masked depth is backprojected, predictions are grouped into object instances, and per-instance multi-view fusion produces a 3D goal estimate. No global 3D map is built. (4) Navigation: a PointNav policy navigates directly to the goal. (Scene point cloud shown for visualization only.)
Scene
Method
Average
Object
Image
Language
4ok
DynaMem [ 28 ]
10.1
13.2
17.1
0.0
VLMaps [ 16 ]
63.3
75.5
41.8
70.8
LagMemo [ 29 ]
86.7
95.2
73.0
88.6
OpenFunGraph [ 26 ]
77.5
73.7
70.7
87.8
HOV-SG [ 17 ]
78.5
73.7
–
82.9
Ours
90.1
92.1
85.4
92.7
TABLE I: GOAT-Core localization. SR@5 (%) on GOAT-Core [ 29 ] . “–” indicates the method does not support that query type.
Sensor
FLAVA [ 32 ]
CLIP-B [ 30 ]
ALIGN [ 31 ]
CLIP-L [ 30 ]
Ours
AR@1 ↑
KV1
5.3
60.0
67.5
62.0
70.5
KV2
26.2
48.4
64.5
69.8
82.2
Xtion
15.2
49.7
58.2
64.7
73.1
RealSense
19.6
66.7
46.6
60.2
62.0
Avg.
16.6
56.2
59.2
64.2
71.9
TABLE III: Text-to-image retrieval. AR@1 and AR@5 on SUN RGB-D [ 58 ] .
Fig. 3: Qualitative retrieval examples. Top-1 retrieved images for natural-language queries across small indoor (HM3D [ 51 ] , MP3D [ 52 ] ), large indoor (LaMAR-CAB [ 59 ] ), and outdoor (LaMAR-LIN [ 59 ] ) scenes.
Fig. 4: Qualitative comparison with HOV-SG [ 17 ] . Our method retrieves the correct view with segmentation mask and produces a fused 3D point cloud, capturing fine-grained details that HOV-SG overlooks.
Fig. 5: Real-world robot navigation. Spot navigates to queried objects in an iPad-scanned indoor scene. (a) Third-person view of the robot at the goal. (b) Top-1 retrieved image with segmentation mask ( green ). (c) Spot’s onboard camera view upon arrival.
Fig. 6: VLM re-ranking ablation. SigLIP2 retrieves semantically similar but incorrect views ( red ); VLM re-ranking corrects all three queries ( green ).
Standard visual localization methods typically require offline pre-processing of scenes to obtain 3D structural information for better performance. This inevitably introduces additional computational and time costs, as well as the overhead of storing scene representations. Can we visually localize in a wild scene without any off-line preprocessing step? In this paper, we leverage the online inference capabilities of feed-forward 3D reconstruction networks to propose a novel map-free visual localization framework L3. Specifically, by performing direct online 3D reconstruction on RGB images, followed by two-stage metric scale recovery and pose refinement based on 2D-3D correspondences, L3 achieves high accuracy without the need to pre-build or store any offline scene representations. Extensive experiments demonstrate L3 not only that the performance is comparable to state-of-the-art solutions on various benchmarks, but also that it exhibits significantly superior robustness in sparse scenes (fewer reference images per scene).
Zero-shot object navigation requires agents to locate unseen targets in unfamiliar environments without prior maps or task-specific training. Despite the commonsense reasoning ability of vision-language models (VLMs), existing mapless navigators often suffer from repeated exploration and premature stopping caused by limited historical context and false-positive target predictions. We propose ReMemNav, a training-free framework that combines lightweight semantic grounding, memory-based decision correction, and target verification. RAM-derived semantic priors guide panoramic direction selection, while a geometry-triggered correction mechanism retrieves historical scene descriptions when the selected direction points toward previously visited regions. When a target is predicted to be visible, ReMemNav verifies the corresponding single-view observation before the final approach. Depth-based action sampling then converts the high-level decision into collision-free motion. Experiments on HM3D and MP3D show that ReMemNav achieves higher success rates and path efficiency than existing training-free zero-shot baselines. Specifically, ReMemNav achieves absolute SR/SPL gains of 1.7%/5.3% on HM3D v0.1, 18.2%/11.1% on HM3D v0.2, and 8.7%/6.9% on MP3D.
Localizing objects and parts from natural language in 3D space is essential for robotics, AR, and embodied AI, yet existing methods face a trade-off between the accuracy and geometric consistency of per-scene optimization and the efficiency of feed-forward inference. We present TrianguLang, a feed-forward framework for 3D localization that requires no camera calibration at inference. Unlike prior methods that treat views independently, we introduce Geometry-Aware Semantic Attention (GASA), which utilizes predicted geometry to gate cross-view feature correspondence, suppressing semantically-plausible but geometrically-inconsistent matches without requiring ground-truth poses. Validated on five benchmarks including ScanNet++ and uCO3D, TrianguLang achieves state-of-the-art feed-forward text-guided segmentation and localization, reducing user effort from O(N) clicks to a single text query. The model processes each frame at 1008x1008 resolution in ∼57ms (∼18 FPS) without optimization, enabling practical deployment for interactive robotics and AR applications. Code and checkpoints are available at https://cwru-aism.github.io/triangulang/.
Bryce Grant, Aryeh Rothenberg, Atri Banerjee +1
Case Western Reserve University, Cleveland, OH, USA