Authors: Hardik Shah, Erica Tevere, Deegan Atha, Marcel Kaufmann, Shehryar Khattak, Manthan Patel, Marco Hutter, Jonas Frey, Patrick Spieler
Organizations: Jet Propulsion Laboratory (JPL), California Institute of Technology (Caltech), Pasadena, CA 91011, USA · Swiss Federal Institute of Technology (ETH Zürich), Robotic Systems Lab, Zürich 8092, Switzerland · FieldAI Inc. 3 Morgan, Irvine, CA, USA. Work Performed at NASA JPL · Stanford University, Autonomous Systems Lab, Stanford, CA 94305, USA · University of California Berkeley, Berkeley, CA 94720, USA
Autonomous navigation in complex, unstructured outdoor environments requires robots to operate over long ranges without prior maps and limited depth sensing. In such settings, relying solely on geometric frontiers for exploration is often insufficient. In such settings, the ability to reason semantically about where to go and what is safe to traverse is crucial for robust, efficient exploration. This work presents WildOS, a unified system for long-range, open-vocabulary object search that combines safe geometric exploration with semantic visual reasoning. WildOS builds a sparse navigation graph to maintain spatial memory, while utilizing a foundation-model-based vision module, ExploRFM, to score frontier nodes of the graph. ExploRFM simultaneously predicts traversability, visual frontiers, and object similarity in image space, enabling real-time, onboard semantic navigation tasks. The resulting vision-scored graph enables the robot to explore semantically meaningful directions while ensuring geometric safety. Furthermore, we introduce a particle-filter-based method for coarse localization of the open-vocabulary target query, that estimates candidate goal positions beyond the robot's immediate depth horizon, enabling effective planning toward distant goals. Extensive closed-loop field experiments across diverse off-road and urban terrains demonstrate that WildOS enables robust navigation, significantly outperforming purely geometric and purely vision-based baselines in both efficiency and autonomy. Our results highlight the potential of vision foundation models to drive open-world robotic behaviors that are both semantically informed and geometrically grounded. Project Page: https://leggedrobotics.github.io/wildos/
Figures & tables
Figure 1: (a) WildOS enables autonomous semantic navigation in diverse unstructured outdoor environments. (b, c) Due to the limited range of geometric sensing, robots can only reliably perceive nearby regions within a depth horizon (blue), leading to myopic exploration and (e) difficulty localizing distant targets (e.g., a “house”) beyond sensing range. (b, c) Conventional exploration (dashed path) relies on geometric frontiers (blue dots) at the boundary between known and unknown space, which ignores long-range semantic and traversability cues. WildOS (green path) augments geometric exploration with long-range visual reasoning using a vision foundation model, defining a visual horizon (red) that extends beyond depth sensing and predicts visual traversability, visual frontiers (red dots), and open-vocabulary object similarity in image space. (d) During deployment, a sparse navigation graph is built from geometry and frontier nodes are scored using vision, while a particle-filter-based goal localization module (yellow particles) estimates candidate goal locations beyond the depth horizon, enabling safe, efficient planning toward distant semantic goals.
Figure 2: Method Overview consisting of five main components: 1) WildOS incrementally builds a sparse navigation graph from geometric sensing to maintain persistent spatial memory and identify geometric frontier nodes for safe exploration (Sec. 4-B ). 2) To reason beyond the limited depth horizon, a learned vision-language module, ExploRFM, processes the current image and text query to predict visual traversability, visual frontiers, and open-vocabulary object similarity over a long-range visual horizon (Sec. 4-C ). 3) Object detections from multiple viewpoints are fused by a probabilistic goal triangulation module to estimate a coarse 3D target location beyond direct sensor range (Sec. 4-D ). 4) Geometric frontier nodes are then projected into the image and scored using the visual-semantic cues and the current goal estimate, producing a semantically scored navigation graph (Sec. 4-E ). 5) Finally, a hierarchical planner selects and executes actions by planning over the scored graph and generating locally safe motions toward intermediate goals (Sec. 4-F ).
Figure 3: Navigation Graph Construction (a) Node Sampling in free cells of Ttgeo , and assigning rif (free radius) . Invalid samples are shown in red. (b) rie (explored radius) for each node. (c) Identifying frontier cells and assigning them to nodes to get Ftgeo . (d) redge for the current node. (e) Dense graph obtained after first iteration of graph construction. (f) Robot pose xt update and shift of Ttgeo . (g) Node Sampling and update rif,rie (h) Detect new frontier cells and remove invalid ones. Invalid ones shown in red - present in known regions, present within explored radius of another node. (i) Assign frontier cells to nodes, and update frontier nodes.
Algorithm 1 Update Navigation Graph
Algorithm 2 Update Nodes
Algorithm 3 Sample New Nodes
Algorithm 4 Detect and Update Frontier Nodes
Algorithm 5 Build Edges
Figure 4: ExploRFM Architecture. ExploRFM builds upon the RADIO vision foundation model to jointly reason about traversability, semantic frontiers, and goal-object localization from a single RGB frame and language query.
Figure 5: Coarse goal query localization . Ray-distance weighting based triangulation. Particles with darker shades get a higher weight.
Algorithm 6 ProjectParticles (V,Np,dmin,dmax)
Algorithm 7 RayWeightedTriangulation (P,V)
Figure 6: Scoring the navigation graph using images. (a) Heuristic Formulation for Distance-to-goal, and heading similarity for approximation of visual frontier to goal cost ( Cgoal ) (b) In-image grid search in the traversable region for approximation of geometric frontier to visual frontier cost ( Ctrans )
Algorithm 8 Scored Frontier Graph Construction
Figure 7: Planning using the Scored Graph. (a) Visualizing edges to the auxiliary goal node. (b) Computing cost of edges from frontier nodes to the goal node using shortest collision-free path search in a coarse grid. A darker color implies a higher traversal cost. (c) Final high-level path after Dijkstra’s algorithm on the graph.
Algorithm 9 High-Level Planning on the Scored Navigation Graph
Figure 8: System Overview . Frontiers and Traversability are thresholded and shown in Jet and Inverse Jet colormaps respectively. Edges of the navigation graph are shown in red, free nodes in green, and frontier nodes in blue. Frontier nodes are surrounded with a score ring in the Scored Navigation Graph, indicating the score ( color according to the Jet colormap) in the goal direction. The Triangulated Goal is shown as a cyan sphere, along with projected particles in white. The planned high-level path is shown in green. A similar color scheme is followed in subsequent figures unless stated otherwise.
Parameter
Description
Symbol
Value
A. Geometric Mapping & Navigation Graph
Local map radius
Radius of reliable geometric sensing
rmax
10 m
Local map resolution
Resolution of Ttgeo
—
0.1 m
Max free radius
Cap on node free-space radius
rmaxf
4.0 m
Node sampling count
Node samples per update step
Nsamples
1000
Traversability clearance
Min obstacle clearance for node
rtrav
0.5 m
Table 1: WildOS System Parameters
Figure 9: Examples of Visual Traversability and Frontiers from ExploRFM in varied terrains. Visual Frontiers are thresholded and shown in the Jet Colormap (Red indicating higher frontier confidence). Visual Traversability is also thresholded and shown in the inverse Jet Colormap (Blue indicating higher traversability confidence). (a),(b): Offoad environments with a dirt-track (c): Narrow trail amidst rocks, bushes and undergrowth (d),(e): Urban environments with paved terrains (f): Vast and open urban terrain
Figure 10: Overview of the Object Search Experiment for the query - “NASA Logo”. (a) Third-Person view of the robot approaching the NASA sign. (b) Expected trajectory of the robot overlaid on a satellite image showing the environment. (c) Scored Navigation Graph after a successful run. The rings around the frontier nodes denote the scores for each bin according to the Jet Colormap - higher scores being red, and the lower ones being blue. (d) Explored regions of the graph - visualizing the explored radius of each node in blue.
Figure 11: ExploRFM Outputs and Triangulation of “NASA Logo”. (a) Object Similarity map from ExploRFM (b) Initial triangulation of the NASA sign (c) Visual Frontiers, Traversability and Projected frontier node scores for all three cameras. Frontier and Traversability Confidence is shown using the Jet Colormap. (d) Final Triangulation of the logo (e) RViZ markers projected in the front camera
Figure 13: Evaluating Vision-Based Scoring for Navigation Efficiency. (a) GPS trajectories for all methods. (b) Third-person view at the start of the experiment with future paths overlaid. Vision-based approaches detect the corridor opening early on and move towards it, whereas Vanilla GraphNav heads straight towards the goal until it reaches the fence and goes around. (c) Quantitative comparison across three runs (mean ± 95% CI). (d), (e) Visual frontier and the confidence map at the start of the experiment F0vis , highlighting the opening between the fences. (f) Tri-fork after exiting the corridor, where LRN [ 2 ] hesitates between competing frontiers. (g) Scored Navigation Graph after a successful WildOS run.
Figure 14: Potential LRN Failure Cases. We visualize the visual frontier and traversability heatmaps (unthresholded) overlaid on top of the RGB image - both in the Jet Colormap i.e. red indicating a higher probability score. Notice how relying purely on frontier heatmaps (LRN [ 2 ] ) can lead to high affordance scores for non-reachable frontiers. WildOS tackles this by incorporating visual traversability in frontier scoring.
Figure 15: Evaluating the Role of the Navigation Graph in Handling dead-ends. (a) Final scored navigation graph visualized after a successful WildOS run. (b) GPS trajectory for WildOS showing successful rerouting around the fence. (c) GPS trajectory for LRN, where human intervention was required. (d) Third-person view showing the blocked trail. (e) Visual frontiers at the fork - WildOS initially takes the opening on the left given it’s better alignment with the goal. After encountering a dead-end it turns back and takes the opening on the right. LRN does not store that it has already explored the opening on the left and keeps choosing the opening on the left at this fork and turning back due to a dead-end leading to an oscillatory behaviour.
Figure 16: WildOS in Urban Environments. Third-person views of the robot at the start of the run, GPS trajectories, and scored navigation graphs at the end of the runs for two scenarios: (a) Goal located within a cluster of buildings. (b) Goal located behind a building. WildOS plans efficient routes around obstacles and navigates to the goal successfully.
Open-world navigation requires robots to make decisions in complex everyday environments while adapting to flexible task requirements. Conventional navigation approaches often rely on dense 3D reconstruction and hand-crafted goal metrics, which limits their generalization across tasks and environments. Recent advances in vision-language navigation (VLN) and vision-language-action (VLA) models enable end-to-end policies conditioned on natural language, but typically require interactive training, large-scale data collection, or task-specific fine-tuning with a mobile agent. We formulate navigation as a sparse subgoal identification and reaching problem and observe that providing visual anchoring targets for high-level semantic priors enables highly efficient goal-conditioned navigation. Based on this insight, we select visual frontiers as semantic anchors and propose OpenFrontier, a navigation framework that requires no task-specific training or fine-tuning and seamlessly integrates diverse vision-language prior models. OpenFrontier enables efficient navigation with a lightweight system design, without dense 3D semantic mapping, task-specific policy training, or model fine-tuning. We evaluate OpenFrontier across multiple navigation benchmarks and demonstrate strong zero-shot performance, as well as effective real-world deployment on a mobile robot.
Esteban Padilla-Cerdio, Boyang Sun, Marc Pollefeys +1
1ETH Zurich · 2Microsoft Spatial AI Lab · University of Bonn
Safe autonomous navigation requires not only geometric obstacle avoidance but also semantic awareness. Different object categories demand different safety clearances, and the set of relevant categories may expand as the robot explores novel environments. Vision-language models (VLMs) offer rich semantic representations for building maps of unknown scenes, yet their confidence scores are poorly calibrated, making raw predictions unreliable for safety-critical planning. Conformal prediction (CP) can provide coverage guarantees, but existing methods assume a fixed label set and stationary data, limiting their use in open-world settings. We propose Conformal Semantic Mapping for Open-World Safe Robot Navigation (CoSMO), a semantic safety mapping framework that transforms VLM embeddings into calibrated coverage sets and safety-aware obstacle dilations over a dynamically expanding hazard vocabulary, even under distribution shift. Experiments on the Matterport3D (MP3D) dataset show that CoSMO reduces semantic collisions by 61.6% relative to a geometry-only baseline, while maintaining formal coverage guarantees even under delayed expert feedback. Real-world experiments on a quadruped robot further demonstrate generalization by achieving long-run empirical coverage of 90.73% against the 90% target. Project website: https://yangming911.github.io/cosmo-page/.
Shuqi Wang, Yiwei Zheng, Pingrui Zhang +2
Shanghai Jiao Tong University · Shanghai Innovation Institute · Fudan University
Vision-language navigation requires embodied agents to navigate environments using natural language instructions and visual observations. Existing approaches typically decompose navigation into sequential language-guided decisions or rely on online exploration without prior environmental knowledge. Scene graph representations offer compact semantic memory but remain decoupled from downstream navigation, which still depends on dense metric maps. To close this gap, we propose an object--path graph that unifies open-vocabulary semantic reasoning with topological navigation. The proposed representation jointly supports semantic grounding, graph-based localization, and navigation within a single lightweight topological framework. Building on this graph, we introduce a navigation strategy that combines global path planning with local inter-node execution through lightweight node localization and semantic visual servoing, enabling navigation directly over the graph without dense metric reconstruction. Experiments on HM3D and Replica demonstrate competitive performance in open-vocabulary object grounding through the proposed hierarchical graph structure, while achieving effective navigation performance. Real-world robot experiments further validate the practicality of the proposed framework.
Linwei Zheng, Daojie Peng, Bingtao Wang +2
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology