cs.ROFeb 22, 2026
SaveWildOS: Open-Vocabulary Object Search in the Wild
Organizations: Jet Propulsion Laboratory (JPL), California Institute of Technology (Caltech), Pasadena, CA 91011, USA · Swiss Federal Institute of Technology (ETH Zürich), Robotic Systems Lab, Zürich 8092, Switzerland · FieldAI Inc. 3 Morgan, Irvine, CA, USA. Work Performed at NASA JPL · Stanford University, Autonomous Systems Lab, Stanford, CA 94305, USA ·
Abstract
Autonomous navigation in complex, unstructured outdoor environments requires robots to operate over long ranges without prior maps and limited depth sensing. In such settings, relying solely on geometric frontiers for exploration is often insufficient. In such settings, the ability to reason semantically about where to go and what is safe to traverse is crucial for robust, efficient exploration. This work presents WildOS, a unified system for long-range, open-vocabulary object search that combines safe geometric exploration with semantic visual reasoning. WildOS builds a sparse navigation graph to maintain spatial memory, while utilizing a foundation-model-based vision module, ExploRFM, to score frontier nodes of the graph. ExploRFM simultaneously predicts traversability, visual frontiers, and object similarity in image space, enabling real-time, onboard semantic navigation tasks. The resulting vision-scored graph enables the robot to explore semantically meaningful directions while ensuring geometric safety. Furthermore, we introduce a particle-filter-based method for coarse localization of the open-vocabulary target query, that estimates candidate goal positions beyond the robot's immediate depth horizon, enabling effective planning toward distant goals. Extensive closed-loop field experiments across diverse off-road and urban terrains demonstrate that WildOS enables robust navigation, significantly outperforming purely geometric and purely vision-based baselines in both efficiency and autonomy. Our results highlight the potential of vision foundation models to drive open-world robotic behaviors that are both semantically informed and geometrically grounded. Project Page: https://leggedrobotics.github.io/wildos/
Figures & tables
Figure 1: (a) WildOS enables autonomous semantic navigation in diverse unstructured outdoor environments. (b, c) Due to the limited range of geometric sensing, robots can only reliably perceive nearby regions within a depth horizon (blue), leading to myopic exploration and (e) difficulty localizing distant targets (e.g., a “house”) beyond sensing range. (b, c) Conventional exploration (dashed path) relies on geometric frontiers (blue dots) at the boundary between known and unknown space, which ignores long-range semantic and traversability cues. WildOS (green path) augments geometric exploration with long-range visual reasoning using a vision foundation model, defining a visual horizon (red) that extends beyond depth sensing and predicts visual traversability, visual frontiers (red dots), and open-vocabulary object similarity in image space. (d) During deployment, a sparse navigation graph is built from geometry and frontier nodes are scored using vision, while a particle-filter-based goal localization module (yellow particles) estimates candidate goal locations beyond the depth horizon, enabling safe, efficient planning toward distant semantic goals.
Figure 2: Method Overview consisting of five main components: 1) WildOS incrementally builds a sparse navigation graph from geometric sensing to maintain persistent spatial memory and identify geometric frontier nodes for safe exploration (Sec. 4-B ). 2) To reason beyond the limited depth horizon, a learned vision-language module, ExploRFM, processes the current image and text query to predict visual traversability, visual frontiers, and open-vocabulary object similarity over a long-range visual horizon (Sec. 4-C ). 3) Object detections from multiple viewpoints are fused by a probabilistic goal triangulation module to estimate a coarse 3D target location beyond direct sensor range (Sec. 4-D ). 4) Geometric frontier nodes are then projected into the image and scored using the visual-semantic cues and the current goal estimate, producing a semantically scored navigation graph (Sec. 4-E ). 5) Finally, a hierarchical planner selects and executes actions by planning over the scored graph and generating locally safe motions toward intermediate goals (Sec. 4-F ).
Figure 3: Navigation Graph Construction (a) Node Sampling in free cells of , and assigning . Invalid samples are shown in red. (b) for each node. (c) Identifying frontier cells and assigning them to nodes to get . (d) for the current node. (e) Dense graph obtained after first iteration of graph construction. (f) Robot pose update and shift of . (g) Node Sampling and update (h) Detect new frontier cells and remove invalid ones. Invalid ones shown in red - present in known regions, present within explored radius of another node. (i) Assign frontier cells to nodes, and update frontier nodes.
Algorithm 1 Update Navigation Graph
Algorithm 2 Update Nodes
Algorithm 3 Sample New Nodes
Algorithm 4 Detect and Update Frontier Nodes
Algorithm 5 Build Edges
Figure 4: ExploRFM Architecture. ExploRFM builds upon the RADIO vision foundation model to jointly reason about traversability, semantic frontiers, and goal-object localization from a single RGB frame and language query.
Figure 5: Coarse goal query localization . Ray-distance weighting based triangulation. Particles with darker shades get a higher weight.
Algorithm 6 ProjectParticles
Algorithm 7 RayWeightedTriangulation
Figure 6: Scoring the navigation graph using images. (a) Heuristic Formulation for Distance-to-goal, and heading similarity for approximation of visual frontier to goal cost ( ) (b) In-image grid search in the traversable region for approximation of geometric frontier to visual frontier cost ( )
Algorithm 8 Scored Frontier Graph Construction
Figure 7: Planning using the Scored Graph. (a) Visualizing edges to the auxiliary goal node. (b) Computing cost of edges from frontier nodes to the goal node using shortest collision-free path search in a coarse grid. A darker color implies a higher traversal cost. (c) Final high-level path after Dijkstra’s algorithm on the graph.
Algorithm 9 High-Level Planning on the Scored Navigation Graph
Figure 8: System Overview . Frontiers and Traversability are thresholded and shown in Jet and Inverse Jet colormaps respectively. Edges of the navigation graph are shown in red, free nodes in green, and frontier nodes in blue. Frontier nodes are surrounded with a score ring in the Scored Navigation Graph, indicating the score ( color according to the Jet colormap) in the goal direction. The Triangulated Goal is shown as a cyan sphere, along with projected particles in white. The planned high-level path is shown in green. A similar color scheme is followed in subsequent figures unless stated otherwise.
| Parameter | Description | Symbol | Value |
| A. Geometric Mapping & Navigation Graph | |||
| Local map radius | Radius of reliable geometric sensing | 10 m | |
| Local map resolution | Resolution of | — | 0.1 m |
| Max free radius | Cap on node free-space radius | 4.0 m | |
| Node sampling count | Node samples per update step | 1000 | |
| Traversability clearance | Min obstacle clearance for node | 0.5 m | |
Table 1: WildOS System Parameters
Figure 9: Examples of Visual Traversability and Frontiers from ExploRFM in varied terrains. Visual Frontiers are thresholded and shown in the Jet Colormap (Red indicating higher frontier confidence). Visual Traversability is also thresholded and shown in the inverse Jet Colormap (Blue indicating higher traversability confidence). (a),(b): Offoad environments with a dirt-track (c): Narrow trail amidst rocks, bushes and undergrowth (d),(e): Urban environments with paved terrains (f): Vast and open urban terrain
Figure 10: Overview of the Object Search Experiment for the query - “NASA Logo”. (a) Third-Person view of the robot approaching the NASA sign. (b) Expected trajectory of the robot overlaid on a satellite image showing the environment. (c) Scored Navigation Graph after a successful run. The rings around the frontier nodes denote the scores for each bin according to the Jet Colormap - higher scores being red, and the lower ones being blue. (d) Explored regions of the graph - visualizing the explored radius of each node in blue.
Figure 11: ExploRFM Outputs and Triangulation of “NASA Logo”. (a) Object Similarity map from ExploRFM (b) Initial triangulation of the NASA sign (c) Visual Frontiers, Traversability and Projected frontier node scores for all three cameras. Frontier and Traversability Confidence is shown using the Jet Colormap. (d) Final Triangulation of the logo (e) RViZ markers projected in the front camera
Figure 12: More Object-Search Experiments (a)”Orange Flag” (b)”Garbage Container” (c)”Golf Cart”
Figure 13: Evaluating Vision-Based Scoring for Navigation Efficiency. (a) GPS trajectories for all methods. (b) Third-person view at the start of the experiment with future paths overlaid. Vision-based approaches detect the corridor opening early on and move towards it, whereas Vanilla GraphNav heads straight towards the goal until it reaches the fence and goes around. (c) Quantitative comparison across three runs (mean 95% CI). (d), (e) Visual frontier and the confidence map at the start of the experiment , highlighting the opening between the fences. (f) Tri-fork after exiting the corridor, where LRN [ 2 ] hesitates between competing frontiers. (g) Scored Navigation Graph after a successful WildOS run.
Figure 14: Potential LRN Failure Cases. We visualize the visual frontier and traversability heatmaps (unthresholded) overlaid on top of the RGB image - both in the Jet Colormap i.e. red indicating a higher probability score. Notice how relying purely on frontier heatmaps (LRN [ 2 ] ) can lead to high affordance scores for non-reachable frontiers. WildOS tackles this by incorporating visual traversability in frontier scoring.
Figure 15: Evaluating the Role of the Navigation Graph in Handling dead-ends. (a) Final scored navigation graph visualized after a successful WildOS run. (b) GPS trajectory for WildOS showing successful rerouting around the fence. (c) GPS trajectory for LRN, where human intervention was required. (d) Third-person view showing the blocked trail. (e) Visual frontiers at the fork - WildOS initially takes the opening on the left given it’s better alignment with the goal. After encountering a dead-end it turns back and takes the opening on the right. LRN does not store that it has already explored the opening on the left and keeps choosing the opening on the left at this fork and turning back due to a dead-end leading to an oscillatory behaviour.
Figure 16: WildOS in Urban Environments. Third-person views of the robot at the start of the run, GPS trajectories, and scored navigation graphs at the end of the runs for two scenarios: (a) Goal located within a cluster of buildings. (b) Goal located behind a building. WildOS plans efficient routes around obstacles and navigates to the goal successfully.
| Method | Vision-based Reasoning | Graph Memory / History |
|---|---|---|
| LRN [ 2 ] | ✓ | ✗ |
| Vanilla GraphNav | ✗ | ✓ |
| WildOS (Ours) | ✓ | ✓ |
Table 2: Comparison of Navigation Capabilities.