Autonomous navigation in diverse real-world environments remains one of the greatest challenges in robotics. It demands both robust low-level motion control to handle diverse terrain and high-level reasoning to interpret environmental cues. We present ROAM (Robust Open-world Autonomous Mobility), a system that enables legged robots to perform kilometer-scale navigation in complex alpine and urban settings. ROAM combines the world knowledge of a Vision-Language Model (VLM), pre-trained on internet-scale data, with robust locomotion and local planning policies learned in simulation via reinforcement learning (RL). The VLM provides common-sense reasoning and semantic interpretation of the scene, assessing risks, interpreting signs, and producing semantic labels with associated navigation costs. An open-vocabulary segmentation module converts these labels into a dense, real-time risk map, which conditions two pre-trained RL policies responsible for agile locomotion and local navigation, effectively coupling high-level VLM reasoning with low-level physical control. Real-world experiments showed the system's ability to autonomously navigate a wide range of complex real-world environments. The robot completes kilometer-long hikes with minimal intervention, interprets signs to avoid danger, and evaluates its environment to identify safe areas on which to step. This work represents a step toward the reliable deployment of systems in unstructured environments. However, it also underscores the challenges that remain in VLM reliability and spatial reasoning, indicating that foundational models are a crucial component but not yet a complete solution for navigation in the wild.
Figures & tables
Figure 1: Overview of the ROAM pipeline. The proposed system bridges high-level reasoning from vision-language models (VLMs) with low-level actuator commands for autonomous robot control. It processes RGB images and robot descriptions through a VLM to generate object labels enriched with risk, preferred area, and steppable values. These labels are converted into semantic images using open-vocabulary semantic segmentation, which operates at a higher frequency than the VLM analysis. The semantic images are then fused with elevation maps to create an environmental representation that reinforcement learning (RL) policies utilize for fast and robust robot control.
Figure 2: Deployment of ROAM across diverse environments. These include natural outdoor settings such as mountain trails (A to E), forests (F), and grassy areas (G), as well as urban environments such as university campuses (H) and roads (I). The system was also tested in indoor settings (J to L). Additionally, we evaluated its performance in the presence of various challenges, such as smoke (M), train tracks (N), and signage (O).
Figure 3: Long-distance hiking experiments in the Swiss Alps. (A) Zermatt experiment. This experiment covered a total distance of 10.1km with an elevation change of 1220m . The route encompassed diverse terrains, including gravel paths, rocky sections, grassy areas, and segments near a lake and train tracks. The steepest segment of the hike had a gradient of approximately 30%, posing a substantial challenge for locomotion. (B) Seealpsee experiment. This experiment covered a total distance of 2.38km with an elevation change of 20m , following a path around the lake. The route included a narrow trail close to the water, encounters with cows, gravel sections, and a forest area near the goal. In (A) and (B), photographs (i) to (vi) show the robot at the correspondingly labeled locations on the map.
Figure 4: Comparison of ROAM with baselines on a course with semantic and contextual challenges. The experiment covered a 1.2km path with challenges such as an inaccurate global path deviating into bushes or grass, a mannequin lying on the path as an obstacle, and a construction zone with warning signs. (A) Robust pathfinding. The global path was offset into bushes [(i) and (ii)] and into grass [(iii) and (iv)]; (i) and (iii) show ROAM, and (ii) and (iv) show the geometry-only baseline. (B) Semantic hazard avoidance with a mannequin lying on the path for ROAM, the geometry-only baseline, and ViNT. (C) Contextual awareness with construction signs. The map shows the paths of the three methods and the global path. The remaining panels show the robot views, the risk and preferred-area images, and the VLM responses before and after ROAM read the “ROAD WORK AHEAD” sign (r1) and the “END ROAD WORK” sign (r2), followed by snapshots of the geometry-only baseline (g1 to g3) and ViNT (v1 to v3). Information icons mark the ViNT interventions.
Figure 5: ROAM brings world knowledge to locomotion, local navigation, and exploration. (A) ROAM reads signs and uses this information in locomotion: (i) metal cans with a “WARNING: EXPLOSIVE CANS” sign and (ii) wooden pallets on grass without a sign, with a sign on the pallet, and with signs on the grass. The table in (i) lists the VLM labels and their values, the boxes in (ii) show the VLM responses, and the map in (ii) shows the wooden pallet (orange outline) and the foot trajectories for the three sign conditions. (B) Comparison of ROAM with a geometry-only baseline on (i) a campus route, (ii) a route next to a railroad, and (iii) a snow-covered path. In (i), images 1 to 4 correspond to the numbered points along the paths taken by the two methods, highlighting the different terrains encountered, such as paved areas, gravel, and grass. (C) Exploration behavior in the presence of an obstacle blocking the path: (i) the blocked path, (ii) the alternative route along the path, (iii) the shortcut through the grass, and (iv) the return to the global path. The robot explored along the path after encountering the obstacle and took a shortcut through the grass to merge back to the global path.
Figure 6: Comparison of VLM output abstractions for navigation. (A) Comparison of VLM-based navigation methods by output type. ROAM outputs semantic labels and values, avoiding the need for spatial understanding. Methods outputting directions or coordinates require spatial reasoning; those generating direct commands (such as velocity) additionally require embodiment understanding. (B) Example trajectories from different methods, including three “select label” variants. ROAM information can augment label placement on ground regions. (C) Quantitative evaluation of the safety and preferred ratios, defined as the fraction of all trajectory pixels, pooled over the scenes (N = 27 scenes; see Statistical analysis), that lie within the hand-labeled “safe” and “preferred” regions. Error bars, 95% bootstrap confidence intervals.
Method
Category
Safety
P
Preferred
P
Goal reached
ROAM A*
ROAM A*
0.991
–
0.868
–
27
Straight line
Baseline
0.900
0.037
0.550
0.001
27
Trajectory label with ROAM
Label with ROAM-risk
0.880
0.037
0.519
0.006
3
Line label with ROAM
Label with ROAM-risk
0.840
0.055
0.668
0.037
5
Uniform label
Label w/o ROAM-risk
0.740
< 0.001
0.418
< 0.001
4
Pixel coordinates from prompt
Pixel coordinates
0.714
0.001
0.398
< 0.001
5
Table 1: Safety and preferred ratios of VLM-based path generation methods. Values are fractions of trajectory pixels pooled across N = 27 hand-labeled scenes (see Statistical analysis). Goal reached denotes scenes ending within 20% of the start-to-goal distance from the goal; P , Holm-corrected two-sided Wilcoxon signed-rank test against ROAM A*.
Figure S1: VLM labels during the Zermatt hike. The labels and their risk values throughout the mission. It shows what label the VLM outputs in different scenarios as a colored bar, where the color represents the risk value and the bar represents the existence of the label.
Figure S2: VLM labels during the Seealpsee hike. The labels and their risk values throughout the mission. It shows what label the VLM outputs in different scenarios, shown as a colored bar, where the color represents the risk value and the bar represents the presence of the label.
Figure S3: VLM labels during navigation from indoor to outdoor environments. The evolution of labels and their associated risk values as the robot transitions from an indoor setting to an outdoor environment, traversing urban areas and open fields. The category distribution plot shows the fraction of indoor-, urban-, and nature-related labels along the route.
Figure S4: Mannequin avoidance. Comparison of mannequin avoidance between our method, a geometry-based baseline, and ViNT. We placed a mannequin between the start and goal positions, ran 10 trials per condition (N = 10), and recorded both goal-reaching and collisions (classified as either a “hard” or a “slight” hit). The condition “ours with random postures” shows our method with the mannequin placed in random postures; collisions for the geometry baseline were not subdivided into hard and slight hits. Rates are the fraction of the 10 trials; the number of trials with each outcome is listed in Data S1.
Figure S5: Full VLM prompt. The full prompt for the VLM. We provide this prompt plus the front and rear camera images to the VLM model.
Figure S6: Effect of memory in the prompt. Comparison of the risk with memory and without memory module in the prompt for (A) the construction sign experiment and (B and C) the no-step sign experiment.
Figure S7: Interventions during the hiking experiments. Near the lake, the operator intervened once as a precautionary measure. Another intervention was required due to a partially obscured path that impeded segmentation. Two further interventions occurred in the construction area, where fences or paths were misclassified, and the global path continued to point to the old route, causing a deviation of 10m20m . Finally, an impassable gate necessitated carrying the robot over by hand.
Figure S8: Qualitative comparison of the fused and raw GNSS trajectories. The red path is the raw measured GNSS position, whereas green depicts the fused trajectory at 400Hz obtained online during the operation (not replayed).
Table S1: Localization accuracy of the fused and single-sensor estimates. Comparison of methods in terms of APE , ARE , RPE , and RRE . RPE and RRE are computed over all snippet pairs of 20m length.
Model
TartanGround [ 85 ]
Sign
IoU
Prec.
Rec.
IoU
Prec.
Rec.
gemini-2.0-flash
0.851
0.901
0.938
0.912
0.944
0.964
gemini-2.0-flash-lite
0.839
0.882
0.945
0.799
0.934
0.846
gpt-4o
0.853
0.895
0.947
0.790
0.863
0.904
gemma-3-27b
0.836
0.887
0.936
0.686
0.795
0.833
gemma-3-4b
0.837
0.886
0.939
0.521
0.697
0.674
Table S2: Performance of different VLMs on the TartanGround and Sign benchmarks. IoU, precision, and recall are micro-averaged over all pixels of the 372 TartanGround images and the 12 sign images.
Figure S11: Comparison with a fixed-label pipeline. This comparison evaluates our dynamic, context-dependent scene-reasoning VLM framework against a Fixed Labels pipeline across evaluation datasets with a variety of indoor and outdoor scenes. Each row shows the original image, the risk maps of our method and the Fixed Labels template, and the labels and risk values estimated by our method.
Figure S12: Prompt ablation examples. This prompt ablation study evaluates the visual and quantitative effect of omitting specific prompt constraints from the VLM prompt: the directional ban, the terrain guidelines, or all guidelines (see Supplementary Methods). The heading of each table names the ablation shown in that row.
Configuration
TartanGround [ 85 ]
Sign
IoU
Prec.
Rec.
IoU
Prec.
Rec.
Baseline (Full Prompts)
0.827
0.840
0.985
0.904
0.924
0.977
Perturbation (Synonyms)
0.842
0.853
0.987
0.867
0.925
0.936
No Robot Description
0.819
0.829
0.988
0.878
0.906
0.952
No Directional Ban
0.808
0.818
0.988
0.803
0.808
0.994
No Terrain Guidelines
0.823
0.836
0.985
0.802
0.868
0.927
Table S3: Prompt ablation and perturbation study. Macro-averaged scores across the TartanGround (N = 372 images) and Sign (N = 12 images) benchmarks.
Figure S13: Effect of the explicit instruction text within the sign images. We removed the highly explicit text instructions (such as “do not step here” and “danger mine”) from the sign regions (creating aligned non-explicit counterparts) and evaluated if the semantic risk prediction changed.
Configuration
Explicit (SignDataset Baseline)
Non-Explicit (signs_non_explicit)
IoU
Prec.
Rec.
IoU
Prec.
Rec.
Baseline (Full Prompts)
0.904
0.924
0.977
0.841
0.929
0.904
Table S4: Performance comparison on explicit and nonexplicit sign datasets. Macro-averaged scores across N = 12 paired sign scenes.
Figure S14: Qualitative examples of VLM risk maps under explicit height constraints. (A to D) Four staircases with measured riser heights of 14.5, 15.5, 16.0, and 18.0 cm, each evaluated by Gemini 3.5 under three prompts: the original ROAM prompt (no height limit) and prompts with explicit 10 cm and 15 cm climbability limits. Blue indicates low assigned risk and red high risk. The text below each risk map states whether the staircase was allowed or blocked and whether this decision was correct.
Figure S15: Quantitative analysis of VLM stair-height reasoning. (A and B) Risk assigned to staircases as a function of measured riser height under an explicit 10 cm (A) and 15 cm (B) safety limit, for three model generations (N = 27 images per model). (C) Directly estimated versus measured riser height without safety constraints. (D) Qualitative estimates from Gemini 3.5.
Table S5: Fixed Labels lookup table used as a baseline in the prompt ablation. Risk, steppable, and preferred-area values assigned to each of the 55 predefined labels.
Label
Risk
Steppable
Pref. Area
Walkable ground
ground
0.0
1.0
1.0
floor
0.0
1.0
1.0
Off-road paths / secondary ground
trail
0.15
1.0
0.7
Walkable structured inclines / caution areas
Table S6: Fixed Labels (Simple) lookup table used as a baseline in the prompt ablation. Risk, steppable, and preferred-area values assigned to each of the 9 predefined labels.
Spatial AI & Robotics Lab, University at Buffalo, USA. · 2Robotics Institute, Carnegie Mellon University, USA. · 1Spatial AI & Robotics Lab, University at Buffalo, USA.