cs.RONov 18, 2026

Bridging vision-language reasoning and learned control for legged navigation in the wild

Authors: Takahiro Miki, Robin Schmid, Manthan Patel, Turcan Tuna, Julia Richter, Julian Nubert, Joonho Lee, Matthias Müller, 

Organizations: ETH Zurich, Zurich, Switzerland · Apple, Cupertino, CA, USA

Abstract

Autonomous navigation in diverse real-world environments remains one of the greatest challenges in robotics. It demands both robust low-level motion control to handle diverse terrain and high-level reasoning to interpret environmental cues. We present ROAM (Robust Open-world Autonomous Mobility), a system that enables legged robots to perform kilometer-scale navigation in complex alpine and urban settings. ROAM combines the world knowledge of a Vision-Language Model (VLM), pre-trained on internet-scale data, with robust locomotion and local planning policies learned in simulation via reinforcement learning (RL). The VLM provides common-sense reasoning and semantic interpretation of the scene, assessing risks, interpreting signs, and producing semantic labels with associated navigation costs. An open-vocabulary segmentation module converts these labels into a dense, real-time risk map, which conditions two pre-trained RL policies responsible for agile locomotion and local navigation, effectively coupling high-level VLM reasoning with low-level physical control. Real-world experiments showed the system's ability to autonomously navigate a wide range of complex real-world environments. The robot completes kilometer-long hikes with minimal intervention, interprets signs to avoid danger, and evaluates its environment to identify safe areas on which to step. This work represents a step toward the reliable deployment of systems in unstructured environments. However, it also underscores the challenges that remain in VLM reliability and spatial reasoning, indicating that foundational models are a crucial component but not yet a complete solution for navigation in the wild.

Figures & tables

Explore similar work

CardsList
  1. VL-Nav: Neuro-Symbolic Reasoning-based Vision-Language Navigation

    Feb 2, 2025Yi Du, Taimeng Fu, Zhipeng Zhao +6Robot NavigationVision-Language Navigation

  2. "Dear LLaVA, Please Drive": A Depth-Aware Vision-Language Agent for Closed-Loop Robotic Control

    Sep 19, 2026Sebastian Berger, Katharina Winter, Fabian B. FlohrVision-Language NavigationVision-Based Robot Control

  3. A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigation

    Jul 9, 2026Liuyi Wang, Kai Sheng, Zongtao He +8Embodied NavigationMobile Robotics