Aerial imagery provides essential global context for autonomous navigation, enabling route planning at scales inaccessible to onboard sensing. We address the problem of generating global costmaps for long-range planning directly from satellite imagery when entities and mission-specific traversal rules are expressed in natural language at test time. This setting is challenging since mission requirements vary, terrain entities may be unknown at deployment, and user prompts often encode compositional traversal logic. Existing approaches relying on fixed ontologies and static cost mappings cannot accommodate such flexibility. While foundation models excel at language interpretation and open-vocabulary perception, no single model can simultaneously parse nuanced mission directives, locate arbitrary entities in large-scale imagery, and synthesize them into an executable cost function for planners. We therefore propose OVerSeeC, a zero-shot modular framework that decomposes the problem into Interpret-Locate-Synthesize: (i) an LLM extracts entities and ranked preferences, (ii) an open-vocabulary segmentation pipeline identifies these entities from high-resolution imagery, and (iii) the LLM uses the user's natural language preferences and masks to synthesize executable costmap code. Empirically, OVerSeeC handles novel entities, respects ranked and compositional preferences, and produces routes consistent with human-drawn trajectories across diverse regions, demonstrating robustness to distribution shifts. This shows that modular composition of foundation models enables open-vocabulary, preference-aligned costmap generation for scalable, mission-adaptive global planning.
Figures & tables
Fig. 1: Overview of OVerSeeC , which uses a satellite image I and a natural language prompt P to generate a preference-aligned costmap C for global planning. The Entity Identifier and Costmap Function Compositor (Sec. IV-A , IV-C ) use an LLM to extract relevant terrain classes C and synthesize a cost function fLLM(⋅) respectively. The Open-Vocabulary Mask Generator (Sec. IV-B1 , IV-B2 ) performs zero-shot semantic segmentation over I , yielding class masks {Mc} and thresholded probability maps {Pcτ} , where c∈C . Finally fLLM(⋅) is executed to generate the final costmap C .
Fig. 2: Open-Vocabulary Mask Generator (Sec. IV-B ). Given a satellite image I and extracted classes C , the pipeline comprises two submodules: (i) Open-Vocabulary Semantic Segmentation (Sec. IV-B1 ), which produces per-class probability maps Pc and coarse masks Mc for open-ontology classes; and (ii) Mask Refinement (Sec. IV-B2 ), which refines them into fine probabilities Pc and masks Mc .
Fig. 3: Planning results for the D2-OOD-OV scenario : Comparison of costmap alignment using RRPI (Sec. VI-B1 ) metric under the user preference: “Prefer the roads and trails, grass should be fine, try to avoid the baseball field as much as possible.” The class ranking used are: road: 1, trail: 1, grass: 2, baseball field : 3, tree: 4, building: 5 . The top row shows RRPI vs. path length scatter plots with KDE contours; the colored pointers in these plots indicates the COM of the KDE, and the solid line represents a linear regression fit. A lower slope for this line is preferable, as it indicates that the RRPI score remains low even as path length increases. The bottom row shows a subset of these trajectories generated from Dijkstra’s algorithm overlaid on the map (start: arrow, goal: star).
Map Name
Objective
Dataset: D2
D2-ID1D2-ID2
ID setting with familiar regions and fixed ontology from baseline training.
D2-OOD
OOD region with fixed ontology.
D2-OOD-OV
OOD region with prompt mentioning ‘baseball field’ requiring OV generalization and understanding its relation to ‘grass’.
Dataset: D3
D3-HE1
Recognizing novel class ‘electric tower’.
TABLE I: Evaluation settings across environment types. ID : In-distribution (same domain as supervised training D1 ). OOD : Out-of-distribution. OV : Open-vocabulary.
Fig. 4: RQ1 — OVerSeeC ’s alignment to geometric preferences. (a) is the satellite image, (b) costmap for the prompt “stay on the center of the road,” (c) costmap for the prompt “stay on the side of the road,” assigning low cost to the road edges. In both cases, the cost bar ranges from 0 (lowest cost) to 1 (highest).
Figure 6Figure 7
Fig. 7: OVerSeeC GUI : A modular interface that enables rapid, zero-shot iteration. (a) Users upload a satellite image or extract imagery via the map tool. (b) OVerSeeC parameters can be adjusted. (c) Natural language prompts are processed by the LLM to generate classes of interest and a costmap function. (d) The open-vocabulary mask generator produces class-specific masks. (e) The finalized costmap is generated and A* is used to plan over it.
Navigating unstructured environments requires assessing traversal risk relative to a robot's physical capabilities, a challenge that varies across embodiments. We present CATNAV, a cost-aware traversability navigation framework that leverages multimodal LLMs for zero-shot, embodiment-aware costmap generation without task-specific training. We introduce a visuosemantic caching mechanism that detects scene novelty and reuses prior risk assessments for semantically similar frames, reducing online VLM queries by 85.7%. Furthermore, we introduce a VLM-based trajectory selection module that evaluates proposals through visual reasoning to choose the safest path given behavioral constraints. We evaluate CATNAV on a quadruped robot across indoor and outdoor unstructured environments, comparing against state-of-the-art vision-language-action baselines. Across five navigation tasks, CATNAV achieves 10 percentage point higher average goal-reaching rate and 33% fewer behavioral constraint violations.
Aditya Potnis, Francisco Affonso, Shreya Gummadi +2
Field Robotics Engineering and Science Hub (FRESH), Illinois Autonomous Farm, University of Illinois at Urbana-Champaign (UIUC), IL
Bird's-eye-view (BEV) images have been demonstrated to provide valuable prior information for navigation. Given the global information provided by such views, two key challenges remain: how to fully exploit this information and how to reliably use it during execution. In this paper, we propose a navigation system that uses BEV images as global priors and is designed for ground and near-ground robotic platforms. The system employs an image generation model to interpret human intent from natural language, identify the target destination, and generate traversability masks. During execution, we introduce cross-view localization to align the robot's odometry with the BEV map and mitigate long-term drift in conventional odometry. We conduct extensive benchmark experiments to evaluate the proposed method and further validate its effectiveness on both UAV and quadruped robot platforms. Using only a conventional local motion planner, the quadruped robot successfully completes a 285-meter outdoor long-range navigation task. This work demonstrates how the world-understanding capabilities of foundation models can be transferred to embodied navigation, enabling robots to benefit from the strong generalization ability of existing image generation models.
Yijin Wang, Yuru Tian, Xijie Huang +5
Zhejiang University · Differential Robotics · Beihang University
Open-world navigation requires robots to make decisions in complex everyday environments while adapting to flexible task requirements. Conventional navigation approaches often rely on dense 3D reconstruction and hand-crafted goal metrics, which limits their generalization across tasks and environments. Recent advances in vision-language navigation (VLN) and vision-language-action (VLA) models enable end-to-end policies conditioned on natural language, but typically require interactive training, large-scale data collection, or task-specific fine-tuning with a mobile agent. We formulate navigation as a sparse subgoal identification and reaching problem and observe that providing visual anchoring targets for high-level semantic priors enables highly efficient goal-conditioned navigation. Based on this insight, we select visual frontiers as semantic anchors and propose OpenFrontier, a navigation framework that requires no task-specific training or fine-tuning and seamlessly integrates diverse vision-language prior models. OpenFrontier enables efficient navigation with a lightweight system design, without dense 3D semantic mapping, task-specific policy training, or model fine-tuning. We evaluate OpenFrontier across multiple navigation benchmarks and demonstrate strong zero-shot performance, as well as effective real-world deployment on a mobile robot.
Esteban Padilla-Cerdio, Boyang Sun, Marc Pollefeys +1
1ETH Zurich · 2Microsoft Spatial AI Lab · University of Bonn