Organizations: Department of Intelligent Construction and Operation, Southeast University, Nanjing 211189, China. · Department of Construction Management and Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China. · Department of Automation, Tsinghua University, Beijing 100084, China.
Mobile robots typically rely on geometric maps for obstacle avoidance and path planning, but the resulting obstacle representation does not always match how an object should affect navigation. A low lying cable may be missed, a flexible curtain may create spurious blockage, and a traffic cone may require an exclusion region larger than its observed footprint. We present NavPatch, an object level correction layer that assigns ADD, REMOVE, or EXTEND to navigation relevant object categories through periodic scene understanding with a vision-language model. Open vocabulary grounding localizes object instances, and LiDAR and RGB-D observations provide 3D support. Observation quality filtering and cross frame maintenance determine when each correction patch is committed, replaced, or revoked. In 50 real robot trials across five layouts, NavPatch achieves an overall success rate of 86.0%. An ablation study of four configurations with 200 runs in total shows that NavPatch improves the success rate from 70.0% to 86.0% and reduces the false commit rate from 68.4% to 40.7% compared with updates based only on the current observation.
Figures & tables
Fig. 1: Three representative geometry and navigation mismatches. (a) A traversable curtain creates spurious blockage, (b) a low profile obstacle is weakly represented or missed, and (c) traffic cones require an exclusion region beyond their physical footprint. NavPatch applies Remove , Add , and Extend , respectively. Dashed and solid paths show the plans before and after map correction.
Fig. 2: System overview of NavPatch . Periodic VLM scene understanding proposes object categories and map actions. GroundingDINO and SAM 2.1 localize instances, while LiDAR and RGB-D observations provide 3D support for local correction regions. Observation quality filtering and cross frame maintenance regulate patch updates before correction patches are fused with the base costmap for navigation.
Fig. 3: Cross frame maintenance of a REMOVE patch under changing viewpoints. A region accepted at t1 is retained as a candidate before sufficient evidence is accumulated, and is committed as a patch at t2 . The patch is retained at t3 while a newly observed region is accumulated as a candidate, and is replaced at t4 . Dashed outlines denote candidate regions under evidence accumulation, whereas filled regions denote committed patches.
Fig. 4: Real robot navigation in five layouts. Panels (a) and (b) show Mixed-A and Mixed-B, while (c), (d), and (e) focus on Add , Remove , and Extend , respectively. Each panel shows the corrected map and a representative navigation path, with an RGB inset displaying detected object instances.
Layout
Trials
SR
FCR
MCR
PUR
Mixed-A
10
70.0%
42.2%
35.0%
8.1%
Mixed-B
10
70.0%
37.3%
34.7%
8.5%
ADD-focused
10
100.0%
47.3%
3.3%
7.8%
REMOVE-focused
10
100.0%
27.3%
20.0%
1.7%
EXTEND-focused
10
90.0%
38.5%
20.0%
5.4%
Overall
50
86.0%
40.7%
24.9%
6.9%
TABLE I: Navigation and map correction results across five layouts
Configuration
SR
FCR
MCR
PUR
Frame-wise
70.0%
68.4%
22.4%
95.4%
Quality-only
66.0%
64.6%
24.1%
88.6%
Temporal-only
54.0%
54.1%
27.6%
5.9%
NavPatch
86.0%
40.7%
24.9%
6.9%
TABLE II: Ablation study of patch update strategies
Vision-based navigation models, particularly foundation models, generate viable trajectories from RGB observations alone. However, even state-of-the-art transformer- and diffusion-based policies struggle to generalize in unfamiliar deployment environments containing unseen obstacles or shifted conditions. The resulting trajectories often remain goal-directed but unsafe. Existing efforts improve safety through external trajectory correction or internal geometric priors, yet the resulting policies are not trained to explicitly represent obstacle boundaries or traversable free-space structure. To address this, we propose a navigation model that incorporates these structures directly into the policy via fine-tuning and is designed to be compatible with diverse RGB-based backbones. Across multiple robot platforms, indoor environments, and static and dynamic obstacle scenarios, our method reduces collision frequency relative to ViNT, NoMaD, and their CARE-augmented variants while maintaining goal-reaching performance.
Geonyeong Ko, Giung Lee, Changjoo Nam
Dept. of Electronic Engineering, Sogang University, Seoul, Korea · Dept. of Computer Science, Rice University, Houston, TX, USA · Vertical Labs, Co., Ltd., Seoul, Korea
Safe autonomous navigation requires not only geometric obstacle avoidance but also semantic awareness. Different object categories demand different safety clearances, and the set of relevant categories may expand as the robot explores novel environments. Vision-language models (VLMs) offer rich semantic representations for building maps of unknown scenes, yet their confidence scores are poorly calibrated, making raw predictions unreliable for safety-critical planning. Conformal prediction (CP) can provide coverage guarantees, but existing methods assume a fixed label set and stationary data, limiting their use in open-world settings. We propose Conformal Semantic Mapping for Open-World Safe Robot Navigation (CoSMO), a semantic safety mapping framework that transforms VLM embeddings into calibrated coverage sets and safety-aware obstacle dilations over a dynamically expanding hazard vocabulary, even under distribution shift. Experiments on the Matterport3D (MP3D) dataset show that CoSMO reduces semantic collisions by 61.6% relative to a geometry-only baseline, while maintaining formal coverage guarantees even under delayed expert feedback. Real-world experiments on a quadruped robot further demonstrate generalization by achieving long-run empirical coverage of 90.73% against the 90% target. Project website: https://yangming911.github.io/cosmo-page/.
Shuqi Wang, Yiwei Zheng, Pingrui Zhang +2
Shanghai Jiao Tong University · Shanghai Innovation Institute · Fudan University
Safe and robust robot navigation across diverse environments requires a high-level understanding of complex scenes and the ability to carry it into stable motion. Recent works tackle this with learning-based models trained at scale and with approaches built on vision-language models (VLMs). However, learning-based models break down outside their training distribution, while VLM-based approaches bring that understanding but rarely ground it in the scene or align the action with it. To address these limitations, we present RECAST, a robot navigation framework that combines the reasoning of a VLM with the spatial grounding of vision foundation models to build an Actionable Cost map. Given the robot's front view and the user's instruction, we first decompose the scene with the VLM, judging which surfaces are traversable, which objects pose a risk, which heading to prefer, and which gaps are passable. Vision foundation models then ground these surfaces and objects in the image, and all four judgments are spatially recast into one compact cost map. This map both conditions the trajectory decoders and scores their proposals to select the one to execute. As the VLM's answers trail the live scene, both steps draw on cost maps from two points in time: the pivot frame the VLM judged, which carries all four judgments, and the current frame, whose terrain and collision costs are rebuilt from the current image. RECAST improves success over the strongest prior method by 13.3 points in simulation and 31.4 points on a real quadruped, and reduces the collision rate relative to it by 9.6 and 14.3 points, reaching the lowest collision rate among all methods. The project page is available at https://recast-nav.github.io/