CoSMO: Conformal Semantic Mapping for Open-World Safe Robot Navigation

Shuqi Wang; Yiwei Zheng; Pingrui Zhang; Yue Gao; Xiang Yin

cs.ROOct 13, 2026

CoSMO: Conformal Semantic Mapping for Open-World Safe Robot Navigation

Authors: Shuqi Wang, Yiwei Zheng, Pingrui Zhang, Yue Gao, Xiang Yin

Organizations: Shanghai Jiao Tong University · Shanghai Innovation Institute · Fudan University

Abstract

Safe autonomous navigation requires not only geometric obstacle avoidance but also semantic awareness. Different object categories demand different safety clearances, and the set of relevant categories may expand as the robot explores novel environments. Vision-language models (VLMs) offer rich semantic representations for building maps of unknown scenes, yet their confidence scores are poorly calibrated, making raw predictions unreliable for safety-critical planning. Conformal prediction (CP) can provide coverage guarantees, but existing methods assume a fixed label set and stationary data, limiting their use in open-world settings. We propose Conformal Semantic Mapping for Open-World Safe Robot Navigation (CoSMO), a semantic safety mapping framework that transforms VLM embeddings into calibrated coverage sets and safety-aware obstacle dilations over a dynamically expanding hazard vocabulary, even under distribution shift. Experiments on the Matterport3D (MP3D) dataset show that CoSMO reduces semantic collisions by 61.6% relative to a geometry-only baseline, while maintaining formal coverage guarantees even under delayed expert feedback. Real-world experiments on a quadruped robot further demonstrate generalization by achieving long-run empirical coverage of 90.73% against the 90% target. Project website: https://yangming911.github.io/cosmo-page/.

Figures & tables

Figure 1: Classical ZSON pipeline [ 4 ] navigates with no semantic safety awareness and collides. VLM-only Obstacle Avoidance considers the single most likely label and still enters the semantic hazard zone when misclassified. CoSMO calibrates coverage sets over the visual embedding, accounting for multiple possible labels, successfully reaching the goal. Closed-set offline conformal prediction breaks when a previously unseen object appears during navigation.

Figure 2: CoSMO module for open-world semantic safety. CoSMO takes an embedding-based semantic map as input, constructs conformal coverage sets, maintains the semantic dilation map, updates the adaptive threshold via open-vocabulary ACI, and expands the safety dictionary using delayed expert feedback. The updated dilation map serves as a safety layer for downstream planning.

Figure 5: Effect of expert delay $\tau$ on CoSMO. (a) Threshold convergence: all delay values converge to the same offline quantile $Q_{90\%}$ , consistent with Proposition 1 . (b) Empirical coverage: all curves converge to the $90\%$ target, but larger $\tau$ produces larger oscillations, consistent with Proposition 2 .

Figure 6: Real-world navigation with a Unitree Go2 robot in an initially unknown $3.5\,\mathrm{m}\times 10\,\mathrm{m}$ indoor environment. The robot searches for a chair while avoiding semantic obstacles, with GCLIP for perception and Grounding-DINO for delayed expert feedback. Different colors denote different argmax labels predicted by the VLM-based perception module. The varying argmax labels across observations show the ambiguity of VLM perception. CoSMO instead constructs calibrated coverage sets through online conformal inference, achieving label coverage with a guarantee.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Category	V1	V2	V3	V4	Category	V1	V2	V3	V4
chair	0	2	5	0	stool	0	2	5	0
table	5	5	5	0	towel	0	2	5	0
picture	0	2	5	0	tv_monitor	2	2	5	0
cabinet	4	4	5	0	shower	3	3	5	0
cushion	0	2	5	0	counter	0	2	5	0
sofa	0	2	5	0	bathtub	0	2	5	4

Appendix

Table 2: Obstacle safety dictionaries (top: per-category safety distances

D_{\text{safe}}

in grid cells,

0.1

m/cell) and results on 330 MP3D val_ablation episodes (bottom: V1–V3 dictionary ablation and the unperturbed V4 sensitivity setting, with CoSMO using

\alpha=0.1

and

\gamma=0.05

). Bold entries: 5 representative hazard categories shared by V1 and V2 (top).

Dict.	Perturbation	SR(%)	SCR(%)	SPL	STUCK(%)
V1	$-1$	13.3	11.2	0.077	40.9
V1	$0$	12.7	10.3	0.072	43.6
V1	$+1$	11.5	10.0	0.068	47.3
V4	$-1$	20.3	4.8	0.108	37.6
V4	$0$	20.3	4.8	0.107	36.7
V4	$+1$	20.0	5.2	0.103	36.7

Appendix

Table 3: Sensitivity to safety distance perturbations on 330 ablation episodes. Perturbations are in grid cells; SCR is evaluated using unchanged reference distances.

Component	Specification
Vision–Language Models
Navigation backbone	ConvNeXt-Large-d-320 (LAION-2B)
Obstacle detection (GCLIP)	ViT-B/16 (OpenAI CLIP)
Geometric detection	YOLO-World/L
Noisy expert ablation	Grounding-DINO (Swin-T)
Simulator and Platform

Appendix

Table 4: Models, software, and hyperparameters.

Scene	Episodes	Argmax	CoSMO	Argmax	CoSMO	Avg.STUCK
		SR		SCR
2azQ1b91cZZ	200	14.5	14.0	12.5	7.5	26.0
TbHJrupSAjP	200	24.5	20.0	26.0	23.0	42.3

Appendix

Table 5: Scene-level comparison on 200 episodes per scene. All rates are percentages.

Panel of Figure 7 — Figure 7: Top-down views of the scenes in Table 5 : a more open layout (left) and a more constrained layout (right).

Figure 8: Empirical coverage vs. vocabulary size.

$\tau$	SR $\uparrow$	SCR $\downarrow$	SPL $\uparrow$
0	12.7%	10.3%	0.071
5	11.8%	10.0%	0.067
10	13.0%	10.3%	0.066
20	14.2%	10.9%	0.076
100	14.2%	10.0%	0.075

Appendix

Table 6: Delay ablation on 330 episodes. SCR and SPL show no systematic dependence on

\tau

, consistent with Proposition 1 .

Panel of Figure 9 — Figure 9: GT oracle vs. Grounding-DINO oracle on nonconformity score distribution, $Q_{t}$ trajectory, per-episode coverage, and real-time miscoverage.

$\tau$	Fallback $d_{\mathrm{wc}}$	SR(%)	SCR(%)	SPL	STUCK(%)	Pre-disc.	Post-disc.
0	0	12.7	10.3	0.070	42.7	0 (0.0%)	34 (100%)
0	2	11.2	12.1	0.069	43.0	0 (0.0%)	40 (100%)
0	5	13.0	10.9	0.069	43.6	0 (0.0%)	36 (100%)
50	0	12.4	9.4	0.068	46.4	2 (6.45%)	29 (93.6%)
50	2	13.3	9.4	0.071	43.9	2 (6.45%)	29 (93.6%)
50	5	12.7	11.8	0.063	45.5	2 (5.13%)	37 (94.9%)

Appendix

Table 8: Cold-start fallback on 330 MP3D ablation episodes. Fallback radii are in grid cells (

0.1

m/cell). Pre-disc. and Post-disc. report semantic collision counts and their percentages of all semantic collisions in each setting.

Environment and Robot
Symbol	Description
$\Omega\subset\mathbb{R}^{2}$	Workspace domain
$x_{t}\!\in\!\mathcal{X}\!\subset\!\mathbb{R}^{n_{x}}$	Robot state at time step $t$
$u_{t}\!\in\!\mathcal{U}\!\subset\!\mathbb{R}^{n_{u}}$	Control input
$(I_{t},Z_{t})$	RGB-D observation: $I_{t}\!\in\!\mathbb{R}^{H\times W\times 3}$ , $Z_{t}\!\in\!\mathbb{R}^{H\times W}$
$\mathcal{J}$	Set of indices for the grid map cells

Appendix

Table 9: Variable definitions.

Safe Robot Navigation

Safe robot navigation develops methods for reaching goals while avoiding collisions and managing uncertainty, dynamic obstacles, and environmental hazards.

See all 134 papers

G2-Nav: Grounded and Guarded Vision-Language Costmaps for Robot Social Navigation
Jul 18, 2026Yuwen Liao, Yihang Lan, Yizhuo Yang +4Robot Navigation Safe Robot Navigation
School of Electrical and Electronic Engineering Nanyang Technological University Singapore
WildOS: Open-Vocabulary Object Search in the Wild
Feb 22, 2026Hardik Shah, Erica Tevere, Deegan Atha +6Visual Robot Navigation Robotic Object Search
Jet Propulsion Laboratory (JPL), California Institute of Technology (Caltech), Pasadena, CA 91011, USA · Swiss Federal Institute of Technology (ETH Zürich), Robotic Systems Lab, Zürich 8092, Switzerland · FieldAI Inc. 3 Morgan, Irvine, CA, USA. Work Performed at NASA JPL +2
SAFER-Nav: Enhancing Safety for Visual Robot Navigation via Segmentation-Aware Fine-Tuning
Jun 10, 2026Geonyeong Ko, Giung Lee, Changjoo NamRobot Policy Learning Trajectory Generation
Dept. of Electronic Engineering, Sogang University, Seoul, Korea · Dept. of Computer Science, Rice University, Houston, TX, USA · Vertical Labs, Co., Ltd., Seoul, Korea

Jul 18, 2026J/K move · Enter open · S save

G2-Nav: Grounded and Guarded Vision-Language Costmaps for Robot Social Navigation

Yuwen Liao, Yihang Lan, Yizhuo Yang, Ruimeng Liu+3

School of Electrical and Electronic Engineering Nanyang Technological University Singapore

Robot Navigation Safe Robot Navigation Social Robot Navigation Robot Safety VLMs for Robotics

Open paper AskPDF arXiv

Social navigation requires the robot to reason and respond in complex real-world environments. While recent works attempt to incorporate human-level intelligence into robot planning using large Vision-Language Models (VLMs), end-to-end frameworks often create an unpredictable black-box, and existing instruction-following methods are not designed for full autonomy. To bridge this gap, we present G2-Nav, a novel framework that grounds abstract social reasoning and guards safe real-world deployment. Instead of asking the VLM for direct planning decisions, G2-Nav translates its semantic reasoning into a vision-language costmap with reliability and interpretability. The VLM evaluates traversable regions and social agents from open-set perception, mapping social context into the costmap. To improve real-world robustness, the VLM performs semantic verification on upstream tracking, and we introduce a high-frequency safety check to guard against system latency prior to trajectory generation. We demonstrate through real-world experiments that G2-Nav delivers safe, efficient, and socially compliant autonomous navigation in unstructured environments. Code is available at https://github.com/centiLinda/G2-Nav.

Abstract

Safe Robot Navigation

Explore similar work

G2-Nav: Grounded and Guarded Vision-Language Costmaps for Robot Social Navigation

WildOS: Open-Vocabulary Object Search in the Wild

SAFER-Nav: Enhancing Safety for Visual Robot Navigation via Segmentation-Aware Fine-Tuning