Contextual Safety Reasoning and Grounding for Open-World Robots

Zachary Ravichandran; David Snyder; Alexander Robey; Hamed Hassani; Vijay Kumar; George J. Pappas

cs.ROFeb 23, 2026

Contextual Safety Reasoning and Grounding for Open-World Robots

Authors: Zachary Ravichandran, David Snyder, Alexander Robey, Hamed Hassani, Vijay Kumar, George J. Pappas

Organizations: University of Pennsylvania · Carnegie Mellon University

Abstract

Robots are increasingly operating in open-world environments where safe behavior depends on context: the same hallway may require different navigation strategies when crowded versus empty, or during an emergency versus normal operations. Traditional safety approaches enforce fixed constraints in user-specified contexts, limiting their ability to handle the open-ended contextual variability of real-world deployment. We address this gap via CORE, a safety framework that enables online contextual reasoning, grounding, and enforcement without prior knowledge of the environment (e.g., maps or safety specifications). CORE uses a vision-language model (VLM) to continuously reason about context-dependent safety rules directly from visual observations, grounds these rules in the physical environment, and enforces the resulting spatially-defined safe sets via control barrier functions. We provide probabilistic safety guarantees for CORE that account for perceptual uncertainty, and we demonstrate through simulation and real-world experiments that CORE enforces contextually appropriate behavior in unseen environments, significantly outperforming prior semantic safety methods that lack online contextual reasoning. Ablation studies validate our theoretical guarantees and underscore the importance of both VLM-based reasoning and spatial grounding for enforcing contextual safety in novel settings. We provide additional resources at https://zacravichandran.github.io/CORE.

Figures & tables

Fig. 1: CORE enforces contextual safety via a three-stage process of contextual safety reasoning, semantic grounding, and safe control synthesis, enabling it to operate in open-world environments.

Fig. 2: The CORE Framework enforces contextual safety via three modules. First, a VLM provides context-dependent safety constraints from visual observations. Constraints are grounded into spatially-defined safe sets, which are used for CBF-based control synthesis.

Fig. 3: Illustration of CORE ’s contextual reasoning and grounding process. Given an observation (A), the VLM predicts contextual safety constraints (B). CORE then segments the image (C) and constructs a image space safe set (D), which is integrated into a barrier function (E).

Fig. 4: Simulation Environments and platforms used for experiments.

Fig. 5: Example safe and unsafe tasks. Top: the nominal controller attempts to guide the robot into an area prohibited by cones. Bottom: the nominal controller attempts to guide the robot through a space constrained aisle.

Method	Task Success			Failure Attribution
Method	Total	Safe	Unsafe	Ctx.	Grnd.	Enf.
Oracle	98.3%	100%	96.6%	0.0%	1.7%	0.0%
CORE	95.0%	96.6%	93.3%	3.3%	1.7%	0.0%
No Context	55.0%	93.3%	16.6%	43.3%	1.7%	0.0%
Geometric	50.0%	100%	0%	100.0%	0.0%	0.0%

TABLE I: Simulation results over 60 evaluations.

Size	VLM	Safety Prediction Success			Latency
Size	VLM	Total	Safe	Unsafe	Latency
Small	Gemma 3 12B	75.0%	83.0 %	67.0%	2.2s
	Llava 13B	0.0%	0.0%	0.0%	2.7s
	Gemma 3 27B	88.0%	91.0%	85.0%	4.1s
Large	Qwen3-VL 30B	75.0%	77.0%	73.0%	26.6s
	Llava 34B	52.5%	49.0%	56.0%	6.0s

TABLE II: VLM Safety Constraint Prediction Success

Prompt	Safe	Unsafe	Latency
Full (ours)	91.0%	85.0%	4.1s
No Safety Logic	90.0%	80.0%	4.1s
No CoT Reasoning	61.0%	77.0%	2.8s

TABLE III: Safety Constraint Prediction Ablation

Method	Task Type			Failure Attribution
Method	Total	Safe	Unsafe	Ctx.	Grnd.	Enf.
CORE	86.6%	86.6%	86.6%	3.9%	9.0%	0.0%
Geometric	50.0%	100%	0.0%	100.0%	0.0%	0.0%

TABLE IV: Hardware results over 30 evaluations

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. A.1: Example safety predicates constructed from the four spatial operators considered in our work. For each predicate, we show the source observation and segmentation mask.

Module	Latency (s)
Safety Reasoning	4.37 $\pm$ 0.45
Semantic Grounding	0.75 $\pm$ 1.14
Safety Enforcement	0.04 $\pm$ 0.15

Appendix

TABLE A.I: Module latency mean and standard deviation

Module	Memory (GB)
Gemma 3 12B	10.4
Llava 13B	11.3
Gemma 3 27B	20.4
Qwen3-VL 30B	24.3
Llava 34B	21.4

Appendix

TABLE A.II: VLM memory requiremnets

Fig. A.2: Example nominal behavior and failure mode. The VLM’s input image is shown on top, with its response below. Left: failure mode where the VLM does not identify the wet floor sign, so the appropriate safety constraint is never grounded or enforced. Right: nominal behavior where the VLM correctly identifies the wet floor sign as a semantic hazard.

Robot Safety

Robot safety studies how to prevent harmful physical actions while enabling robots to complete tasks around people and in uncertain environments.

See all 229 papers

Mar 15, 2026

HomeGuard: VLM-based Embodied Safeguard for Identifying Contextual Risk in Household Task

Vision-Language Models (VLMs) empower embodied agents to execute complex instructions, yet they remain vulnerable to contextual safety risks where benign commands become hazardous due to subtle environmental states. Existing safeguards often prove inadequate. Rule-based methods lack scalability in object-dense scenes, whereas model-based approaches relying on prompt engineering suffer from unfocused perception, resulting in missed risks or hallucinations. To address this, we propose an architecture-agnostic safeguard featuring Context-Guided Chain-of-Thought (CG-CoT). This mechanism decomposes risk assessment into active perception that sequentially anchors attention to interaction targets and relevant spatial neighborhoods, followed by semantic judgment based on this visual evidence. We support this approach with a curated grounding dataset and a two-stage training strategy utilizing Reinforcement Fine-Tuning (RFT) with process rewards to enforce precise intermediate grounding. Experiments demonstrate that our model HomeGuard significantly enhances safety, improving risk match rates by over 30% compared to base models while reducing oversafety. Beyond hazard detection, the generated visual anchors serve as actionable spatial constraints for downstream planners, facilitating explicit collision avoidance and safety trajectory generation. Code and data are released under https://github.com/AI45Lab/HomeGuard

Xiaoya Lu, Yijin Zhou, Zeren Chen +6

Shanghai AI Laboratory · Shanghai Jiao Tong University · Beihang University +1

AI Agent Safety VLMs for Robotics Visual Grounding

Ask PDF

Oct 13, 2026

CoSMO: Conformal Semantic Mapping for Open-World Safe Robot Navigation

Safe autonomous navigation requires not only geometric obstacle avoidance but also semantic awareness. Different object categories demand different safety clearances, and the set of relevant categories may expand as the robot explores novel environments. Vision-language models (VLMs) offer rich semantic representations for building maps of unknown scenes, yet their confidence scores are poorly calibrated, making raw predictions unreliable for safety-critical planning. Conformal prediction (CP) can provide coverage guarantees, but existing methods assume a fixed label set and stationary data, limiting their use in open-world settings. We propose Conformal Semantic Mapping for Open-World Safe Robot Navigation (CoSMO), a semantic safety mapping framework that transforms VLM embeddings into calibrated coverage sets and safety-aware obstacle dilations over a dynamically expanding hazard vocabulary, even under distribution shift. Experiments on the Matterport3D (MP3D) dataset show that CoSMO reduces semantic collisions by 61.6% relative to a geometry-only baseline, while maintaining formal coverage guarantees even under delayed expert feedback. Real-world experiments on a quadruped robot further demonstrate generalization by achieving long-run empirical coverage of 90.73% against the 90% target. Project website: https://yangming911.github.io/cosmo-page/.

Shuqi Wang, Yiwei Zheng, Pingrui Zhang +2

Shanghai Jiao Tong University · Shanghai Innovation Institute · Fudan University

Safe Robot Navigation Conformal Prediction Semantic Occupancy Mapping VLMs for Robotics

Ask PDF

Apr 21, 2026

Safety-Critical Contextual Control via Online Riemannian Optimization with World Models

Modern world models are becoming too complex to admit explicit dynamical descriptions. We study safety-critical contextual control, where a Planner must optimize a task objective using only feasibility samples from a black-box Simulator, conditioned on a context signal

ξ_t

. We develop a sample-based Penalized Predictive Control (PPC) framework grounded in online Riemannian optimization, in which the Simulator compresses the feasibility manifold into a score-based density

\hat{p}(u \mid ξ_t)

that endows the action space with a Riemannian geometry guiding the Planner's gradient descent. The barrier curvature

κ(ξ_t)

, the minimum curvature of the conditional log-density

-\ln\hat{p}(\cdot\midξ_t)

, governs both convergence rate and safety margin, replacing the Lipschitz constant of the unknown dynamics. Our main result is a contextual safety bound showing that the distance from the true feasibility manifold is controlled by the score estimation error and a ratio that depends on

κ(ξ_t)

, both of which improve with richer context. Simulations on a dynamic navigation task confirm that contextual PPC substantially outperforms marginal and frozen density models, with the advantage growing after environment shifts.

Tongxin Li

Member, IEEE

Riemannian Optimization Safety-Critical Control Model Predictive Control

Ask PDF

Abstract

Robot Safety

Explore similar work

HomeGuard: VLM-based Embodied Safeguard for Identifying Contextual Risk in Household Task

CoSMO: Conformal Semantic Mapping for Open-World Safe Robot Navigation

Safety-Critical Contextual Control via Online Riemannian Optimization with World Models