Robots are increasingly operating in open-world environments where safe behavior depends on context: the same hallway may require different navigation strategies when crowded versus empty, or during an emergency versus normal operations. Traditional safety approaches enforce fixed constraints in user-specified contexts, limiting their ability to handle the open-ended contextual variability of real-world deployment. We address this gap via CORE, a safety framework that enables online contextual reasoning, grounding, and enforcement without prior knowledge of the environment (e.g., maps or safety specifications). CORE uses a vision-language model (VLM) to continuously reason about context-dependent safety rules directly from visual observations, grounds these rules in the physical environment, and enforces the resulting spatially-defined safe sets via control barrier functions. We provide probabilistic safety guarantees for CORE that account for perceptual uncertainty, and we demonstrate through simulation and real-world experiments that CORE enforces contextually appropriate behavior in unseen environments, significantly outperforming prior semantic safety methods that lack online contextual reasoning. Ablation studies validate our theoretical guarantees and underscore the importance of both VLM-based reasoning and spatial grounding for enforcing contextual safety in novel settings. We provide additional resources at https://zacravichandran.github.io/CORE.
Figures & tables
Fig. 1: CORE enforces contextual safety via a three-stage process of contextual safety reasoning, semantic grounding, and safe control synthesis, enabling it to operate in open-world environments.
Fig. 2: The CORE Framework enforces contextual safety via three modules. First, a VLM provides context-dependent safety constraints from visual observations. Constraints are grounded into spatially-defined safe sets, which are used for CBF-based control synthesis.
Fig. 3: Illustration of CORE ’s contextual reasoning and grounding process. Given an observation (A), the VLM predicts contextual safety constraints (B). CORE then segments the image (C) and constructs a image space safe set (D), which is integrated into a barrier function (E).
Fig. 4: Simulation Environments and platforms used for experiments.
Fig. 5: Example safe and unsafe tasks. Top: the nominal controller attempts to guide the robot into an area prohibited by cones. Bottom: the nominal controller attempts to guide the robot through a space constrained aisle.
Fig. A.1: Example safety predicates constructed from the four spatial operators considered in our work. For each predicate, we show the source observation and segmentation mask.
Module
Latency (s)
Safety Reasoning
4.37 ± 0.45
Semantic Grounding
0.75 ± 1.14
Safety Enforcement
0.04 ± 0.15
Appendix
TABLE A.I: Module latency mean and standard deviation
Module
Memory (GB)
Gemma 3 12B
10.4
Llava 13B
11.3
Gemma 3 27B
20.4
Qwen3-VL 30B
24.3
Llava 34B
21.4
Appendix
TABLE A.II: VLM memory requiremnets
Fig. A.2: Example nominal behavior and failure mode. The VLM’s input image is shown on top, with its response below. Left: failure mode where the VLM does not identify the wet floor sign, so the appropriate safety constraint is never grounded or enforced. Right: nominal behavior where the VLM correctly identifies the wet floor sign as a semantic hazard.