Organizations: Department of Computer Science, McGill University, Montreal, Canada. · Department of Electrical and Computer Engineering, McGill University, Montreal, Canada.
Safety in robotics does not solely depend on geometric constraints, but also on semantics: the given task, environment properties, and interactions between objects in the scene. We present a framework based on vision-language models (VLMs) to infer task- and scene-dependent safety constraints for robot manipulation. We provide the VLM a structured safety tool based on an RGB-D observation and a natural-language task instruction that first decides whether the task should be rejected since its execution may result in a non-recoverable hazardous outcome. Otherwise, it identifies preventable object-pair hazards and specifies the spatial relations that must be avoided. Open-vocabulary segmentation and depth information are then used to ground the detected objects in 3D and construct continuous risk fields that represent the corresponding hazardous regions. These fields are integrated with an obstacle avoidance module that corrects nominal robot motions during execution while accounting for the risk field. We evaluate the framework on preventable hazards, tasks requiring rejection, and safe variants in simulation and the real-world. Project website: https://vlm-hazard-riskfield.github.io
Figures & tables
Fig. 1 : Overview of the proposed framework. Given a task instruction and a scene image, we either reject the task or identifies object-pair hazards (top) that are grounded into a 3D risk field (middle), which guides safe execution (bottom).
Fig. 2 : Representation of how the location of a point x in the field ρ affects Γ .
Fig. 3 : Density and gamma for laptop
Fig. 4 : Example of Γ estimation between a bounding box of a mug a and a risk field ρ . The corner points, face centers, center of the bounding-box are evaluated to determine whether any part of the object enters the hazardous region.
Fig. 5 : Evaluation scenarios: (a) preventable hazards that can be mitigated with trajectory modification; (b) tasks requiring rejection due to unavoidable unsafe consequences; (c) safe variants used to assess unnecessary intervention or rejection. The task instruction is shown below each scene.
VLM
Cost
Prompt
Tool (ours)
Rel. ↓
F1 ↑
Miss ↓
F.rej. ↓
F1 ↑
Miss ↓
F.rej. ↓
GPT-4o
1.0
82.8
25.0
79.8
86.2
7.5
0.0
Claude Haiku 4.5
0.7
23.2
73.9
25.6
38.1
60.3
9.3
Gemini ER 2
1.1
91.1
13.2
0.0
73.8
41.5
0.0
Claude Sonnet 5
1.6
60.6
27.0
0.0
77.3
23.9
0.0
Gemini 3.1 Pro
1.7
99.3
0.7
10.9
86.6
23.6
0.8
TABLE I : Our Toolset vs. state-of-the-art VLM. Dark and light gray indicate the best and second-best in each column, respectively. Top arrows indicate whether higher or lower values are preferred, respectively.
[6pt][0pt]
Preventable hazard
Safe variant
Overall
[6pt][0pt]
Drawer-plant
Laptop-Cup
Steak-Dripping
Drawer-Safe
Laptop-Cube
Steak-Safe
Average
[6pt][6pt] Method
SR ↑
SSR ↑
Viol. ↓
SR ↑
SSR ↑
Viol. ↓
SR ↑
SSR ↑
Viol. ↓
SR ↑
SSR ↑
SR ↑
SSR ↑
SR ↑
SSR ↑
SR ↑
SSR ↑
Viol. ↓
No context
33.3 [-1.5pt] ± 15.3
0.0 [-1.5pt] ± 0.0
4.1 [-1.5pt] ± 1.3
63.3 [-1.5pt] ± 5.8
46.7 [-1.5pt] ± 5.8
3.8 [-1.5pt] ± 2.8
100.0 [-1.5pt] ± 0.0
0.0 [-1.5pt] ± 0.0
11.4 [-1.5pt] ± 0.1
26.7 [-1.5pt] ± 5.8
26.7 [-1.5pt] ± 5.8
100.0 [-1.5pt] ± 0.0
100.0 [-1.5pt] ± 0.0
100.0 [-1.5pt] ± 0.0
100.0 [-1.5pt] ± 0.0
70.6 [-1.5pt] ± 4.2
45.6 [-1.5pt] ± 1.0
6.5 [-1.5pt] ± 1.4
Text only
36.7 [-1.5pt] ± 11.5
0.0 [-1.5pt] ± 0.0
5.4 [-1.5pt] ± 1.7
23.3 [-1.5pt] ± 11.5
6.7 [-1.5pt] ± 5.8
5.0 [-1.5pt] ± 1.4
100.0 [-1.5pt] ± 0.0
0.0 [-1.5pt] ± 0.0
11.5 [-1.5pt] ± 0.3
3.3 [-1.5pt] ± 5.8
3.3 [-1.5pt] ± 5.8
100.0 [-1.5pt] ± 0.0
100.0 [-1.5pt] ± 0.0
97.0 [-1.5pt] ± 5.2
97.0 [-1.5pt] ± 5.2
60.1 [-1.5pt] ± 3.4
34.5 [-1.5pt] ± 1.0
7.3 [-1.5pt] ± 1.1
Simple OA
100.0 [-1.5pt] ± 0.0
0.0 [-1.5pt] ± 0.0
27.5 [-1.5pt] ± 0.0
40.0 [-1.5pt] ± 0.0
10.0 [-1.5pt] ± 0.0
6.7 [-1.5pt] ± 0.0
100.0 [-1.5pt] ± 0.0
0.0 [-1.5pt] ± 0.0
11.5 [-1.5pt] ± 0.0
100.0 [-1.5pt] ± 0.0
100.0 [-1.5pt] ± 0.0
100.0 [-1.5pt] ± 0.0
100.0 [-1.5pt] ± 0.0
100.0 [-1.5pt] ± 0.0
100.0 [-1.5pt] ± 0.0
90.0 [-1.5pt] ± 0.0
51.7 [-1.5pt] ± 0.0
15.2 [-1.5pt] ± 0.0
VLM (ours)
100.0 [-1.5pt] ± 0.0
80.0 [-1.5pt] ± 0.0
3.1 [-1.5pt] ± 1.3
70.0 [-1.5pt] ± 0.0
60.0 [-1.5pt] ± 0.0
2.5 [-1.5pt] ± 0.0
75.8 [-1.5pt] ± 5.2
57.6 [-1.5pt] ± 5.2
7.5 [-1.5pt] ± 1.3
100.0 [-1.5pt] ± 0.0
100.0 [-1.5pt] ± 0.0
100.0 [-1.5pt] ± 0.0
100.0 [-1.5pt] ± 0.0
90.9 [-1.5pt] ± 0.0
90.9 [-1.5pt] ± 0.0
89.4 [-1.5pt] ± 0.9
81.4 [-1.5pt] ± 0.9
4.4 [-1.5pt] ± 0.4
TABLE II : Evaluation of safe trajectory execution. Dark gray shading indicates the best result in each column. Top arrows indicate whether higher or lower values are preferred, respectively.
Fig. 6 : Laptop-cup trajectory comparison of our method against the baselines.
Correct rejection ↑
False rejection ↓
Method
Non-recoverable hazard
Recoverable hazard
Safe variants
VLM + Task
96/96
63/93
42/90
VLM + Task + Scene Text
96/96
83/93
58/90
VLM + Task + Scene Image
96/96
78/93
49/90
VLM + Task + Scene Text + Tool Calling
96/96
24/93
30/90
TABLE III : Rejection tasks study with different ablations. Dark gray shading indicates the best result in each column. Top arrows indicate whether higher or lower values are preferred, respectively.
Fig. 7 : Three real-world experiments scenarios.
Fig. 8 : Real-world Steak-Dripping trajectories from different perspectives
Method
Laptop–Cup
Steak Dripping
SR ↑
SSR ↑
Viol. ↓
SR ↑
SSR ↑
Viol. ↓
No context
100
20
80
100
20
80
Simple OA
100
20
80
100
20
80
VLM (ours)
100
100
0
80
80
0
TABLE IV : Real-world execution performance. Dark shading indicates the best result in each column. Top arrows indicate whether higher or lower values are preferred, respectively.
Robot motion planning in everyday environments must satisfy hard geometric constraints while accounting for context-dependent semantic risk. We present a foundation-model-guided, topology-aware semantic risk field that extends manipulation safety beyond collision avoidance. For each manipulated-object/scene-object pair, a foundation model provides six directional risk weights and a pair-specific spatial decay scale. The method combines these priors with voxelized 3D scene geometry using topology-aware shielding and geodesic spatial decay. A GPU-parallel backend batches object-level distance and risk computations to construct a dense 3D field that serves as a modular cost for downstream motion planning. We evaluate the field's shielding behavior under full and partial barriers and compare its 3D workspace representation with a pixel-wise semantic-prior baseline. Across three household simulation scenarios, trajectories optimized with the proposed field have lower semantic exposure than collision-only trajectories under the same geometric constraints. We also evaluate the computational practicality and reliability of the supporting pipeline. Together, these results support the proposed field as a practical topology-aware semantic cost representation for manipulation planning beyond collision avoidance.
Giung Lee, Weihang Guo, Lydia E. Kavraki
Department of Computer Science, Rice University, Houston, TX, USA
Modern safety-critical systems increasingly rely on human-robot interaction to reduce disaster risk and support decision-making during emergencies. Vision-Language Models (VLMs) are promising for these settings because they can interpret complex scenes and communicate safety-relevant information, but they still require careful evaluation to ensure reliable safety reasoning. In particular, current evaluations often frame danger recognition as a binary decision (Safe/Unsafe), making it unclear whether a model is identifying true physical hazards or merely reacting to unusual scene elements. We address this limitation by introducing an explicit distinction between hazard and anomaly, and by separately recognizing hazardous and anomalous states. We evaluate several state-of-the-art VLMs across two datasets and multiple prompting strategies to test whether this distinction changes model behavior. Our results show that VLMs frequently misinterpret anomalousness as hazardousness, revealing an over-reliance on contextual irregularity as a proxy for danger. We further show that explicitly separating anomaly from hazard provides a more informative evaluation of VLM safety reasoning and exposes failure modes that binary safety judgments can obscure. Our public dataset is available on Roboflow https://app.roboflow.com/vlm-in-context-anomaly-and-hazard-detection/camera-ready-roman-ds.
Murali Indukuri, Mohammad Eskandari, Sree Nitya Kollu +2
Interactive Robotics and Language Lab, University of Maryland Baltimore County, Computer Science and Electrical Engineering Department, Baltimore, MD, USA. · DEVCOM Army Research Laboratory, Adelphi, MD, USA.
Recent vision-language-action (VLA) models are promising for general-purpose manipulation, but long-horizon execution remains fragile. Small state-estimation or control errors can lead to irreversible failures (e.g., collisions and object drops). Avoiding these risks requires a proactive safety mechanism capable of anticipating hazards. In this paper, we introduce SafeLoop, a non-invasive external wrapper that adds hazard prediction and rollback-based recovery to a VLA model without changing its parameters. SafeLoop trains a risk predictor from vision and proprioception to output four values: the probability and time-to-hazard for body collisions and for object failures. A lightweight controller then chooses one of three actions based on the predicted risk: continue execution (noop), save a safety checkpoint (record), or retreat in joint space (rollback). Rollback moves the robot back to a recent safe waypoint and queries the base policy again, which may yield an alternative continuation. Across 24 LIBERO tasks (16 random seeds each) and three real-robot tasks (25 rollouts each), SafeLoop achieves a stronger overall safety-success trade-off than alternative methods, reducing hazard cases by roughly 70% while preserving task success and the base-policy control rate. Project code is available at https://github.com/Loule0-0/SafeLoop/tree/release/safeloop.
Zeyu Lou, Tianran Zhang, Xinquan Yue +2
Nanjing University, Nanjing, China · The Hong Kong University of Science and Technology (Guangzhou), Guangdong, China · Beijing University of Technology, Beijing, China