cs.CVJan 31, 2026

Toward Autonomous Laboratory Safety Monitoring with Vision Language Models: Learning to See Hazards Through Scene Structure

Authors: Trishna Chakraborty, Udita Ghosh, Aldair Ernesto Gongora, Ruben Glatt, Yue Dong, Jiachen Li, Amit K. Roy-Chowdhury, Chengyu Song

Organizations: Computer Science and Engineering, University of California, Riverside, USA · Lawrence Livermore National Laboratory · Electrical and Computer Engineering, University of California, Riverside, USA

Abstract

Laboratories are prone to severe injuries from unsafe actions, yet continuous manual safety monitoring, beyond mandatory pre-lab safety training, is infeasible. Vision-language models (VLMs) offer promise for autonomous laboratory safety monitoring, but their effectiveness in realistic settings is unclear due to the lack of visual evaluation data. Safety incidents are mostly documented as text, and existing studies are largely limited to textual understanding. Since hazardous scenes are unsafe to stage or capture, synthetic visual data becomes the only practical way to study this problem. This motivates us to introduce a structured data generation pipeline that converts textual laboratory scenarios into aligned quadruples using LLM as scene graph architect and image generation model as renderer, yielding \dataset: a synthetic dataset of 1,207 ⟨\langleimage, scene graph, ground truth, hazard relevance mask⟩\rangle quadruples across 362 unique scenarios. Experiments on \dataset and seven open- and closed-source models show that VLMs perform effectively given a textual scene graph but degrade substantially in visual-only settings (i.e., where no scene graph is involved), indicating difficulty in extracting structured object relationships directly from pixels. We thus propose \saga, a post-training framework that leverages reinforcement learning to first structure the visual scene as a scene graph, and then predict hazards while aligning attention with relevant image regions. Evaluations with Qwen 3.5 and Gemma 4 show that our method nearly doubles the F1F_1 score over the baseline on our synthetic dataset and achieves an improvement of ∼3×\sim 3\times on a mini real-world dataset, while preserving general visual understanding in visual question answering.

Figures & tables

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. TSHA: A Benchmark for Visual Language Models in Trustworthy Safety Hazard Assessment Scenarios

    Mar 31, 2026Qiucheng Yu, Ruijie Xu, Mingang Chen +2VLM EvaluationVLM Robustness

  2. EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards

    Jun 30, 2026Siddhant Panpatil, Arth Singh, Mijin Koo +3VLM RobustnessEgocentric Video Understanding

  3. Hazard or Anomaly? Evaluating VLMs for Understanding Dangers and Discrepancies

    Jul 18, 2026Murali Indukuri, Mohammad Eskandari, Sree Nitya Kollu +2VLM Evaluation