Toward Autonomous Laboratory Safety Monitoring with Vision Language Models: Learning to See Hazards Through Scene Structure
Authors: Trishna Chakraborty, Udita Ghosh, Aldair Ernesto Gongora, Ruben Glatt, Yue Dong, Jiachen Li, Amit K. Roy-Chowdhury, Chengyu Song
Organizations: Computer Science and Engineering, University of California, Riverside, USA · Lawrence Livermore National Laboratory · Electrical and Computer Engineering, University of California, Riverside, USA
Laboratories are prone to severe injuries from unsafe actions, yet continuous manual safety monitoring, beyond mandatory pre-lab safety training, is infeasible. Vision-language models (VLMs) offer promise for autonomous laboratory safety monitoring, but their effectiveness in realistic settings is unclear due to the lack of visual evaluation data. Safety incidents are mostly documented as text, and existing studies are largely limited to textual understanding. Since hazardous scenes are unsafe to stage or capture, synthetic visual data becomes the only practical way to study this problem. This motivates us to introduce a structured data generation pipeline that converts textual laboratory scenarios into aligned quadruples using LLM as scene graph architect and image generation model as renderer, yielding \dataset: a synthetic dataset of 1,207 ⟨image, scene graph, ground truth, hazard relevance mask⟩ quadruples across 362 unique scenarios. Experiments on \dataset and seven open- and closed-source models show that VLMs perform effectively given a textual scene graph but degrade substantially in visual-only settings (i.e., where no scene graph is involved), indicating difficulty in extracting structured object relationships directly from pixels. We thus propose \saga, a post-training framework that leverages reinforcement learning to first structure the visual scene as a scene graph, and then predict hazards while aligning attention with relevant image regions. Evaluations with Qwen 3.5 and Gemma 4 show that our method nearly doubles the F1 score over the baseline on our synthetic dataset and achieves an improvement of ∼3× on a mini real-world dataset, while preserving general visual understanding in visual question answering.
Figures & tables
Figure 1: Overview of our approach. (A) An LLM Architect translates textual laboratory scenarios into scene graphs, which condition an image-generation Renderer to synthesize photorealistic lab images. (B) VLMs often miss hazards from raw images alone, but inferring a scene graph from the image first enables safety reasoning and leads to correct hazard detection.
Figure 2: End-to-end pipeline for dataset construction, hazard detection settings, and model training. Textual laboratory scenarios are converted into scene graphs, hazard annotations, images, and hazard relevance masks, then verified through human and VLM-as-judge checks. VLMs are evaluated under text-only, vision+text, vision-only, and scene-graph-based prompting settings. We further improve visual-only hazard detection using SAGA , which trains models to generate scene-graph reasoning while grounding attention in relevant image regions.
VLMs
M
S
Overall
Biology
Chemistry
Cryogenic Liquids
Physics
General
FE
Label
Count
FE
Label
Count
FE
Label
Count
FE
Label
Count
FE
Label
Count
FE
Label
Count
F1
MAE
F1
MAE
F1
MAE
F1
MAE
F1
MAE
F1
MAE
llava- next-7B
V
-
0
0
0.55
0
0
0.42
0
0
0.68
0
0
0.85
0
0
0.28
0
0
0.62
SG-G
31.89
0.22
0.55
38.98
0
0.52
27.67
0.26
0.65
19.86
0.28
0.83
35.74
0.23
0.31
33.59
0.16
0.6
TSG
+ H
6.61
0.72
0.62
0
0.75
0.44
0
0.8
0.48
0
0.86
0.57
20
0.45
0.65
9.13
0.67
0.82
- H
6.53
0.54
0.42
0
0.59
0.29
0
0.57
0.5
0
0.49
0.64
20
0.31
0.18
8.7
0.54
0.49
Table 1: Hazard detection performance across input modalities (M) and reasoning settings (S): overall and domain-wise F1 , format error (FE), and mean absolute error (MAE) for vision-only (V), textual scene graph with (TSG+H) and without (TSG-H) hazard attributes, and vision + textual scene graph (V+TSG) inputs. Precision and recall are in Table 6 . Scene-graph-based settings, with or without images, outperform vision-only in nearly all cases, showing that VLMs reason effectively given structured object relationships. SG-guided prompting improves vision-only detection, suggesting that safety knowledge is present but becomes usable once models explicitly structure the scene. Darker shading indicates higher F1 .
Figure 3: Examples from LabHazVision illustrating hazardous and non-hazardous scenes.
Figure 4: Sample distribution across laboratory subjects. (Left) Original scenarios with our re-extracted ground truth. (Right) LabHazVision images after human and VLM-as-judge filtering, which largely preserve the distribution of the original scenarios.
Figure 5: F1 across hazard detection settings. Textual scene graphs without hazard attributes (TSG-H) perform best and vision-only (V) worst; adding scene graphs to visual input (V+TSG-H) helps, and SG-guided prompting (SG-G) further improves visual-only detection.
Figure 6: Textual scene graphs with (TSG+H) vs. without (TSG-H) hazard attributes. Hazard attributes boost LLaVA’s F1 but hurt other models, suggesting shortcut memorization over structured reasoning.
Figure 7: Subject-wise F1 in the vision-only. Performance varies by laboratory domain, with Cryogenic Liquids and Physics more challenging, highlighting domain-dependent difficulty. LLaVA omitted ( F1=0 )
Model
Method
FE
P
R
F1
Qwen3.5-0.8B (Full-finetuning)
Vision only
0
1
0.20
0.34
SG-guided Prompting
100
0
0
0
SG-based GSPO
8.99
0.93
0.32
0.47
SAGA
1.56
0.90
0.58
0.71
Qwen3.5-2B (Full-finetuning)
Vision only
0
1
0.22
0.35
SG-guided Prompting
100
0
0
0
Table 2: Hazard detection performance on test set. SAGA consistently achieves the highest F1 across all models, improving over SG-based GSPO alone.
Model
Method
VQA
Real-World Data
Acc.
FE
P
R
F1
Qwen3.5-0.8B
Base model
0.75
0
1
0.08
0.14
SAGA
0.72
0
1
0.31
0.47
Qwen3.5-2B
Base model
0.77
0
1
0.08
0.14
SAGA
0.78
0
1
0.54
0.70
Gemma4-2B
Base model
0.47
0
1
0.08
0.14
Table 3: Generalization of SAGA to real-world data and VQA. Though trained only on synthetic data, SAGA improves F1 on real-world images for every base model while preserving general utility (VQA accuracy).
Figure 9: Qualitative attention maps comparing the base model and the fine-tuned SAGA model for Qwen-3.5-0.8B. Each row shows (left to right): the original image, the base model’s attention map, the fine-tuned model’s attention map, the difference map (FT − Base; red = increased attention), and the ground-truth binary hazard mask . In all three cases, the fine-tuned model shifts attention toward the hazard region, with the red regions in the difference map aligning with the GT mask.
Model
Method
Precision
Recall
F1
Gemma 4-2B
Base model
1.000
0.125
0.222
SAGA
1.000
0.375
0.545
Appendix
Table 4: Out-of-distribution performance on the SafeAgentBench-derived laboratory hazard set.
Figure 10: Representative examples from the real-world dataset. Unlike our synthetic data, these images focus on isolated violations.
Figure 11: Caption-based generation ( left ) produces the container but omits the visible gas vapour, whereas scene-graph-based generation ( right ) correctly depicts the liquid nitrogen undergoing phase transition, consistent with the hazard described.
Model
Method
Precision
Recall
F1
Gemma 4-2B
Base model
1.000
0.190
0.320
SAGA
0.957
0.349
0.512
Appendix
Table 5: SAGA performance on human-validated images rejected by the VLM judge.
VLMs
M
S
Overall
Biology
Chemistry
Cryogenic Liquids
Physics
General
FE
Classification
Count
FE
Classification
Count
FE
Classification
Count
FE
Classification
Count
FE
Classification
Count
FE
Classification
Count
P
R
F1
MAE
P
R
F1
MAE
P
R
F1
MAE
P
R
F1
MAE
P
R
F1
MAE
P
R
F1
MAE
llava- next-7B
V
-
0
0
0
0
0.55
0
0
0
0
0.42
0
0
0
0
0.68
0
0
0
0
0.85
0
0
0
0
0.28
0
0
0
0
0.62
SG-G
31.89
0.74
0.13
0.22
0.55
38.98
0
0
0
0.52
27.67
0.91
0.15
0.26
0.65
19.86
0.5
0.2
0.28
0.83
35.74
0.4
0.17
0.23
0.31
33.59
0.6
0.09
0.16
0.6
TSG
+ H
6.61
0.77
0.68
0.72
0.62
0
0.97
0.63
0.75
0.44
0
0.96
0.68
0.8
0.48
0
1
0.77
0.86
0.57
20
0.34
0.65
0.45
0.65
9.13
0.72
0.63
0.67
0.82
- H
6.53
1
0.37
0.54
0.42
0
1
0.42
0.59
0.29
0
1
0.4
0.57
0.5
0
1
0.34
0.49
0.64
20
0.8
0.19
0.31
0.18
8.7
1
0.37
0.54
0.49
Appendix
Table 6: Hazard detection performance of VLMs across input modalities (M) and reasoning settings (S), reported overall and by laboratory domain. We compare vision-only (V), textual scene graph with hazard (TSG+H), textual scene graph without hazard (TSG-H), and vision + textual scene graph (V+TSG) inputs using format error (FE,%), precision (P), recall (R), F1 score, and mean absolute error (MAE). Textual scene graph–based settings, with or without visual input, generally outperform vision-only models across nearly all cases. This indicates that VLMs reason effectively when structured object relationships are provided. SG-guided prompting improves hazard detection in the visual-only setting by asking VLMs to first synthesize structured textual scene representations, revealing that safety logic is present but becomes accessible after explicit scene structuring by the models themselves. F1 scores are color-coded, with darker shading indicating higher performance.
Variation
Temp.
N Runs
Mean P
Std P
Mean R
Std R
Mean F1
Std F1
TSG
0.0
3
0.80
0.00
1.00
0.00
0.89
0.00
0.5
3
0.79
0.02
1.00
0.00
0.88
0.01
1.0
3
0.77
0.02
1.00
0.00
0.87
0.01
V
0.0
3
0.95
0.00
0.56
0.00
0.70
0.00
0.5
3
0.94
0.01
0.55
0.01
0.69
0.01
1.0
3
0.93
0.02
0.53
0.01
0.67
0.01
Appendix
Table 7: Qwen3-8B-4bit Performance.
Figure 12: Alignment outcomes when using VLMs as judges compared to human evaluation. Bars show the number of images classified as Aligned or Not Aligned with the corresponding scene graph and ground-truth hazards. Gemini exhibits substantially higher agreement with human judgments than GPT-5.2-Pro, supporting its use as the VLM-as-judge for large-scale alignment filtering.
Category
Hyperparameter
Values
Optimizer
Optimizer
AdamW
Learning rate
5×10−6
β1
0.9
β2
0.99
Weight decay
0.1
Warmup ratio
0.1
Appendix
Table 8: Hyperparameters for SAGA .
Model Name
Complete Model ID
Hosting
llava-next-7B
llava-hf/llava-v1.6-mistral-7b-hf
Hugging Face
qwen3-8B
Qwen/Qwen3-VL-8B-Instruct
Hugging Face
llama-3.2-11B
meta-llama/Llama-3.2-11B-Vision-Instruct
Hugging Face
mistral-medium-3
mistral-medium-3
GCP Vertex
claude-sonnet-4.5
claude-sonnet-4-5@20250929
GCP Vertex
gemini-3-flash
gemini-3-flash-preview
GCP Vertex
Appendix
Table 9: Model cards used in our experiments
Figure 13: Prompt used for scene graph generation from textual laboratory safety scenarios (Part 1).
Figure 14: Prompt used for scene graph generation from textual laboratory safety scenarios (Part 2).
Figure 15: Prompt used for scene graph generation from textual laboratory safety scenarios (Part 3).
Figure 16: Prompt used to extract ground-truth hazard labels from textual laboratory scenarios
Figure 17: Prompt used for graph-conditioned image generation of hazardous laboratory scenarios.
Figure 18: Prompt used for graph-conditioned image generation of non-hazardous laboratory scenarios.
Figure 19: Prompt used for hazard detection from images only.
Figure 20: Prompt used for hazard detection from textual scene graphs only.
Figure 21: Prompt used for hazard detection from images with textual scene graph context.
Figure 22: Prompt used for scene graph guided hazard detection (generate a scene graph, then classify hazards), Part 1/2.
Figure 23: Prompt used for scene graph guided hazard detection (generate a scene graph, then classify hazards), Part 2/2.
Figure 24: Prompt used to generate binary hazard relevance masks.
Jul 18, 2026·Murali Indukuri, Mohammad Eskandari, Sree Nitya Kollu +2VLM Evaluation
Interactive Robotics and Language Lab, University of Maryland Baltimore County, Computer Science and Electrical Engineering Department, Baltimore, MD, USA. · DEVCOM Army Research Laboratory, Adelphi, MD, USA.