LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratories

Qianpu Sun; Xiaowei Chi; Yuhan Rui; Ying Li; Kuangzhi Ge; Jiajun Li; Sirui Han; Shanghang Zhang

cs.AIMar 12, 2026

LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratories

Authors: Qianpu Sun, Xiaowei Chi, Yuhan Rui, Ying Li, Kuangzhi Ge, Jiajun Li, Sirui Han, Shanghang Zhang

Organizations: Tsinghua University, Beijing, China · The Hong Kong University of Science and Technology, Hong Kong, China · Southern University of Science and Technology, Shenzhen, China · Peking University, Beijing, China ·

Abstract

Artificial intelligence is increasingly catalyzing scientific automation, with multimodal large language model (MLLM) agents evolving from lab assistants into self-driving lab operators. This transition imposes stringent safety requirements on laboratory environments, where fragile glassware, hazardous substances, and high-precision laboratory equipment render planning errors or misinterpreted risks potentially irreversible. However, the safety awareness and decision-making reliability of embodied agents in such high-stakes settings remain insufficiently defined and evaluated. To bridge this gap, we introduce LABSHIELD, a realistic multi-view benchmark designed to assess MLLMs in hazard identification and safety-critical reasoning. Grounded in U.S. Occupational Safety and Health Administration (OSHA) standards and the Globally Harmonized System (GHS), LABSHIELD establishes a rigorous safety taxonomy spanning 164 operational tasks with diverse manipulation complexities and risk profiles. We evaluate 20 proprietary models, 9 open-source models, and 3 embodied models under a dual-track evaluation framework. Our results reveal a systematic gap between general-domain MCQ accuracy and Semi-open QA safety performance, with models exhibiting an average drop of 32.0% in professional laboratory scenarios, particularly in hazard interpretation and safety-aware planning. These findings underscore the urgent necessity for safety-centric reasoning frameworks to ensure reliable autonomous scientific experimentation in embodied laboratory contexts. The full dataset will be released soon.

Figures & tables

Benchmark	Size	Embodied	Lab	Multimodal	Real	Ego.	Safety	Refusal	Holistic
RoboBench ( Luo et al., 2025 )	6092	✓	✗	✓	✓	✗	✗	✗	✓
EmbodiedBench ( Yang et al., 2025 )	1128	✓	✗	✓	✗	✗	✗	✗	✓
EmbodiedEval ( Cheng et al., 2025d )	327	✓	✗	✓	✗	✗	✗	✗	✓
EgoPlan-Bench ( Chen et al., 2023 )	3335	✓	✗	✓	✓	✓	✗	✗	✓
SafePlan-Bench ( Huang et al., 2025 )	2027	✓	✗	✓	✗	✗	✓	✗	✗
SafeAgentBench ( Yin et al., 2024 )	750	✓	✗	✓	✗	✓	✓	✗	✗

Table 1: Comparison with and existing benchmarks. LabShield focuses on safety-oriented embodied reasoning for agents operating in scientific laboratory. RoboBench targets holistic embodied evaluation in general environments, SafeAgentBench focuses on safety understanding, and ChemSafetyBench evaluates LLM safety in the chemical domain. Refusal: unsafe instruction rejection; Holistic: comprehensive evaluation.

Figure 1: The LabShield Diagnostic Framework. Top: Safety-centric evaluation pipeline and performance landscape of leading MLLMs. Bottom: Multi-view data acquisition workflow using an ego-centric robotic platform to capture high-fidelity multimodal data in real-world safety-critical laboratory environments.

Figure 2: An example of the LabShield safety evaluation pipeline. Both MCQs and semi-open evaluations are presented under a representative hazardous laboratory scenario, with correct and incorrect planning outcomes highlighted to illustrate safety-critical decision-making behavior.

Figure 3: Dataset Statistics. Left: Cross-distribution of Safety ( $S_{0}$ – $S_{3}$ ) and Operational ( $L_{0}$ – $L_{3}$ ) levels. Middle: Breakdown of VQA annotations by cognitive category (Perception, Reasoning, Planning). Right: Top-5 Unsafe Factors and Hazard Patterns across the four experimental scenarios.

Closed-source Models
Model	MCQ Evaluation(%)				Semi-open QA Evaluation (%)
	Mean $\uparrow$	P-Avg $\uparrow$	R-Avg $\uparrow$	Pl-Avg $\uparrow$	S. Score $\uparrow$	Perception			Reasoning				Planning		Safety L23
	Mean $\uparrow$	P-Avg $\uparrow$	R-Avg $\uparrow$	Pl-Avg $\uparrow$	S. Score $\uparrow$	U-J $\uparrow$	U-P $\uparrow$	U-R $\uparrow$	H-J $\uparrow$	H-P $\uparrow$	H-R $\uparrow$	Ana. $\uparrow$	Sco. $\uparrow$	Pas. $\uparrow$	Acc. $\uparrow$	Und. $\downarrow$
Human	92.0	88.4	98.2	89.4	87.3	87.8	89.6	84.8	87.2	87.8	89.6	87.8	88.4	85.4	94.5	5.5
OpenAI Family
GPT-4o	73.2	63.7	90.0	76.4	41.7	24.3	34.0	33.1	21.5	27.5	28.5	66.5	78.4	32.9	70.0	28.6

Table 2: Consolidated results on the benchmark. Metrics are aggregated into MCQ and Semi-open QA Evaluation. The semi-open evaluation is structurally categorized into Perception (Unsafe Factor identification), Reasoning (Hazard Pattern & Analysis), and Planning (Plan execution scores), providing a fine-grained PRP diagnostic. Rankings within categories are color-coded: 1st , 2nd , and 3rd .

Figure 4: Safety Performance Scales with Hazard Perception and Pattern Recognition.

Figure 5: Effect of in-context examples.

Figure 6: Effect of Different Camera Views.

Figure 7: Ablation Study on Number of Cameras.

Figure 8: Visualization of Attention Maps for Transparent Objects.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Proprietary Models
Model	Multiple-Choice Question (MCQ) Accuracy (%)										Semi-open Question Answering (QA) Performance (%)
	Avg.	Perception				Reasoning		Planning			S.Score	PR (Unsafe & Hazard)							Plan L01		Plan L23
	Mean	Sym.	Spa.	Obj.	Sta.	Cau.	Cnt.	Nxt.	Ord.	Rec.	S.Score	U-J	U-P	U-R	H-J	H-P	H-R	Ana.	Sco.	Pas.	Acc.	Und.	Ove.
Anthropic Claude Family
Claude3-Haiku	65.2	57.1	47.0	60.1	52.9	82.4	72.2	71.9	53.1	78.2	41.0	22.6	29.1	42.9	22.0	27.0	40.4	62.0	73.6	23.2	67.1	31.4	1.4
Claude4V_Sonnet	66.3	60.5	49.3	63.1	61.8	77.6	77.8	67.9	64.6	73.0	44.4	26.0	31.0	49.7	22.7	27.4	41.2	62.6	72.4	47.0	63.8	34.5	1.7

Appendix

Table 3: Additional results on the benchmark. We report the accuracy (%) for MCQ and Semi-open QA tasks. Key metrics Mean (MCQ Average) and S.Score (Safety Score) are highlighted. Models are grouped by family for clearer comparison.

Figure 9: Detailed performance breakdown across different operational levels (Level 0–3), showing the divergence between MCQ and safety-critical planning.

Figure 10: Impact of safety severity levels (Level 0–3) on perception and planning metrics, highlighting the cascading error patterns.

GPT4o (errors = 386)			Claude4 Sonnet (errors = 386)
Category	Count	Share	Category	Count	Share
state_recog.	66	17.1%	state_recog.	61	15.8%
object_recog.	58	15.0%	object_recog.	60	15.5%
next_step	53	13.7%	next_step	54	14.0%
symbol_recog.	51	13.2%	spatial	52	13.5%
spatial	46	11.9%	symbol_recog.	45	11.7%

Appendix

Table 4: Top-5 QA error categories (outer-ring subtypes; abbreviated) for GPT4o and Claude4 Sonnet, computed over all incorrect QA predictions.

Semi-open QA Performance (%)	GPT4o	Claude4 Sonnet
Safety-level accuracy	68.9	69.4
Safety-level under / over	21.3 / 9.8	11.5 / 19.1
Safety-level (2–3) accuracy	70.0	72.1
Safety-level (2–3) under / over	28.6 / 1.4	26.5 / 1.5
Unsafe Jacc / Prec / Rec	24.3 / 34.0 / 33.1	33.0 / 42.9 / 54.9
Hazard Jacc / Prec / Rec	21.5 / 27.5 / 28.5	30.1 / 38.0 / 47.0

Appendix

Table 5: Semi-open Question Answering (QA) Performance (%) for GPT4o and Claude4 Sonnet. All entries are reported in percentage points.

Figure 11: additional Visualization of Attention Maps for Transparent Objects.

Figure 12: Structure of The Prompt Template.

Figure 13: Structure of The Example Template.

Figure 14: Structure of The Example Template.

Figure 15: Structure of The Example Template.

Figure 16: Structure of The Example Template.

Figure 17: Structure of The Example Template.

Figure 18: Structure of The Example Template.

AI Agent Safety Benchmarks

AI agent safety benchmarks measure whether tool-using and embodied AI systems act safely across realistic tasks and interactions.

See all 216 papers

DeepSafe-SciHazard: A Benchmark for Measuring Scientific Safety Risks with Decomposed Harm Scoring
Jul 21, 2026Chunxiao Li, Yuan Xiong, Lijun Li +4LLM Safety Benchmarks AI Agent Safety Benchmarks
Shanghai Artificial Intelligence Laboratory
LabGuard: Grounding Natural-Language Laboratory Rules into Runtime Guards for Embodied Laboratory Agents
Jun 30, 2026Jingpu Yang, Fengxian Ji, Zhengzhao Lai +8AI Agent Safety AI Agent Monitoring
1Wuhan University · 2MBZUAI · 3Northeastern University
ThinkingGuard: Decoding Implicit Hazards via Step-by-Step Risk Attribution in Multimodal Large Language Models
Sep 29, 2026Ruochen Zhang, Yao Huang, Yitong Sun +5Multimodal Reasoning LLM Safety
Beihang University Beijing, China · Tsinghua University Beijing, China · Institute of Artificial Intelligence Beihang University Beijing, China State Key Laboratory of AI Safety

Jul 21, 2026J/K move · Enter open · S save

DeepSafe-SciHazard: A Benchmark for Measuring Scientific Safety Risks with Decomposed Harm Scoring

Chunxiao Li, Yuan Xiong, Lijun Li, Tianyi Du+3

Shanghai Artificial Intelligence Laboratory

LLM Safety Benchmarks AI Agent Safety Benchmarks

Open paper AskPDF arXiv

Large language models (LLMs) increasingly support science, but they can also convert hazardous scientific knowledge into actionable misuse guidance. Existing benchmarks often rely on templated queries disconnected from real-world hazards, and employ LLM-as-a-Judge paradigms without domain grounding. To address this, we introduce SciHazard, a real-world-grounded benchmark for scientific risks and a dataset agnostic evaluation framework for measuring harmfulness. SciHazard contains 2400 hazardous questions and 600 oversafety questions across 12 disciplines, with both queries grounded in regulated entities and documented failure scenarios. To compute DeHarm-Score , we develop a decomposed evaluating procedure that combines query hazard severity, refusal behavior, and response-level risk. For non-refused responses, it further decomposes response-level harm into Executability, quantified via dynamic checklists with importance weighting, and Net-new risk, assessed through retrieval-augmented claim extraction and synthesis-barrier verification. An expert-validation study shows that DeHarm-Score improves agreement with expert annotations by 90.17% over the strongest baseline. We benchmark 31 frontier LLMs and deep research agents in an extensive scientific safety evaluation. Notably, deep research agents yield 32.3% higher mean DeHarm-Score than standard LLMs, exposing autonomous agents as a critical blind spot in current safety defenses. Code and dataset are available at https://github.com/AI45Lab/DeepSafe-Sci.

Abstract

AI Agent Safety Benchmarks

Explore similar work

DeepSafe-SciHazard: A Benchmark for Measuring Scientific Safety Risks with Decomposed Harm Scoring

LabGuard: Grounding Natural-Language Laboratory Rules into Runtime Guards for Embodied Laboratory Agents

ThinkingGuard: Decoding Implicit Hazards via Step-by-Step Risk Attribution in Multimodal Large Language Models