The rise of unmanned ``dark factories'' operating without human presence demands autonomous safety systems capable of detecting and responding to multiple hazard types. We present SafeGuard ASF (Agentic Security Fleet), a comprehensive framework deploying humanoid robots for autonomous hazard detection in industrial environments. Our system integrates multi-modal perception (RGB-D imaging), a ReAct-based agentic reasoning framework, and learned locomotion policies on the Unitree G1 humanoid platform. We address three critical hazard scenarios: fire and smoke detection, abnormal temperature monitoring in pipelines, and intruder detection in restricted zones. Our perception pipeline achieves 94.2% mAP for fire or smoke detection with 127ms latency. We train multiple locomotion policies, including dance motion tracking and velocity control, using Unitree RL Lab with PPO, demonstrating stable convergence within 80,000 training iterations. We validate our system in both simulation and real-world environments, demonstrating autonomous patrol, human detection with visual perception, and obstacle avoidance capabilities. The proposed ToolOrchestra action framework enables structured decision-making through perception, reasoning, and actuation tools.
Figures & tables
Fig. 1 : Six-layer hierarchical architecture of SafeGuard ASF. Sensor data flows through perception (L1) and understanding (L2) layers to inform decision-making (L4), which coordinates planning (L5) and action (L6) for hazard response.
Fig. 2 : Locomotion policy training pipeline using Unitree RL Lab. Training in Isaac Sim with 4096 parallel environments and domain randomization, followed by sim-to-sim validation in MuJoCo and sim-to-real deployment on the physical G1 robot.
Category
Count
Examples
Perception
8
fire_smoke, thermal_scan, depth_localize
Reasoning
5
fire_severity, thermal_hazard, spill_hazard
Knowledge
4
facility_map, thermal_baselines, personnel_db
Actuation
6
remote_valve, fire_suppression, locomotion
TABLE I : ToolOrchestra Tool Categories
Fig. 3 : Fire detection and response scenario. The robot detects fire via visual perception, assesses severity, coordinates suppression activation, and monitors until extinguishment. Timeline shows detection at T=0, alert at T+3s, suppression at T+18s, and confirmation at T+48s.
Hazard Type
Precision
Recall
F1
Latency
Fire
0.962
0.924
0.942
127ms
Smoke
0.918
0.891
0.904
127ms
Thermal Anomaly
0.895
0.936
0.915
83ms
Intruder
0.971
0.948
0.959
152ms
Average
0.937
0.925
0.930
122ms
TABLE II : Detection Performance by Hazard Type
Fig. 4 : Qualitative results of dance training.
Fig. 5 : Qualitative results of Gangnam training.
Fig. 6 : Qualitative results of velocity training.
Fig. 7 : Qualitative results of IsaacSim simulation
Phase
Fire
Thermal
Intruder
Detection
0.13
0.08
0.15
Reasoning
0.85
1.23
0.72
Alert Transmission
0.42
0.38
0.35
Navigation (avg)
8.50
6.20
5.80
Intervention
2.50
3.40
1.80
Total
12.40
11.29
8.82
TABLE III : Response Time Breakdown (seconds)
Scenario
Success
Partial
Failure
Fire/Smoke Response
92%
6%
2%
Thermal Anomaly
88%
8%
4%
Intruder Detection
88%
10%
2%
Overall
89.3%
8.0%
2.7%
TABLE IV : Scenario Success Rate (N=20 per scenario)
System
Coverage
Blind Spots
Response
Success
Rule-Based
100%
0%
14.2s
76%
SafeGuard ASF
100%
0%
12.4s
89%
TABLE V : Comparison with Baseline Systems
Fig. 8 : Thermal anomaly detection and response. Temperature profile along pipe PP-C-15 shows localized peak at stuck valve PV-C-15-M. Following remote reset, temperature returns to baseline within 5 minutes.
Fig. 9 : Qualitative results of the real-world environment
Module
Latency
FPS
GPU Util.
YOLOv8-m (Fire/Smoke)
33ms
30.3
45%
YOLOv8-n (Person)
12ms
83.3
18%
Thermal Processing
8ms
125.0
12%
Depth Localization
15ms
66.7
8%
OSNet Re-ID
25ms
40.0
22%
Full Pipeline
38ms
26.3
68%
TABLE VI : Computational Performance on Jetson Orin
This report presents a structured Robotics Physical Safety Framework based on explicit asset declaration, systematic vulnerability enumeration, and hazard-driven synthetic data generation. The approach bridges classical risk engineering with modern machine learning pipelines, enabling safety envelope learning grounded in a formalized hazard ontology. The key contribution of this framework is the alignment between classical safety engineering, digital twin simulation, synthetic data generation, and machine learning model training.
As AI systems increasingly assist humans in physical tasks, ensuring safety becomes paramount -- physical actions carry immediate and irreversible consequences that digital errors do not. We introduce the Vision-Language Embodied Safety Agent (VLESA), a framework that monitors human activities from egocentric video and triggers real-time safety interventions when dangerous actions are predicted. VLESA addresses intent-dependent safety where identical actions can be safe or dangerous depending on context. A dataset pairing egocentric frames with goal-conditioned safety annotations is introduced, enabling a goal-conditioned safety Q-filter trained via GRPO that evaluates actions with respect to inferred intent without retraining. On top of that, an intent-action prediction agent is proposed to jointly infer goals and predict future actions from video. On the ASIMOV-2.0 benchmark, VLESA achieves higher intervention accuracy at the exact ground-truth frame compared to baselines, while the GRPO-trained Q-filter improves action safety by over 41 percentage points through goal-conditioned constrained decoding. Code is available at https://github.com/HanjiangHu/VLESA.
Hanjiang Hu, Yiyuan Pan, Jiaxing Li +5
Carnegie Mellon University · Harvard University · Mitsubishi Electric Research Laboratories
Ensuring functional safety in human-robot interaction is challenging because AI perception is inherently probabilistic, whereas industrial standards require deterministic behavior. We present an LLM-guided safety agent for edge robotics, built on an ISO-compliant low-latency perception-compute-control architecture. Our method translates natural-language safety regulations into executable predicates and deploys them through a redundant heterogeneous edge runtime. For fault-tolerant closed-loop execution under edge constraints, we adopt a symmetric dual-modular redundancy design with parallel independent execution for low-latency perception, computation, and control. We prototype the system on a dual-RK3588 platform and evaluate it in representative human-robot interaction scenarios. The results demonstrate a practical edge implementation path toward ISO 13849 Category 3 and PL d using cost-effective hardware, supporting practical deployment of safety-critical embodied AI.