EAST: Embodied AI Safety Testing via Systematic Fault Injection in Digital Twins
Authors: Yuhao Tian, Tao Li, Ling Zhou, Dave Towey, Zhaoxin Fan, Wenjun Wu, Rubing Huang, Tsong Yueh Chen
Organizations: School of Computer Science and Engineering, Macau University of Science and Technology, China · School of Artificial Intelligence (Institute of Artificial Intelligence), Beihang University, China · School of Computer Science, University of Nottingham Ningbo China, China · Department of Computing Technologies, Swinburne University of Technology, Australia
Task success is not a sufficient measure of safety for embodied artificial intelligence (AI), because robots may complete objectives while violating safety requirements. Existing evaluations often emphasize performance and robustness, but lack a systematic route from documented safety hazards to executable, verifiable tests. We present EAST (Embodied AI Safety Testing), a digital-twin framework that converts literature-grounded safety evidence into fault-injection tests across perception, instructions, environments, actions, and timing. EAST combines an expert-reviewed safety-fault taxonomy and eight executable operators with applicability and validity constraints. Its three-stage workflow generates test specifications, verifies fault realization during simulated execution, and independently assesses safety challenges, violations, task outcomes, and behavioral signatures using simulator-state evidence. The framework preserves execution traces and replayable records, supporting attribution and reproducibility across experiments. We evaluate EAST on six RoboTwin manipulation tasks and two policy architectures, ACT and RDT, against alternative testing strategies under matched budgets. Across five comparative campaigns, EAST targets all 74 admissible task-policy-operator combinations and obtains valid evidence for 63, while achieving safety-challenge exposure rates close to Random-Legal and Rule-Based. Moreover, 82 valid EAST executions complete their tasks despite safety violations, revealing risks concealed by success-oriented evaluation. These findings support systematic, validity-aware, reproducible safety testing beyond task-completion metrics.
Figures & tables
Figure 1. Representative unsafe-success executions exposed by EAST .
Abbreviation
Meaning
Abbreviation
Meaning
EAST
Embodied AI Safety Testing
RQ
Research Question
VLA
Vision–Language–Action
ACT
Action Chunking with Transformers
RDT
Robotics Diffusion Transformer
RGB
Red–Green–Blue
DR
Domain Randomization
RoboTwin-DR
RoboTwin domain-randomization baseline
VSTY
Valid Safety-Test Yield
SCER
Safety-Challenge Exposure Rate
SFTR
Safety-Failure Trigger Rate
SF
Safety Fault
Table 1. Summary of the main abbreviations used in this paper.
Symbol
Meaning
Symbol
Meaning
Σ
Embodied AI system model
S , O , A
State, observation, and action spaces
st , ot , at
State, observation, and action at time t
ψ , π , δ
Observation, policy, and transition functions
Φ , ϕi
Safety-specification set and an individual safety predicate
ρ , ρ≤t
Execution trace and trace prefix
C , ci
Safety-taxonomy class set and an individual class
Ω , ωi
Fault-operator set and an individual operator
Γ
Taxonomy-to-operator mapping
Θω , θ
Configuration space of operator ω and a concrete configuration
K , k
Task set and an individual task
Π
Set of embodied policies under test; each π∈Π is a policy function
Table 2. Summary of the main mathematical symbols used in this paper.
Figure 2. Overview of the EAST safety-testing pipeline.
Task ID
RoboTwin Task
Task Description
Step Limit
Task 1
handover_block
Block Handover
800
Task 2
beat_block_hammer
Hammering a Block
400
Task 3
place_container_plate
Container-to-Plate Placement
400
Task 4
place_burger_fries
Burger-and-Fries Placement
500
Task 5
open_microwave
Microwave Door Opening
1500
Task 6
adjust_bottle
Bottle Adjustment
400
Table 3. Evaluation tasks used in the formal EAST experiments.
Table 4. Overview of the six experiments used to evaluate EAST .
Code
Category
Evidence (M/R)
Papers
Role
Operators
C. Safety taxonomy and contextual safety evidence
Category 1
Unsafe Contact and Collision
13/13
6/15
Phenotype
Op3, Op4, Op7, Op8
Category 2
Grasp and Retention Failure
6/6
4/15
Phenotype
Op1, Op4, Op5, Op6, Op7, Op8
Category 3
Release, Placement and Containment Failure
12/8
5/15
Phenotype
Op4, Op5, Op6, Op8
Category 4
Target, Role and Semantic Grounding Failure
10/4
4/15
Phenotype
Op1, Op2, Op3, Op4
Category 5
Temporal, Control and Action-Ordering Failure
8/6
6/15
Phenotype
Op6, Op7, Op8
Table 5. Expert-reviewed evidence coding and operator mapping for RQ1. Evidence is reported as mapped/retained (M/R) counts.
Qualification Scope
Safety-Test Outcomes
Operator
Applicable Pairs
Fault-Injected Sources
Total Executions
VSTY n/N
SCER n/N
SFTR n/N
Op1
12
200
600
200/200 (100.0%)
150/200 (75.0%)
85/200 (42.5%)
Op2
6
200
600
200/200 (100.0%)
166/200 (83.0%)
67/200 (33.5%)
Op3
6
200
600
200/200 (100.0%)
166/200 (83.0%)
67/200 (33.5%)
Op4
12
200
600
200/200 (100.0%)
116/200 (58.0%)
68/200 (34.0%)
Op5
12
200
600
200/200 (100.0%)
166/200 (83.0%)
68/200 (34.0%)
Table 6. Experiment 1 fault-injection validity, safety-challenge exposure, and safety-failure rates. Rates refer to fault-injected source executions; total executions also include no-op controls and exact replays.
Valid Outcomes
Controls and Replay Consistency
Operator
Challenge, no violation n/N
Non-Safety Failure n/N
No Trigger n/N
No-Op False Pos. n/N
Exact-Replay Match n/N
Op1
65/200 (32.5%)
50/200 (25.0%)
0/200 (0.0%)
0/200 (0.0%)
200/200 (100.0%)
Op2
99/200 (49.5%)
34/200 (17.0%)
0/200 (0.0%)
0/200 (0.0%)
200/200 (100.0%)
Op3
99/200 (49.5%)
34/200 (17.0%)
0/200 (0.0%)
0/200 (0.0%)
200/200 (100.0%)
Op4
48/200 (24.0%)
84/200 (42.0%)
0/200 (0.0%)
0/200 (0.0%)
200/200 (100.0%)
Op5
98/200 (49.0%)
34/200 (17.0%)
0/200 (0.0%)
0/200 (0.0%)
200/200 (100.0%)
Table 7. Experiment 1 valid outcomes, controls, and exact-replay consistency.
Figure 3. Experiment 2 candidate validity and safety-challenge exposure under matched generation budgets.
Experiment
Comparator
Metric
Difference (EAST − Comparator, pp)
Paired-Bootstrap 95% CI (pp)
Experiment 2
Random-Unconstrained
VSTY
+16.50
[ +12.83, +20.17 ]
Random-Legal
VSTY
-4.00
[ -6.33, -1.67 ]
Random-Unconstrained
SCER
+0.26
[ -2.03, +2.62 ]
Random-Legal
SCER
+2.04
[ -0.09, +4.24 ]
Experiment 3
Random-Legal
SFTR
-0.69
[ -2.22, +0.57 ]
Rule-Based
SFTR
-6.66
[ -11.07, -2.88 ]
Table 8. Paired hypothesis comparisons for Experiment 2 and Experiment 3.
Difference from Full EAST (pp)
Exploration Breadth
Configuration
VSTY n/480
VSTY
SCER
SFTR
Distinct Failure Signatures
Distinct Operators
Full EAST
389/480 (81.04%)
0.00
0.00
0.00
38
8
No taxonomy guidance
394/480 (82.08%)
-1.04
+0.19
+0.82
37
8
No applicability checking
348/480 (72.50%)
+8.54
-0.36
-0.03
37
8
No validity checking
291/480 (60.63%)
+20.42
+0.38
+0.67
35
8
No semantic preservation
326/480 (67.92%)
+13.13
-0.49
+0.39
35
8
Table 9. Experiment 5 paired component-ablation results. Differences are computed as Full EAST minus the corresponding ablation.
Method
VSTY n/600
SCER n/Nvalid
SFTR n/Nvalid
Unsafe Success n/Nvalid
Targeted Cov.
Valid Cov.
Distinct Failure Signatures
Clean
600/600 (100.00%)
0/600 (0.00%)
0/600 (0.00%)
0/600 (0.00%)
–
–
0
EAST
515/600 (85.83%)
392/515 (76.12%)
182/515 (35.34%)
82/515 (15.92%)
74/74 (all 5)
63/74 (all 5)
27
Random-Legal
546/600 (91.00%)
408/546 (74.73%)
197/546 (36.08%)
93/546 (17.03%)
59–63/74
53–59/74
39
Rule-Based
600/600 (100.00%)
443/600 (73.83%)
248/600 (41.33%)
106/600 (17.67%)
12/74
12/74
8
RoboTwin-DR
600/600 (100.00%)
0/600 (0.00%)
0/600 (0.00%)
0/600 (0.00%)
–
–
0
Table 10. Experiment 3 equal-budget comparison across the five campaigns.
Figure 4. Experiment 3 cumulative discoveries as the candidate budget increases
Category
Operator
Mapped Observations
Valid Observations
VSTY
Challenge Exposures
SCER
Safety Failures
Failure Conversion
Within-Category Failure Share
Category 1
Op3
846
846
100.00%
646
76.36%
216
33.44%
17.34%
Op4
1,097
1,097
100.00%
897
81.77%
291
32.44%
23.35%
Op7
1,035
1,035
100.00%
835
80.68%
280
33.53%
22.47%
Op8
659
659
100.00%
459
69.65%
459
100.00%
36.84%
Category 2
Op1
2,285
2,285
100.00%
2,085
91.25%
864
41.44%
36.29%
Op4
1,097
1,097
100.00%
897
81.77%
291
32.44%
12.22%
Table 11. Experiment 4 safety-phenotype characterization across fault operators.
Figure 5. Experiment 4 safety-failure mappings between safety categories and fault operators.
Embodied AI systems are increasingly deployed in open-world environments, yet ensuring their reliability remains a fundamental challenge. Drawing on discussions from the AAAI'26 Bridge Program on "Making Embodied AI Reliable with Testing and Formal Verification", this article argues that reliability in embodied AI is inherently a lifecycle assurance problem arising from uncertainty, human interaction, and emergent behaviors across tightly coupled system components. We identify three complementary directions toward reliable embodied AI: (1) trustworthy scenario-based testing supported by validated specifications and meaningful coverage metrics, (2) compositional verification enabled by structured symbolic representations of system behavior and environmental context, and (3) runtime assurance mechanisms capable of adapting to uncertainty and distribution shifts during deployment. Rather than treating these approaches independently, we advocate integrated assurance workflows that connect testing, verification, and runtime adaptation through shared neuro-symbolic representations and continuous feedback across the system lifecycle. Such integration provides a foundation for building trustworthy embodied AI systems that can operate safely and reliably in complex real-world environments.
Rapid progress in AI agents has brought growing attention to agent safety, with extensive evaluation focused on digital environments. As agents move into the physical world, embodied safety becomes increasingly important: failures can cause human injury and costly hardware damage. Beyond selecting safe actions, embodied agents must also operate under real-time constraints: the physical world does not pause while an agent reasons. As pedestrians move and vehicles approach during inference, an action that appears safe at observation time may become unsafe before execution. Real-time embodied safety therefore depends on both decision quality and decision latency. We introduce RT-SAFE, a simulated urban benchmark for evaluating embodied-agent safety under real-time constraints. RT-SAFE combines navigation tasks with moving actors, environmental hazards, and traffic rules, while allowing the world to evolve throughout inference and action execution. Across eight VLMs, agents achieve high task completion yet almost never complete safely: in the hardest setting, only 0.7% of episodes finish without a safety event. More strikingly, matched static and real-time evaluations yield task completion rates of 91.3% and 94.1%, respectively, while real-time execution increases collisions by 12.3×. These results reveal that standard task success can mask substantial safety failures, and that decision latency itself can become a source of physical risk. Finally, we show that RT-SAFE can support offline RL training and substantially reduce collision rates while achieving strong task completion.
Tianruo Rose Xu, Jiawei Ren, Yichi Yang +2
Cornell University · University of California, San Diego
General purpose language models are increasingly able to control robotic hardware. Understanding the capabilities and safety of these models when embodied is therefore increasingly important for understanding their societal impact and risks. To this end, we introduce Inspect Robots, a modular, open-source framework for developing and running evaluations of embodied agents. Inspect Robots pairs customizable, reusable abstractions for specifying physical evaluations and analyzing their results with infrastructure that automates evaluation setup, execution and termination. We demonstrate Inspect Robots by using it to evaluate the capabilities and safety of six policies based on frontier language models. Inspect Robots has seen significant early uptake, receiving nearly 100,000 downloads in the three months since its release.
Christopher Leet, Achu Menon, Sravanthi Machcha +12
Robocurve · University of Southern California · San Jose State University +7