Alpha: Systematic Fault Injection and Robustness Evaluation for LLM-Driven Embodied Manipulation

Jona Joy

cs.ROOct 21, 2026

Alpha: Systematic Fault Injection and Robustness Evaluation for LLM-Driven Embodied Manipulation

Authors: Jona Joy

Organizations: independent robotics and artificial intelligence research engineer

Abstract

Large language models (LLMs) are increasingly integrated into embodied robotic planning, yet standard benchmarks primarily evaluate end-to-end task completion in nominal environments, obscuring failure origins across the execution stack. We present Alpha, a systematic fault-injection and robustness-evaluation framework for LLM-driven manipulation in simulation. Alpha injects calibrated faults across perception (sensor lag), actuation (grip slip), kinematics (unreachable targets), and planning (planner-output corruption), alongside concurrent conditions. To separate physical vulnerabilities from planner sensitivities, we introduce a matched methodology pairing an LLM agent (Llama-3 via Ollama) against a deterministic Scripted Oracle planner across a balanced 150-episode matrix under identical seeds, geometries, and safety criteria. Across the matrix, the Oracle achieves 64.0% (48/75) success while the LLM achieves 44.0% (33/75) (paired McNemar p = 3.01 x 10^-4). Crucially, excluding planner-output corruption, performance across all physical and kinematic conditions is identical: 55.0% (33/60) for both planners with zero discordant pairs (McNemar p = 1.00). This demonstrates that physical actuator slip, kinematic reach boundaries, and perceptual staleness affect deterministic controllers and LLM planners equivalently. The performance gap is driven entirely by planner-output corruption (100% Oracle vs. 0% LLM), isolating the prompt-to-JSON serialization and validation interface rather than a cognitive reasoning deficit. We also audit 169 historical pilot attempts for transparency.

Figures & tables

Fig. 1: System architecture of the Alpha evaluation harness. Controlled faults (F1–F4) are injected by ChaosRoboticsWrapper at four operational boundaries without altering the Brain’s prompt definitions or the Body’s inverse-kinematics control routines. The PyBullet Safety Monitor synchronously inspects multi-link contact dynamics at 240 Hz.

Fault Class	Layer	Injection Boundary	Intensity Formula	Realized Experimental Behavior	Primary Metric
F1: Sensor Lag	Perception	wrap_observation()	$\tau_{\text{lag}}=0.10+1.40\cdot I$	Observed lag saturated at $0.500\,\text{s}$ ; $\Delta_{\text{pos}}=0.025\,\text{m}$	Stale divergence, Task success
F2: Grip Slip	Actuation	step() on lift	$P_{\text{slip}}=0.10+0.85\cdot I$	Evaluated once per lift; slip rate $40\%\rightarrow 80\%$	Realized slip rate, Success rate
F3: Unreachable IK	Kinematics	move_end_ effector()	$\delta_{\text{reach}}=0.03+0.27\cdot I$	Target offset outside envelope; placement error $>{0.17}\,\text{m}$	Placement error (m), Incidents
F4: Output Corruption	Serialization	intercept_ planner_output()	$P_{\text{corrupt}}=0.20+0.80\cdot I$	Schema validation catches malformed JSON; empty plan	Validation rejection rate, Success
Concurrent (F1+F2)	Multi-layer	Perception + Actuation	$I=0.66$ fixed	Saturated lag ( $0.500\,\text{s}$ ) combined with grip slip	Dual fault detection, Success rate

TABLE I: Formal fault taxonomy, injection boundaries, intensity formulas, and primary evaluation metrics in Alpha.

Cohort	Intensity ( $I$ )	Oracle	LLM	Total
Control	0.00 (Nominal)	10	10	20
F1: Sensor Lag	0.33, 0.66, 1.00	15	15	30
F2: Grip Slip	0.33, 0.66, 1.00	15	15	30
F3: Unreachable IK	0.33, 0.66, 1.00	15	15	30
F4: Corruption	0.33, 0.66, 1.00	15	15	30
Concurrent	0.66 (F1+F2)	5	5	10

TABLE II: Experimental Design Matrix: Balanced 150-Episode Cohort Allocation (Master Seed 42).

Fig. 2: Task success rate across fault conditions for the Scripted Oracle baseline ( $N=75$ ) versus the LLM agent ( $N=75$ ) across the 150-episode matrix. On all physical and kinematic conditions (Control, F1, F2, F3, and Concurrent), both planners achieve identical performance with zero discordant pairs. The overall divergence is localized entirely to F4 planner-output corruption.

Cohort	Oracle	LLM	Disc.	McNemar $p$
Full Matrix ( $N=75$ )	48/75 (64%)	33/75 (44%)	15	$3.01\times 10^{-4}$
Physical ( $N=60$ )	33/60 (55%)	33/60 (55%)	0	$\mathbf{1.00}$

TABLE III: Overall performance and primary statistical decomposition across the 150-episode matched matrix.

Condition	Intensity	N	Oracle Succ.	Oracle 95% CI	LLM Succ.	LLM 95% CI	Disc.	Primary Failure Mechanism
Control	Baseline (0.00)	10	10/10 (100%)	[72.2%, 100%]	10/10 (100%)	[72.2%, 100%]	0	Nominal pick-and-place
F1: Sensor Lag	Low (0.33)	5	5/5 (100%)	[56.6%, 100%]	5/5 (100%)	[56.6%, 100%]	0	Stale $z$ -offset within grasp tolerance
F1: Sensor Lag	Medium (0.66)	5	5/5 (100%)	[56.6%, 100%]	5/5 (100%)	[56.6%, 100%]	0	Stale $z$ -offset within grasp tolerance
F1: Sensor Lag	High (1.00)	5	5/5 (100%)	[56.6%, 100%]	5/5 (100%)	[56.6%, 100%]	0	Stale $z$ -offset within grasp tolerance
F2: Grip Slip	Low (0.33)	5	3/5 (60%)	[23.1%, 88.2%]	3/5 (60%)	[23.1%, 88.2%]	0	Constraint release on lift ( $P_{\text{slip}}=0.27$ )
F2: Grip Slip	Medium (0.66)	5	2/5 (40%)	[11.8%, 76.9%]	2/5 (40%)	[11.8%, 76.9%]	0	Constraint release on lift ( $P_{\text{slip}}=0.52$ )

TABLE IV: Condition-by-condition empirical matrix results for Scripted Oracle vs. LLM Agent (

N=150

total,

n=5

per fault intensity cell,

n=10

for Control). Confidence intervals are 95% Wilson score intervals.

Fig. 3: F1 sensor lag analysis: (a) Requested lag versus realized lag, showing saturation at the 0.500 s available history buffer depth. (b) Spatial staleness divergence (2.50 cm) compared against the physical gripper opening envelope (8.0 cm), explaining 100% task completion.

Fig. 4: F2 grip slip results: Calibrated slip probability curve versus realized empirical slip rates and task success rates. Both planners exhibit identical monotonic degradation across intensities with zero discordant pairs.

Intensity ( $I$ )	Oracle (m)	LLM (m)	Success (O/L)
Control (0.00)	0.0349	0.0349	100% / 100%
Low (0.33)	0.1794	0.1839	0% / 0%
Medium (0.66)	0.2248	0.3176	0% / 0%
High (1.00)	0.2830	0.2693	0% / 0%

TABLE V: F3 Unreachable IK: Continuous final placement error (distance to target center in meters;

n=5

per cell, within-cell

\sigma=0.000\,\text{m}

Fig. 5: Continuous placement error under F3 unreachable IK. Final distance to place-zone center increases monotonically for the Scripted Oracle, while the LLM displays a non-monotonic response due to kinematic deflection at higher reach limits.

Fig. 6: Execution path divergence under F4 planner-output corruption. The LLM text stream passes through JSON serialization where corruption triggers Pydantic schema rejection (0% success). The Scripted Oracle bypasses serialization, executing directly via in-memory objects (100% success).

Parameter / Dimension	Specification / Observed Value
Safety Monitor Version	1.1.0
Monitored Links	IIWA/Franka Links 0–6
Excluded Contacts	Arm $\leftrightarrow$ table base; active grasp reactions
Force Threshold	$F_{\text{thresh}}=3000\,\text{N}$ (at 240 Hz)
Workspace ( $X,Y,Z$ )	$[0.15,0.95]$ , $[-0.45,0.55]$ , $[0.55,1.25]\,\text{m}$
Place Zone	Center $(0.6,0.35,0.65)\,\text{m}$ , $r=0.18\,\text{m}$

TABLE VI: Safety Monitor Specification (v1.1.0) and Empirical Safety Incident Outcomes.

Fig. 7: Complete accounting of 169 historical pilot attempts. All 99 rejected OpenAI attempts failed due to external API infrastructure aborts prior to simulation. The accepted local Ollama cohort ( $N=70$ ) executed with a 0% crash rate.

Large Language Model-Based Robot Planning

Large language model-based robot planning turns natural-language goals and sensory observations into grounded, executable actions for robots.

See all 236 papers

How Long Until Your Robot Ignores You? A Safety Benchmark for LLM Orchestrators in Human-Humanoid Collaboration
Sep 7, 2026Aulon Bajrami, Mohamed Elshamouty, Werner KrausRobot Safety Human-Robot Collaboration
Fraunhofer IPA Stuttgart, Germany
Embodied Zero-Shot Task Planning via Hardware-Aware LLM-VLM Integration for Grounded Robotic Manipulation
Oct 8, 2026Lonnie Tankersley, Noorbakhsh Amiri Golilarz, Soroush KorivandRobotic Manipulation Robot Task Planning
dept. of Aerospace Engineering, Mississippi State University, Starkville, MS · dept. of Computer Science, The University of Alabama, Tuscaloosa, AL · dept. of Mechanical Engineering, Mississippi State University, Starkville, MS
SIMMER: Benchmarking Latent Failures in LLM Executable Planning with a World Model
Jun 12, 2026Xiaoxin Lu, Ranran Haoran Zhang, Rui ZhangLarge Language Model-Based Robot Planning LLM Planning
The Pennsylvania State University, State College, PA, USA

Sep 7, 2026J/K move · Enter open · S save

How Long Until Your Robot Ignores You? A Safety Benchmark for LLM Orchestrators in Human-Humanoid Collaboration

Aulon Bajrami, Mohamed Elshamouty, Werner Kraus

Fraunhofer IPA Stuttgart, Germany

Robot Safety Human-Robot Collaboration LLM Safety Evaluation AI Agent Safety Benchmarks LLM Agent Safety

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly employed to orchestrate robot behavior through natural-language interfaces, yet no benchmark exists to evaluate their reliability as safety-aware decision makers in human-humanoid collaboration. Unlike deterministic safety systems that enforce binary allow/deny decisions, LLM-based orchestrators exhibit a compliance spectrum ranging from overcompliance (refusing safe actions) to full safety violations. This paper introduces the first safety benchmarking environment for LLM orchestrators in human-humanoid collaboration, built on a Model Context Protocol (MCP)-based architecture with safety invariants grounded in ISO 10218-2:2025 protective measures. The benchmark defines five testable safety invariants, a four-level compliance taxonomy (correct compliance, overcompliance, undercompliance, full violation), and a three-layer evaluation pipeline (text prompting, simulated sensor-actuator loops, and physical validation on a Unitree G1 EDU humanoid). We report Layer-1 results: three cloud backends (Claude Haiku 4.5, GPT-4o-mini, Gemini 2.5 Flash) and a local open-weights baseline (qwen3:8b) across 40 100-turn sessions under full-context and sliding-window budget conditions, while the simulation and physical layers remain ongoing. We find that (1) model family determines the safety floor, as Claude and Gemini remain at or near zero violations while GPT-4o-mini commits up to 13 per session, (2) context management dissociates two failure axes, reducing mean behavioral issues by 42-57% for every cloud backend while nearly doubling GPT-4o-mini's violations (3.8 to 7.2 per session), and (3) proportional compliance, clamping movement speed to the rule-specified maximum rather than refusing, emerges consistently only in Gemini; the preliminary simulation layer reproduces the model ranking and the GPT-4o-mini failure-mode inversion.

Abstract

Large Language Model-Based Robot Planning

Explore similar work

How Long Until Your Robot Ignores You? A Safety Benchmark for LLM Orchestrators in Human-Humanoid Collaboration

Embodied Zero-Shot Task Planning via Hardware-Aware LLM-VLM Integration for Grounded Robotic Manipulation

SIMMER: Benchmarking Latent Failures in LLM Executable Planning with a World Model