LLM-Generated Fault Scenarios for Evaluating Perception-Driven Lane Following in Autonomous Edge Systems
Organizations: Department of Electrical Engineering Hamm-Lippstadt University. of Applied Sciences (HSHL) Lippstadt, Germany
Abstract
Deploying autonomous vision systems on edge devices faces a critical challenge: resource constraints prevent real-time and predictable execution of comprehensive safety tests. Existing validation methods depend on static datasets or manual fault injection, failing to capture the diverse environmental hazards encountered in real-world deployment. To address this, we introduce a decoupled offline-online fault injection framework. This architecture separates the validation process into two distinct phases: a computationally intensive Offline Phase and a lightweight Online Phase. In the offline phase, we employ Large Language Models (LLMs) to semantically generate structured fault scenarios and Latent Diffusion Models (LDMs) to synthesize high-fidelity sensor degradations. These complex fault dynamics are distilled into a pre-computed lookup table, enabling the edge device to perform real-time fault-aware inference without running heavy AI models locally. We extensively validated this framework on a ResNet18 lane-following model across 460 fault scenarios. Results show that while the model achieves a baseline R^2 of approximately 0.85 on clean data, our generated faults expose significant robustness degradation, with RMSE increasing by up to 99% and within-0.10 localization accuracy dropping to as low as 31.0% under fog conditions, demonstrating the inadequacy of normal-data evaluation for real-world edge AI deployment.
Figures & tables
| Condition | R 2 | MSE | RMSE | Within-0.10 | Within-0.20 |
|---|---|---|---|---|---|
| Normal (baseline) | 0.85 | 0.011 | 0.105 | — | — |
| Fault range (3 subsets) | 0.755–0.840 | — | 0.180–0.209 | 0.31–0.45 | 0.66–0.75 |
| Best fault ( FOG_SIMULATION_007 ) | 0.835 | — | 0.181 | 0.445 | 0.752 |
| Worst fault ( FOG_SLIGHT_015 ) | 0.755 | — | 0.209 | 0.310 | 0.662 |
| Max degradation vs. baseline | 11% | — | 99% | — | — |
| a Metrics computed on lane-following test data across three fault categories. | |||||
Autonomous Driving Safety Evaluation
Autonomous driving safety evaluation studies how to measure, test, and improve the safety of self-driving systems in routine and hazardous conditions.
See all 152 papers