As AI assistants become integrated into safety engineering workflows for Physical AI systems, a critical question emerges: does AI assistance improve safety analysis quality, or introduce systematic blind spots that surface only through post-deployment incidents? This paper develops a formal framework for AI assistance in safety analysis. We first establish why safety engineering resists benchmark-driven evaluation: safety competence is irreducibly multidimensional, constrained by context-dependent correctness, inherent incompleteness, and legitimate expert disagreement. We formalize this through a five-dimensional competence framework capturing domain knowledge, standards expertise, operational experience, contextual understanding, and judgment. We introduce the competence shadow: the systematic narrowing of human reasoning induced by AI-generated safety analysis. The shadow is not what the AI presents, but what it prevents from being considered. We formalize four canonical human-AI collaboration structures and derive closed-form performance bounds, demonstrating that the competence shadow compounds multiplicatively to produce degradation far exceeding naive additive estimates. The central finding is that AI assistance in safety engineering is a collaboration design problem, not a software procurement decision. The same tool degrades or improves analysis quality depending entirely on how it is used. We derive non-degradation conditions for shadow-resistant workflows and call for a shift from tool qualification toward workflow qualification for trustworthy Physical AI.
Figures & tables
Dim.
Description
D
Domain knowledge. System physics, component interactions, failure propagation mechanisms, and domain-specific engineering principles.
S
Standards expertise. Knowledge of applicable safety standards (e.g., IEC 61508 [ 5 ] , ISO 26262 [ 29 ] , ISO 13482 [ 3 ] ), compliance procedures, and regulatory approval processes.
E
Operational experience. Accumulated knowledge from debugging production systems, investigating incidents, and observing how systems actually fail rather than how theory predicts they should.
C
Contextual understanding. Recognition of how the same technical failure has different severity depending on deployment environment, user population, and available mitigations.
J
Judgment. Risk calibration developed through empirical feedback loops between predictions and outcomes. Ability to assess severity and likelihood based on how predictions map to real-world consequences.
TABLE I: Dimensions of Safety Engineering Competence
Fig. 1: Complementary competence profiles forming a complete team. Panel (a): consider a certified safety engineer, strong in standards (S) and judgment (J), limited in operational exposure. Panel (b): field expert, strong in domain knowledge (D) and contextual understanding (C). Panel (c): test engineer, strong in operational experience (E) and contextual knowledge (C). Panel (d): combined team coverage (dashed envelope).
Fig. 2: Four canonical human-AI collaboration structures for AI-assisted safety analysis, differing in information flow and active shadow mechanisms.
Fig. 3: Waterfall analysis of the compounding shadow. Starting from the idealized case at 0.948, each mechanism sequentially reduces quality: scope framing (0.888), attention allocation (0.721), confidence asymmetry (0.700), and time compression (0.680). Final quality is 20% below human baseline.
Generative AI can produce engineering artefacts, but generation alone does not determine whether or how those artefacts should enter safety-critical workflows. This paper develops a formal framework for assigning AI participation and assurance at the level of individual workflow units. Each unit has a participation and assurance record covering its engineering requirement, an approved operational formalization where applicable, the applicable mechanism, fallback where applicable, evidence obligations and the applicable guarantee, plus a deployment-readiness status. The framework distinguishes deterministic verification, statistically calibrated admission, authorized human judgement supported by AI advice, authorized human adjudication of AI-produced artefacts, retained deterministic tool paths and explicit non-participation; these arrangements carry different kinds of guarantee rather than levels on a common scale. The framework also separates formalization fidelity from verifier soundness, provides a staged classification and readiness procedure, and derives conditions for comparing a gated AI-assisted unit with an incumbent process under recurring-population assumptions. We instantiate and apply the framework in an executed 17-unit wing-spar structural-analysis workflow combining deterministically gated AI-generated CAD, retained deterministic computation and human judgement. The AI-generated CAD program passed all 23 deterministic checks and was admitted at the first attempt. Favourable stress magnitudes did not suffice to pass the stress criteria where the predeclared mesh-convergence evidence was insufficient; those criteria were instead referred to engineering judgement. The case demonstrates selective AI participation and explicit evidence handling at unit level; no claim is made of workflow-level dependability, certification, structural safety or productivity.
Puxue Tan
School of Mechanical and Aerospace Engineering Queen’s University Belfast 50 Malone Road, Belfast, UK
Human feedback is critical for aligning AI systems to human values. As AI capabilities improve and AI is used to tackle more challenging tasks, verifying quality and safety becomes increasingly challenging. This paper explores how we can leverage AI to improve the quality of human oversight. We focus on an important safety problem that is already challenging for humans: fact-verification of AI outputs. We find that combining AI ratings and human ratings based on AI rater confidence is better than relying on either alone. Giving humans an AI fact-verification assistant further improves their accuracy, but the type of assistance matters. Displaying AI explanation, confidence, and labels leads to over-reliance, but just showing search results and evidence fosters more appropriate trust. These results have implications for Amplified Oversight -- the challenge of combining humans and AI to supervise AI systems even as they surpass human expert performance.
Rishub Jain, Sophie Bridgers, Lili Janzer +3
1Google DeepMind · 2Work done while previously at Google DeepMind
Incidents show that AI safety failures often arise across multiple layers. We present a safety-by-design assurance architecture combining model-level supervision, such as Scientist AI, with system-level controls over scaffolds and harnesses, independent verification, monitoring, and evidence infrastructure, supported by governance for accountability and evidence interoperability.
Qinghua Lu, Yoshua Bengio
CSIRO, Australia · LawZero, Canada · Université de Montréal, Canada +1