The Competence Shadow: Theory and Bounds of AI Assistance in Safety Engineering

Umair Siddique

cs.AIMar 26, 2026

The Competence Shadow: Theory and Bounds of AI Assistance in Safety Engineering

Authors: Umair Siddique

Organizations: Independent Research, Ottawa, ON, Canada

Abstract

As AI assistants become integrated into safety engineering workflows for Physical AI systems, a critical question emerges: does AI assistance improve safety analysis quality, or introduce systematic blind spots that surface only through post-deployment incidents? This paper develops a formal framework for AI assistance in safety analysis. We first establish why safety engineering resists benchmark-driven evaluation: safety competence is irreducibly multidimensional, constrained by context-dependent correctness, inherent incompleteness, and legitimate expert disagreement. We formalize this through a five-dimensional competence framework capturing domain knowledge, standards expertise, operational experience, contextual understanding, and judgment. We introduce the competence shadow: the systematic narrowing of human reasoning induced by AI-generated safety analysis. The shadow is not what the AI presents, but what it prevents from being considered. We formalize four canonical human-AI collaboration structures and derive closed-form performance bounds, demonstrating that the competence shadow compounds multiplicatively to produce degradation far exceeding naive additive estimates. The central finding is that AI assistance in safety engineering is a collaboration design problem, not a software procurement decision. The same tool degrades or improves analysis quality depending entirely on how it is used. We derive non-degradation conditions for shadow-resistant workflows and call for a shift from tool qualification toward workflow qualification for trustworthy Physical AI.

Figures & tables

Dim.	Description
D	Domain knowledge. System physics, component interactions, failure propagation mechanisms, and domain-specific engineering principles.
S	Standards expertise. Knowledge of applicable safety standards (e.g., IEC 61508 [ 5 ] , ISO 26262 [ 29 ] , ISO 13482 [ 3 ] ), compliance procedures, and regulatory approval processes.
E	Operational experience. Accumulated knowledge from debugging production systems, investigating incidents, and observing how systems actually fail rather than how theory predicts they should.
C	Contextual understanding. Recognition of how the same technical failure has different severity depending on deployment environment, user population, and available mitigations.
J	Judgment. Risk calibration developed through empirical feedback loops between predictions and outcomes. Ability to assess severity and likelihood based on how predictions map to real-world consequences.

TABLE I: Dimensions of Safety Engineering Competence

Fig. 1: Complementary competence profiles forming a complete team. Panel (a): consider a certified safety engineer, strong in standards (S) and judgment (J), limited in operational exposure. Panel (b): field expert, strong in domain knowledge (D) and contextual understanding (C). Panel (c): test engineer, strong in operational experience (E) and contextual knowledge (C). Panel (d): combined team coverage (dashed envelope).

Fig. 2: Four canonical human-AI collaboration structures for AI-assisted safety analysis, differing in information flow and active shadow mechanisms.

Fig. 3: Waterfall analysis of the compounding shadow. Starting from the idealized case at 0.948, each mechanism sequentially reduces quality: scope framing (0.888), attention allocation (0.721), confidence asymmetry (0.700), and time compression (0.680). Final quality is 20% below human baseline.

Metric	Serial	Indep.	Tool Aug.	HIE
Expected Quality	68%	88%	84%	90%
$\Delta$ vs. Baseline	$-17$ pp	$+3$ pp	$-1$ pp	$+5$ pp
Shadow Mechs. Active	4/4	1/4	$\varepsilon$ only	1/4

TABLE II: Four-structure comparison (

q_{h}=0.85

q_{AI}=0.65

)

Explore similar work

Dependable AI-Assisted Engineering: A Formal Framework for AI Participation and Assurance in Safety-Critical Workflows
Oct 2, 2026Puxue TanAI Assurance Human Oversight of AI
School of Mechanical and Aerospace Engineering Queen’s University Belfast 50 Malone Road, Belfast, UK
Human-AI Complementarity: A Goal for Amplified Oversight
Oct 30, 2025Rishub Jain, Sophie Bridgers, Lili Janzer +3Claim Verification Trust in AI
1Google DeepMind · 2Work done while previously at Google DeepMind
AI Safety: Not Optional, Not Later
Sep 9, 2026Qinghua Lu, Yoshua BengioAI Assurance AI Accountability
CSIRO, Australia · LawZero, Canada · Université de Montréal, Canada +1

Oct 2, 2026J/K move · Enter open · S save

Dependable AI-Assisted Engineering: A Formal Framework for AI Participation and Assurance in Safety-Critical Workflows

Puxue Tan

School of Mechanical and Aerospace Engineering Queen’s University Belfast 50 Malone Road, Belfast, UK

AI Assurance Human Oversight of AI

Open paper AskPDF arXiv

Generative AI can produce engineering artefacts, but generation alone does not determine whether or how those artefacts should enter safety-critical workflows. This paper develops a formal framework for assigning AI participation and assurance at the level of individual workflow units. Each unit has a participation and assurance record covering its engineering requirement, an approved operational formalization where applicable, the applicable mechanism, fallback where applicable, evidence obligations and the applicable guarantee, plus a deployment-readiness status. The framework distinguishes deterministic verification, statistically calibrated admission, authorized human judgement supported by AI advice, authorized human adjudication of AI-produced artefacts, retained deterministic tool paths and explicit non-participation; these arrangements carry different kinds of guarantee rather than levels on a common scale. The framework also separates formalization fidelity from verifier soundness, provides a staged classification and readiness procedure, and derives conditions for comparing a gated AI-assisted unit with an incumbent process under recurring-population assumptions. We instantiate and apply the framework in an executed 17-unit wing-spar structural-analysis workflow combining deterministically gated AI-generated CAD, retained deterministic computation and human judgement. The AI-generated CAD program passed all 23 deterministic checks and was admitted at the first attempt. Favourable stress magnitudes did not suffice to pass the stress criteria where the predeclared mesh-convergence evidence was insufficient; those criteria were instead referred to engineering judgement. The case demonstrates selective AI participation and explicit evidence handling at unit level; no claim is made of workflow-level dependability, certification, structural safety or productivity.

Abstract

Explore similar work

Dependable AI-Assisted Engineering: A Formal Framework for AI Participation and Assurance in Safety-Critical Workflows

Human-AI Complementarity: A Goal for Amplified Oversight

AI Safety: Not Optional, Not Later