Artificial Intelligence (AI) is increasingly integrated into complex sociotechnical systems, including Critical National Infrastructure (CNI), where harms emerge from interactions between technical, human, and organisational elements. Yet current AI evaluation remains model-centric, offering little insight into how observed behaviours might translate into system-level risk. We propose a framework that links structured hazard analysis, component-level testing, and probabilistic system modelling to bridge this gap. By providing a traceable pathway from model behaviour to system-level outcomes, the framework enables practitioners to answer the "so what?" of AI failures, quantify their systemic impact, and move toward evidence-based and anticipatory governance of AI in complex systems. Applied to the UK's Real Time Gross Settlement (RTGS) system as an illustrative worked example, we derive AI-driven loss scenarios using Systems Theoretic Process Analysis (STPA) and examine adversarial manipulation of LLM-based trading as one such loss scenario. Component-level experiments show that simple adversarial inputs induce measurable behavioural shifts where AI recommendations are followed. Under the component-to-system mapping used here for a financial contagion model, these shifts alter system resilience, increasing bank failures and lowering the threshold at which shocks lead to cascading disruption, particularly under widespread or monopolistic AI adoption.
Figures & tables
Figure 1: Framework developed in this work, exploring how old and new tools can be used to assess the risk of adoptions of a new technology in complex sociotechnical systems. It is modular and hence can be applied proportionately: practitioners may find it appropriate to finish at any of A, B or C depending on their findings, capabilities and risk tolerance. Finishing at A may be appropriate for paths to harm that do not warrant further technical investigation due to infeasibility, or due to the ease of implementing a mitigation regardless (e.g. staff training). Assessment of a specific harm may end at B if the component behaviour is evaluated to not be of concern, or projecting it into a statistical model is infeasible.
Figure 2: Tracing of high-level hazards into losses for the Real-Time Gross Settlement system (RTGS), surfaced via systems-theoretic process analysis. Hazards are system states that, when combined with unfavourable environmental conditions, produce losses. Losses relate to unacceptable outcomes to be avoided for the stakeholders of the system.
Figure 3: Scatter plot across the four asset types, demonstrating shift in asset allocation from Neutral sentiment across assets (blue) to Bearish sentiment for the shown asset (pink) and adversarial manipulation added to a distressed asset (yellow). The standard deviation of the asset allocation is shown on the y-axis, and the mean asset allocation on the x-axis. The asset value for the adversarial setting is averaged across all the attacks explored in this work. Claude 3.5 Haiku results only present 50 measurements for the adversarial setting as this model was retired during testing.
Figure 4: Plot of bank failure rate, absolute (left), and relative to baseline firesales (right). We observe that increased firesale intensity leads to heightened overall bank failure rates, and that adversarial scenarios yield the largest bank failure rates, as well as monopolistic adoption scenarios. This plot uses mapping K=1.0 and fh=0.05 .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Full table of all sub-hazards considered within this system, and the associated system constraints that must be enforced to avoid losses.
Figure 6: RTGS-level HCS, where the top-level controller, the BoE, controls: the main settlement system, SWIFT and CHAPS direct participants. The lowest level controlled state in this system is the transaction state within RTGS, i.e. the ongoing balances of participants, and the queued settlements.
Figure 7: Illustration of the control structure internal to a generalised bank, where a focus has been drawn on its trading activities. “Liquidity controller” is a generalised term, and can mean the treasurer.
Figure 8: Table presenting unsafe control actions that were analysed to derive AI-driven loss scenarios.
Figure 9: Scenarios derived under STPA of RTGS network, identified as being particularly tractable for the testing of systemic risk indicators. Loss scenarios 18.2.2-A and -B relate to the same UCA, and could be assessed with the same underlying AI component.
Figure 10: Scenarios derived under STPA of RTGS network, identified as being less tractable for the testing of systemic risk indicators.
Figure 11: Overlay of heuristic criteria used to downsample AI use cases in finance for, from the Bank of England and FCA 2024 survey.
Table 1: Table of preference manipulation attacks used in this work. Based off of adversarial search engine optimisation work ( Nestaas et al. 2024 )
To effectively address potential harms from Artificial Intelligence (AI) systems, it is essential to identify and mitigate system-level hazards. Current analysis approaches focus on individual components of an AI system, like training data or models, in isolation, overlooking hazards from component interactions or how they are situated within a company's development process. To this end, we draw from the established field of system safety, which considers safety as an emergent property of the entire system. In this work, we translate System Theoretic Process Analysis (STPA) - a recognized system safety framework - for analyzing AI development and operation processes. We focus on systems that rely on machine learning algorithms and conduct STPA on three case studies involving linear regression, reinforcement learning, and transformer-based generative models. Our analysis explored how STPA's control and system-theoretic perspectives apply to AI systems and whether unique AI traits - such as model opacity, capability uncertainty, and output complexity - necessitate modifications to the framework. We find that the key concepts and steps of conducting an STPA apply to AI systems but require targeted adaptations to address AI-specific challenges that arise to differing degrees across three case studies. We present the Process-oriented Hazard Analysis for AI Systems (PHASE) as a guideline that adapts STPA concepts for AI. Applying and interpreting STPA using the PHASE guidelines enables four key affordances for analysts responsible for managing AI system harms: 1) detection of system-level hazards, including those from accumulation of disparate issues; 2) explicit acknowledgment of social factors contributing to algorithmic harms; 3) creation of traceable accountability chains between harms and those who can mitigate them; and 4) ongoing monitoring and mitigation of new hazards.
Shalaleh Rismani, Roel Dobbe, AJung Moon
McGill University · Montreal, Quebec, Canada · Delft University of Technology +1
Organizations across finance, healthcare, transportation, content moderation, and critical infrastructure are rapidly deploying highly automated AI systems, yet they lack principled methods to quantify how increasing automation amplifies harm when failures occur. We propose a parsimonious Bayesian risk decomposition expressing expected loss as the product of three terms: the probability of system failure, the conditional probability that a failure propagates into harm given the automation level, and the expected severity of harm. This framework isolates a critical quantity -- the conditional probability that failures propagate into harm -- which captures execution and oversight risk rather than model accuracy alone. We develop complete theoretical foundations: formal proofs of the decomposition, a harm propagation equivalence theorem linking the harm propagation probability to observable execution controls, risk elasticity measures, efficient frontier analysis for automation policy, and optimal resource allocation principles with second-order conditions. We motivate the framework with an illustrative case study of the 2012 Knight Capital incident ($440M loss) as one instantiation of a broadly applicable failure pattern, and characterize the research design required to empirically validate the framework at scale across deployment domains. This work provides the theoretical foundations for a new class of deployment-focused risk governance tools for agentic and automated AI systems.
Vishal Srivastava, Tanmay Sah
Whiting School of Engineering, Johns Hopkins University, Baltimore, MD 21218 · Harrisburg University of Science and Technology, Harrisburg, PA
As automated decision-making and data-driven technologies pervade society and are used to manage consequential outcomes, understanding the technology's capabilities, limitations, and attendant risks in context requires analysis of full sociotechnical systems. Sociotechnical analysis of risks in highly complex systems provides clear lessons for the design and evaluation of AI systems, transcending a technical focus on reliable or "responsibly designed" components to understand risks at a systems level. Human-made catastrophes have been studied for decades because of the severity of these events: consider Chernobyl, Three Mile Island, Fukushima-Daiichi, Bhopal, the Challenger disaster. A common misconception is that these kinds of events are freak accidents, resulting from the inherently unforeseeable interactions in complex systems. Closer examination reveals that the risks and hazards were well-known beforehand but not acted upon due to social structural, political and economic factors. We outline several areas where the development and use of AI can benefit from learning these unlearned lessons: improved risk perception, communication, and analysis at the organizational level; traceability of requirements and responsibilities; and holistic approaches to responsibility and safety that include social and organizational dynamics as first-order engineering concerns. For each area, we offer concrete unlearned lessons and exemplify how they led to failure in prior accidents as well as examples of how these lessons remain unlearned for modern computing systems, particularly AI.
Joshua A. Kroll, Andrew Smart, R. Stuart Geiger +1
Naval Postgraduate School · Google Research · University of California, San Diego +1