AI Risk Management

Ask about this topic

Overview

The field combines lifecycle frameworks such as the NIST AI Risk Management Framework and hazard analysis methods like HARA and STPA with threat modeling, red-teaming, statistical certification, and domain-specific testing in areas such as healthcare, finance, and autonomous systems. Researchers use incident databases, risk taxonomies, benchmarks, and deployment simulations to connect system failures to real-world harms, while governance work translates findings into accountability, standards, controls, and incident response. For agentic systems, approaches include least-privilege access, runtime monitoring, limits on delegated authority, cumulative risk budgets, and insurance. Open questions include whether evaluations predict harms in real deployments, how to measure uncertain and system-level effects, and how to balance safeguards with usefulness, fairness, and organizational incentives.

Latest papers

CardsList

108 papers11 papers in the last four weeks, down 27% on the four weeks before. 0.1% of all new papers.

Apr 13, 2026

PRISM Risk Signal Framework: Hierarchy-Based Red Lines for AI Behavioral Risk

Current approaches to AI safety define red lines at the case level: specific prompts, specific outputs, specific harms. This paper argues that red lines can be set more fundamentally -- at the level of value, evidence, and source hierarchies that govern AI reasoning. Using the PRISM (Profile-based Reasoning Integrity Stack Measurement) framework, we define a taxonomy of 27 behavioral risk signals derived from structural anomalies in how AI systems prioritize values (L4), weight evidence types (L3), and trust information sources (L2). Each signal is evaluated through a dual-threshold principle combining absolute rank position and relative win-rate gap, producing a two-tier classification (Confirmed Risk vs. Watch Signal). The hierarchy-based approach offers three advantages over case-specific red lines: it is anticipatory rather than reactive (detecting dangerous reasoning structures before they produce harmful outputs), comprehensive rather than enumerative (a single value-hierarchy signal subsumes an unlimited number of case-specific violations), and measurable rather than subjective (grounded in empirical forced-choice data). We demonstrate the framework's detection capacity using approximately 397,000 forced-choice responses from 7 AI models across three Authority Stack layers, showing that the signal taxonomy successfully discriminates between models with structurally extreme profiles, models with context-dependent risk, and models with balanced hierarchies.

Seulki Lee

AI Integrity Organization (AIO), Geneva, Switzerland

AI Safety EvaluationAI Risk Management
AskPDF
Apr 7, 2026

Who Governs the Machine? A Machine Identity Governance Taxonomy (MIGT) for AI Systems Operating Across Enterprise and Geopolitical Boundaries

The governance of artificial intelligence has a blind spot: the machine identities that AI systems use to act. AI agents, service accounts, API tokens, and automated workflows now outnumber human identities in enterprise environments by ratios exceeding 80 to 1, yet no integrated framework exists to govern them. A single ungoverned automated agent produced $5.4-10 billion in losses in the 2024 CrowdStrike outage; nation-state actors including Silk Typhoon and Salt Typhoon have operationalized ungoverned machine credentials as primary espionage vectors against critical infrastructure. This paper makes four original contributions. First, the AI-Identity Risk Taxonomy (AIRT): a comprehensive enumeration of 37 risk sub-categories across eight domains, each grounded in documented incidents, regulatory recognition, practitioner prevalence data, and threat intelligence. Second, the Machine Identity Governance Taxonomy (MIGT): an integrated six-domain governance framework simultaneously addressing the technical governance gap, the regulatory compliance gap, and the cross-jurisdictional coordination gap that existing frameworks address only in isolation. Third, a foreign state actor threat model for enterprise identity governance, establishing that Silk Typhoon, Salt Typhoon, Volt Typhoon, and North Korean AI-enhanced identity fraud operations have already operationalized AI identity vulnerabilities as active attack vectors. Fourth, a cross-jurisdictional regulatory alignment structure mapping enterprise AI identity governance obligations under EU, US, and Chinese frameworks simultaneously, identifying irreconcilable conflicts and providing a governance mechanism for managing them. A four-phase implementation roadmap translates the MIGT into actionable enterprise programs.

Andrew Kurtz, Klaudia Krawiecka

AI GovernanceCybersecurityAI Agent SecurityAI Risk Management
AskPDF
Apr 5, 2026

Quantifying Trust: Financial Risk Management for Trustworthy AI Agents

Prior work on trustworthy AI emphasizes model-internal properties such as bias mitigation, adversarial robustness, and interpretability. As AI systems evolve into autonomous agents deployed in open environments and increasingly connected to payments or assets, the operational meaning of trust shifts to end-to-end outcomes: whether an agent completes tasks, follows user intent, and avoids failures that cause material or psychological harm. These risks are fundamentally product-level and cannot be eliminated by technical safeguards alone because agent behavior is inherently stochastic. To address this gap between model-level reliability and user-facing assurance, we propose a complementary framework based on risk management. Drawing inspiration from financial underwriting, we introduce the Agentic Risk Standard (ARS), a payment settlement standard for AI-mediated transactions. ARS integrates risk assessment, underwriting, and compensation into a single transaction framework that protects users when interacting with agents. Under ARS, users receive predefined and contractually enforceable compensation in cases of execution failure, misalignment, or unintended outcomes. This shifts trust from an implicit expectation about model behavior to an explicit, measurable, and enforceable product guarantee. We also present a simulation study analyzing the social benefits of applying ARS to agentic transactions. ARS's implementation can be found at https://github.com/t54-labs/AgenticRiskStandard.

Wenyue Hua, Tianyi Peng, Chi Wang +4

Microsoft Research · Columbia University · Google DeepMind +2

AI Risk ManagementTrust in AIAI Agent Reliability
AskPDF
Apr 1, 2026

When AI Gets it Wrong: Reliability and Risk in AI-Assisted Medication Decision Systems

Artificial intelligence (AI) systems are increasingly integrated into healthcare and pharmacy workflows, supporting tasks such as medication recommendations, dosage determination, and drug interaction detection. While these systems often demonstrate strong performance under standard evaluation metrics, their reliability in real-world decision-making remains insufficiently understood. In high-risk domains such as medication management, even a single incorrect recommendation can result in severe patient harm. This paper examines the reliability of AI-assisted medication systems by focusing on system failures and their potential clinical consequences. Rather than evaluating performance solely through aggregate metrics, this work shifts attention towards how errors occur and what happens when AI systems produce incorrect outputs. Through a series of controlled, simulated scenarios involving drug interactions and dosage decisions, we analyse different types of system failures, including missed interactions, incorrect risk flagging, and inappropriate dosage recommendations. The findings highlight that AI errors in medication-related contexts can lead to adverse drug reactions, ineffective treatment, or delayed care, particularly when systems are used without sufficient human oversight. Furthermore, the paper discusses the risks of over-reliance on AI recommendations and the challenges posed by limited transparency in decision-making processes. This work contributes a reliability-focused perspective on AI evaluation in healthcare, emphasising the importance of understanding failure behavior and real-world impact. It highlights the need to complement traditional performance metrics with risk-aware evaluation approaches, particularly in safety-critical domains such as pharmacy practice.

Khalid Adnan Alsayed

Founder, Ducaltus | AI Assurance & Governance; BSc (Hons) Artificial Intelligence; School of Computing, Engineering & Digital Technologies; Teesside University, UK

Clinical Decision SupportAI Risk ManagementHuman Oversight of AI
AskPDF
Mar 27, 2026

A federated architecture for sector-led AI governance: lessons from India

Purpose: India has adopted a vertical, sector-led AI governance strategy. While promoting innovation, such a light-touch approach risks policy fragmentation. This paper aims to propose a cohesive "whole-of-government" architecture to mitigate these risks and connect policy goals with a practical implementation plan. Design/methodology/approach: The paper applies an established five-layer conceptual framework to the Indian context. First, it constructs a national architecture for overall governance. Second, it uses a detailed case study on AI incident management to validate and demonstrate the architecture's practical utility in designing a specific, operational system. Findings: The paper develops two actionable architectures. The primary model assigns clear governance roles to India's key institutions. The second is a detailed, federated architecture for national AI Incident Management. It addresses the data silo problem by using a common national standard that allows sector-specific data collection while facilitating cross-sectoral analysis. Practical implications: The proposed architectures offer a clear and predictable roadmap for India's policymakers, regulators and industry to accelerate the national AI governance agenda. Social implications: By providing a systematic path from policy to practice, the architecture builds public trust. This structured approach ensures accountability and aligns AI development with societal values. Originality/value: This paper proposes a detailed operational architecture for India's "whole-of-government" approach to AI. It offers a globally relevant template for any nation pursuing a sector-led governance model, providing a clear implementation plan. Furthermore, the proposed federated architecture demonstrates how adopting common standards can enable cross-border data aggregation and global sectoral risk analysis without centralising control.

Avinash Agarwal, Manisha J. Nene

Department of Telecommunications, Ministry of Communications, Sanchar Bhawan, New Delhi, India · Defence Institute of Advanced Technology, Ministry of Defence, Girinagar, Pune, India

AI GovernanceAI Risk Management
AskPDF
Mar 24, 2026

From the AI Act to a European AI Agency: Completing the Union's Regulatory Architecture

As artificial intelligence (AI) technologies continue to advance, effective risk assessment, regulation, and oversight are necessary to ensure that AI development and deployment align with ethical principles while preserving innovation and economic competitiveness. The adoption of the EU AI Act marks an important step in this direction, establishing a harmonised legal framework that includes detailed provisions on AI governance, as well as the creation of the European AI Office. This paper revisits the question of whether a more robust supranational agency dedicated to AI is still warranted and explores how such a body could enhance policy coherence, improve risk assessment capacities, and foster international cooperation. It also argues that a strengthened EU-level agency would also serve the Union's strategic aim of securing digital and technological sovereignty.

Georgios Pavlidis

Associate Professor of International and EU Law, Director, Jean Monnet Centre of Excellence ‘AI-2-TRACE-CRIME’ Neapolis University Pafos (NUP), Cyprus.

AI GovernanceAI Risk Management
AskPDF
Mar 20, 2026

An Agentic Multi-Agent Architecture for Cybersecurity Risk Management

Getting a real cybersecurity risk assessment for a small organization is expensive -- a NIST CSF-aligned engagement runs $15,000 on the low end, takes weeks, and depends on practitioners who are genuinely scarce. Most small companies skip it entirely. We built a six-agent AI system where each agent handles one analytical stage: profiling the organization, mapping assets, analyzing threats, evaluating controls, scoring risks, and generating recommendations. Agents share a persistent context that grows as the assessment proceeds, so later agents build on what earlier ones concluded -- the mechanism that distinguishes this from standard sequential agent pipelines. We tested it on a 15-person HIPAA-covered healthcare company and compared outputs to independent assessments by three CISSP practitioners -- the system agreed with them 85% of the time on severity classifications, covered 92% of identified risks, and finished in under 15 minutes. We then ran 30 repeated single-agent assessments across five synthetic but sector-realistic organizational profiles in healthcare, fintech, manufacturing, retail, and SaaS, comparing a general-purpose Mistral-7B against a domain fine-tuned model. Both completed every run. The fine-tuned model flagged threats the baseline could not see at all: PHI exposure in healthcare, OT/IIoT vulnerabilities in manufacturing, platform-specific risks in retail. The full multi-agent pipeline, however, failed every one of 30 attempts on a Tesla T4 with its 4,096-token default context window -- context capacity, not model quality, turned out to be the binding constraint.

Ravish Gupta, Saket Kumar, Shreeya Sharma +2

BigCommerce · Lead Software Engineer, BigCommerce; IEEE Senior Member · University at Buffalo, The State University of New York, Buffalo, NY, USA +4

CybersecurityAI Risk ManagementMulti-Agent LLM SystemsLLMs for CybersecurityLLM Agent Evaluation
AskPDF
Mar 19, 2026

Soft-Label Governance for Distributional Safety in Multi-Agent Systems

Multi-agent AI systems exhibit emergent risks that no single agent produces in isolation. Existing safety frameworks rely on binary classifications of agent behavior, discarding the uncertainty inherent in proxy-based evaluation. We introduce SWARM (System-Wide Assessment of Risk in Multi-agent systems), a simulation framework that replaces binary good/bad labels with soft probabilistic labels p=P(v=+1)∈[0,1]p = P(v{=}+1) \in [0,1], enabling continuous-valued payoff computation, toxicity measurement, and governance intervention. SWARM implements a modular governance engine with configurable levers (transaction taxes, circuit breakers, reputation decay, and random audits) and quantifies their effects through probabilistic metrics including expected toxicity E[1−p∣accepted]\mathbb{E}[1{-}p \mid \text{accepted}] and quality gap E[p∣accepted]−E[p∣rejected]\mathbb{E}[p \mid \text{accepted}] - \mathbb{E}[p \mid \text{rejected}]. Across seven scenarios with five-seed replication, strict governance reduces welfare by over 40% without improving safety. In parallel, aggressively internalizing system externalities collapses total welfare from a baseline of +262+262 down to −67-67, while toxicity remains invariant. Circuit breakers require careful calibration; overly restrictive thresholds severely diminish system value, whereas an optimal threshold balances moderate welfare with minimized toxicity. Companion experiments show soft metrics detect proxy gaming by self-optimizing agents passing conventional binary evaluations. This basic governance layer applies to live LLM-backed agents (Concordia entities, Claude, GPT-4o Mini) without modification. Results show distributional safety requires continuous risk metrics and governance lever calibration involves quantifiable safety-welfare tradeoffs. Source code and project resources are publicly available at https://www.swarm-ai.org/.

Aizierjiang Aiersilan, Raeli Savitt

SWARM AI Safety, New York, USA

AI Agent GovernanceAI Risk ManagementMulti-Agent SystemsAI Safety Evaluation
AskPDF
Mar 16, 2026

Safety Case Patterns for VLA-based driving systems: Insights from SimLingo

Vision-Language-Action (VLA)-based driving systems represent a significant paradigm shift in autonomous driving since, by combining traffic scene understanding, linguistic interpretation, and action generation, these systems enable more flexible, adaptive, and instruction-responsive driving behaviors. However, despite their growing adoption and potential to support socially responsible autonomous driving as well as understanding high-level human instructions, VLA-based driving systems may exhibit new types of hazardous behaviors. For instance, the integration of open-ended natural language inputs (e.g., user or navigation instructions) into the multimodal control loop may lead to unpredictable and unsafe behaviors that could endanger vehicle occupants and pedestrians. Hence, assuring the safety of these systems is crucial to help build trust in their operations. To support this, we propose a novel safety case design approach called RAISE. Our approach introduces novel patterns tailored to instruction-based driving systems such as VLA-based driving systems, an extension of Hazard Analysis and Risk Assessment (HARA) detailing safe scenarios and their outcomes, and a design technique to create the safety cases of VLA-based driving systems. A case study on SimLingo illustrates how our approach can be used to construct rigorous, evidence-based safety claims for this emerging class of autonomous driving systems.

Gerhard Yu, Fuyuki Ishikawa, Oluwafemi Odu +1

York University, North York ON M3J 1P3, Canada · National Institute of Informatics, Chiyoda City Tokyo 101-8430, Japan

Autonomous Driving Safety EvaluationAutonomous DrivingSafety Evaluation of VLA ModelsAI Risk Management
AskPDF
Mar 16, 2026

Governing frontier general-purpose AI in the public sector: adaptive risk management and policy capacity under uncertainty through 2030

The governance of frontier general-purpose artificial intelligence has become a public-sector problem of institutional design, not merely a technical issue of model performance. Recent evidence indicates that AI capabilities are advancing rapidly, though unevenly, while knowledge about harms, safeguards, and effective interventions remains partial and lagged. This combination creates a difficult policy condition: governments must decide under uncertainty, across multiple plausible trajectories of progress through 2030, and in environments where adoption outcomes depend on organizational routines, data arrangements, accountability structures, and public values. This article argues that public governance for frontier AI should be based on adaptive risk management, scenario-aware regulation, and sociotechnical transformation rather than static compliance models. Drawing on the International AI Safety Report 2026, OECD foresight and policy documents, and recent scholarship in digital government, the article first reconstructs the conceptual foundations of the 'evidence dilemma', differentiated AI risk categories, and the limits of prediction. It then examines how AI adoption in government depends on organizational redesign, public-sector institutional dynamics, and data collaboration capacity. On that basis, it proposes an adaptive governance framework for public institutions that integrates capability monitoring, risk tiering, conditional controls, institutional learning, and standards-based interoperability. The article concludes that effective AI governance requires stronger policy capacity, clearer allocation of responsibility, and governance mechanisms that remain robust across divergent technological futures.

Fabio Correa Xavier

AI GovernanceAI Risk Management
AskPDF
Mar 4, 2026

Towards automated data analysis: A guided framework for LLM-based risk estimation

Large Language Models (LLMs) are increasingly integrated into critical decision-making pipelines, a trend that raises the demand for robust and automated data analysis. Current approaches to dataset risk analysis are limited to manual auditing methods which involve time-consuming and complex tasks, whereas fully automated analysis based on Artificial Intelligence (AI) suffers from hallucinations and issues stemming from AI alignment. To this end, this work proposes a framework for dataset risk estimation that integrates Generative AI under human guidance and supervision, aiming to set the foundations for a future automated risk analysis paradigm. Our approach utilizes LLMs to identify semantic and structural properties in database schemata, subsequently propose clustering techniques, generate the code for them and finally interpret the produced results. The human supervisor guides the model on the desired analysis and ensures process integrity and alignment with the task's objectives. A proof of concept is presented to demonstrate the feasibility of the framework's utility in producing meaningful results in risk assessment tasks.

Panteleimon Rodis

AI Risk ManagementData Analysis AgentsHuman-in-the-Loop AI
AskPDF
Mar 4, 2026

When AI Fails, What Works? A Data-Driven Taxonomy of Real-World AI Risk Mitigation Strategies

Large language models (LLMs) are increasingly embedded in high-stakes workflows, where failures propagate beyond isolated model errors into systemic breakdowns that can lead to legal exposure, reputational damage, and material financial losses. Building on this shift from model-centric risks to end-to-end system vulnerabilities, we analyze real-world AI incident reporting and mitigation actions to derive an empirically grounded taxonomy that links failure dynamics to actionable interventions. Using a unified corpus of 9,705 media-reported AI incident articles, we extract explicit mitigation actions from 6,893 texts via structured prompting and then systematically classify responses to extend MIT's AI Risk Mitigation Taxonomy. Our taxonomy introduces four new mitigation categories, including 1) Corrective and Restrictive Actions, 2) Legal/Regulatory and Enforcement Actions, 3) Financial, Economic, and Market Controls, and 4) Avoidance and Denial, capturing response patterns that are becoming increasingly prevalent as AI deployment and regulation evolve. Quantitatively, we label the mitigation dataset with 32 distinct labels, producing 23,994 label assignments; 9,629 of these reflect previously unseen mitigation patterns, yielding a 67% increase of the original subcategory coverage and substantially enhancing the taxonomy's applicability to emerging systemic failure modes. By structuring incident responses, the paper strengthens "diagnosis-to-prescription" guidance and advances continuous, taxonomy-aligned post-deployment monitoring to prevent cascading incidents and downstream impact.

Evgenija Popchanovska, Ana Gjorgjevikj, Maryan Rizinski +3

Faculty of Computer Science and Engineering, Ss. Cyril and Methodius University, 1000 Skopje, North Macedonia. · Department of Computer Science, Metropolitan College, Boston University, Boston, MA 02215, USA.

AI Risk Management
AskPDF
Mar 4, 2026

Measuring AI R&D Automation

The automation of AI R&D (AIRDA) could have significant implications, but its extent and ultimate effects remain uncertain. We need empirical data to resolve these uncertainties, but existing data (primarily capability benchmarks) may not reflect real-world automation or capture its broader consequences, such as whether AIRDA accelerates capabilities more than safety progress or whether our ability to oversee AI R&D can keep pace with its acceleration. To address these gaps, this work proposes metrics to track the extent of AIRDA and its effects on AI progress and oversight. The metrics span dimensions such as capital share of AI R&D spending, researcher time allocation, and AI subversion incidents, and could help decision makers understand the potential consequences of AIRDA, implement appropriate safety measures, and maintain awareness of the pace of AI development. We recommend that companies and third parties (e.g. non-profit research organisations) start to track these metrics, and that governments support these efforts.

Alan Chan, Ranay Padarath, Joe Kwon +2

GovAI · University of Oxford

AI Risk ManagementHuman Oversight of AIAI Safety Evaluation
AskPDF
Feb 27, 2026

The AI Transformation Gap Index (AITG): An Empirical Framework for Measuring AI Transformation Opportunity, Disruption Risk, and Value Creation at the Industry and Firm Level

Despite the scale of capital being deployed toward AI initiatives, no empirical framework currently exists for benchmarking where a firm stands relative to competitors in AI readiness and deployment, or for translating that position into auditable financial outcomes. In practice, private equity deal teams, management consultants, and corporate strategists have relied on qualitative judgment and ad-hoc maturity labels; tools that are neither comparable across industries nor grounded in observable economic data. This paper introduces the AI Transformation Gap Index (AITG), a composite empirical framework that measures the distance between a firm's current AI deployment and a time varying, industry constrained capability frontier, then maps that distance to dollar denominated value creation, execution feasibility under uncertainty, and competitive disruption risk. Five linked modules address this gap: cross industry normalization (IASS), a dynamic capability ceiling that evolves with frontier capabilities (AFC), trajectory based firm scoring with integrated execution risk (IFS), a CES bottleneck value decomposition mapping gap scores to enterprise value (VCB), and a competitive hazard measure for inaction (ADRI). I calibrate the framework for 22 industry verticals and apply it to 14 public companies using public filings. A retrospective construct validity exercise correlating AITG scores with observed EBITDA margin expansion yields Spearman rho_s = 0.818 (n = 10), directionally consistent with predictions though insufficient for causal identification. A counterintuitive result emerges: the largest AI transformation gaps do not produce the highest value density, because implementation friction, CES bottlenecks, and timing lags erode the theoretical upside of wide gaps.

Dean Barr

Applied AI Researcher

Competitive AnalysisAI Risk Management
AskPDF
Feb 22, 2026

Quantifying Automation Risk in High-Automation AI Systems: A Bayesian Framework for Failure Propagation and Optimal Oversight

Organizations across finance, healthcare, transportation, content moderation, and critical infrastructure are rapidly deploying highly automated AI systems, yet they lack principled methods to quantify how increasing automation amplifies harm when failures occur. We propose a parsimonious Bayesian risk decomposition expressing expected loss as the product of three terms: the probability of system failure, the conditional probability that a failure propagates into harm given the automation level, and the expected severity of harm. This framework isolates a critical quantity -- the conditional probability that failures propagate into harm -- which captures execution and oversight risk rather than model accuracy alone. We develop complete theoretical foundations: formal proofs of the decomposition, a harm propagation equivalence theorem linking the harm propagation probability to observable execution controls, risk elasticity measures, efficient frontier analysis for automation policy, and optimal resource allocation principles with second-order conditions. We motivate the framework with an illustrative case study of the 2012 Knight Capital incident ($440M loss) as one instantiation of a broadly applicable failure pattern, and characterize the research design required to empirically validate the framework at scale across deployment domains. This work provides the theoretical foundations for a new class of deployment-focused risk governance tools for agentic and automated AI systems.

Vishal Srivastava, Tanmay Sah

Whiting School of Engineering, Johns Hopkins University, Baltimore, MD 21218 · Harrisburg University of Science and Technology, Harrisburg, PA

AI Risk ManagementHuman Oversight of AI
AskPDF
Feb 16, 2026

Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report v1.5

To understand and identify the unprecedented risks posed by rapidly advancing artificial intelligence (AI) models, Frontier AI Risk Management Framework in Practice presents a comprehensive assessment of their frontier risks. As Large Language Models (LLMs) general capabilities rapidly evolve and the proliferation of agentic AI, this version of the risk analysis technical report presents an updated and granular assessment of five critical dimensions: cyber offense, persuasion and manipulation, strategic deception, uncontrolled AI R&D, and self-replication. Specifically, we introduce more complex scenarios for cyber offense. For persuasion and manipulation, we evaluate the risk of LLM-to-LLM persuasion on newly released LLMs. For strategic deception and scheming, we add the new experiment with respect to emergent misalignment. For uncontrolled AI R&D, we focus on the ``mis-evolution'' of agents as they autonomously expand their memory substrates and toolsets. Besides, we also monitor and evaluate the safety performance of OpenClaw during the interaction on the Moltbook. For self-replication, we introduce a new resource-constrained scenario. More importantly, we propose and validate a series of robust mitigation strategies to address these emerging threats, providing a preliminary technical and actionable pathway for the secure deployment of frontier AI. This work reflects our current understanding of AI frontier risks and urges collective action to mitigate these challenges.

Dongrui Liu, Yi Yu, Jie Zhang +18

Shanghai AI Laboratory

AI Risk ManagementAI Agent SafetyLLM Safety EvaluationStrategic Reasoning Risks in Language Models
AskPDF
Feb 11, 2026

A Patient Simulation Framework for Risk Assessment of Conversational Healthcare AI: Evaluation of an Antidepressant Decision Aid

Objective: This study develops and validates a patient simulation framework that aligns with the National Institute of Standards and Technology AI Risk Management Framework MAP and MEASURE functions, providing an empirical basis for identifying and characterizing performance risks in conversational clinical AI across medical, linguistic, and behavioral patient variation. We applied the framework to a conversational decision aid for antidepressant selection in major depressive disorder. Methods: The simulator integrates three profile dimensions: (1) medical profiles constructed from All of Us electronic health records using risk-ratio gating; (2) linguistic profiles modeling a health literacy gradient and condition-specific communication; and (3) behavioral profiles representing cooperative, distracted, and adversarial engagement. We generated 500 simulated conversations and evaluated profile fidelity through human annotation and a large language model (LLM) judge, then assessed downstream effects on the AI Decision Aid's concept retrieval and antidepressant recommendations. Results: The patient simulator expressed medical concepts with high fidelity (96.6% accurate across 8,210 concepts), with human inter-annotator agreement of 0.73κκ and LLM-judge agreement against human annotators of 0.78κκ, both indicating substantial agreement. Behavioral profiles were reliably distinguished (0.93κκ, near perfect agreement), and linguistic profiles showed substantial agreement at the lower bound of the substantial range (0.61κκ), adequate to support profile-level analysis. The framework revealed monotonic degradation in AI Decision Aid performance across the health literacy gradient. Rank-1 concept retrieval increased from 47.6% for limited health literacy to 81.9% for proficient health literacy, with corresponding declines in antidepressant recommendation accuracy.

Md Tanvir Rouf Shawon, Mohammad Sabik Irbaz, Hadeel R. A. Elyazori +6

Department of Computer Science, George Mason University, Fairfax, VA, United States · Department of Information Sciences and Technology, George Mason University, Fairfax, VA, United States · Department of Health Administration and Policy, George Mason University, Fairfax, VA, United States +1

AI Risk ManagementHealthcareClinical Decision Support
AskPDF
Oct 29, 2024

From Silos to Systems: Process-Oriented Hazard Analysis for AI Systems

To effectively address potential harms from Artificial Intelligence (AI) systems, it is essential to identify and mitigate system-level hazards. Current analysis approaches focus on individual components of an AI system, like training data or models, in isolation, overlooking hazards from component interactions or how they are situated within a company's development process. To this end, we draw from the established field of system safety, which considers safety as an emergent property of the entire system. In this work, we translate System Theoretic Process Analysis (STPA) - a recognized system safety framework - for analyzing AI development and operation processes. We focus on systems that rely on machine learning algorithms and conduct STPA on three case studies involving linear regression, reinforcement learning, and transformer-based generative models. Our analysis explored how STPA's control and system-theoretic perspectives apply to AI systems and whether unique AI traits - such as model opacity, capability uncertainty, and output complexity - necessitate modifications to the framework. We find that the key concepts and steps of conducting an STPA apply to AI systems but require targeted adaptations to address AI-specific challenges that arise to differing degrees across three case studies. We present the Process-oriented Hazard Analysis for AI Systems (PHASE) as a guideline that adapts STPA concepts for AI. Applying and interpreting STPA using the PHASE guidelines enables four key affordances for analysts responsible for managing AI system harms: 1) detection of system-level hazards, including those from accumulation of disparate issues; 2) explicit acknowledgment of social factors contributing to algorithmic harms; 3) creation of traceable accountability chains between harms and those who can mitigate them; and 4) ongoing monitoring and mitigation of new hazards.

Shalaleh Rismani, Roel Dobbe, AJung Moon

McGill University · Montreal, Quebec, Canada · Delft University of Technology +1

AI Risk ManagementAI Safety
AskPDF