cs.CYApr 9, 2026
SaveKeeping an Eye on AI: A Framework for Effective Human Oversight of AI Systems
Organizations: University College London · University of Freiburg · University of Queensland · Saarland University ·
Abstract
The use of Artificial Intelligence (AI) in high-risk, decision-making scenarios presents technical, safety, and normative challenges; problems that may only be ameliorated by human oversight. However, notions of human oversight lack a common foundational understanding: oversight architectures are not well defined, the roles involved remain unclear, and implementation steps are opaque. Hence, researchers and practitioners struggle to determine how to design, implement, and evaluate systems that enable effective human oversight. This paper advances a practical framework for effective human oversight of AI systems, based on a cross-disciplinary perspective that draws on insights from computer science, human-computer interaction, psychology, philosophy, and law. The core contributions are: (1) a foundational framework, with a working definition, architecture and processes for effective human oversight of AI systems; (2) an initial template for documenting oversight architectures and processes, applied to diverse domains; and (3) a synthesis of open research challenges that need to be considered in the emerging field of effective human oversight of AI systems.
Figures & tables
Figure 1: An architecture for human oversight. The two components of the architecture are the oversight layer and the task layer . The task layer (lower-left) reflects the socio-technical system that is to be overseen, involving some AI system executing or assisting a task with human users/operators (optional), and the organisation of the task. The oversight layer (upper-left) reflects the socio-technical system performing the oversight with a human overseer(s) who may be supported by technology (optional) to achieve oversight goals within an organisational setting. To the right we see boxes with illustrative examples of parts of both of these layers.
| Component | Example |
| Task Layer | |
| Object-Level Task | The task goal is to detect an acute haemorrhage in a head CT scan and provide a diagnosis, where potential risks include missed findings, delayed diagnosis, and unnecessary medical interventions |
| AI System | The AI tool that analyses CT scans, highlighting suspect regions, and providing haemorrhage-probability scores |
| Human(s) | The users of the AI tool, the radiologists or emergency clinicians who review AI flags, interpret images, and make time-critical decisions on medical intervention (n.b., fully-automated triage there could be minimal human involvement) |
| Organisation | The associated clinical protocols, IT systems (e.g., PACS – Picture Archiving and Communication System), and governance for the AI deployment |
Table 1: Simplified Diagnostic Decision-Making Scenario: Task & Oversight Layers
Figure 2: An Oversight Layer may need to be unpacked into different layers of oversight; there may be a Real-Time Layer (i.e., with overseers monitoring/intervening on outputs in real-time), a Systemic Layer (i.e., with developers monitoring/intervening, e.g. by overseeing AI functioning on an aggregate level), and a Compliance Layer (i.e., with compliance officers monitoring/intervening to enforce regulations/contracts). Note, these different layers of oversight will be affected by other even broader layers of risk-mitigation in AI deployment; e.g., legal provisions.
Figure 3: Human Oversight Process. Oversight layer with an observe-think-act process of monitoring, deciding, and intervention. While specific responsibilities change between layers in specific contexts, the general process is the same in all layers.
| Process | Design Options for Human Oversight | Example |
|---|---|---|
| Monitor | Support detection and interpretation of inaccurate and inadequate outputs; support data aggregation and visualisation | Radiologist monitors AI outputs and detects false positives/negatives; performance dashboards summarise trends for Clinical Safety Officers |
| Assess Risk | Tools to simulate consequences of inaccurate AI support for a larger scale of cases | Radiologist decides that inaccurate output is a patient safety risk; Clinical Safety Officer reviews emerging patterns, considers the clinical and operational impact of false positives/negatives (e.g., over/under treatment) |
| Decide | Offer clear decision guidelines for whether and when to intervene | Radiologist decides to override the AI output; Radiology Governance Committee limits AI use to certain patient groups |
| Intervene | Enable effective communication across oversight layers | Radiologist changes report generated by the AI tool; Clinical Lead suspends AI use for affected indications, informs Digital Health team, and reports to the vendor |
Table 2: Human oversight processes, their descriptions, and design options.
| Component | Description | |
| Task | Describe the object-level task to be performed: • primary goals : intended outcomes of the task such as timely decision making. • risks : potential failures or harms associated with the task, such as safety hazards. • outcomes : measurable results such as operational efficiency. | |
| AI system | Describe the AI system that supports or automates parts of this task: • core properties : main parts of the system architecture, such as models, platforms, etc. • functionality : what the system does, such as risk prediction. • inputs and outputs : data/information that it requires and produces. • level of autonomy : the degree of independence that the AI system has to perform functions without human intervention. • known limitations or failure modes : conditions under which the system may underperform. | |
| Human(s) | Identify the people directly interacting with, using, or affected by the AI system, such as operators or end-users. Describe their: • abilities/competencies : relevant skills and knowledge. • degree of discretion or authority : their power to make decisions independently. • responsibilities : duties and obligations assigned to the people. • intentions : goals and priorities when using the system. | |
| Organisation | Describe the organisational setting in which the task takes place, including: • hierarchical structures : reporting lines, responsibilities, team composition. • relevant rules, processes, and procedural constraints : standard operating procedures, compliance requirements, resource availabilities. |
| Component | Description | |
| Oversight Task | Describe the oversight task to be performed: • purpose : why oversight is needed, e.g. ensuring safety. • goals : specific objectives that oversight aims to achieve, such as detecting anomalies. • outcomes : measurable results of the oversight activity, such as reduced error rates. • risks : potential harms associated with the oversight task, such as unclear escalation. | |
| Oversight Technology (Optional) | Describe the technologies that support oversight, such as monitoring dashboards, warning systems, or audit logs. Describe their: • properties : key characteristics such as real-time monitoring capability. • functionalities : what the system does, such as display performance metrics. • failure modes : conditions under which these tools might fail, such as data latency. • system interfaces : how these tools interface with the AI system, such as API interaction. • signals : how they communicate relevant information to the human overseers, such as visual alerts, reports etc. | |
| Oversight Personnel | Describe the personnel responsible for oversight, including: • abilities/competencies : knowledge, skills, abilities required, such as technical expertise. • authority : decision-making power, such as the ability to halt the system. • responsibility : tasks overseers must perform, such as monitoring and intervention duties, decision thresholds, and escalation procedures. • intentions : goals and priorities guiding actions, such as complying with regulations. | |
| Oversight Organisation | Describe the organisational setting in which the oversight takes place, including: • composition of oversight : individual or team oversight; single organisation or distributed oversight. • rules and procedural constraints : standard operating procedures and audit protocols; • governance structures : committees or bodies allocating oversight responsibilities. • reporting mechanisms : incident logs, escalation pathways. • coordination : how oversight activities align with other teams (IT, legal, operations). • constraints : resource limitations or technological dependencies affecting feasibility. |
| Component | Description | |
| Monitor | Specify the types of monitoring activities and their temporal frequency: • signals : the signals received from the overseen layer. • interpretation : the process for interpreting signals. • frequency : how often monitoring occurs, such as continuous/real-time, scheduled, or conditional on other events. | |
| Assess Risk | Define the risk assessment process to be used by the oversight personnel. | |
| Decide | Define the decision-making process that the oversight personnel follow, including: • decision protocols : predefined rules or human judgment. • documentation : record rationale for accountability and audit purposes. | |
| Intervene | Specify intervention types and their temporal frequency (real-time, scheduled, or conditional). |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Object-Level Task | Describe the object-level task to be performed: • primary goals : The task is to assess applicant fit against job requirements. • risks : unfair discrimination of individual applicants or groups of applicants, i.e., applicant fit is judged to be lower than the respective applicants’ actual suitability. • outcomes : Time-to-hire; job performance of hired individuals in the future. | |
| AI system | Describe the AI system that supports or automates parts of this task: • core properties : The AI system is an automated job interview evaluation system. Applicants record videos of themselves responding to interview questions. There is a transcription module that transcribes applicant responses and an evaluation module that was trained on 1000 prior applicants and their responses. • functionality : The AI automatically assesses applicant suitability from 0 to 100 percent fit. • inputs and outputs : Input: applicant videos; Output: applicant fit score. • level of autonomy : The AI fully automates the applicant evaluation. • known limitations or failure modes : Low audio quality can undermine transcription accuracy; low audio quality paired with applicants whose first language is not English leads to even worse transcription accuracies which can reduce applicant suitability ratings. | |
| Human(s) | Identify the people directly interacting with, using, or affected by the AI system, such as operators or end-users. Describe their: • abilities/competencies : The operator is a hiring manager trained in assessing applicant fit. They have also received some training in the use of the AI system, e.g., on how to interpret the applicant suitability score. • degree of discretion or authority : The operator uses the outputs of the AI system for deciding who to invite to an in-person interview. They can also ignore the AI output for this task. • responsibilities : They are responsible for evaluating applicants, for ensuring that hiring processes follow legal standards, e.g., of non-discrimination. They also communicate with invited and rejected applicants. | |
| Organisation | Describe the organisational setting in which the task takes place, including: • hierarchical structures : The hiring manager is responsible for the successful and quick task completion, and they report to their group leader. • relevant rules and procedural constraints : The organization has a human resources group that defines the hiring processes. |
| Oversight Task | Describe the oversight task to be performed: • purpose : The evaluation AI may provide inaccurate results and under- or overestimate the actual applicant suitability especially when audio quality of applicant interviews is low. The evaluation AI may also produce discriminatory outputs if applicants do not have English as their first language. • goals : 1. Assess the adequacy of the AI evaluation module for individual applicants; 2. Assess whether the AI evaluation module is functioning properly on a group level; and 3. Assess whether the AI evaluation module provides discriminatory outputs. • risks : Inadequacy of using the AI evaluation may remain undetected, for instance, due to the large number of applicants applying and due to the limited time provided for real-time oversight. | |
| Oversight Technology (Optional) | Describe the technologies that support oversight, such as monitoring dashboards, warning systems, or audit logs. Describe their: • functionalities : Real-time oversight (i.e., hiring managers) has access to a warning tool that monitors word error rates in transcriptions and audio quality. Systemic oversight (i.e., technicians) has access to the benchmark dashboard that includes warning modules that highlight benchmarks that are out of bounds and that assess different fairness metrics. | |
| Oversight Personnel | Describe the personnel responsible for oversight, including: • composition : Real-time oversight personnel: a hiring manager overseeing AI use for individual applicants. Systemic oversight personnel: a technician overseeing the aggregate functioning of the AI system. Compliance oversight personnel: compliance officers assessing whether AI behavior reflects fairness issues. • abilities/competencies : Real-time oversight personnel is trained in assessing applicant fit and in using the AI system as decision support. Systemic oversight personnel is trained in assessing technical AI benchmarks. Compliance oversight personnel is trained in assessing AI fairness challenges. • responsibility : Real-time oversight: Their duty is to assess for each individual applicant whether the AI evaluation system is adequate to use. Systemic oversight: Their duty is to assess on a group level whether the system is functioning adequately. They monitor whether AI benchmarks are critically out of bounds. Compliance oversight: Their duty is to assess whether reported challenges reflect fairness issues. | |
| Oversight Organisation | Describe the organisational setting in which the oversight takes place, including: • reporting mechanisms : Real-time oversight reports to the systemic- and compliance oversights if they detect issues for individual applicants. Systemic oversight reports to the real-time and to compliance oversight personnel if they detect issues on a group level. Compliance oversight reports back to the other layers if critical fairness challenges arise. |
| Monitor | Specify the types of monitoring activities and their temporal frequency: • signals : Real-time oversight monitoring duties: Examine the word error rate for single individuals, check for audio quality issues in videos, check for accents and dialects that may undermine the quality of the AI evaluation module. Systemic oversight monitoring duties: monitor benchmarks over time such as shifts in suitability ratings depending on different subgroups of applicants. Compliance oversight monitoring duties: monitor and assess whether reported issues indeed represent fairness issues. | |
| Assess Risk | Define the risk assessment process to be used by the oversight personnel: • The main risk is that the AI evaluation module is inadequate to assess specific applicants due to the transcription module producing higher error rates for certain applicants and certain technical circumstances. | |
| Decide | Define the decision making process that the oversight personnel follow, including: • decision protocols : All oversight personnel is instructed to decide for interventions based on their respective oversight training, duties and based on their own discretion. • documentation : Oversight personnel is instructed to document by reporting to the other oversight layers. | |
| Intervene | Specify intervention types and their temporal frequency (real-time, scheduled, or conditional). • Real-time oversight: Ignore AI evaluation module and assess applicant suitability independently. • Systemic oversight: Provide a warning signal to real-time and compliance oversight; stop the AI evaluation module. • Compliance oversight: Provide a warning signal to the systemic and real-time oversight, stop the AI evaluation module and suggest solutions to fix fairness issues. |
| Object-Level Task | Describe the object-level task to be performed: • primary goals : The goals of object-level task include allocating aircraft to airspace to ensure the safety of people on board and on the ground, while also ensuring a smooth flow of air traffic (e.g. avoiding delays). Individually, controllers allocate paths to aircraft within their own air sector, but are also responsible for the safe handover of aircraft between their sector and that of other controllers. The task is accomplished through a distributed system which spans multiple controllers, interactive artifacts (e.g., flight strips, radar), and organisational structure. • risks : The main risks are: potential accidents, and delays in flights, with the constraint that this is a real-time scheduling problem that must be solved "in flight"—aircraft cannot be parked or paused. • outcomes : The outcomes are whether aircraft land safely and on time, and a successful transformation of the continuous flow of air traffic into a manageable sequence. | |
| AI system | Describe the AI system that supports or automates parts of this task: • core properties : There are several technological components in ATC, but some critical examples using AI include prediction of air traffic 20-30 minutes in advance, prediction of aircraft trajectory, collision prediction, and collision resolution (recommending conflict-free trajectories). • inputs and outputs : inputs such as flight schedules, information on delays, information on aircraft; outputs: prediction of aircraft trajectory. • level of autonomy : The AI can work completely autonomously, such that controllers do not have to rely on mental simulations of flight trajectories. • known limitations or failure modes : Failure can result when incorrect information is provided to the AI, when predictions of traffic, trajectories, and collisions are incorrect, or when collision resolution fails; e.g., solves one conflict but introduces another, or does not have access to necessary information that the controller has, including controller preferences. Moreover, automating some tasks can disrupt the working division of labor if it does not account for how different stakeholders coordinate through shared artifacts that they use to make decisions. | |
| Human(s) | Identify the people directly interacting with, using, or affected by the AI system, such as operators or end-users. Describe their: • abilities/competencies : Air traffic controllers, whose decisions are in real time and can have major safety and air traffic flow consequences. These are highly-trained professionals in the core task of air traffic management. They have also received some training on the AI system, for instance regarding its known limitations. • degree of discretion or authority : They have the ability to change flight departure time, arrival time, and flight paths, and the authority to tell pilots to do these. • responsibilities : They are responsible for ensuring safety and air traffic flow. | |
| Organisation | Describe the organisational setting in which the task takes place, including: • hierarchical structures : The organisations involved include airlines, air services, and airports. Air services have authority to dictate—to airlines and airports—flight departure, arrival, paths, etc. Pilots and air traffic controllers report regularly to each other on flight updates. • relevant rules and procedural constraints : Procedurally, aviation is one of the most heavily regulated and compliancy-based sectors. |
| Oversight Task | Describe the oversight task to be performed: • purpose and goals : The primary oversight task is to ensure safety by monitoring AI system outputs (trajectory predictions, conflict alerts, resolution recommendations) and intervening when these systems provide inaccurate, incomplete, or unsafe guidance. Secondary goals include maintaining efficient traffic flow and ensuring regulatory compliance. Oversight is not a discrete check but is woven into the ongoing work of controlling: monitoring occurs continuously as controllers interact with AI-generated predictions and recommendations; each decision to accept or override an AI suggestion constitutes human oversight activity. • risks : The oversight may fail due to unclear responsibilities between the ATC and their supervisor; the oversight may fail due to the significant workload for ATCs. | |
| Oversight Technology (Optional) | Describe the technologies that support oversight, such as monitoring dashboards, warning systems, or audit logs. Describe their: • functionalities : The real-time oversight technology include warning systems for potential collisions or congestion, load-balancing dashboards for sector merging for supervisors to monitor sector loads per pilot, intervene in high- or low-load sectors by merging contiguous sectors, or splitting sectors. These oversee the great socio-technical system, not just the software. These tools function as cognitive artifacts that off-load some monitoring burden (e.g., automated scanning for conflicts) while introducing new coordination demands (e.g., understanding why a conflict alert was triggered, distinguishing true alerts from false positives). • system interfaces : This oversight technology interfaces with AI systems by receiving real-time predictions and comparing them against actual aircraft behavior captured by radar and data links. • signals : Warning systems for human overseer(s) may use visual (color-coded alerts, flashing indicators) and auditory signals to communicate urgency. | |
| Oversight Personnel | Describe the personnel responsible for oversight, including: • composition : The two main oversight personnel are the front-line controllers and their supervisors. Oversight responsibility is distributed across multiple roles operating at different timescales and hierarchies. • abilities/competencies : Front-line controllers handle real-time oversight through continuous interaction with the AI task systems. They possess deep expertise in traffic patterns, aircraft performance characteristics, local airspace constraints, and weather impacts. Supervisors maintain shift-level oversight by monitoring system performance at an aggregate level across multiple sectors, tracking both automated tools and controller workload. They have expertise in system behavior patterns, organisational coordination, and resource allocation. • authority : Controllers and supervisors have authority to immediately override or ignore AI recommendations in their sector. They hold authority to disable any malfunctioning systems, reallocate workload, and escalate issues to on-call or offline technical teams. • responsibility : Front-line controllers are responsible for monitoring the air-traffic control systems and intervening in situations that have the potential to cause an incident within their sector or at the boundary between their airspace and a neighbouring sector. Supervisors are expected to identify potential conflict across different airspaces and ensuring even spread of traffic at an aggregate level across multiple sectors, tracking both automated tools and controller workload. | |
| Oversight Organisation | As above. |
| Oversight Task | Describe the object-level task to be performed: • purpose and goals : The systemic oversight task considers factors beyond the final predictions (e.g., of trajectories, load balancing) to intermediate computational steps that would enable auditing. For example, for trajectory prediction, this may be done by logging the aircraft performance, wind estimates used, waypoint calculations, etc. – such that domain experts can use a model’s explanatory output to trace why a prediction was (in)accurate. This level of oversight must also be responsive to real-time alternatives (e.g., through counterfactual predictions); multiple reports (e.g., when controllers in different sectors encounter similar problems); and added situational factors (e.g., updating the model weights to prioritize weather information). These intermediate computational steps also contextualise the design task for AI system developers who work with the controllers to make updates to the system. | |
| Oversight Technology | Describe the technologies that support oversight, such as monitoring dashboards, warning systems, or audit logs. Describe their: • functionalities : These would typically be immutable logging systems for traceability of runtime decisions, tools for aggregating individual flight information, and replay tools to replay incidents that occur, as well as tools for safety audit, such as root-cause analysis and interpretability tools. | |
| Oversight Personnel | Describe the personnel responsible for oversight, including: • composition : Experienced controllers (e.g., supervisors) continually test AI performance, coordinate across sector controllers to identify common themes of AI errors, and make decisions about systematic AI use in different scenarios. System and safety engineers (e.g., at Air Navigation Service Providers) bridge operations and development. AI developers and data scientists analyze logged data (e.g., accepted/rejected recommendations, computational trace data) to identify potential algorithmic failures and retrain models. • abilities/competencies : For effective oversight, these may be controllers with technical training, which gives them a combination of prior domain expertise and acquired technical skills to make changes. This requires expertise in AI and software development, supported by design technology such as explanatory outputs. • responsibility : Experienced controllers (e.g., supervisors) continually test AI performance, coordinate across sector controllers to identify common themes of AI errors, and make decisions about systematic AI use in different scenarios. System and safety engineers (e.g., at Air Navigation Service Providers) bridge operations and development: they diagnose root causes of controller-reported issues, translating operational problems into technical specifications. AI developers and data scientists analyze logged data (e.g., accepted/rejected recommendations, computational trace data) to identify potential algorithmic failures and retrain models. | |
| Organisation | Describe the organisational setting in which the oversight takes place, including: assessment and support technology. • rules and procedural constraints : Regulatory bodies set formal reporting and performance guidelines (as above). Aviation software certification standards are rigid due to the safety-critical system needs. They require formal verification, hazard analysis, and extensive safety documentation that can extend design timelines. Legacy infrastructure dependencies can be an issue, which require modern AI systems to interface with decades-old radar, communication, and data processing and presentation systems. • governance structures : These are operationalized as hierarchies within the ATC teams and supervisors, with different tasks for local vs. global AI assessment and support technology. |
| Oversight Task | Describe the oversight task to be performed: • purpose : Assurance that an organisation is complying with external and internal regulations in air-traffic control. • primary goals : Overseers are typically tasked with the goal of demonstrating and maintaining compliance to safety regulation, as well as ensuring air traffic flow requirements set internally by their organisation. • risks : The risks are that they fail to monitor compliance appropriately, misinterpret regulation, and do not have sufficient information to identify when to intervene. • outcomes : Verified compliance, correct interventions such as requesting software improvements. | |
| Oversight Technology | Describe the technologies that support oversight, such as monitoring dashboards, warning systems, or audit logs. Describe their: • properties : traceability; auditability, transparency, and legally robust. • functionalities : immutable logging systems that capture, aggregate and interpret relevant events to determine compliance; replay tools to view potential non-compliant incidents; dashboards to provide real-time views of status and potential gaps, risk registers; compliance tracking; auditing packages. Certification platforms would be used to certify people, systems, and organisations. • failure modes : Delayed data logging and aggregated data access, inability to inspect non-compliant behaviour, inability to trace data across systems. • system interfaces : dashboards for logged and aggregated data; telemetry data; process and workflow definition and monitoring; checklists; attestation tools; data exchange between systems. • signals : compliance reports; interventions in the systemic and realtime layers; incident reports. | |
| Oversight Personnel | Describe the personnel responsible for oversight, including: • composition : Internal compliance officers, safety assurance personnel; audit teams; quality assurance teams. • abilities/competencies : Aviation standards and safety expertise; regulatory expertise; ability to make judgements under uncertainty. • authority : Demand evidence; mandate corrective interventions; approve/suspend systems or individuals; mandate reviews; escalate issues; prepare audit materials. • responsibility : Monitoring and maintaining compliance against requirements; data and documentation maintenance; identifying compliance gaps; coordinating corrective actions; supporting external regulatory. • intentions : Demonstrate compliance; reduce safety and operational risk; cooperate with external regulators; ensure AI-driven ATC systems align with standards and regulations. | |
| Oversight Organisation | Describe the organisational setting in which the oversight takes place, including: • rules and procedural constraints : Procedures for compliance, safety management, auditing, documentation, and escalation. • governance structures : Audit committees; compliance and safety functions; clear separation between compliance and operational roles. • reporting mechanisms : Compliance reports; safety reports; audit fundings; incident reports; dashboards. • coordination : Coordinate with air services and ATC organisations to maintain compliance; with external regulators to understand requirements and demonstrate compliance; with systemic and realtime layer personnel to ensure requirements interpreted and implemented correctly. • constraints : Aviation commission requirements and other regulations; time and real-time pressures. |
| Monitor | Specify the types of monitoring activities and their temporal frequency: • signals : Monitoring happens through multiple channels, for AI and other artifacts in the ATC control room: (a) direct observation of flight conflict alerts and trajectory predictions (local, case-by-case monitoring); (b) peripheral awareness of unusual information (unexpected weather alerts, real-time error reports for the system); and (c) comparing to physical instrumentation and artifacts (e.g., flight strips, ground staff’s alerts) to identify discrepancies and potential AI system errors. • interpretation : Controllers are highly training in how to interpret signals, and what potential failure modes could occur. • frequency : monitoring within sectors is continuous and real-time. | |
| Assess Risk | Define the risk assessment process to be used by the oversight personnel: • Controllers use the evidence from monitoring to establish the severity of safety risks. Effectively assessing risks requires pattern recognition on the part of the overseer(s): observing when AI recommendations deviate from expected ATC functions, typical pilot decisions, or local airspace considerations. Controllers use information about false alerts, implausible predictions, or inappropriate resolutions from AI to capture a growing severity of safety risks. Assessment also includes the controllers confidence in their own judgment (compared to AI predictions), recent history of AI system reliability, and cognitive capabilities and workload at the time of judgment. | |
| Decide | Define the decision-making process that the oversight personnel follow, including: • decision protocols : Decision options include: (a) accepting the AI recommendation as-is, (b) overriding the AI recommendation and using the controller’s own judgment instead; (c) modifying the AI recommendation (e.g., adjusting the parameters of a specific solution); (d) seeking verification from other stakeholders (e.g., pilots, supervisors, ground staff); (e) if high risk assessment severity, temporarily disabling specific AI functions for some situations and reverting to manual procedures; and (f) depending on severity, escalating to on-call technical support or logging for offline evaluation. Decision authority may also be distributed, with controllers making immediate decisions for their sectors, planning controllers coordinating effective handovers and traffic flow, and supervisors making decisions about system (de)activation. • documentation : Air-traffic control systems log every action made by controllers, including voice communications between controllers, pilots, and supervisors. | |
| Intervene | Specify intervention types and their temporal frequency (real-time, scheduled, or conditional). | |
| See ‘Decide’ above. |
| Monitor | Specify the types of monitoring activities and their temporal frequency: • signals : system performance monitoring, including accuracy rates, error patterns, response times, and any updates to these (global system monitoring), incident & safety reports, changes in performance, system downtime (if any), feedback from trials; • interpretation : system designers and maintainers interpret these signals by determine severity of individual incidents and software failures, identifying trends over time in incidents, air traffic flow, and late arrivals; • frequency : conditional in the case of safety incidents, scheduled (e.g., weekly or monthly review) for monitoring trends. | |
| Assess Risk | Define the risk assessment process to be used by the oversight personnel: • Risk assessment at this level is about immediate safety issues that could result in serious harm, as well as the potential for future issues in safety and air traffic flow. Risk severity would be focused on safety, yet balanced against air traffic flow. Oversight personnel assess risks such as model drift, data limitations, unsuitable interface design, alert fatigue, experience of controllers, and controller training. | |
| Decide | Define the decision-making process that the oversight personnel follow, including: • decision protocols : systemic decisions would be guided by ATC regulations and governance processes at high level, and protocols would be defined by national/international air services, as well as individual organisations producing ATC software; • documentation : incident reports, investigation reports, audit reports, software and process design decisions, software verification results, usability test results, and change logs. | |
| Intervene | Specify intervention types and their temporal frequency (real-time, scheduled, or conditional): In the case of serious safety incidents, interventions can be real-time; e.g. switch to the backup system. Most others are conditional, based on reporting, decisions, software upgrades, etc., and some scheduled; e.g. monitoring of safety and air traffic flow over time. |
| Monitor | Specify the types of monitoring activities and their temporal frequency: • signals : reports on the systemic layers (internal audits, etc), safety incident reports, compliance dashboard information such as control status; aggregated data logs; evidence completeness metrics. • interpretation : signals are interpreted against requirements such as standards, regulations, governance requirements, internatl control objectives, etc; • frequency : continuous or periodic for ongoing controls; scheduled routine audits; conditional in response to serious incidents, system changes, and audit findings. | |
| Assess Risk | Define the risk assessment process to be used by the oversight personnel: At this level, risk that the lower layers do not comply with the relevant regulations, standards, etc. at the organisational level, considering changes to processes and AI/software/ | |
| Decide | Define the decision-making process that the oversight personnel follow, including: • decision protocols : internal compliance and procedures define thresholds for intervention; some decisions are governed by specific processes, often by regulation; • documentation : records of compliance assessment; incident reports; audit reports; mandates to lower levels; reports on enforcement; etc., will be recorded and in most cases are mandated by regulation. | |
| Intervene | Specify intervention types and their temporal frequency (real-time, scheduled, or conditional): initiating corrective actions when required (conditional); updating controls or procedures (scheduled/conditions); mandating additional evidence (conditional/scheduled); system access/user restrictions (realtime, conditional); escalation of issues (conditional). |
| Information flows | Information flows regularly between controllers, and from controllers to supervisors, during runtime operation. When incidents occur, supervisors communicate these to design, management, safety, and legal teams within the air services organisation, who then, if necessary, report to regulators. Similarly, regulators provide information on risks and new regulations, and the design, management, safety and legal teams communicate to controllers and supervisors about changes in procedure, protocol, etc. | |
| Feedback loops | Incidents that affect safety and traffic flow are reported upwards, and design teams manage incident logs, then are discussed and may become new features with an AI and software system. In most organisations, changes to systems are carefully monitored, and often need to go through simulation testing for several weeks without an incident before being put into practice. | |
| Evaluation metrics | Safety incidents, including the number and severity are the primary measure, with late arrival of aircraft a secondary measure. Errors within the AI system that do not lead to safety incidents are also logged and measured. At the regulatory level, metrics include certification compliance rates, audit findings, incident rates per flight operation, and serious incident investigations involving AI system failures. | |
| Dependencies and feasibility | Infeasible oversight is almost unheard of in ATC, however, in serious situations, such as ATC systems being unusable, many organisations have basic backup systems that provide just enough functionality for all aircraft to land safely as soon as possible. At such times, oversight of controllers is more difficult, and different procedures are put into place. When these infeasibility conditions are identified (e.g., through workload studied, cognitive task analysis, or incident investigations), findings are communicated through multiple channels: air service safety personnel inform ATC controllers and ground staff to implement operational restrictions; inform AI developers and vendors to redesign the algorithm or interfaces; and inform regulators through safety reports that may lead to certification requirement changes. |
| Object-Level Task | Describe the object-level task to be performed: • primary goals : To maintain blood glucose levels within a target range (Time-in-Range) to avoid acute hypo- and hyperglycemia, prevent long-term micro- and macrovascular complications, and support normal daily activities as well as exercise. • risks : Severe hypoglycemia (leading to loss of consciousness/seizures) and diabetic ketoacidosis (DKA); risks are exacerbated by inaccurate dosing, delayed trend detection, or hardware/software malfunctions. • outcomes : Short-term safety (avoidance of extremes), long-term metabolic health (in the form of stable HbA1c), and the successful conversion of complex biological regulation into a manageable daily therapy where the cognitive workload remains sustainable for the patient. | |
| AI system | Describe the AI system that supports or automates parts of this task: • core properties : Hybrid closed-loop systems integrate glucose sensing and delivery using control algorithms (e.g., PID or Model Predictive Control) to adjust basal insulin and predict future trajectories. • functionality : The system automatically modulates basal insulin delivery and provides recommendations for bolus dosing based on real-time and predicted glucose levels. • inputs and outputs : Inputs consist of real-time CGM data, carbohydrate estimates, and exercise markers; Outputs consist of automated insulin delivery adjustments and specific bolus dosing recommendations. • level of autonomy : High autonomy for basal control (often functioning while the user sleeps), but relies on user-initiated inputs for meal-time boluses and correction doses. • known limitations or failure modes : Failure modes include sensor inaccuracies (e.g., compression lows), communication drops (Bluetooth failure), incorrect trajectory predictions, or inappropriate recommendations that increase hypoglycemia risk. Automation may decrease user vigilance or mask physiological symptoms. | |
| Human(s) | Identify the people directly interacting with, using, or affected by the AI system: • abilities/competencies : End-users with T1D who must interpret trends and symptoms; HCPs with clinical expertise in therapy optimization. In pediatric settings, caretakers without formal medical education may act as the primary operators. • degree of discretion or authority : Users have the final authority to accept or override dosing suggestions and can suspend automation at any time to revert to manual mode. • responsibilities : Ensuring safe device operation, providing sufficiently correct carbohydrate estimation, and responding to system alerts and alarms. • intentions : Optimizing glycemic outcomes while maintaining immediate physical safety and long-term health. | |
| Organisation | Describe the organisational setting in which the task takes place: • hierarchical structures : Clinical care teams prescribe and configure the systems; manufacturers provide the technology, technical support, and initial training. • relevant rules and procedural constraints : Usage is heavily regulated by medical device authorities (e.g., FDA, EMA/MDR); clinical treatment protocols guide system settings. |
| Oversight Task | Describe the oversight task to be performed: • purpose : To prevent severe hypoglycemia and DKA by reviewing algorithm outputs and intervening when therapy appears unsafe or misaligned with the user’s actual physiological state. • goals : Safety-critical monitoring of insulin adjustments and immediate response to glycemic excursions. • outcomes : Successful avoidance of acute medical emergencies and maintenance of time-in-range. • risks : Risks include alarm fatigue, automation bias (trusting the system over physical symptoms), and failure to detect mechanical issues like pump occlusions. | |
| Oversight Technology (Optional) | Describe the technologies that support oversight: • properties : Integrated software interfaces on insulin pumps and mobile applications. • functionalities : Predictive alert systems for hypo/hyperglycemia, "Insulin on Board" (IOB) tracking, and real-time CGM trend visualization (e.g., trend arrows). • failure modes : Warning systems may fail due to sensor inaccuracies, signal loss, or depleted device batteries. • system interfaces : Displays on the pump or smartphone showing current glucose, predicted trends, and insulin delivery logs. • signals : Auditory alarms (with increasing volume), haptic vibrations, and color-coded visual indicators. | |
| Oversight Personnel | Describe the personnel responsible for oversight: • composition : Primarily the end-user (patient) and occasionally caregivers/parents or school nurses. • abilities/competencies : Proficiency in interpreting CGM trends and correlating device data with physical symptoms (e.g., shakiness or sweating). • authority : Direct authority to override suggestions, adjust targets, or disable the automated insulin delivery entirely. • responsibility : Monitoring system integrity (e.g., validity of sensor data) and responding to alerts. • intentions : Ensuring safety while minimizing the cognitive burden of disease management. | |
| Oversight Organisation | As above. |
| Object-level Task | Describe the systemic oversight task: • purpose : Design oversight for traceability and safety analysis. It involves visibility into intermediate algorithm states (e.g., constraint limits) to understand the logic behind specific adjustments. • goals : Identifying systematic errors (e.g., recurrent post-meal hyperglycemia) and refining model assumptions to better fit the user’s insulin sensitivity. • outcomes : Optimized clinical parameters and refined software performance. • risks : Risk of incorrect parameter tuning leading to chronic over- or under-delivery of insulin. | |
| Oversight Technology | Describe the technologies that support oversight: • properties : Secure cloud-based data aggregation platforms. • functionalities : Retrospective data dashboards (e.g., Glooko, Libreview), immutable logs for auditing, and clinical simulators for testing parameter changes. • failure modes : Incomplete data logs due to syncing errors or unrecorded contextual factors like exercise or illness. • system interfaces : Web portals providing Ambulatory Glucose Profile (AGP) reports. • signals : Reports highlighting periods of "Time Below Range" or excessive system-initiated suspensions. | |
| Oversight Personnel | Describe the personnel responsible for oversight: • composition : Clinicians with advanced technology training, safety engineers at device manufacturers, and AI data scientists. • abilities/competencies : Expertise in retrospective data analysis, root-cause analysis of device failures, and translating clinical insights into algorithmic refinements. • authority : Clinicians adjust foundational therapy settings; manufacturers push firmware updates. • responsibility : Analyzing aggregated performance patterns to ensure long-term system safety and efficacy. • intentions : Improving the accuracy and reliability of the automated therapy. |
| Monitor | Specify the types of monitoring activities and their temporal frequency: • signals : Direct observation of CGM trends (visual arrows), tracking insulin delivery logs (IOB), and monitoring for physical symptoms of dysglycemia. • interpretation : Comparing AI dosing recommendations against physical sensations and recent contextual factors (e.g., meal size, activity level). • frequency : Monitoring is continuous and real-time, occurring multiple times per hour via device displays and alerts. | |
| Assess Risk | Define the risk assessment process to be used by the oversight personnel: • Users use pattern recognition to establish the severity of safety risks, observing when AI recommendations deviate from expected therapy or physical state. Risks are assessed by correlating device alerts with physiological symptoms. Assessment also includes the user’s confidence in the current sensor accuracy and their own cognitive capacity at the time of judgment. | |
| Decide | Define the decision-making process that the oversight personnel follow: • decision protocols : Decision options include: (a) Accepting the AI recommendation; (b) Overriding with a manual bolus; (c) Adjusting temporary targets; (d) Performing a fingerstick check; or (e) Disabling automation. • documentation : Interactions are documented automatically via the pump’s event log and cloud-synced therapy data. | |
| Intervene | Specify intervention types and their temporal frequency: • Execution of decisions occurs in real-time. This involves changing dosing, suspending the pump, or confirming values manually. Frequency is conditional based on the occurrence of glycemic excursions or system alerts. |
| Monitor | Specify the types of monitoring activities and their temporal frequency: • signals : Monitoring global system performance via metrics like Time-in-Range, GMI, and incident reports. • interpretation : System maintainers and clinicians interpret these signals to identify trends in failure or "model drift" over time. • frequency : Scheduled on a quarterly basis for routine clinical reviews, and conditional upon the reporting of a safety incident. | |
| Assess Risk | Define the risk assessment process: • Personnel assess risks such as model drift, unsuitable interface design leading to errors, and user alert fatigue. Assessment focuses on potential safety issues that could lead to severe hypoglycemia or DKA across the user population. | |
| Decide | Define the decision-making process: • decision protocols : Decisions are guided by clinical protocols and software design standards. • documentation : Incident reports, audit logs, and documentation of all clinical settings changes. | |
| Intervene | Specify intervention types and their temporal frequency: • Interventions include clinical parameter updates, software firmware patches, and user retraining. Frequency is typically scheduled (visits) or conditional (patches). |
| Object-Level Task | Describe the object-level task to be performed: • primary goals : Timely and accurate interpretation of head CT images to support clinical decisions (e.g., identifying intracranial haemorrhage, early ischaemic change, mass effect, hydrocephalus), appropriate escalation of critical findings, and production of a clinically useful radiology report. • risks : Missed findings (false negatives) or incorrect findings (false positives) leading to delayed or inappropriate treatment (e.g., delayed neurosurgical referral, inappropriate stroke pathway activation), communication failures for critical results, and downstream patient harm; risks are exacerbated by time pressure, high caseloads, interruptions, limited clinical information, and image artefacts. • outcomes : High diagnostic accuracy for clinically important findings, good patient outcomes, minimum time-to-report (especially for emergency head CT), and consistency and clarity of reporting (e.g., following structured reporting templates, such as RadReport by RSNA). |
| AI System | Describe the AI system that supports or automates parts of this task: • core properties : AI-enabled decision-support integrated into PACS/RIS or a vendor platform; typical components include (a) image-processing and pre-processing pipeline (e.g., slice normalisation, artefact handling), (b) one or more trained models for detection/triage (e.g., haemorrhage detection, large vessel occlusion proxy markers, early ischaemic change support), and (c) user interface elements (e.g., overlays, flags, worklist prioritisation). • functionality : Triage and prioritisation of suspected abnormal head CTs, detection and localisation of acute findings (e.g., intracranial haemorrhage subtypes, midline shift, mass effect), and presentation of decision-support outputs to guide review. • inputs and outputs : Inputs include DICOM head CT image series, acquisition parameters, and optionally limited clinical metadata; outputs include worklist flags (e.g., “suspected ICH”), probability or confidence scores, saliency/heatmap overlays or bounding boxes or segmentations, structured prompts (e.g., “consider mass effect”), and pre-worded patient report. • level of autonomy : The AI system operates autonomously in generating alerts and prioritisation recommendations, but is intended as decision support; the radiologist retains authority for interpretation, changing and signing of reports, and escalation. • known limitations or failure modes : Underperformance in cases with motion, beam-hardening, post-operative change or or poorly acquired imaging; atypical presentations; confounding anatomy (e.g., calcifications); distribution shift across scanners/protocols/patient populations; miscalibrated confidence leading to overtrust or underuse; false positives contributing to alert fatigue and workflow disruption; false negatives creating unwarranted reassurance if users rely on AI triage. |
| Human(s) | Identify the people directly interacting with, using, or affected by the AI system, such as operators or end-users. Describe their: • abilities/competencies : Radiologists’ expertise in neuroanatomy, pathology recognition, and clinical reasoning under uncertainty; ability to integrate imaging with clinical context and priors; competence in using PACS/RIS and interpreting AI outputs (including their limits) through formal trainging and/or experience working with decision support systems. • degree of discretion or authority : Radiologists have discretion to accept, ignore, or override AI outputs, determine report content, and initiate escalation (e.g., critical results communication, stroke team notification, neurosurgical referral); referrers act on reports. • responsibilities : Responsibility for accurate diagnosis and appropriate communication of urgent findings; maintaining professional standards in reporting; using AI outputs appropriately as supportive information rather than definitive diagnosis. |
| Organisation | Describe the organisational setting in which the task takes place, including: • hierarchical structures : radiology teams organised by shift and subspecialty; consultants/attendings, registrars/residents, and reporting radiographers (where applicable); clinical leads and modality leads; interfaces with emergency medicine, stroke services, neurosurgery, and critical care; on-call structures for out-of-hours reporting (can include outsourced teleradiology providers, potentially using different AI systems) and second opinions. • relevant rules, processes and procedural constraints : protocols for head CT acquisition and interpretation (e.g., stroke pathways), outsourcing policies, critical result communication policies, reporting standards and documentation requirements, service-level turnaround targets, information governance, and local policies regarding AI use. |
| Oversight Task | Describe the oversight task to be performed: • purpose : Continuously monitor AI-supported head CT interpretation during active clinical work to detect emerging risks and prevent patient harm through intervention before it occurs. • goals : Identify inaccurate, misleading, or incomplete AI outputs at the patient case level; maintain alignment between AI output and clinical judgement. • outcomes : safe diagnostic decisions for individual patients, effective correction or override of erroneous AI outputs; timely escalation of AI failure patterns. • risks : Automation bias, alert fatigue, inappropriate reliance on AI under time pressure, delayed escalation when AI and human judgement conflict, and ambiguity about when intervention is required. |
| Oversight Technology (Optional) | Describe the technologies that support oversight, such as monitoring dashboards, warning systems, or audit logs. Describe their: • properties : Real-time availability, seamless integration into existing IT infrastructure, low interaction cost (e.g., minimum amount of additional clicks to access relevant information). • functionalities : Explainability features (e.g., sliency maps, bounding boxes, confidence indicators) which should help to identify inaccurate AI outputs more easily. • failure modes : Misleading saliency indicators, poorly calibrated confidence scores, excessive false positives contributing to alert fatigue, technical failures (e.g., where imaging does not reach the AI server), and poorly acquired images that do not meet minimum quality standards. • system interfaces : Direct integration with PACS viewers and reporting software. • signals : See functionalities. |
| Oversight Personnel | Describe the personnel responsible for oversight, including: • composition : reporting radiologists. • abilities/competencies : expertise in head CT interpretation, understanding of AI technology (intended use and performance boundaries/limitations of specific tools), ability to recognise AI failure modes, and capacity to integrate AI outputs with clinical context under time pressure. • authority : authority to override, ignore, or correct AI outputs and to initiate immediate clinical escalation. • responsibility : continuous case-level monitoring and intervention (i.e., ignore, overrride, stop displaying AI outputs) to prevent patient-level harm. |
| Oversight Organisation | Describe the organisational setting in which the oversight takes place, including: • rules and procedural constraints : local/national reporting standards and communication pathways/policies. • governance structures : departmental clinical governance arrangements. • reporting mechanisms : immediate discrepancy reporting; • coordination : coordination emergency medicine, stroke services, and neurosurgery (primary), as well as with IT (secondary). • constraints : time pressure, interruptions, and high workload. |
| Oversight Task | Describe the oversight task to be performed: • purpose : monitor AI-supported head CT workflows across cases and over time to detect systemic risks that are not visible at the individual case level (e.g., systematic bias among specific patient groups). • goals : Identify recurring error patterns, performance drift, misalignment between AI behaviour and clinical practice / clinical guidelines, or workflow disruptions introduced by AI use (e.g., exessive time spent on (a) false alersts; (b) fixing technical problems, or (c) workarounds). • outcomes : System-level interventions such as recalibration, retraining, redesign of AI models or interfaces, and modification of task (e.g., screening order), human (e.g., advanced training), or organisational (e.g., reporting protocols) attributes to prevent future harm. • risks : Delayed recognition of systemic failure, normalisation of deviance, diffusion of responsibility, and reliance on incomplete or poorly contextualised performance metrics. |
| Oversight Technology (Optional) | Describe the technologies that support oversight, such as monitoring dashboards, warning systems, or audit logs. Describe their: • properties : Longitudinal, aggregate, and auditable; • functionalities : performance dashboards, audit logs of AI outputs and clinician responses, case replay tools, and incident aggregation. • failure modes : Incomplete logging, lack of clinical granularity, aggregation masking clinically meaningful errors. • system interfaces : Data pipelines connecting PACS/RIS, EHR, AI vendor platforms, and quality and safety systems. • signals : Changes in alert rates, clusters of discrepancies, recurrent incident reports/tickets, unusal alorithm processing time, and performance shifts (e.g., changes in turn-around-times), unusual interaction patterns following updates or protocol changes. |
| Oversight Personnel | Describe the personnel responsible for oversight, including: • composition : Senior radiologists, modality leads, clinical safety and quality teams, and system designers. • abilities/competencies : Clinical risk assessment, audit and incident investigation skills, and understanding of AI system behaviour and performance limitations. • authority : Authority to modify deployment scope, suspend or restrict AI use, and mandate system updates or retraining. • responsibility : Identifying systemic risks, coordinating system-level interventions, and ensuring organisational learning. • intentions : Preventing future harm and maintaining alignment between AI systems and clinical practice. |
| Oversight Organisation | Describe the organisational setting in which the oversight takes place, including: • rules and procedural constraints : audit schedules, incident investigation protocols, and change-management processes. • governance structures : AI oversight committees and clinical safety boards; • reporting mechanisms : formal audit reports and safety reviews. • coordination : coordination between radiology, IT, vendors, and organisational leadership. • constraints : limited resources, vendor dependency, and poor IT infrastructure. |
| Oversight Task | Describe the oversight task to be performed: • purpose : Monitor AI-supported head CT systems to enforce regulatory, legal, and contractual obligations. • goals : Ensure compliance with medical device regulation (e.g., FDA or EMA), professional standards, and contractual requirements for safe and appropriate AI use. • outcomes : Verified compliance, mandated corrective actions (e.g., retraining or withdrawal), and sustained public trust. • risks : Regulatory lag, insufficient clinical specificity in compliance criteria, and misalignment between regulatory requirements and operational realities. |
| Oversight Technology (Optional) | Describe the technologies that support oversight, such as monitoring dashboards, warning systems, or audit logs. Describe their: • properties : Standardised, auditable, and legally robust. • functionalities : Compliance dashboards, certification records, audit trails, and adverse event reporting systems. • failure modes : Delayed reporting, incomplete data, or lack of traceability across systems. • system interfaces : Interfaces between healthcare organisations, vendors, and regulatory bodies. • signals : Compliance reports, mandated notifications, patient claims/medical negligence claims, and adverse event alerts. |
| Oversight Personnel | Describe the personnel responsible for oversight, including: • composition : Regulators, compliance officers, and accreditation bodies. • abilities/competencies : Regulatory expertise, audit capability, and safety evaluation. • authority : Authority to mandate retraining, restrict deployment, impose sanctions, or withdraw certification. • responsibility : Enforcing compliance and responding to safety signals that exceed organisational control. |
| Oversight Organisation | • rules and procedural constraints : Medical device regulation, professional standards, and contractual obligations. • governance structures : National and international regulatory frameworks. • reporting mechanisms : Mandatory incident reporting and audits. • coordination : Coordination with healthcare providers and AI vendors. • constraints : Jurisdictional boundaries and regulatory update cycles. |
| Monitor | Specify the types of monitoring activities and their temporal frequency: • signals : AI-generated case-level outputs (e.g., predictions, flags, saliency maps, bounding boxes, confidence scores), image features observed during review, discrepancies between AI output and radiologist judgement; • interpretation : radiologists interpret signals through expert pattern recognition, triangulating AI outputs with image evidence, patient information, clinical context, prior imaging, and known AI failure modes; • frequency : continuous and real-time during active clinical practice. |
| Assess Risk | Define the risk assessment process to be used by the oversight personnel: • Risk assessment focuses on the likelihood and severity of immediate diagnostic harm. Radiologists assess whether AI outputs plausibly reflect true pathology, the clinical consequences of acting or not acting on AI suggestions, and their own confidence under current cognitive and workload constraints. Disagreement between AI and human judgement increases perceived risk and triggers heightened scrutiny or verification. |
| Decide | Define the decision-making process that the oversight personnel follow, including: • decision protocols : primarily rely on human judgement informed by AI as a supportive signal; predefined escalation rules apply for critical findings, but accepting, rejecting, or ignoring AI outputs remains discretionary; • documentation : clinically significant discrepancies, critical result communications, and escalation actions are documented through reporting systems and local discrepancy mechanisms where required. |
| Intervene | Specify intervention types and their temporal frequency (real-time, scheduled, or conditional): Interventions are immediate and case-specific. They include overriding or ignoring AI outputs, re-examining image regions highlighted by AI, adjusting the order or priority of case review, seeking a second opinion, and initiating urgent clinical escalation based on clinical judgement. |
| Monitor | Specify the types of monitoring activities and their temporal frequency: • signals : aggregated AI performance metrics (e.g., alert rates, false-positive/false-negative patterns), clusters of clinician overrides or discrepancies, rates of IT tickes, time taken to process the AI output, incident and near-miss reports, changes in operational context (e.g., software updates, protocol changes), unusal algorithm processing times, performance changes (e.g., changes in turn-around time); • interpretation : senior clinicians and safety teams interpret signals by identifying trends, correlations, and deviations from expected performance, and by distinguishing random variation from systematic bias or drift; • frequency : scheduled (e.g., weekly or monthly review) and conditional, triggered by incidents, sudden performance shifts, or system updates. |
| Risk Assess | Define the risk assessment process to be used by the oversight personnel: • Risk assessment at this layer focuses on the potential for repeated or widespread harm. Oversight personnel assess whether observed issues indicate model drift, data mismatch, interface-induced misuse, training data limitations, or organisational factors such as workload or insufficient training. Risk severity is evaluated in terms of scale, persistence, and clinical impact. |
| Decide | Define the decision-making process that the oversight personnel follow, including: • decision protocols : decisions are guided by predefined governance processes (e.g., safety review boards) and may include human judgement informed by technical evidence and clinical expertise; • documentation : findings, rationales, and decisions are formally documented through audit reports, incident analyses, and change-management records. |
| Intervene | Specify intervention types and their temporal frequency (real-time, scheduled, or conditional): • Interventions are system-level and preventative. They include recalibrating or retraining models, redesigning interfaces or alerts, modifying deployment scope or use conditions, updating local protocols and training, and, where necessary, suspending AI use pending further investigation. |
| Monitor | Specify the types of monitoring activities and their temporal frequency: • signals : Compliance reports, audit findings, adverse event notifications, evidence of deviation from approved use or certified performance, and outputs from real-time and systemic oversight layers. • interpretation : Regulators and compliance officers interpret signals against formal requirements, approved indications, and risk classifications, rather than local clinical preferences. • frequency : Scheduled (e.g., periodic audits) and conditional (triggered by serious incidents or regulatory notifications). |
| Risk Assess | Define the risk assessment process to be used by the oversight personnel: • Risk assessment focuses on regulatory non-compliance and unacceptable safety risk at scale. This includes assessing whether failures exceed allowable risk thresholds, indicate breaches of approved use, or undermine certification requirements. |
| Decide | Define the decision-making process that the oversight personnel follow, including: • decision protocols : Decisions are governed by formal regulatory procedures and legal mandates rather than discretionary clinical judgement. • documentation : Compliance decisions, enforcement actions, and rationales are formally recorded through regulatory and contractual documentation. |
| Intervene | Specify intervention types and their temporal frequency (real-time, scheduled, or conditional): • Interventions enforce compliance and may include mandating retraining or re-certification, restricting or withdrawing approval, imposing reporting or monitoring requirements, or requiring organisational changes to ensure ongoing compliance. |
More in this topic
Human Oversight of AI
Human oversight of AI studies how people supervise, verify, and control AI decisions, especially as systems become more autonomous and consequential.
See all 184 papers