Organizations: Department of Industrial Engineering and Operations Research, UC Berkeley, United States. · Stoa, United States. · Department of Cybersecurity, University at Albany, SUNY, United States.
Deploying AI systems requires deciding when to delegate tasks and when humans should intervene to monitor and mitigate risk induced by AI operations. These decisions are challenging when failures cluster: a hallucination or harmful output can trigger further errors, creating periods of elevated risk. We introduce a continuous-time framework for learning adaptive human oversight under such turbulent AI risk. Existing oversight and delegation formulations condition on history but do not model incident clustering, or its suppression by supervision effort, jointly with the delegation decision and this study fixes this gap. The self-exciting dynamics capture how risk events increase the likelihood of subsequent events, making their timing and history central to decision-making. We formulate a stochastic control problem combining human actions, monitoring effort, and switching between human-AI-assisted operation and full AI delegation, balancing operational rewards against oversight costs, and cascading AI-failures and induced uncertainty. Human participation is an endogenous component of risk management: the policy determines both when oversight is needed and how much effort to allocate. We study a relaxed switching formulation and propose Hawkes-PPO, a policy-gradient method that uses a bank of exponential filters of observed incident times. In a synthetic environment it attains a higher risk-adjusted objective than either fixed regime and approaches an approximate full-information oracle. We illustrate our results with numerical simulations by examining how cascade risks influence intervention and delegation, connecting reinforcement learning with adaptive human oversight of AI systems. In particular, we illustrate the benefit of our switching strategy and Hawkes-PPO algorithm to monitor the project efficiently along time, reducing turbulent risks occurrences and costs.
Figures & tables
Figure 1: Four-method comparison. Error bars are 90% confidence intervals. Higher objective and terminal state are better; lower realized volatility indicates less quadratic variation.
Figure 2: Vertical dashed lines mark regime switches. From left to right, top to bottom: paths of X , net running reward and terminal reward, intensity of state and action turbulence (Hawkes intensities), quadratic variation of the project, regime activation, optimal switching intensity, action of the agent human or AI, human effort.
Configuration
Human time (%)
Effort (Human)
Switches
Incidents
Burst (%)
Terminal quality
Objective
Panel A. AI initial regime
Baseline
46.4
0.896
1.46
5.40
45.9
1.81
-93.61
Excitation 0.5
0.9
0.893
0.27
5.59
45.9
1.17
-85.96
Excitation 1.5
73.6
0.842
1.22
5.13
38.6
1.88
-95.85
Human cost 0.75
76.0
0.731
1.15
5.05
36.7
1.82
-89.68
Human cost 1.25
13.7
0.973
1.01
6.20
57.4
1.30
-96.33
Table 1: Supervision and project outcomes across model configurations.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Numerical choice
Parameter
Numerical choice
(T,Δ)
(8,0.05)
(X0,R0,Z0)
(0,0,0)
(a,e,ν)
[0,1]2×[0,4]
ε
0.01
δ
0.18
(σ0,σ1)
(0.16,0.04)
f(x)
0.1x+0.05tanh(x−1)
g(x)
1.2x+0.6tanh(x−1)
(k0,ka,ke)
(3.25,0.75,1.25)
kAI
1.5
(η,κ,χ)
(50,0.30,0.20)
μ
(0.35,0.25)
Appendix
Table 2: Physical, AI and simulation parameters. Kernel rows/columns follow (S,A) . Learning settings are in the appendix.
Setting
Choice
Network and optimizer
Width 64, one gated DGM block, Adam, gradient norm cap 10.
Training
6,000 updates; batch 2,048; learning rates 10−3 for updates 1–3,000, 3⋅10−4 for 3,001–4,800, and 10−4 for 4,801–6,000.
Collocation and selection
t uniform on [0,8] ; x sampled 65% from [−4,6] and 35% from [−16,16] . Residual validation on 8,192 points every 250 updates; selected update 6,000.
Exported policy
Grid [0,8]×[−12,12] , spacings Δt=Δx=0.01 , bilinear interpolation; analytic affine-tail action outside the x -range.
Table 3: Project-only recommender training and deployment.
Setting
Choice
Regime networks
Actor 46−128−128−6 ; value 46−128−128−1 , independently for each regime.
Initialization
Orthogonal weights, hidden gain 2 ; actor output gain 0.01 , value output gain 1; log-standard-deviation bias −0.7 . Switching-mean bias atanh(−0.95) ; initial a,e≈0.5 , ν≈0.1 .
Parallel sampling
1,024 environments, exactly 512 AI starts and 512 Human starts. Each reset restores the assigned initial regime.
Rollout and update
32 decisions per environment; four shuffled epochs; minibatch 4,096; Adam; actor/value gradient norm caps 1 and 5.
Budget
480 rollouts, 15,728,640 transitions, 15,360 actor and 15,360 value updates. No replay buffer or target-network soft updates.
Initial learning rates
Actor 3⋅10−4 , value 5⋅10−4 ; common schedule below.
Appendix
Table 4: Hawkes-PPO training configuration.
Method
AI start
Human start
Oracle (HJB/DGM)
-93.262 [-93.508, -93.015]
-94.230 [-94.468, -93.991]
Hawkes-PPO
-93.608 [-93.861, -93.354]
-94.676 [-94.915, -94.437]
Pure AI
-98.127 [-98.537, -97.718]
—
Pure Human (HJB/DGM)
—
-96.027 [-96.191, -95.864]
Other RL methods:
PPO
-94.954 [-95.198, -94.710]
-97.219 [-97.407, -97.031]
Appendix
Table 5: Objective values and 90% Monte Carlo confidence intervals. Each reported entry uses 4096 paths; a dash indicates an incompatible initial regime for a fixed-regime benchmark. Higher is better.
Configuration
Human time (%)
Conditional effort
Switches / path
Incidents / path
Burst prob. (%)
Terminal quality
Objective
Panel A: AI initial regime
Baseline
46.436 [45.694,47.179]
0.896 [0.893,0.898]
1.457 [1.430,1.484]
5.396 [5.335,5.458]
45.874 [44.597,47.157]
1.811 [1.787,1.835]
-93.608 [-93.861,-93.354]
Excitation 0.5
0.946 [0.857,1.034]
0.893 [0.882,0.904]
0.271 [0.251,0.291]
5.595 [5.523,5.666]
45.923 [44.645,47.206]
1.168 [1.143,1.193]
-85.962 [-86.204,-85.720]
Excitation 1.5
73.607 [73.045,74.168]
0.842 [0.838,0.846]
1.222 [1.204,1.241]
5.127 [5.068,5.187]
38.647 [37.404,39.906]
1.881 [1.856,1.905]
-95.849 [-96.069,-95.628]
Human cost 0.75
75.998 [75.463,76.533]
0.731 [0.726,0.735]
1.151 [1.135,1.166]
5.054 [4.996,5.113]
36.670 [35.441,37.917]
1.817 [1.793,1.841]
-89.677 [-89.868,-89.487]
Human cost 1.25
13.716 [13.237,14.196]
0.973 [0.972,0.974]
1.013 [0.981,1.045]
6.202 [6.126,6.277]
57.446 [56.171,58.712]
1.297 [1.271,1.323]
-96.334 [-96.669,-96.000]
Appendix
Table 6: Supervision and project outcomes under the original numerical protocol. Means appear above conditional 90% evaluation-path Monte Carlo intervals.
AI systems increasingly delegate decisions to specialized models, evaluators, tools, and supervisory controllers. The central AI problem is no longer only model accuracy, but uncertainty-aware governance: how much autonomy to grant, which evidence should calibrate trust, what performance ceiling a delegated AI system can sustain, and when human intervention becomes necessary. We propose the Minimum Sufficient Oversight Principle (MSO), a variational principle for principled autonomy delegation: minimize governance burden on the Fisher information manifold subject to a delivery constraint. The resulting Euler-Lagrange solution yields a water-filling allocation of governed delegation across the task space. Building on a revealed-action governed delegation channel model, we prove a capacity theorem for stationary symbolwise review policies, derive a local first-order approximation relating workflow complexity to quality degradation, and give a drift-dominated autonomy-time scaling law linking intervention timing to effective capacity, complexity, and drift. Within this framework, masking appears as a structural AI-governance pathology: corrected performance can hide the competence signal needed to calibrate trust. Synthetic simulations and a semi-real reconstructed workflow support design prescriptions including upstream-first correction, sensitivity-based intervention, and explicit feasibility checks before autonomy is expanded. The result is a computable framework for uncertainty, planning, and oversight in delegated AI systems. A companion Python package is available at https://github.com/crbazevedo/delegation-lab.
AI agents pose significant risks as they are granted increasing autonomy. A commonly proposed solution is human oversight and keeping a ''human in the loop'', but this is not a simple solution: Not only do current approaches to AI agent design impede effective human oversight, but the cognitive capacities required for it are also themselves degraded by extended use of AI systems. This position paper argues that current approaches to the development and deployment of AI agent systems do not support effective human oversight -- they contribute to its degradation. To address this, a top priority in the advancement of AI agents should be supporting the situated goals and cognitive requirements of effective human oversight, treating the human needs of overseers at the same level of importance as AI agent capability. To put this idea into practice, we connect work on automation and human-computer interaction to AI agent processes, outlining design-level affordances and organizational protocols that (1) support overseers in exercising critical judgement and (2) counteract the skill atrophy that arises from extended use of automation. We urge developers and deployers to adopt these or similar approaches. Without explicit support for the cognitive demands of effective human-agent interaction, AI agent systems will continue to passively incentivize the degradation of the very human skills they rely on.
As LLM agents begin to take real, irreversible actions (shell commands, file edits, deploys), the standard safety pattern is a human-in-the-loop approval gate: risky actions pause and wait for a person. We argue the gate is the easy part; the hard part is the judgment - which actions to stop - which the field evaluates against two false assumptions: that there is a ground-truth notion of "risky," and that the human reviewer is a perfect, infinitely-available oracle. On a hand-labeled set of 125 adversarially-weighted agent actions we show that (i) reviewers only moderately agree on what is risky (Fleiss' kappa = 0.52), so there is no single correct label; (ii) framing the guard as selective classification under asymmetric cost makes its operating limits measurable, and on hard inputs the guard cannot safely auto-decide; and (iii) when the reviewer is modeled as endogenous (fatiguing as escalation load grows), realized safety becomes an inverted-U in the escalation rate: more human oversight can make a system less safe, and the safety-optimal guard escalates below full escalation - a setting a load-aware policy also uses to resist a flooding attack that slips a malicious action past a fatigued reviewer. Agent oversight, framed this way, is not only a classification problem but a resource-allocation one: human attention is finite, and the guard's escalation policy spends it. We claim none of these mechanisms as novel - fatigue-aware learning-to-defer (FALCON), cost-sensitive deferral under workload constraints (DeCCaF), trajectory-level guarding, and reviewer-fatigue/flooding attacks are all prior art we cite. Our contribution is an open-source agent-oversight system that operationalizes and measures them in the LLM-agent action-gating setting, turning "is my guard good?" from a guess into a curve. The inverted-U and the flooding attack are modeling results that motivate a human study.