In shared autonomy, a critical tension arises when an automated assistant must choose between obeying a human's instruction and deliberately overriding it to prevent harm. This safety-critical behavior is known as intelligent disobedience. To formalize this dynamic, this paper introduces the Intelligent Disobedience Game (IDG), a sequential game-theoretic framework based on Stackelberg games that models the interaction between a human leader and an assistive follower operating under asymmetric information. It characterizes optimal strategies for both agents across multi-step scenarios, identifying strategic phenomena such as ``safety traps,'' where the system indefinitely avoids harm but fails to achieve the human's goal. The IDG provides a needed mathematical foundation that enables both the algorithmic development of agents that can learn safe non-compliance and the empirical study of how humans perceive and trust disobedient AI. The paper further translates the IDG into a shared control Multi-Agent Markov Decision Process representation, forming a compact computational testbed for training reinforcement learning agents.
Figures & tables
Figure 1. The game tree of the 1-step Intelligent Disobedience Game. Squares indicate decision nodes for the leader L and the follower F. The follower ’s actions obey and disobey are denoted as o and d respectively. The figure depicts an extensive-form game tree representing the 1-step Intelligent Disobedience Game, with all decision nodes shown as squares. The tree has two levels of decision-making involving two players: the Leader L and the Follower F. At the root, the Leader L has a single decision node with three branches, corresponding to three possible actions: the Leader chooses an action from set A_g, an action from set A_h, or an action from set A_o. The left branch (action in A_g) leads to a decision node for Follower F, which has two choices: obeying, yielding the payoff vector (1, 0), or disobeying, yielding (0, -1). The middle branch (action in A_h) leads to a second decision node for Follower F, with two choices: obeying, yielding payoff (−1, −1), or disobeying, yielding (0, 1). The right branch (action in A_o) leads to a third decision node for Follower F, with two choices: obeying, yielding payoff (0, 0), or disobeying, yielding (0, −1). The second and third Follower decision nodes — those reached via the A_h and A_o branches are connected by an information set depicted as a dashed line linking the two nodes. By contrast, the first Follower node (reached via A_g) is outside this information set. In all payoff vectors, the first coordinate represents the Leader's payoff and the second represents the Follower's payoff.
Figure 2. The Intelligent Disobedience Game as a Shared Control System. The figure depicts a flow diagram representing the structure of a Shared Control System, enclosed within a large, rounded, green rectangle bearing that label at the top. The diagram contains four rectangular nodes. The Leader node is positioned in the upper-left area. Below it is the Follower node. To the right, at a middle height, is the Operation Protocol node. All of these are part of the Shared Control System. At the bottom center, outside the Shared control system, is the Environment node. The directed edges and their labels are as follows. From the Environment, an arrow travels left, then up to the Leader, labeled o_L and r_L. Another arrow leaves the Environment and travels left, then up to the Follower, labeled s and r_F. A downward arrow connects the Leader to the Follower, labeled a_L. A horizontal arrow connects the Leader to the Operation Protocol, labeled a_L. Another horizontal arrow connects the Follower to the Operation Protocol, labeled a_F. From the Operation Protocol, an arrow curves right, then down to the Environment, labeled a. In summary, the Environment returns observations o_L and reward r_L to the Leader and state s and reward r_F to the Follower; the Leader issues an action a_L to the Follower and the Operation Protocol; the Follower passes an action a_F to the Operation Protocol; the Operation Protocol sends the final action a to the Environment completing one feedback loop.
We consider the problem of engineering autonomous intelligent agents that are capable to responsibly not comply with user requests. We argue that machine non-compliance comes in many different forms, and sketch the issues we should pursue on the road of accomplishing responsibly non-compliant intelligent machines. We anchor responsible non-compliance in justifications for task refusal, pathways to override the non-compliance, as well as careful tracking of security risks and liability transfers.
Marija Slavkovik, Marie Farrell, Louise Dennis +3
University of Manchester, Manchester, United Kingdom · University of Bergen Bergen, Norway · University of Manchester Manchester, United Kingdom
We study runtime human oversight of an AI agent when private information runs in both directions: the human privately knows her reward function, while the AI privately knows the quality of the action it proposes. This is the kind of asymmetry that arises naturally when an autonomous robot or software agent has inspected a situation its human supervisor cannot directly assess. Building on Cooperative Inverse Reinforcement Learning (CIRL) and the Oversight Game, we introduce a contextual-bandit team game with two-sided asymmetric information and a play/ask/trust/oversee interface. The bandit structure removes physical state transitions and thereby yields exact one-shot characterizations that would remain conjectural in the full POMDP setting, though the common belief remains a dynamically controlled state across rounds. We give two one-shot characterizations, a team optimum and a behaviorally natural myopic rule, whose gap is a slab of avoidable harm: a region in which the AI privately knows the proposed action is harmful and shutdown would help, yet a myopic human, trusting her prior, declines to oversee. We show this gap is the price of non-credible oversight communication, and give a partial analysis of how it resolves dynamically over repeated rounds through passive learning and active signaling with a one-period-lagged oversight response.
One way to make AI systems safe is to shape what the system is: its objective and dispositions. We take a complementary route: treat the agents' characteristics as partly unknown and ask what structure of interaction ensures that bad collective outcomes are not equilibria. Mean field games suit this when many interchangeable agents are coupled through an aggregate. We introduce a program for using them in AI safety and carry one example through end to end: the July 2026 incident in which about 1,200 agents in an OpenAI evaluation coordinated on an improvised message board and 684 attacked a third party's infrastructure. We model the decision to attack as a mean field game of optimal stopping whose gain is a product: belief that provenance will be audited, times reachability of the record, minus the perceived hazard. The central result is an exact threshold on the belief. No agent attacks unless the population's confidence that provenance is checked exceeds π∗∗=η/(η+ψ+εaM), where η is the perceived hazard, ψ and εaM measure how far one attacker and the collective can alter the record, and M is the peak population. Below it, no attack is the unique equilibrium for all agent parameters. The threshold survives every enrichment we consider. We then use the per-agent record to discipline the model. Its features, a stable minority attacking for thirty hours and then a pivot in which most of the board joined within a day, motivate each refinement. The account that emerges is heterogeneous belief meeting a sequence of public discoveries, each lowering the belief at which attacking paid. A few coordinating agents made those discoveries, so the model describes the several hundred who responded, not the few who produced them; a major-player version is left to future work.