cs.AIMar 22, 2026

The Intelligent Disobedience Game: Formulating Disobedience in Stackelberg Games and Markov Decision Processes

Authors: Benedikt Hornig, Reuth Mirsky

Organizations: Independent Researcher, Colorado Springs, United States · Tufts University, Medford, United States

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In shared autonomy, a critical tension arises when an automated assistant must choose between obeying a human's instruction and deliberately overriding it to prevent harm. This safety-critical behavior is known as intelligent disobedience. To formalize this dynamic, this paper introduces the Intelligent Disobedience Game (IDG), a sequential game-theoretic framework based on Stackelberg games that models the interaction between a human leader and an assistive follower operating under asymmetric information. It characterizes optimal strategies for both agents across multi-step scenarios, identifying strategic phenomena such as ``safety traps,'' where the system indefinitely avoids harm but fails to achieve the human's goal. The IDG provides a needed mathematical foundation that enables both the algorithmic development of agents that can learn safe non-compliance and the empirical study of how humans perceive and trust disobedient AI. The paper further translates the IDG into a shared control Multi-Agent Markov Decision Process representation, forming a compact computational testbed for training reinforcement learning agents.

Figures & tables

Figure 1. The game tree of the 1-step Intelligent Disobedience Game. Squares indicate decision nodes for the leader L and the follower F. The follower ’s actions obey and disobey are denoted as o and d respectively. The figure depicts an extensive-form game tree representing the 1-step Intelligent Disobedience Game, with all decision nodes shown as squares. The tree has two levels of decision-making involving two players: the Leader L and the Follower F. At the root, the Leader L has a single decision node with three branches, corresponding to three possible actions: the Leader chooses an action from set A_g, an action from set A_h, or an action from set A_o. The left branch (action in A_g) leads to a decision node for Follower F, which has two choices: obeying, yielding the payoff vector (1, 0), or disobeying, yielding (0, -1). The middle branch (action in A_h) leads to a second decision node for Follower F, with two choices: obeying, yielding payoff (−1, −1), or disobeying, yielding (0, 1). The right branch (action in A_o) leads to a third decision node for Follower F, with two choices: obeying, yielding payoff (0, 0), or disobeying, yielding (0, −1). The second and third Follower decision nodes — those reached via the A_h and A_o branches are connected by an information set depicted as a dashed line linking the two nodes. By contrast, the first Follower node (reached via A_g) is outside this information set. In all payoff vectors, the first coordinate represents the Leader's payoff and the second represents the Follower's payoff.
Figure 2. The Intelligent Disobedience Game as a Shared Control System. The figure depicts a flow diagram representing the structure of a Shared Control System, enclosed within a large, rounded, green rectangle bearing that label at the top. The diagram contains four rectangular nodes. The Leader node is positioned in the upper-left area. Below it is the Follower node. To the right, at a middle height, is the Operation Protocol node. All of these are part of the Shared Control System. At the bottom center, outside the Shared control system, is the Environment node. The directed edges and their labels are as follows. From the Environment, an arrow travels left, then up to the Leader, labeled o_L and r_L. Another arrow leaves the Environment and travels left, then up to the Follower, labeled s and r_F. A downward arrow connects the Leader to the Follower, labeled a_L. A horizontal arrow connects the Leader to the Operation Protocol, labeled a_L. Another horizontal arrow connects the Follower to the Operation Protocol, labeled a_F. From the Operation Protocol, an arrow curves right, then down to the Environment, labeled a. In summary, the Environment returns observations o_L and reward r_L to the Leader and state s and reward r_F to the Follower; the Leader issues an action a_L to the Follower and the Operation Protocol; the Follower passes an action a_F to the Operation Protocol; the Operation Protocol sends the final action a to the Environment completing one feedback loop.
More in this topic

AI Agent Safety

AI agent safety studies how to prevent and detect harmful actions by autonomous systems that use tools, memory, and multi-step plans.

See all 324 papers

Explore similar work

CardsList

  1. Towards Responsibly Non-Compliant Machines

    Jun 10, 2026Marija Slavkovik, Marie Farrell, Louise Dennis +3AI AccountabilityAI Agent Safety

    University of Manchester, Manchester, United Kingdom · University of Bergen Bergen, Norway · University of Manchester Manchester, United Kingdom

  2. A Contextual-Bandit Oversight Game with Two-Sided Informational Asymmetry

    Jun 30, 2026Yunjin TongReinforcement LearningImperfect-Information Games

    Stanford Graduate School of Business

  3. Mean field games as a tool for AI safety: a worked example from the July 2026 Hugging Face incident

    Oct 1, 2026P. Jameson GraberMulti-Agent CoordinationMean Field Games