Artificial Intelligent Disobedience: Rethinking the Agency of Our Artificial Teammates
Abstract
Artificial intelligence has made remarkable strides in recent years, achieving superhuman performance across a wide range of tasks. Yet despite these advances, most cooperative AI systems remain rigidly obedient, designed to follow human instructions without question and conform to user expectations, even when doing so may be counterproductive or unsafe. This paper argues for expanding the agency of AI teammates to include intelligent disobedience, empowering them to make meaningful and autonomous contributions within human-AI teams. It introduces a scale of AI agency levels and uses representative examples to highlight the importance and growing necessity of treating AI autonomy as an independent research focus in cooperative settings. The paper then explores how intelligent disobedience manifests across different autonomy levels and concludes by proposing initial boundaries and considerations for studying disobedience as a core capability of artificial agents.
Figures & tables
| Autonomy Level | Example of Autonomous Agent | Example of Intelligent Disobedience or Override |
|---|---|---|
| L0: No Autonomy | A factory robot arm is programmed to follow a fixed sequence of motions to assemble parts, without any sensing or decision-making. | Perceived disobedience: a factory robot arm stops its fixed routine due to a faulty motor. |
| L1: Support | A grammar-checking tool that suggests corrections as a user types, but doesn’t change anything unless the user approves. | A grammar-checking tool automatically corrects a critical error in a legal document, even after the user dismisses the suggestion, to preserve document integrity. |
| L2: Occasional Autonomy | A robotic vacuum that maps a room and autonomously cleans it, but needs human input to avoid new obstacles or to resume after a pause. | A robotic vacuum refuses a user’s command to continue cleaning when a small animal is detected in its path, choosing to avoid potential harm. |
| L3: Limited Autonomy | A surgical robot that can autonomously suture a wound during a procedure, but requires a human surgeon to handle complex decisions, unexpected bleeding, or any deviation from the standard procedure. | A surgical robot pauses a suturing task performed by a human operator upon detecting unexpected bleeding to ensure patient safety. |
| L4: Full Autonomy under Constraints | A warehouse robot that autonomously navigates and manages inventory within a fenced facility, but allows supervisors to override or redirect it if needed. | A warehouse robot disobeys a constraint to avoid a restricted safety zone, rerouting its task through that zone in case of an emergency. |
| L5: Full Autonomy | An AI-powered digital agronomist that makes all decisions on planting schedules, irrigation, fertilization, pest management, and harvesting. | The digital agronomist decides to change the designation of the farm to a completely different type of plant, which is not defined in its original mission description. |
Agentic AI
Agentic artificial intelligence studies systems that plan, use tools, learn from feedback, and act toward goals with varying degrees of autonomy.
See all 219 papers