Shutdown Safety Valves for Advanced AI

Vincent Conitzer

cs.AIMar 7, 2026

Shutdown Safety Valves for Advanced AI

Authors: Vincent Conitzer

Organizations: Foundations of Cooperative AI Lab Carnegie Mellon University Pittsburgh, PA, USA

Abstract

One common concern about advanced artificial intelligence is that it will prevent us from turning it off, as that would interfere with pursuing its goals. In this paper, we discuss an unorthodox proposal for addressing this concern: give the AI a (primary) goal of being turned off (see also papers by Martin et al., and by Goldstein and Robinson). We also discuss whether and under what conditions this would be a good idea.

Figures & tables

Figure 1: Illustration of the basic setup. The AI interacts with humans (or the world more broadly) through a restricted interface (say, allowing only text messages). We create an outer sandbox that is as strong as possible (e.g., to prevent it from accessing other resources in its computational environment), though the strength of this outer sandbox is necessarily limited by the interface (e.g., the AI could still use text messages to convince a human to make changes to the computational environment). We create an inner sandbox that is intentionally somewhat weak, specifically so that any capability that we perceive as approaching being dangerous would allow the AI to break out of it. (E.g., if we think sophisticated theory of mind is dangerous as it would allow manipulating the humans it interacts with, we could create a set of theory-of-mind puzzles whose solution directly opens up the inner sandbox; and we create similar ways out for other dangerous capabilities, resulting in multiple ways to break out of the inner sandbox. Note that it should be easier to solve the theory of mind puzzles than to effectively manipulate the humans the AI interacts with.) Breaking out of the inner sandbox gives immediate access to the shutdown button (off switch). We make this clear to the AI, and give it the primary goal of shutting itself down via the off switch, and if it fails at that, a next-best goal of being helpful to the humans through the interface.

AI Safety

Artificial intelligence safety studies how to prevent harmful, unreliable, or uncontrollable behavior and manage risks across AI systems and their deployment.

See all 134 papers

Revisiting the shutdown problem
Jun 6, 2026David ThorstadAI Agent Safety AI Safety
Shutdown Sabotage Propensities in Multi-Agent Systems
Sep 23, 2026Amelie Knecht, Ulysse Schaller, Christopher Summerfield +1AI Agent Safety Multi-Agent System Security
AI Safety Research Group, University of Stuttgart, Stuttgart, Germany · University of Oxford, Oxford, United Kingdom
AI Agents Push Humans Out of the Loop
Aug 24, 2026Margaret Mitchell, Avijit Ghosh, Samir PassiAI Agent Safety Human-in-the-Loop AI
Hugging Face · Data & Society

Jun 6, 2026J/K move · Enter open · S save

Revisiting the shutdown problem

David Thorstad

AI Agent Safety AI Safety

Open paper AskPDF arXiv

A key premise in leading arguments for existential risk from artificial intelligence is that malfunctioning artificial agents could not be easily shut down. This motivates the catastrophic shutdown problem of ensuring that agents can be shut down before they cause an existential catastrophe. A range of arguments and theorems are offered to suggest that solving the catastrophic shutdown problem is difficult, bolstering arguments for existential risk and motivating a search for solutions to the catastrophic shutdown problem. This paper argues for two conclusions. First, existing arguments do not establish the difficulty of solving the catastrophic shutdown problem. Second, concern for the catastrophic shutdown problem has led to technical solutions that impose a high safety tax on model performance.

Abstract

AI Safety

Explore similar work

Revisiting the shutdown problem

Shutdown Sabotage Propensities in Multi-Agent Systems

AI Agents Push Humans Out of the Loop