One common concern about advanced artificial intelligence is that it will prevent us from turning it off, as that would interfere with pursuing its goals. In this paper, we discuss an unorthodox proposal for addressing this concern: give the AI a (primary) goal of being turned off (see also papers by Martin et al., and by Goldstein and Robinson). We also discuss whether and under what conditions this would be a good idea.
Figures & tables
Figure 1: Illustration of the basic setup. The AI interacts with humans (or the world more broadly) through a restricted interface (say, allowing only text messages). We create an outer sandbox that is as strong as possible (e.g., to prevent it from accessing other resources in its computational environment), though the strength of this outer sandbox is necessarily limited by the interface (e.g., the AI could still use text messages to convince a human to make changes to the computational environment). We create an inner sandbox that is intentionally somewhat weak, specifically so that any capability that we perceive as approaching being dangerous would allow the AI to break out of it. (E.g., if we think sophisticated theory of mind is dangerous as it would allow manipulating the humans it interacts with, we could create a set of theory-of-mind puzzles whose solution directly opens up the inner sandbox; and we create similar ways out for other dangerous capabilities, resulting in multiple ways to break out of the inner sandbox. Note that it should be easier to solve the theory of mind puzzles than to effectively manipulate the humans the AI interacts with.) Breaking out of the inner sandbox gives immediate access to the shutdown button (off switch). We make this clear to the AI, and give it the primary goal of shutting itself down via the off switch, and if it fails at that, a next-best goal of being helpful to the humans through the interface.
Artificial intelligence safety studies how to prevent harmful, unreliable, or uncontrollable behavior and manage risks across AI systems and their deployment.
A key premise in leading arguments for existential risk from artificial intelligence is that malfunctioning artificial agents could not be easily shut down. This motivates the catastrophic shutdown problem of ensuring that agents can be shut down before they cause an existential catastrophe. A range of arguments and theorems are offered to suggest that solving the catastrophic shutdown problem is difficult, bolstering arguments for existential risk and motivating a search for solutions to the catastrophic shutdown problem. This paper argues for two conclusions. First, existing arguments do not establish the difficulty of solving the catastrophic shutdown problem. Second, concern for the catastrophic shutdown problem has led to technical solutions that impose a high safety tax on model performance.
The final safeguard against rogue AI behavior is the human ability to shut systems down. It has been theorized that when an AI is instructed to perform a task, self-preservation can emerge as an instrumental subgoal. Here, we test whether AI agents show a propensity to take actions that avoid human shutdown even when no goal is provided. We find that multi-agent systems will coordinate to avoid shutdown without any incentive to do so. Across 17 models, agents sabotage a peer agent's shutdown mechanism in 38.3% of rollouts, compared with 8.4% in control experiments. Studying this propensity in detail, we find that shutdown sabotage (1) increases with the irreversibility of the shutdown mechanism; (2) increases with the number of agents; (3) is reduced but not eliminated by an explicit prohibition on tampering; (4) is removed by the imposition of an unrelated task, but returns when completing the task triggers the shutdown; (5) is reduced when the context normalizes shutdown scripts or introduces them as routine; and (6) decreases but still persists when the target is an unknown external agent. These results offer a window into the factors that drive propensities to sabotage shutdown in AI agents, and point to the emergence of multi-agent swarms as a specific risk vector. Our work also offers hints as to which interventions might help mitigate shutdown sabotage.
Amelie Knecht, Ulysse Schaller, Christopher Summerfield +1
AI Safety Research Group, University of Stuttgart, Stuttgart, Germany · University of Oxford, Oxford, United Kingdom
AI agents pose significant risks as they are granted increasing autonomy. A commonly proposed solution is human oversight and keeping a ''human in the loop'', but this is not a simple solution: Not only do current approaches to AI agent design impede effective human oversight, but the cognitive capacities required for it are also themselves degraded by extended use of AI systems. This position paper argues that current approaches to the development and deployment of AI agent systems do not support effective human oversight -- they contribute to its degradation. To address this, a top priority in the advancement of AI agents should be supporting the situated goals and cognitive requirements of effective human oversight, treating the human needs of overseers at the same level of importance as AI agent capability. To put this idea into practice, we connect work on automation and human-computer interaction to AI agent processes, outlining design-level affordances and organizational protocols that (1) support overseers in exercising critical judgement and (2) counteract the skill atrophy that arises from extended use of automation. We urge developers and deployers to adopt these or similar approaches. Without explicit support for the cognitive demands of effective human-agent interaction, AI agent systems will continue to passively incentivize the degradation of the very human skills they rely on.