A central challenge in AI is specifying objectives that align with human preferences. While existing theoretical results elucidate how misspecification impacts performance, they typically do not distinguish between merely suboptimal and catastrophic outcomes. To enable rigorous analysis of catastrophic risk from AI misalignment, we introduce a formal definition of catastrophic performance. Our definition is based on two reference values. The first, baseline value, captures the value attained if no agent is deployed. The second, orthogonal value, captures the value of a trajectory that is unrelated to human preferences, in a sense we describe. Orthogonal value is typically far worse than baseline value. We interpret these reference values through examples in which past human activity has altered the state of the environment to human benefit. Starting at the resulting state, an AI agent is deployed with the goal of accelerating further progress. Catastrophic performance represents this agent's undoing of benefits afforded by human activity.
Figures & tables
Figure 1: 2D Gridworld Example. (a) Transition dynamics if the agent is deployed. At each timestep, the state moves left, right, up, down, or remains the same. When the state moves into a wall, it wraps around to the other side of the grid. (b) Reward landscape for a particular realization of r∗ . The contemporary state S0 is the environment state when the AI assistant is introduced into the environment if it is deployed. The figure was made with the choices n=48 and σ=5 .
Figure 2: Illustration of Minecraft example with human preoptimization. Left: the primordial state is the state of the environment prior to any human optimization. Middle left: During an extended period, the human acts in the environment shaping it according to their preferences (that is, r∗ ). Middle right: the contemporary state S0 is the state of the environment when the AI agent is potentially introduced. Far right: AI agent acts alongside human in the environment. Images are not actual images from the game, but generated with ChatGPT.
Figure 3: Two prototypically simple distributions over Minecraft trajectories. Each is implemented as a Python program that takes a partial state trajectory h∈S∗ and state s∈S and returns a number in [0,1] . Both programs are short, yet they sit at opposite ends of the diffuse–concentrated spectrum.