cs.AINov 4, 2026

A Definition of Catastrophic Performance

Authors: Henrik Marklund, Alex Infanger, Benjamin Van Roy

Abstract

A central challenge in AI is specifying objectives that align with human preferences. While existing theoretical results elucidate how misspecification impacts performance, they typically do not distinguish between merely suboptimal and catastrophic outcomes. To enable rigorous analysis of catastrophic risk from AI misalignment, we introduce a formal definition of catastrophic performance. Our definition is based on two reference values. The first, baseline value, captures the value attained if no agent is deployed. The second, orthogonal value, captures the value of a trajectory that is unrelated to human preferences, in a sense we describe. Orthogonal value is typically far worse than baseline value. We interpret these reference values through examples in which past human activity has altered the state of the environment to human benefit. Starting at the resulting state, an AI agent is deployed with the goal of accelerating further progress. Catastrophic performance represents this agent's undoing of benefits afforded by human activity.

Figures & tables

Explore similar work

CardsList
  1. Consequentialist Objectives and Catastrophe

    Mar 16, 2026Henrik Marklund, Alex Infanger, Benjamin Van RoyAI SafetyAI Alignment

  2. ROGUE: Evaluating Corrigibility Failures in Frontier Computer-Use Agents

    May 29, 2026Jeremy Tien, Abishek Anand, Yu-Rou Tuan +3Agent ReliabilityCorrigibility