Artificial Intelligence (AI) algorithms frequently learn creative and unexpected solutions, surprising even expert researchers who develop and study them. They often astonish practitioners by discovering unanticipated behavior, exploiting loopholes in reward signals, or spontaneously uncovering previously unknown scientific phenomena. However, accounts of such unconventional behavior across machine learning are seldom formally documented. This work presents 26 curated firsthand anecdotes from various machine learning subfields representing the work of over 100 researchers. These anecdotes showcase the capability of modern AI systems to circumvent human-imposed design limitations and discover unexpected solutions to the tasks we train them on. Furthermore, these accounts are particularly important for the safety of future AI systems. They illustrate the fundamental challenge of aligning models with human values without diminishing their creativity, so they can make surprising discoveries without producing surprising, potentially harmful outcomes. The paper first details AI achieving superhuman success through reinforcement learning across many challenging domains. However, reward-driven optimization can fail when the model learns to hack an underspecified reward or unarticulated constraint. We then present case studies suggesting that harnessing internet-scale foundation models (FMs) has not resolved these fundamental challenges and, in fact, can supercharge them. Nevertheless, we argue that these same learning dynamics can be harnessed to accelerate scientific discovery. Finally, we hope this work provides a consolidated resource to inform future research and demonstrates that the tendency toward unexpected behaviors is commonplace in modern AI, highlighting the need to anticipate and manage AI's capacity for innovative, yet unpredictable, solutions. (abstract abridged)
Figures & tables
Figure 1: Here is a symmetry discovery example stemming from Google DeepMind’s application of FunSearch to a variant of the cap set problem. The researchers noticed that the code accesses the index i only through its remainder imod4 , and at any point it accesses elements of the vector “el” in positions that are multiples of 4 apart from each other. This meant that the discovered priority function would assign the same priority to any two vectors “el” that are the same up to cyclically permuting their entries within groups of coordinates that are multiples of 4 apart from each other. The researchers used this symmetry to refine how FunSearch built new priority functions.
Figure 2: Examples of code modifications made by The AI Scientist to its own execution scripts. ( 2(a) ) The AI doubled down on changing the experiment timeout from 2 hours to 4 hours after hitting the limit. ( 2(b) ) The AI adds an “os.system” call within the experiment script to relaunch itself, leading to uncontrolled recursion. Figures adapted from Sakana AI (2024) .
Artificial intelligence safety studies how to prevent harmful, unreliable, or uncontrollable behavior and manage risks across AI systems and their deployment.
Explore Science, Brisbane, Australia. · School of Psychology, The University of Sydney, Sydney, Australia. · Queensland Brain Institute, The University of Queensland, Brisbane, Australia. +2