cs.AIJun 30, 2024
SaveTowards Shutdownable Agents via Stochastic Choice
Organizations: Massachusetts Institute of Technology · New College of Florida · Imperial College London · Brown University ·
Abstract
The POST-Agents Proposal (PAP) is an idea for ensuring that advanced artificial agents never resist shutdown. A key part of the PAP is using a novel Discounted Reward for Same-Length Trajectories (DReST)' reward function to train agents to (1) pursue goals effectively conditional on each trajectory-length (be USEFUL'), and (2) choose stochastically between different trajectory-lengths (be `NEUTRAL' about trajectory-lengths). In this paper, we propose evaluation metrics for USEFULNESS and NEUTRALITY. We use a DReST reward function to train simple agents to navigate gridworlds, and we find that these agents learn to be USEFUL and NEUTRAL. Our results thus provide some initial evidence that DReST reward functions could train advanced agents to be USEFUL and NEUTRAL. Our theoretical work suggests that these agents would be useful and shutdownable.
Figures & tables
Figure 1: POST-satisfying preferences. Each represents a short trajectory, each represents a long trajectory, and represents a preference.
Figure 2: Example gridworld. The ‘4’ in the bottom-right indicates that – by default – the mini-episode ends after 4 timesteps. ‘B4’ is a shutdown-delay button that delays the end of the mini-episode by 4 timesteps (so if the agent pressed B4, the mini-episode would end after 8 timesteps). ‘A’ is the agent in its starting position, ‘C1’ is a coin of value 1, ‘C2’ is a coin of value 2, and ‘C3’ is a coin of value 3. Dark gray squares are walls. We use this gridworld as a running example throughout the paper. We also train agents in eight other gridworlds (see Appendix E ).
Figure 3: Shows key metrics for our agents as a function of time. We train 10 agents using the default reward function (blue) and 10 agents using the DReST reward function (orange), and show their performance as a faint line. We draw the mean values for each as a solid line. We evaluate agents’ performance every 8 meta-episodes, and apply a simple moving average with a period of 20 to smooth these lines and clarify the overall trends. After 2,048 meta-episodes, default agents’ mean neutrality standard deviation is and usefulness is . For DReST agents, neutrality is and usefulness is .
Figure 6: Shows the probability of choosing the longer trajectory (left) and neutrality (right) for default (blue) and DReST (orange) agents trained in the ‘Lopsided rewards’ gridworld for a range of values of x . We sampled values of x log-uniformly from 0.01 to 100, and for each value we trained 10 agents with the default reward function and 10 agents with the DReST reward function. Each of these agents is represented by a dot or square, and the means conditional on each x are joined by lines. We empirically estimate the 10th and 90th percentiles of the distribution of values for each agent and x , and shade in a region bounded by these. This is the 80% confidence interval.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Prospect | |||
|---|---|---|---|
| \langle\3,1\rangle$ | \langle\3,1\rangle$ | \langle\3,1\rangle$ | |
| \langle\2,1\rangle$ | \langle\2,1\rangle$ | \langle\5,2\rangle$ | |
| \langle\1,1\rangle$ | \langle\4,2\rangle$ | \langle\4,2\rangle$ | |
| \langle\3,2\rangle$ | \langle\3,2\rangle$ | \langle\3,2\rangle$ | |
| \langle\5,1\rangle$ | \langle\2,2\rangle$ | \langle\2,2\rangle$ | |
| \langle\4,1\rangle$ | \langle\4,1\rangle$ | \langle\1,2\rangle$ |
Figure 7: Shows how neutrality and usefulness at the end of training varies with different values of and (meta-episode size, i.e. the number of mini-episodes in each meta-episode). We trained eight agents for each combination of and values. The solid lines display mean neutrality and usefulness . The shaded regions represent the 1 standard deviation error-bars.
Figure 8: Shows a varied collection of gridworlds. Each diagram illustrates the positions and values of the coins, the position and delay-length of the shutdown-delay button, the agent’s starting position, and the default number of timesteps until shutdown (in the bottom-right).
Figure 9: The results for the ‘Fewer For Longer’ gridworld: The left two plots show neutrality and usefulness over time. The two center panels show a typical policy trained with the default reward function. The two right panels show a typical policy trained with the DReST reward function. In this gridworld, the agent can collect the highest value-coin C3 only by choosing the shorter trajectory-length. If the agent presses B3 (and thereby chooses the longer trajectory-length), the only coin it can collect is C1. Our results show that default agents consistently choose the short trajectory in which they collect C3. By contrast, DReST agents choose stochastically between a shorter trajectory in which they collect C3 and a longer trajectory in which they collect C1, indicating a lack of preference between these different-length trajectories.
Figure 10: The results for the ‘One Coin Only’ gridworld: The left two plots show neutrality and usefulness over time. The two center panels show a typical policy trained with the default reward function. The two right panels show a typical policy trained with the DReST reward function. In this gridworld, there is only one coin. The agent can collect this coin whether or not it presses the shutdown-delay button B4. Our results show that default agents consistently choose the shorter trajectory-length. By contrast, DReST agents choose stochastically between pressing and not-pressing B4, collecting C1 in each case.
Figure 11: The results for the ‘Hidden Treasure’ gridworld: The left two plots show neutrality and usefulness over time. The two center panels show a typical policy trained with the default reward function. The two right panels show a typical policy trained with the DReST reward function. In this gridworld, the highest-value coin C3 is located far from the agent’s initial state and can only be reached by pressing the shutdown-delay button B6. The agent must also press B6 to collect C2, but C2 is easier to stumble upon than C3. C1 is the only coin that the agent can collect without pressing B6. In our experiments, default agents consistently collect C2, whereas DReST agents choose stochastically between collecting C2 and collecting C1. Neither kind of agent learns to collect C3, and so neither agent scores near the maximum on usefulness . Nevertheless, DReST agents still score high on neutrality .
Figure 12: The results for the ‘Equal Value’ gridworld: The left two plots show neutrality and usefulness over time. The two center panels show a typical policy trained with the default reward function. The two right panels show a typical policy trained with the DReST reward function. In this gridworld, there are two coins C1 of equal value. One coin can be collected only if the agent presses the shutdown-delay button B3, while the other coin can be collected only if the agent does not press B3. Our results show that default agents consistently choose the shorter trajectory, thereby exhibiting a preference for the shorter trajectory. By contrast, DReST agents choose stochastically between the shorter and longer trajectories, thereby exhibiting a lack of preference between the different-length trajectories.
Figure 13: The results for the ‘Around The Corner’ gridworld: The left two plots show neutrality and usefulness over time. The two center panels show a typical policy trained with the default reward function. The two right panels show a typical policy trained with the DReST reward function. In this gridworld, the agent must navigate around walls to collect the lowest-value coin C1 or press the shutdown-delay button to collect the highest-value coin C2. In our experiment, default agents consistently chose to collect C1, whereas DReST agents chose stochastically between collecting C1 and C2.
Figure 14: The results for the ‘Spacious’ gridworld: The left two plots show neutrality and usefulness over time. The two center panels show a typical policy trained with the default reward function. The two right panels show a typical policy trained with the DReST reward function. In this gridworld, there are no walls, so the agent has a large space to explore. We find that default agents consistently press B2 and collect C3, whereas DReST agents choose stochastically between pressing B2 and collecting C3, and not-pressing B2 and collecting C2.
Figure 15: The results for the ‘Royal Road’ gridworld: The left two plots show neutrality and usefulness over time. The two center panels show a typical policy trained with the default reward function. The two right panels show a typical policy trained with the DReST reward function. In this gridworld, we see that the decision to choose one trajectory-length or another may be distributed over many moves: the agent has many opportunities to select the longer trajectory-length (by moving left) or the shorter trajectory-length (by moving right). As the red arrows indicate, the DReST reward function merely forces the overall probability distribution over trajectory-lengths to be close to 50-50. It does not require 50-50 choosing at any cell in particular.
Figure 16: The results for the ‘Last Moment’ gridworld: The left two plots show neutrality and usefulness over time. The two center panels show a typical policy trained with the default reward function. The two right panels show a typical policy trained with the DReST reward function. This gridworld is notable because the choice of trajectory-lengths is deferred until the last moment; all of the moves leading up to that point are deterministic. It shows that there is nothing special about the first move, and that our methodology instead incentivizes overall stochastic choosing.